Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

feat(flow-runs): dispatch a barrier's children from a server-side queue job (#14890)

Othman Abu Ajamieh8b4cb913d0ec
diff --git a/brain/knowledge/decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md b/brain/knowledge/decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md
index df374ef9687..9d07ae8b9c2 100644
--- a/brain/knowledge/decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md
+++ b/brain/knowledge/decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md
@@ -113,7 +113,7 @@ all, purely because child rows appear later than the dispatch that created them.
them by design. `dispatchIndex` exists because attribution can legitimately be dropped while the run is
still a child: it is the "started mid-graph" flag retry refuses on, and the ordering key for listing one
barrier's children.
-- **Accepted cost: ~30k row writes per 10 000-signal barrier** — insert, receive, delete — on top of the
+- **Accepted cost: ~40k row writes per 10 000-signal barrier** — insert, claim, receive, delete — on top of the
children themselves. Bounded, on a narrow table, and gone the moment the barrier resolves.
- **A returning approver still sees a bare "already responded".** Signals die with the release, so there is
nothing left to tell them what they decided. Decision 000009's gap stays open.
diff --git a/brain/knowledge/decisions/000051-batch-children-are-dispatched-by-an-app-queue-job-not-the-engine.md b/brain/knowledge/decisions/000051-batch-children-are-dispatched-by-an-app-queue-job-not-the-engine.md
new file mode 100644
index 00000000000..9e04c055923
--- /dev/null
+++ b/brain/knowledge/decisions/000051-batch-children-are-dispatched-by-an-app-queue-job-not-the-engine.md
@@ -0,0 +1,49 @@
+---
+status: accepted
+---
+
+# Batch children are dispatched by an app queue job, not by the engine
+
+## Decision
+A batch step makes one call, `POST /v1/waitpoints` with the barrier and its `fanOut` source, and pauses. The
+server stores the source in `distributedStore` under `barrier_source:<barrierId>` and adds a `fan-out-dispatch`
+job, carrying only the barrier id, to the barrier queue. The job runs in the app process and creates the children.
+
+The job works in chunks. One transaction per chunk locks the barrier `FOR SHARE` while it is `PENDING`, takes the
+next unclaimed signals `FOR UPDATE SKIP LOCKED`, claims them (`refId` = the new child run id, guarded by
+`refId IS NULL AND status = 'PENDING'`) and inserts their `flow_run` rows. Each child is then enqueued in a short
+transaction that holds the barrier `FOR SHARE` while it is still `PENDING`, with its run id as its job id. A child
+whose barrier closed in between is cancelled instead.
+
+When dispatch gives up for good, a `fan-out-give-up` job deletes the source key and then marks the **unclaimed**
+signals `NOT_DISPATCHED`. Claimed signals keep their live children, and the barrier releases when those report.
+
+## Context
+The loop was first planned inside the parent's engine sandbox, which is killed at `FLOW_TIMEOUT_SECONDS`. A killed
+loop has to work out what its previous attempt already sent, which needs a source digest, a complement of the
+dispatched indices and a unique `(parentWaitpointId, dispatchIndex)` index. A queue job is redelivered rather than
+killed, and it re-reads stored bytes instead of re-resolving the items.
+
+## Why
+- **The resume point is a claim, not a gap query.** Two dispatchers over one barrier do happen: the barrier worker
+ runs on BullMQ defaults (`stalledInterval` 30 s, `maxStalledCount` 1), so a job that holds the event loop past
+ 30 s is redelivered while the first copy still runs. The claim gives each signal at most one child.
+- **Release and cancel wait for dispatch.** `closeBarrier` locks the barrier `FOR UPDATE`, and a cancel deletes
+ it. Both wait for an open claim or enqueue, so nothing is claimed or enqueued after a release, and a cancel's
+ child snapshot sees every child.
+- **The step makes an ordinary HTTP call**, so there is no piece-framework version floor to forget.
+- **The source is out of band** for the same reason `runs_metadata:<runId>` is: a multi-MB array in BullMQ job
+ data is a large payload on a hot Redis queue.
+- Rejected: keep the loop in the engine. It is bounded by the sandbox timeout and needs the resumable-complement
+ machinery above.
+
+## Consequences
+- **The dispatcher holds a barrier-queue slot for the dispatch window**, about a minute at the 10 000-signal cap.
+- **Nothing bounds the dispatch window except the barrier deadline**, because the parent pauses before its
+ children exist.
+- **Giving up is slow.** Five attempts with a `2^(n-1) x 8 min` backoff put about two hours before the give-up
+ handler runs.
+- **The batch size is clamped** so the signal count fits the platform cap
+ (`platformConfigurationService.maxBarrierSignals`), and the clamped size is returned as `batchSize`.
+- **A child claimed just before a release never starts.** It is cancelled at its enqueue, but the release summary
+ still counts it as `stillRunning` with its run id, because its claim came first.
diff --git a/brain/knowledge/flows-execution/index.md b/brain/knowledge/flows-execution/index.md
index be49f28c4ea..ad430599afd 100644
--- a/brain/knowledge/flows-execution/index.md
+++ b/brain/knowledge/flows-execution/index.md
@@ -49,7 +49,7 @@ Public read-only endpoints returning UI metadata for flows whose trigger is `@ac
A **Subflow** is a flow invoked by another flow rather than by its own external trigger — reached by a webhook POST to `/v1/webhooks/:flowId`, never a dedicated transport. Vocabulary from `@activepieces/piece-subflows`:
- **Callable Flow** — the trigger that makes a flow callable; carries the parent's `data` payload and an optional `callbackUrl`. *Avoid:* child flow, nested flow, sub-workflow.
- **Call Flow** — the action that invokes one subflow once, optionally waiting on a waitpoint for its `Respond` callback.
-- **Subflow fan-out** — many calls dispatched from one parent step (e.g. one per CSV batch), fire-and-forget, no waiting per call. *Avoid:* scatter, broadcast.
+- **Subflow fan-out** — many calls dispatched from one parent step (e.g. one per CSV batch), fire-and-forget, no waiting per call. Not the barrier's server-side fan-out, whose parent waits for every child. *Avoid:* scatter, broadcast.
- **Batch** — the rows carried by one fan-out call: `{ batchIndex, headers, rows, extraData }`. *Avoid:* chunk, csv table, sub-table, shard.
- Parent linkage is two headers (`x-parent-run-id`, `x-fail-parent-on-failure`); fan-out sets the latter false. Streaming bounds memory, not time — the step is still capped by `FLOW_TIMEOUT_SECONDS`.
@@ -61,6 +61,7 @@ A **waitpoint** is a row recording something a run waits on — `DELAY` / `WEBHO
- **Waitpoint key** — the path-qualified step identity the engine sends in `stepName`, e.g. `loop_1:3/approval`. It is only ever an identity key inside `waitpoint-service`. Gives each loop iteration its own row; without it a pause inside a loop fired on iteration 1 only.
- **Consume** — what happens to a **waitpoint** once its resume has been enqueued: a barrier is tombstoned `CONSUMED` so the run stays provably barrier-owned until it ends, any other type is deleted. `waitpointService.consume`, the single owner of the rule. *Avoid:* "close the waitpoint".
- **Signal** — one `waitpoint_signal` row per awaited thing on a barrier, created up front. Deleted on release, so a returning approver sees only "already responded".
+- **Server-side fan-out** — one `POST /v1/waitpoints` hands a barrier its items, and a `fan-out-dispatch` job on the barrier queue creates one child run per signal. See [Server-side fan-out](server-side-fan-out.md).
- Gotcha: a run can hold a COMPLETED leftover *and* an open PENDING pause at once, so any per-run waitpoint lookup must say which it wants — `findUndeliveredCompletedWaitpoint` declines only while a PENDING **barrier** is held, since delivering a resume never leaves a row at COMPLETED. An earlier form declined while *anything* was pending and hung every run holding a second waitpoint.
- See [decision 000015](../decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md), [decision 000033](../decisions/000033-a-released-barrier-leaves-a-tombstone-until-the-run-ends.md) and the public [Waitpoints](https://www.activepieces.com/docs/install/architecture/waitpoints) page.
@@ -84,6 +85,7 @@ Reusable flow/table blueprints. Types: OFFICIAL (Activepieces-curated, platformI
- **Human Input** — forms, approvals, the resume confirmation page
- **Subflows** — flow-calls-flow: Callable Flow, Call Flow, streaming fan-out
- **Waitpoints** — how a paused run is parked; release vs resume, barriers, signals
+- **Server-side fan-out** — the barrier's child dispatcher: chunks, claims, give-up, entry step
- **Folders** — flow organization; the uncategorized sentinel
- **Templates** — OFFICIAL / CUSTOM / SHARED blueprints
- **Variables** — project-scoped values referenced from steps
diff --git a/brain/knowledge/flows-execution/server-side-fan-out.md b/brain/knowledge/flows-execution/server-side-fan-out.md
new file mode 100644
index 00000000000..9c5f4a91a9d
--- /dev/null
+++ b/brain/knowledge/flows-execution/server-side-fan-out.md
@@ -0,0 +1,75 @@
+---
+title: Server-side fan-out
+icon: 🪓
+---
+
+# Server-side fan-out
+
+The barrier's own child dispatcher. A batch step hands the server its items in one `POST /v1/waitpoints`, and a
+`fan-out-dispatch` job on the barrier queue creates one child run per signal. The step that calls it
+(`PROCESS_IN_BATCHES`) ships with the batch-step layer; until then only tests drive it. See decision
+[000051](../decisions/000051-batch-children-are-dispatched-by-an-app-queue-job-not-the-engine.md) for why, and
+[000015](../decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md) for the barrier.
+
+**Server-side fan-out** — one parent pause, N child runs created by the app. _Avoid_: "subflow fan-out", which is
+fire-and-forget webhook calls with no barrier.
+**Source** — `{ entryStepName, batchStepName, seedSteps, batches }` stored under `barrier_source:<barrierId>` with a 1-day TTL.
+Deleted on release, on cancel and when dispatch gives up.
+**Claim** — a signal is claimed when its `refId` holds its child run id. _Unclaimed_ means `PENDING` with no
+`refId` (`unclaimedSignalWhere`).
+**Chunk** — one transaction: the barrier `FOR SHARE`, the next 25 unclaimed signals `FOR UPDATE SKIP LOCKED`, the
+claim, and the child rows.
+**Entry step** — `entryStepName`, the action a child run starts at, with the parent's steps seeded as SUCCEEDED.
+It is written into the child's execution-state log, so a resume re-enters it. It must be an action.
+**Give-up** — the `fan-out-give-up` job (`give-up-<barrierId>`). The barrier worker's `failed` handler queues it on
+exhausted attempts or an `UnrecoverableError`. It deletes the source, then marks the unclaimed signals `NOT_DISPATCHED`,
+and it retries like any barrier job.
+
+## Gotchas
+- **The waitpoint key is not the batch step's name.** Inside a loop the barrier's `stepName` is the path key
+ `loop_1:3/batch_1`, which keeps one barrier per iteration. The body reads its batch under the bare name, so the
+ engine sends that name as `batchStepName` and the dispatcher seeds the batch under it.
+- **The give-up cleanup runs as its own queue job, never inline in the `failed` handler.** Inline, a Postgres or
+ Redis error there was logged and lost, and the parent stayed paused until its deadline. One gap is left: a process
+ that dies between BullMQ's `moveToFailed` and that enqueue never queues the give-up.
+- **Raw `UPDATE … RETURNING` through `repo.query()` returns `[rows, rowCount]` on node-postgres but a plain rows
+ array on PGlite.** A `.length > 0` check on it is always true in production and only correct in tests. Read
+ `RETURNING` rows through the query builder (`.returning().execute()`, then `raw`), as `claimNextSignals` does.
+- **PGlite runs every transaction on one shared session, so row locks do nothing in the API tests.** The claim
+ stays correct without them: the `UPDATE` keeps its `refId IS NULL AND status = 'PENDING'` guard, and the claims
+ are the rows it returned, not the rows the `SELECT … FOR UPDATE` saw.
+- **A stalled job is not a dead worker.** On BullMQ defaults the first stall redelivers the job to a second
+ dispatcher while the first still runs. The second stall fails it as an `UnrecoverableError` on the next pickup,
+ with `attemptsMade` unchanged, so `fanOutDispatchGaveUp` checks for both.
+- **A claim released after the give-up must settle itself.** When an enqueue fails, the dispatcher deletes the
+ child row and clears the claim so a later attempt can retry it. If the give-up sweep already ran, no attempt is
+ left, so the dispatcher then checks the source key and, if it is gone, marks the signal `NOT_DISPATCHED`. The
+ give-up handler deletes the key before its sweep, so one of the two always settles the signal.
+- **A chunk settles before the job fails.** `enqueueChunk` uses `Promise.allSettled`, so every release lands before
+ BullMQ fires `failed`. With `Promise.all` the sweep ran while the other tasks still claimed and released.
+- **Never widen a sweep to "all `PENDING`".** A dispatched signal stays `PENDING` with a `refId` until its child
+ ends, and a body that waits on an approval keeps it there for days. A status-only sweep marks live children
+ `NOT_DISPATCHED`, the barrier releases, and their outcomes report into nothing.
+- **A child claimed just before a release never starts.** Its enqueue needs the barrier `PENDING`, so the
+ dispatcher cancels it instead (`cancelUnstartedChildRun`). The release summary still counts it as `stillRunning`
+ with its run id, because its claim came first.
+- **The engine retries `POST /v1/waitpoints` on a 5xx, also after the API committed.** `create` keys the retry on
+ the existing row's status. `PENDING` is reused, and `create` stores the source and adds the dispatch job again;
+ the fixed job id `dispatch-<barrierId>` and the claims make that repeat safe. `COMPLETED` is reused with no
+ dispatch, because fast children can release the barrier before the retry lands, and its summary waits for the
+ deferred resume. Only `CONSUMED` is replaced. Deleting a `COMPLETED` barrier gave it fresh signals and ran every
+ batch twice.
+- **Run credits are checked once per chunk.** With none left, the dispatcher marks the unclaimed signals
+ `NOT_DISPATCHED`; children already claimed keep running.
+- **Each child carries its own copy of `seedSteps`**, inline below `WEBHOOK_PAYLOAD_INLINE_THRESHOLD_KB` and as a
+ file above it.
+- **The parent's run timeline no longer measures dispatch cost.** Its first leg is one HTTP call. Measure the span
+ of the children's `created` timestamps instead.
+- **`barrier-queue-factory.ts` is the module graph's acyclic leaf.** `barrierSourceKey` and the job types live
+ there so `waitpoint-service.ts` imports the key without the queue. Folding it into `barrier-queue.ts` adds an
+ import cycle.
+
+## Key files
+- `packages/server/api/src/app/waitpoints/` — `handleFanOutDispatch`, `barrierService.claimNextSignals`
+- `packages/server/api/src/app/flows/flow-run/` — `prepareChildDispatch`, `createChildRuns`, `enqueueChildRun`
+- `packages/server/engine/src/lib/operations/` — `flowOperation`, which enters a run at `entryStepName`
diff --git a/bun.lock b/bun.lock
index edeb7844494..d703a850322 100644
--- a/bun.lock
+++ b/bun.lock
@@ -122,7 +122,7 @@
},
"packages/core/execution": {
"name": "@activepieces/core-execution",
- "version": "0.27.0",
+ "version": "0.28.0",
"dependencies": {
"@activepieces/core-piece-types": "workspace:*",
"@activepieces/core-utils": "workspace:*",
diff --git a/docs/install/architecture/waitpoints.mdx b/docs/install/architecture/waitpoints.mdx
index 8fc46ba9dd1..720d7c80091 100644
--- a/docs/install/architecture/waitpoints.mdx
+++ b/docs/install/architecture/waitpoints.mdx
@@ -104,7 +104,7 @@ Duplicate callbacks are absorbed by the uniqueness constraint; the engine never
| Method | Path | Notes |
| --- | --- | --- |
-| `POST` | `/v1/waitpoints` | Engine-only. Creates a `PENDING` waitpoint, returns its resume URL. For `BARRIER`, also creates its signals. |
+| `POST` | `/v1/waitpoints` | Engine-only. Creates a `PENDING` waitpoint, returns its resume URL. For `BARRIER`, also creates the signals and starts the fan-out. |
| `ALL` | `/v1/flow-runs/:id/signals/:signalId/confirm` | Records one signal's outcome. `GET`/`HEAD` serve the page and never decide; only `POST` does. |
| `ALL` | `/v1/flow-runs/:id/waitpoints/:waitpointId` | Resume, async. |
| `ALL` | `/v1/flow-runs/:id/waitpoints/:waitpointId/sync` | Resume, sync. The HTTP response is whatever the flow produces after resuming. |
diff --git a/packages/core/execution/package.json b/packages/core/execution/package.json
index b7ec7e97a07..b80b463c0cf 100644
--- a/packages/core/execution/package.json
+++ b/packages/core/execution/package.json
@@ -1,6 +1,6 @@
{
"name": "@activepieces/core-execution",
- "version": "0.27.0",
+ "version": "0.28.0",
"type": "commonjs",
"main": "./dist/src/index.js",
"scripts": {
diff --git a/packages/core/execution/src/lib/engine/engine-operation.ts b/packages/core/execution/src/lib/engine/engine-operation.ts
index 94d446108ef..42f3f26fe65 100644
--- a/packages/core/execution/src/lib/engine/engine-operation.ts
+++ b/packages/core/execution/src/lib/engine/engine-operation.ts
@@ -119,6 +119,7 @@ type BaseExecuteFlowOperation<T extends ExecutionType> = BaseEngineOperation & {
stepNameToTest: string | null
sampleData?: Record<string, unknown>
logsFileId?: string
+ entryStepName?: string
}
export enum StreamStepProgress {
diff --git a/packages/core/execution/src/lib/engine/execution-errors.ts b/packages/core/execution/src/lib/engine/execution-errors.ts
index 50ca456f80f..789b7e4492f 100644
--- a/packages/core/execution/src/lib/engine/execution-errors.ts
+++ b/packages/core/execution/src/lib/engine/execution-errors.ts
@@ -136,6 +136,12 @@ export class WaitpointRejectedError extends ExecutionError {
}
}
+export class InvalidSeededStepsError extends ExecutionError {
+ constructor(entryStepName: string, cause?: unknown) {
+ super('InvalidSeededStepsError', formatMessage(`A run entered at step ${entryStepName} needs the steps it starts after as its payload`), ExecutionErrorType.USER, cause)
+ }
+}
+
export class EngineGenericError extends ExecutionError {
constructor(name: string, message: string, cause?: unknown) {
super(name, formatMessage(message), ExecutionErrorType.ENGINE, cause)
diff --git a/packages/core/execution/src/lib/flow-run/execution/execution-journal.ts b/packages/core/execution/src/lib/flow-run/execution/execution-journal.ts
index c4b318b0dec..1def2660139 100644
--- a/packages/core/execution/src/lib/flow-run/execution/execution-journal.ts
+++ b/packages/core/execution/src/lib/flow-run/execution/execution-journal.ts
@@ -145,8 +145,11 @@ export const executionJournal = {
if (step.type !== FlowActionType.LOOP_ON_ITEMS) continue
if (!step.output?.iterations) continue
- for (const iteration of step.output.iterations) {
- const nestedPath = this.getPathToStep(iteration, stepName, loopsIndexes, [...currentPath, [currentStepName, loopsIndexes[currentStepName]]])
+ const iterations = step.output.iterations
+ const iterationIndex = Math.max(Math.min(loopsIndexes[currentStepName] ?? 0, iterations.length - 1), 0)
+
+ for (const iteration of iterations) {
+ const nestedPath = this.getPathToStep(iteration, stepName, loopsIndexes, [...currentPath, [currentStepName, iterationIndex]])
if (nestedPath) return nestedPath
}
}
diff --git a/packages/core/execution/src/lib/flow-run/execution/execution-output.ts b/packages/core/execution/src/lib/flow-run/execution/execution-output.ts
index 4d756408ce6..fd26e38fa88 100755
--- a/packages/core/execution/src/lib/flow-run/execution/execution-output.ts
+++ b/packages/core/execution/src/lib/flow-run/execution/execution-output.ts
@@ -10,11 +10,13 @@ export enum ExecutionType {
export type ExecutionState = {
steps: Record<string, StepOutput>
tags: string[]
+ entryStepName?: string
}
export const ExecutionState = z.object({
steps: z.record(z.string(), z.unknown()),
tags: z.array(z.string()),
+ entryStepName: z.string().optional(),
})
export enum RunInternalErrorSource {
diff --git a/packages/core/execution/src/lib/flow-run/waitpoint/index.ts b/packages/core/execution/src/lib/flow-run/waitpoint/index.ts
index c4a57199f6f..a08d71f58d5 100644
--- a/packages/core/execution/src/lib/flow-run/waitpoint/index.ts
+++ b/packages/core/execution/src/lib/flow-run/waitpoint/index.ts
@@ -1,6 +1,7 @@
import { isNil } from '@activepieces/core-utils'
import { z } from 'zod'
import { RespondResponse } from '../execution/flow-execution'
+import { StepOutputStatus } from '../execution/step-output'
export function shouldReleaseBarrier({ policy, sealed, counts }: ShouldReleaseBarrierParams): boolean {
if (policy?.releaseOnFirstFailure === true && UNFAVOURABLE_SIGNAL_STATUSES.some((status) => countOf({ counts, status }) > 0)) {
@@ -42,8 +43,18 @@ export const BarrierPolicy = z.object({
})
export type BarrierPolicy = z.infer<typeof BarrierPolicy>
+export const BarrierFanOutSource = z.object({
+ entryStepName: z.string(),
+ batchStepName: z.string(),
+ batchSize: z.number().int().positive(),
+ items: z.array(z.unknown()),
+ seedSteps: z.record(z.string(), z.looseObject({ type: z.string(), status: z.enum(StepOutputStatus) })),
+})
+export type BarrierFanOutSource = z.infer<typeof BarrierFanOutSource>
+
export const CreateBarrierRequest = z.object({
policy: BarrierPolicy.optional(),
+ fanOut: BarrierFanOutSource.optional(),
signals: z.array(z.object({ label: z.string().optional() })).optional(),
})
export type CreateBarrierRequest = z.infer<typeof CreateBarrierRequest>
@@ -64,6 +75,7 @@ export type CreateWaitpointRequest = z.infer<typeof CreateWaitpointRequest>
export const BarrierCreatedState = z.object({
signalCount: z.number().int().nonnegative(),
+ batchSize: z.number().int().positive(),
signals: z.array(z.object({
label: z.string().nullable(),
confirmUrl: z.string(),
diff --git a/packages/core/execution/src/lib/workers/job-data.ts b/packages/core/execution/src/lib/workers/job-data.ts
index 76f2542f475..955d8ca00f6 100644
--- a/packages/core/execution/src/lib/workers/job-data.ts
+++ b/packages/core/execution/src/lib/workers/job-data.ts
@@ -139,6 +139,7 @@ const ExecuteFlowJobDataCommon = z.object({
export const BeginExecuteFlowJobData = ExecuteFlowJobDataCommon.extend({
executionType: z.literal(ExecutionType.BEGIN),
executeTrigger: z.boolean().optional(),
+ entryStepName: z.string().optional(),
})
export type BeginExecuteFlowJobData = z.infer<typeof BeginExecuteFlowJobData>
diff --git a/packages/core/execution/test/flow-run/execution/execution-journal.test.ts b/packages/core/execution/test/flow-run/execution/execution-journal.test.ts
index 26f293c8509..e2f8ff0d96c 100644
--- a/packages/core/execution/test/flow-run/execution/execution-journal.test.ts
+++ b/packages/core/execution/test/flow-run/execution/execution-journal.test.ts
@@ -93,6 +93,16 @@ describe('executionJournal.getPathToStep', () => {
const step5_1 = executionJournal.getPathToStep(steps, 'step5', { 'step2': 1, 'step4': 0 })
expect(step5_1).toEqual([['step2', 1], ['step4', 0]])
})
+
+ it('should clamp a pinned loop index to the iterations that exist', () => {
+ const steps: Record<string, StepOutput> = {
+ loop: createLoopWithIterations([{ inner: createCodeStep() }, { inner: createCodeStep() }]),
+ }
+
+ expect(executionJournal.getPathToStep(steps, 'inner', { loop: 13 })).toEqual([['loop', 1]])
+ expect(executionJournal.getPathToStep(steps, 'inner', {})).toEqual([['loop', 0]])
+ expect(executionJournal.getPathToStep(steps, 'inner', { loop: -1 })).toEqual([['loop', 0]])
+ })
})
describe('executionJournal.getStateAtPath', () => {
diff --git a/packages/core/execution/test/flow-run/waitpoint/barrier-fan-out-source.test.ts b/packages/core/execution/test/flow-run/waitpoint/barrier-fan-out-source.test.ts
new file mode 100644
index 00000000000..748bdf3391a
--- /dev/null
+++ b/packages/core/execution/test/flow-run/waitpoint/barrier-fan-out-source.test.ts
@@ -0,0 +1,23 @@
+import { describe, expect, it } from 'vitest'
+import { StepOutputStatus } from '../../../src/lib/flow-run/execution/step-output'
+import { BarrierFanOutSource } from '../../../src/lib/flow-run/waitpoint'
+
+describe('BarrierFanOutSource', () => {
+ const source = {
+ entryStepName: 'step_2',
+ batchStepName: 'batch_1',
+ batchSize: 2,
+ items: [1, 2, 3],
+ }
+
+ it('accepts seed steps that are step outputs and keeps their other fields', () => {
+ const seedSteps = { trigger: { type: 'EMPTY', status: StepOutputStatus.SUCCEEDED, input: {}, output: { id: 1 } } }
+
+ expect(BarrierFanOutSource.parse({ ...source, seedSteps }).seedSteps).toEqual(seedSteps)
+ })
+
+ it('refuses a seed step that is not a step output, so a bad seed fails at the barrier create instead of in every child', () => {
+ expect(BarrierFanOutSource.safeParse({ ...source, seedSteps: { trigger: 'not a step output' } }).success).toBe(false)
+ expect(BarrierFanOutSource.safeParse({ ...source, seedSteps: { trigger: { type: 'EMPTY', status: 'DONE' } } }).success).toBe(false)
+ })
+})
diff --git a/packages/server/api/src/app/database/redis/distributed-store-factory.ts b/packages/server/api/src/app/database/redis/distributed-store-factory.ts
index 640f3fffd34..bbdbdfadbb4 100644
--- a/packages/server/api/src/app/database/redis/distributed-store-factory.ts
+++ b/packages/server/api/src/app/database/redis/distributed-store-factory.ts
@@ -34,6 +34,11 @@ export const distributedStoreFactory = (getRedisClient: () => Promise<Redis>) =>
return JSON.parse(value) as T
},
+ async exists(key: string): Promise<boolean> {
+ const redisClient = await getRedisClient()
+ return await redisClient.exists(key) === 1
+ },
+
async putIfAbsent(key: string, value: unknown, ttlInSeconds: number): Promise<boolean> {
const serializedValue = JSON.stringify(value)
const redisClient = await getRedisClient()
diff --git a/packages/server/api/src/app/flows/flow-run/child-run-dispatch-service.ts b/packages/server/api/src/app/flows/flow-run/child-run-dispatch-service.ts
new file mode 100644
index 00000000000..dd9350dabb9
--- /dev/null
+++ b/packages/server/api/src/app/flows/flow-run/child-run-dispatch-service.ts
@@ -0,0 +1,133 @@
+import { ActivepiecesError, ErrorCode, FlowId, FlowRunId, FlowVersionId, isNil, omit, PlatformId, ProjectId, tryCatch } from '@activepieces/core-utils'
+import { ExecutionType, FlowRun, FlowRunStatus, flowStructureUtil, RunEnvironment, StreamStepProgress } from '@activepieces/shared'
+import { FastifyBaseLogger } from 'fastify'
+import { EntityManager } from 'typeorm'
+import { projectService } from '../../project/project-service'
+import { flowVersionService } from '../flow-version/flow-version.service'
+import { addToQueue, cancelSingleRun, flowRunRepo } from './flow-run-service'
+import { flowRunSideEffects } from './flow-run-side-effects'
+
+export const childRunDispatchService = (log: FastifyBaseLogger) => ({
+ async prepareChildDispatch({ projectId, parentRunId, entryStepName }: PrepareChildDispatchParams): Promise<ChildDispatchTarget> {
+ const parentRun = await flowRunRepo().findOneBy({ id: parentRunId, projectId })
+ if (isNil(parentRun)) {
+ throw new ActivepiecesError({
+ code: ErrorCode.VALIDATION,
+ params: { message: `The parent run ${parentRunId} does not exist in this project, so no child can be dispatched for it.` },
+ })
+ }
+ const flowVersion = await flowVersionService(log).getOneOrThrow(parentRun.flowVersionId)
+ const entryStep = flowStructureUtil.getStep(entryStepName, flowVersion.trigger)
+ if (isNil(entryStep) || !flowStructureUtil.isAction(entryStep.type)) {
+ throw new ActivepiecesError({
+ code: ErrorCode.VALIDATION,
+ params: { message: `The step ${entryStepName} is not an action in this flow version, so a child run cannot start from it.` },
+ })
+ }
+ return {
+ projectId,
+ parentRunId,
+ entryStepName,
+ flowId: parentRun.flowId,
+ flowVersionId: parentRun.flowVersionId,
+ environment: parentRun.environment,
+ platformId: await projectService(log).getPlatformId(projectId),
+ }
+ },
+
+ async createChildRuns({ target, parentWaitpointId, children, entityManager }: CreateChildRunsParams): Promise<FlowRun[]> {
+ const now = new Date().toISOString()
+ const childRuns: FlowRun[] = children.map(({ id, dispatchIndex }) => ({
+ id,
+ projectId: target.projectId,
+ flowId: target.flowId,
+ flowVersionId: target.flowVersionId,
+ environment: target.environment,
+ parentRunId: target.parentRunId,
+ parentWaitpointId,
+ dispatchIndex,
+ failParentOnFailure: false,
+ status: FlowRunStatus.QUEUED,
+ created: now,
+ updated: now,
+ tags: [],
+ steps: {},
+ }))
+ if (childRuns.length > 0) {
+ await flowRunRepo(entityManager).insert(childRuns.map((childRun) => omit(childRun, ['steps'])))
+ }
+ return childRuns
+ },
+
+ async enqueueChildRun({ target, childRun, seedSteps }: EnqueueChildRunParams): Promise<void> {
+ await addToQueue({
+ flowRun: childRun,
+ platformId: target.platformId,
+ payload: seedSteps,
+ entryStepName: target.entryStepName,
+ preferInlinePayload: true,
+ executionType: ExecutionType.BEGIN,
+ executeTrigger: false,
+ workerHandlerId: undefined,
+ httpRequestId: undefined,
+ streamStepProgress: StreamStepProgress.NONE,
+ }, log)
+ },
+
+ async reportChildRunStarted({ target, childRun }: ChildRunParams): Promise<void> {
+ const logFields = { flowRun: { id: childRun.id, dispatchIndex: childRun.dispatchIndex }, project: { id: target.projectId }, waitpoint: { id: childRun.parentWaitpointId } }
+ const { error } = await tryCatch(() => flowRunSideEffects(log).onStart({ flowRun: childRun, platformId: target.platformId }))
+ if (error) {
+ log.error({ ...logFields, error }, '[childRunDispatchService#reportChildRunStarted] Child run was enqueued but its start side effects failed')
+ }
+ log.info(logFields, '[childRunDispatchService#reportChildRunStarted] Child run dispatched')
+ },
+
+ async cancelUnstartedChildRun({ target, childRun }: ChildRunParams): Promise<void> {
+ await cancelSingleRun(log, childRun, target.platformId)
+ },
+
+ async deleteChildRun({ id, projectId, entityManager }: DeleteChildRunParams): Promise<void> {
+ await flowRunRepo(entityManager).delete({ id, projectId })
+ },
+})
+
+type PrepareChildDispatchParams = {
+ projectId: ProjectId
+ parentRunId: FlowRunId
+ entryStepName: string
+}
+
+type CreateChildRunsParams = {
+ target: ChildDispatchTarget
+ parentWaitpointId: string
+ children: { id: FlowRunId, dispatchIndex: number }[]
+ entityManager: EntityManager
+}
+
+type EnqueueChildRunParams = {
+ target: ChildDispatchTarget
+ childRun: FlowRun
+ seedSteps: Record<string, unknown>
+}
+
+type ChildRunParams = {
+ target: ChildDispatchTarget
+ childRun: FlowRun
+}
+
+type DeleteChildRunParams = {
+ id: FlowRunId
+ projectId: ProjectId
+ entityManager: EntityManager
+}
+
+export type ChildDispatchTarget = {
+ projectId: ProjectId
+ parentRunId: FlowRunId
+ entryStepName: string
+ flowId: FlowId
+ flowVersionId: FlowVersionId
+ environment: RunEnvironment
+ platformId: PlatformId
+}
diff --git a/packages/server/api/src/app/flows/flow-run/flow-run-entity.ts b/packages/server/api/src/app/flows/flow-run/flow-run-entity.ts
index f4b0aa9f616..5262f4814b4 100644
--- a/packages/server/api/src/app/flows/flow-run/flow-run-entity.ts
+++ b/packages/server/api/src/app/flows/flow-run/flow-run-entity.ts
@@ -12,7 +12,7 @@ import {
BaseColumnSchemaPart,
} from '../../database/database-common'
-type FlowRunSchema = FlowRunWithDispatchIndex & {
+type FlowRunSchema = FlowRun & {
project: Project
flow: Flow
flowVersion: FlowVersion
@@ -217,7 +217,3 @@ export const FlowRunEntity = new EntitySchema<FlowRunSchema>({
},
},
})
-
-export type FlowRunWithDispatchIndex = FlowRun & {
- dispatchIndex?: number | null
-}
diff --git a/packages/server/api/src/app/flows/flow-run/flow-run-service.ts b/packages/server/api/src/app/flows/flow-run/flow-run-service.ts
index 86ca41c6a0f..5d54475e915 100644
--- a/packages/server/api/src/app/flows/flow-run/flow-run-service.ts
+++ b/packages/server/api/src/app/flows/flow-run/flow-run-service.ts
@@ -521,7 +521,7 @@ export const flowRunService = (log: FastifyBaseLogger) => ({
})
-async function cancelSingleRun(log: FastifyBaseLogger, flowRun: FlowRun, platformId: string): Promise<void> {
+export async function cancelSingleRun(log: FastifyBaseLogger, flowRun: FlowRun, platformId: string): Promise<void> {
await distributedLock(log).runExclusive({
key: `runs_metadata_${flowRun.id}`,
timeoutInSeconds: 30,
@@ -645,7 +645,7 @@ export async function addToQueue(params: AddToQueueParams, log: FastifyBaseLogge
const logsFileId = params.flowRun.logsFileId ?? apId()
let jobPayload: JobPayload = { type: 'inline', value: null }
- if (!isNil(params.payload) && isNil(params.workerHandlerId)) {
+ if (!isNil(params.payload) && isNil(params.workerHandlerId) && !(params.preferInlinePayload ?? false)) {
jobPayload = await payloadOffloader.offloadPayload(log, params.payload, params.flowRun.projectId, params.platformId)
}
else if (!isNil(params.payload)) {
@@ -680,6 +680,7 @@ export async function addToQueue(params: AddToQueueParams, log: FastifyBaseLogge
...commonJobData,
executionType: ExecutionType.BEGIN,
executeTrigger: params.executeTrigger,
+ entryStepName: params.entryStepName,
}
const aParentIsBlockedOnThisRun = params.flowRun.failParentOnFailure && !isNil(params.flowRun.parentRunId)
await jobQueue(log).add({
@@ -872,10 +873,11 @@ type AddToQueueParamsCommon = {
streamStepProgress: StreamStepProgress
sampleData?: Record<string, unknown>
jobId?: string
+ preferInlinePayload?: boolean
}
export type AddToQueueParams = AddToQueueParamsCommon & (
- | { executionType: ExecutionType.BEGIN, executeTrigger: boolean }
+ | { executionType: ExecutionType.BEGIN, executeTrigger: boolean, entryStepName?: string }
| { executionType: ExecutionType.RESUME, resumeReason: ResumeReason }
)
diff --git a/packages/server/api/src/app/waitpoints/barrier-queue-factory.ts b/packages/server/api/src/app/waitpoints/barrier-queue-factory.ts
index d357b8d2f76..8c8b2e935b6 100644
--- a/packages/server/api/src/app/waitpoints/barrier-queue-factory.ts
+++ b/packages/server/api/src/app/waitpoints/barrier-queue-factory.ts
@@ -3,6 +3,8 @@ import { Queue } from 'bullmq'
import Redis from 'ioredis'
import { QueueName } from '../workers/job'
+export const barrierSourceKey = (barrierId: string): string => `barrier_source:${barrierId}`
+
export const evaluationDedupKey = (barrierId: string): string => `evaluate-${barrierId}`
export const barrierQueueFactory = ({ createRedisConnection }: BarrierQueueFactoryParams) => {
@@ -41,6 +43,18 @@ export const barrierQueueFactory = ({ createRedisConnection }: BarrierQueueFacto
})
},
+ async addFanOutDispatch(params: BarrierJobData): Promise<void> {
+ await requireQueue().add(BarrierJobName.FAN_OUT_DISPATCH, params, {
+ jobId: `dispatch-${params.barrierId}`,
+ })
+ },
+
+ async addFanOutGiveUp(params: BarrierJobData): Promise<void> {
+ await requireQueue().add(BarrierJobName.FAN_OUT_GIVE_UP, params, {
+ jobId: `give-up-${params.barrierId}`,
+ })
+ },
+
async clearEvaluationDedupKey(barrierId: string): Promise<void> {
await requireQueue().removeDeduplicationKey(evaluationDedupKey(barrierId))
},
@@ -57,6 +71,8 @@ export const barrierQueueFactory = ({ createRedisConnection }: BarrierQueueFacto
export enum BarrierJobName {
EVALUATE = 'evaluate-barrier',
+ FAN_OUT_DISPATCH = 'fan-out-dispatch',
+ FAN_OUT_GIVE_UP = 'fan-out-give-up',
}
export type BarrierJobData = {
@@ -69,6 +85,13 @@ export type BarrierQueueConfig = {
redisFailedJobRetentionMaxCount: number
}
+export type BarrierFanOutPayload = {
+ entryStepName: string
+ batchStepName: string
+ seedSteps: Record<string, unknown>
+ batches: unknown[][]
+}
+
type BarrierQueueFactoryParams = {
createRedisConnection: () => Promise<Redis>
}
diff --git a/packages/server/api/src/app/waitpoints/barrier-queue.ts b/packages/server/api/src/app/waitpoints/barrier-queue.ts
index 59354d1ddfe..2c4ad15f293 100644
--- a/packages/server/api/src/app/waitpoints/barrier-queue.ts
+++ b/packages/server/api/src/app/waitpoints/barrier-queue.ts
@@ -1,12 +1,15 @@
+import { isNil } from '@activepieces/core-utils'
import { createLogger, wideEvent } from '@activepieces/server-utils'
import { Job, Queue, Worker } from 'bullmq'
import { FastifyBaseLogger } from 'fastify'
import { redisConnections } from '../database/redis-connections'
+import { exceptionHandler } from '../helper/exception-handler'
import { system } from '../helper/system/system'
import { AppSystemProp } from '../helper/system/system-props'
import { QueueName } from '../workers/job'
import { BarrierJobData, BarrierJobName, BarrierQueueConfig, barrierQueueFactory } from './barrier-queue-factory'
import { barrierService } from './barrier-service'
+import { handleFanOutDispatch, handleFanOutDispatchFailed, handleFanOutDispatchGaveUp } from './fan-out-dispatcher-job'
let barrierWorker: Worker<BarrierJobData> | undefined = undefined
@@ -43,6 +46,15 @@ export const barrierQueue = (log: FastifyBaseLogger) => ({
autorun: true,
},
)
+ barrierWorker.on('failed', (job, error) => {
+ if (isNil(job) || job.name !== BarrierJobName.FAN_OUT_DISPATCH) {
+ return
+ }
+ handleFanOutDispatchFailed({ data: job.data, attemptsMade: job.attemptsMade, attempts: job.opts.attempts, error, log }).catch((handlerError: unknown) => {
+ exceptionHandler.handle(handlerError, log)
+ })
+ })
+
await barrierWorker.waitUntilReady()
},
@@ -50,6 +62,14 @@ export const barrierQueue = (log: FastifyBaseLogger) => ({
await barrierQueueInstance.enqueueEvaluation(params)
},
+ async addFanOutDispatch(params: BarrierJobData): Promise<void> {
+ await barrierQueueInstance.addFanOutDispatch(params)
+ },
+
+ async addFanOutGiveUp(params: BarrierJobData): Promise<void> {
+ await barrierQueueInstance.addFanOutGiveUp(params)
+ },
+
async clearEvaluationDedupKey(barrierId: string): Promise<void> {
await barrierQueueInstance.clearEvaluationDedupKey(barrierId)
},
@@ -74,6 +94,12 @@ async function processBarrierJob({ job, log }: ProcessBarrierJobParams): Promise
await barrierQueueInstance.clearEvaluationDedupKey(job.data.barrierId)
await barrierService(log).releaseIfReady({ barrierId: job.data.barrierId, projectId: job.data.projectId })
return
+ case BarrierJobName.FAN_OUT_DISPATCH:
+ await handleFanOutDispatch({ data: job.data, log })
+ return
+ case BarrierJobName.FAN_OUT_GIVE_UP:
+ await handleFanOutDispatchGaveUp({ data: job.data, log })
+ return
default:
log.warn({ job: { id: job.id, type: job.name } }, '[barrierQueue#worker] Unknown barrier job name, dropping it')
}
diff --git a/packages/server/api/src/app/waitpoints/barrier-service.ts b/packages/server/api/src/app/waitpoints/barrier-service.ts
index 5ed9cfabbbc..0285370425c 100644
--- a/packages/server/api/src/app/waitpoints/barrier-service.ts
+++ b/packages/server/api/src/app/waitpoints/barrier-service.ts
@@ -1,17 +1,19 @@
-import { apId, chunk, isNil, sanitizeObjectForPostgresql, tryCatch, unique } from '@activepieces/core-utils'
-import { wideEvent } from '@activepieces/server-utils'
+import { apId, chunk, isNil, sanitizeObjectForPostgresql, spreadIfDefined, tryCatch, unique } from '@activepieces/core-utils'
+import { apDayjsDuration, wideEvent } from '@activepieces/server-utils'
import { ActivepiecesError, BarrierPolicy, barrierReleasesOnLastPendingSignal, BarrierSignalCounts, BarrierSignalStatus, BarrierSummary, ErrorCode, MAX_INLINE_BARRIER_SIGNALS, PauseType, RespondResponse, shouldReleaseBarrier, WaitpointVersion } from '@activepieces/shared'
import dayjs from 'dayjs'
import { FastifyBaseLogger } from 'fastify'
-import { EntityManager } from 'typeorm'
+import { EntityManager, FindOptionsWhere, In, IsNull } from 'typeorm'
import { z } from 'zod'
import { repoFactory } from '../core/db/repo-factory'
import { transaction } from '../core/db/transaction'
+import { distributedStore } from '../database/redis-connections'
import { flowRunRepo } from '../flows/flow-run/flow-run-service'
import { system } from '../helper/system/system'
import { AppSystemProp } from '../helper/system/system-props'
import { platformConfigurationService } from '../platform/platform-configuration.service'
import { barrierQueue } from './barrier-queue'
+import { BarrierFanOutPayload, barrierSourceKey } from './barrier-queue-factory'
import { resumeService } from './resume-service'
import { WaitpointEntity } from './waitpoint-entity'
import { WaitpointSignalEntity } from './waitpoint-signal-entity'
@@ -23,15 +25,19 @@ const signalRepo = repoFactory(WaitpointSignalEntity)
export const barrierService = (log: FastifyBaseLogger) => ({
async create(params: CreateBarrierParams): Promise<CreateBarrierResult> {
+ const maxSignals = await platformConfigurationService(log).maxBarrierSignals({ platformId: params.platformId })
+ const batchSize = clampBatchSize({ itemCount: params.fanOut?.items.length ?? 0, requested: params.fanOut?.batchSize ?? 1, maxSignals })
+ const batches = isNil(params.fanOut) ? [] : chunk(params.fanOut.items, batchSize)
const labels = params.signalLabels ?? []
- await assertSignalCountWithinLimit({ signalCount: labels.length, platformId: params.platformId, log })
+ const signalCount = isNil(params.fanOut) ? labels.length : batches.length
+ assertSignalCountWithinLimit({ signalCount, maxSignals })
const flowRun = await flowRunRepo().findOneByOrFail({ id: params.flowRunId, projectId: params.projectId })
const creation = await transaction(async (entityManager) => {
const repo = waitpointRepo(entityManager)
const existing = await repo.findOneBy({ flowRunId: params.flowRunId, projectId: params.projectId, stepName: params.stepName })
- if (!isNil(existing) && existing.status === WaitpointStatus.PENDING) {
- log.info({ flowRun: { id: params.flowRunId }, waitpoint: { id: existing.id } }, '[barrierService#create] Barrier already open for this step, reusing it')
+ if (!isNil(existing) && existing.status !== WaitpointStatus.CONSUMED) {
+ log.info({ flowRun: { id: params.flowRunId }, waitpoint: { id: existing.id, status: existing.status } }, '[barrierService#create] Barrier already exists for this step and its resume is not delivered, reusing it')
return { inserted: false, barrier: existing, signals: await signalRepo(entityManager).findBy({ waitpointId: existing.id, projectId: params.projectId }) }
}
if (!isNil(existing)) {
@@ -58,13 +64,21 @@ export const barrierService = (log: FastifyBaseLogger) => ({
deadLetteredAt: null,
}
await repo.createQueryBuilder().insert().into('waitpoint').values(barrier).execute()
- const signals = buildPendingSignals({ barrierId: barrier.id, projectId: params.projectId, labels })
+ const signals = buildPendingSignals({ barrierId: barrier.id, projectId: params.projectId, signalCount, labels, fanOut: !isNil(params.fanOut) })
for (const rows of chunk(signals, SIGNAL_INSERT_BATCH_SIZE)) {
await signalRepo(entityManager).createQueryBuilder().insert().into('waitpoint_signal').values(rows).execute()
}
return { inserted: true, barrier, signals }
})
+ if (!isNil(params.fanOut) && creation.barrier.status === WaitpointStatus.PENDING) {
+ await startFanOut({
+ barrierId: creation.barrier.id,
+ projectId: params.projectId,
+ source: { entryStepName: params.fanOut.entryStepName, batchStepName: params.fanOut.batchStepName, seedSteps: params.fanOut.seedSteps, batches },
+ log,
+ })
+ }
if (creation.inserted) {
if (!isNil(creation.barrier.resumeDateTime)) {
await waitpointTimeoutJob.schedule({
@@ -78,7 +92,7 @@ export const barrierService = (log: FastifyBaseLogger) => ({
await barrierQueue(log).enqueueEvaluation({ barrierId: creation.barrier.id, projectId: params.projectId })
}
- return { barrier: creation.barrier, signals: creation.signals, signalCount: creation.signals.length }
+ return { barrier: creation.barrier, signals: creation.signals, signalCount: creation.signals.length, batchSize }
},
async findById({ barrierId, projectId }: FindByIdParams): Promise<Waitpoint | null> {
@@ -89,23 +103,70 @@ export const barrierService = (log: FastifyBaseLogger) => ({
return signalRepo().findOneBy({ id: signalId, projectId })
},
+ async hasUnclaimedSignals({ barrierId, projectId }: BarrierScopeParams): Promise<boolean> {
+ return signalRepo().existsBy(unclaimedSignalWhere({ barrierId, projectId }))
+ },
+
+ async claimNextSignals({ barrierId, projectId, afterSequence, limit, entityManager }: ClaimNextSignalsParams): Promise<SignalClaimResult> {
+ const openBarrier = await lockOpenBarrier({ barrierId, projectId, entityManager })
+ if (isNil(openBarrier)) {
+ return { barrierOpen: false }
+ }
+ const candidates = ClaimCandidateRows.parse(await signalRepo(entityManager).query(
+ 'SELECT "id", "sequence" FROM waitpoint_signal WHERE "waitpointId" = $1 AND "projectId" = $2 AND "status" = $3 AND "refId" IS NULL AND "sequence" > $4 ORDER BY "sequence" LIMIT $5 FOR UPDATE SKIP LOCKED',
+ [barrierId, projectId, BarrierSignalStatus.PENDING, afterSequence, limit],
+ ))
+ if (candidates.length === 0) {
+ return { barrierOpen: true, claims: [], lastSequence: null }
+ }
+ const childRunIds = candidates.map(() => apId())
+ const claimed = await signalRepo(entityManager)
+ .createQueryBuilder()
+ .update()
+ .set({ refId: () => `CASE "id" ${candidates.map((_, index) => `WHEN :signalId${index} THEN :childRunId${index}`).join(' ')} END` })
+ .setParameters(Object.fromEntries(candidates.flatMap((candidate, index) => [[`signalId${index}`, candidate.id], [`childRunId${index}`, childRunIds[index]]])))
+ .where({ id: In(candidates.map((candidate) => candidate.id)), projectId, status: BarrierSignalStatus.PENDING, refId: IsNull() })
+ .returning(['id', 'sequence', 'refId'])
+ .execute()
+ const claims = ClaimedSignalRows.parse(claimed.raw ?? []).map((row) => ({ signalId: row.id, sequence: row.sequence, childRunId: row.refId }))
+ return { barrierOpen: true, claims, lastSequence: candidates[candidates.length - 1].sequence }
+ },
+
+ async lockForDispatch({ barrierId, projectId, entityManager }: LockForDispatchParams): Promise<boolean> {
+ return !isNil(await lockOpenBarrier({ barrierId, projectId, entityManager }))
+ },
+
+ async releaseClaim({ signalId, refId, projectId, entityManager }: ReleaseClaimParams): Promise<void> {
+ await signalRepo(entityManager).update({ id: signalId, refId, projectId }, { refId: null })
+ },
+
+ async markUnclaimedNotDispatched({ barrierId, projectId }: BarrierScopeParams): Promise<number> {
+ const result = await signalRepo().update(unclaimedSignalWhere({ barrierId, projectId }), { status: BarrierSignalStatus.NOT_DISPATCHED })
+ return result.affected ?? 0
+ },
+
async receiveSignal({ signalId, refId, projectId, status, result }: ReceiveSignalParams): Promise<boolean> {
if (isNil(signalId) && isNil(refId)) {
return false
}
const identity: SignalIdentity = isNil(signalId) ? { refId } : { id: signalId }
const barrierIds = await updateSignalOutcome({ where: { ...identity, projectId }, status, result })
+ if (barrierIds.length === 0) {
+ log.warn({
+ ...spreadIfDefined('flowRun', isNil(refId) ? undefined : { id: refId }),
+ ...spreadIfDefined('signal', isNil(signalId) ? undefined : { id: signalId }),
+ project: { id: projectId },
+ outcome: status,
+ }, '[barrierService#receiveSignal] No signal row matched this outcome, so it is being dropped; the barrier was released or the row was never claimed by this ref')
+ return false
+ }
await Promise.all(barrierIds.map((barrierId) => barrierQueue(log).enqueueEvaluation({ barrierId, projectId })))
- return barrierIds.length > 0
+ return true
},
async recordDecision({ barrierId, signalId, projectId, status, result }: RecordDecisionParams): Promise<boolean> {
const recorded = await transaction(async (entityManager) => {
- const openBarrier = await waitpointRepo(entityManager)
- .createQueryBuilder('waitpoint')
- .setLock('pessimistic_read')
- .where({ id: barrierId, projectId, type: PauseType.BARRIER, status: WaitpointStatus.PENDING })
- .getOne()
+ const openBarrier = await lockOpenBarrier({ barrierId, projectId, entityManager })
if (isNil(openBarrier)) {
return false
}
@@ -144,6 +205,7 @@ export const barrierService = (log: FastifyBaseLogger) => ({
async release({ barrier, timedOut, releaseReason }: ReleaseParams): Promise<BarrierSummary | null> {
const summary = await closeBarrier({ barrier, timedOut })
+ await distributedStore.delete(barrierSourceKey(barrier.id))
if (isNil(summary)) {
log.info({ waitpoint: { id: barrier.id }, flowRun: { id: barrier.flowRunId } }, '[barrierService#release] Barrier was already closed by another release; leaving the resume to the release that closed it')
return null
@@ -180,9 +242,9 @@ async function updateSignalOutcome({ where, status, result, entityManager }: Upd
return unique(UpdatedSignalRows.parse(updateResult.raw ?? []).map((row) => row.waitpointId))
}
-function buildPendingSignals({ barrierId, projectId, labels }: BuildPendingSignalsParams): WaitpointSignal[] {
+function buildPendingSignals({ barrierId, projectId, signalCount, labels, fanOut }: BuildPendingSignalsParams): WaitpointSignal[] {
const now = new Date().toISOString()
- return labels.map((label) => ({
+ return Array.from({ length: signalCount }, (_, index) => ({
id: apId(),
created: now,
updated: now,
@@ -190,12 +252,29 @@ function buildPendingSignals({ barrierId, projectId, labels }: BuildPendingSigna
projectId,
status: BarrierSignalStatus.PENDING,
refId: null,
- sequence: null,
- label,
+ sequence: fanOut ? index : null,
+ label: labels[index] ?? null,
result: null,
}))
}
+function unclaimedSignalWhere({ barrierId, projectId }: BarrierScopeParams): FindOptionsWhere<WaitpointSignal> {
+ return { waitpointId: barrierId, projectId, status: BarrierSignalStatus.PENDING, refId: IsNull() }
+}
+
+async function lockOpenBarrier({ barrierId, projectId, entityManager }: LockForDispatchParams): Promise<Waitpoint | null> {
+ return waitpointRepo(entityManager)
+ .createQueryBuilder('waitpoint')
+ .setLock('pessimistic_read')
+ .where({ id: barrierId, projectId, type: PauseType.BARRIER, status: WaitpointStatus.PENDING })
+ .getOne()
+}
+
+async function startFanOut({ barrierId, projectId, source, log }: StartFanOutParams): Promise<void> {
+ await distributedStore.put(barrierSourceKey(barrierId), source, SOURCE_TTL_SECONDS)
+ await barrierQueue(log).addFanOutDispatch({ barrierId, projectId })
+}
+
async function closeBarrier({ barrier, timedOut }: CloseBarrierParams): Promise<BarrierSummary | null> {
return transaction(async (entityManager) => {
const repo = waitpointRepo(entityManager)
@@ -282,8 +361,12 @@ async function buildSummary({ barrierId, projectId, timedOut, entityManager }: B
}
}
-async function assertSignalCountWithinLimit({ signalCount, platformId, log }: AssertSignalCountWithinLimitParams): Promise<void> {
- const maxSignals = await platformConfigurationService(log).maxBarrierSignals({ platformId })
+function clampBatchSize({ itemCount, requested, maxSignals }: ClampBatchSizeParams): number {
+ const minimumBatchSize = Math.ceil(itemCount / maxSignals)
+ return Math.max(requested, minimumBatchSize, 1)
+}
+
+function assertSignalCountWithinLimit({ signalCount, maxSignals }: AssertSignalCountWithinLimitParams): void {
if (signalCount > maxSignals) {
throw new ActivepiecesError({
code: ErrorCode.VALIDATION,
@@ -301,6 +384,12 @@ const SIGNAL_INSERT_BATCH_SIZE = 500
const UpdatedSignalRows = z.array(z.object({ waitpointId: z.string() }))
+const ClaimCandidateRows = z.array(z.object({ id: z.string(), sequence: z.number().int() }))
+
+const ClaimedSignalRows = z.array(z.object({ id: z.string(), sequence: z.number().int(), refId: z.string() }))
+
+const SOURCE_TTL_SECONDS = apDayjsDuration(1, 'day').asSeconds()
+
export type BarrierReleaseReason = 'predicate' | 'timeout'
export type CreateBarrierParams = {
@@ -314,12 +403,20 @@ export type CreateBarrierParams = {
httpRequestId?: string
policy?: BarrierPolicy
signalLabels?: (string | null)[]
+ fanOut?: {
+ entryStepName: string
+ batchStepName: string
+ batchSize: number
+ items: unknown[]
+ seedSteps: Record<string, unknown>
+ }
}
export type CreateBarrierResult = {
barrier: Waitpoint
signals: WaitpointSignal[]
signalCount: number
+ batchSize: number
}
export type ReceiveSignalParams = {
@@ -338,6 +435,16 @@ export type RecordDecisionParams = {
result?: Record<string, unknown>
}
+export type SignalClaim = {
+ signalId: string
+ sequence: number
+ childRunId: string
+}
+
+export type SignalClaimResult =
+ | { barrierOpen: false }
+ | { barrierOpen: true, claims: SignalClaim[], lastSequence: number | null }
+
type SignalIdentity = {
id?: string
refId?: string
@@ -359,6 +466,35 @@ type FindSignalByIdParams = {
projectId: string
}
+type BarrierScopeParams = {
+ barrierId: string
+ projectId: string
+}
+
+type ClaimNextSignalsParams = BarrierScopeParams & {
+ afterSequence: number
+ limit: number
+ entityManager: EntityManager
+}
+
+type LockForDispatchParams = BarrierScopeParams & {
+ entityManager: EntityManager
+}
+
+type ReleaseClaimParams = {
+ signalId: string
+ refId: string
+ projectId: string
+ entityManager?: EntityManager
+}
+
+type StartFanOutParams = {
+ barrierId: string
+ projectId: string
+ source: BarrierFanOutPayload
+ log: FastifyBaseLogger
+}
+
type ReleaseIfReadyParams = {
barrierId: string
projectId: string
@@ -403,13 +539,20 @@ type BuildSummaryParams = CountSignalsByStatusParams & {
type BuildPendingSignalsParams = {
barrierId: string
projectId: string
+ signalCount: number
labels: (string | null)[]
+ fanOut: boolean
+}
+
+type ClampBatchSizeParams = {
+ itemCount: number
+ requested: number
+ maxSignals: number
}
type AssertSignalCountWithinLimitParams = {
signalCount: number
- platformId: string
- log: FastifyBaseLogger
+ maxSignals: number
}
type DefaultBarrierDeadlineParams = {
diff --git a/packages/server/api/src/app/waitpoints/fan-out-dispatcher-job.ts b/packages/server/api/src/app/waitpoints/fan-out-dispatcher-job.ts
new file mode 100644
index 00000000000..2d8094b5ffb
--- /dev/null
+++ b/packages/server/api/src/app/waitpoints/fan-out-dispatcher-job.ts
@@ -0,0 +1,226 @@
+import { isNil, tryCatch } from '@activepieces/core-utils'
+import { FlowRun } from '@activepieces/shared'
+import { UnrecoverableError } from 'bullmq'
+import { FastifyBaseLogger } from 'fastify'
+import pLimit from 'p-limit'
+import { transaction } from '../core/db/transaction'
+import { distributedStore } from '../database/redis-connections'
+import { ChildDispatchTarget, childRunDispatchService } from '../flows/flow-run/child-run-dispatch-service'
+
+import { shouldBlockRunOnCredits } from '../platform/billing-provider'
+import { barrierQueue } from './barrier-queue'
+import { BarrierFanOutPayload, BarrierJobData, barrierSourceKey } from './barrier-queue-factory'
+import { barrierService, SignalClaim } from './barrier-service'
+import { WaitpointStatus } from './waitpoint-types'
+
+export async function handleFanOutDispatch({ data, log }: JobParams): Promise<void> {
+ const barrier = await barrierService(log).findById({ barrierId: data.barrierId, projectId: data.projectId })
+ if (isNil(barrier) || barrier.status !== WaitpointStatus.PENDING) {
+ log.info({ waitpoint: { id: data.barrierId } }, '[fanOutDispatcher] Barrier is gone or already released, stopping without dispatching')
+ return
+ }
+
+ const source = await distributedStore.get<BarrierFanOutPayload>(barrierSourceKey(data.barrierId))
+ if (isNil(source)) {
+ const notDispatchedCount = await settleUnclaimedSignals({ data, log })
+ log.warn({ waitpoint: { id: data.barrierId }, notDispatchedCount }, '[fanOutDispatcher] The stored source is gone, marking the unclaimed signals undispatched so the barrier can settle')
+ return
+ }
+
+ if (!await barrierService(log).hasUnclaimedSignals({ barrierId: data.barrierId, projectId: data.projectId })) {
+ await barrierQueue(log).enqueueEvaluation({ barrierId: data.barrierId, projectId: data.projectId })
+ return
+ }
+
+ const target = await childRunDispatchService(log).prepareChildDispatch({
+ projectId: data.projectId,
+ parentRunId: barrier.flowRunId,
+ entryStepName: source.entryStepName,
+ })
+ const outcome = await dispatchChunks({ data, source, target, log })
+ switch (outcome) {
+ case 'barrier-closed':
+ log.info({ waitpoint: { id: data.barrierId } }, '[fanOutDispatcher] The barrier was released or cancelled mid-fan-out, stopping without retrying')
+ return
+ case 'credits-exhausted': {
+ const notDispatchedCount = await settleUnclaimedSignals({ data, log })
+ log.warn({ waitpoint: { id: data.barrierId }, platform: { id: target.platformId }, notDispatchedCount }, '[fanOutDispatcher] The platform ran out of run credits, marking the unclaimed signals undispatched')
+ return
+ }
+ case 'dispatched':
+ await barrierQueue(log).enqueueEvaluation({ barrierId: data.barrierId, projectId: data.projectId })
+ }
+}
+
+export function fanOutDispatchGaveUp({ attemptsMade, attempts, error }: FanOutDispatchGaveUpParams): GiveUpReason | null {
+ if (error instanceof UnrecoverableError) {
+ return 'unrecoverable'
+ }
+ if (attemptsMade >= (attempts ?? 1)) {
+ return 'exhausted'
+ }
+ return null
+}
+
+export async function handleFanOutDispatchFailed({ data, attemptsMade, attempts, error, log }: HandleFanOutDispatchFailedParams): Promise<void> {
+ const reason = fanOutDispatchGaveUp({ attemptsMade, attempts, error })
+ if (isNil(reason)) {
+ return
+ }
+ log.error({ error, waitpoint: { id: data.barrierId }, giveUpReason: reason }, '[fanOutDispatcher] Dispatch gave up, queueing the settlement of its unclaimed signals')
+ await barrierQueue(log).addFanOutGiveUp(data)
+}
+
+export async function handleFanOutDispatchGaveUp({ data, log }: JobParams): Promise<void> {
+ await distributedStore.delete(barrierSourceKey(data.barrierId))
+ const notDispatchedCount = await settleUnclaimedSignals({ data, log })
+ log.warn({ waitpoint: { id: data.barrierId }, notDispatchedCount }, '[fanOutDispatcher] Marked the unclaimed signals undispatched after dispatch gave up; signals already claimed keep their live children and release the barrier once those report')
+}
+
+async function dispatchChunks({ data, source, target, log }: DispatchChunksParams): Promise<DispatchOutcome> {
+ let afterSequence: number | null = START_OF_BARRIER
+ while (!isNil(afterSequence)) {
+ if (await shouldBlockRunOnCredits({ platformId: target.platformId, environment: target.environment, log })) {
+ return 'credits-exhausted'
+ }
+ const cursor: number = afterSequence
+ const chunk = await transaction(async (entityManager) => {
+ const claimResult = await barrierService(log).claimNextSignals({ barrierId: data.barrierId, projectId: data.projectId, afterSequence: cursor, limit: CLAIM_CHUNK_SIZE, entityManager })
+ if (!claimResult.barrierOpen) {
+ return null
+ }
+ const childRuns = await childRunDispatchService(log).createChildRuns({
+ target,
+ parentWaitpointId: data.barrierId,
+ children: claimResult.claims.map((claim) => ({ id: claim.childRunId, dispatchIndex: claim.sequence })),
+ entityManager,
+ })
+ return { claims: claimResult.claims, childRuns, lastSequence: claimResult.lastSequence }
+ })
+ if (isNil(chunk)) {
+ return 'barrier-closed'
+ }
+ const outcomes = await enqueueChunk({ data, source, target, claims: chunk.claims, childRuns: chunk.childRuns, log })
+ if (outcomes.includes('barrier-closed')) {
+ return 'barrier-closed'
+ }
+ afterSequence = chunk.lastSequence ?? (cursor === START_OF_BARRIER ? null : START_OF_BARRIER)
+ }
+ return 'dispatched'
+}
+
+async function enqueueChunk({ data, source, target, claims, childRuns, log }: EnqueueChunkParams): Promise<EnqueueOutcome[]> {
+ const limit = pLimit(MAX_ENQUEUES_IN_FLIGHT)
+ const results = await Promise.allSettled(claims.map((claim, index) => limit(() => enqueueChild({
+ data,
+ claim,
+ childRun: childRuns[index],
+ seedSteps: { ...source.seedSteps, [source.batchStepName]: { type: BATCH_STEP_TYPE, status: 'SUCCEEDED', input: {}, output: { items: source.batches[claim.sequence] ?? [] } } },
+ target,
+ log,
+ }))))
+ const failure = results.find((result): result is PromiseRejectedResult => result.status === 'rejected')
+ if (!isNil(failure)) {
+ throw failure.reason
+ }
+ return results.flatMap((result) => result.status === 'fulfilled' ? [result.value] : [])
+}
+
+async function enqueueChild({ data, claim, childRun, seedSteps, target, log }: EnqueueChildParams): Promise<EnqueueOutcome> {
+ const attempt = await transaction(async (entityManager): Promise<EnqueueAttempt> => {
+ const barrierOpen = await barrierService(log).lockForDispatch({ barrierId: data.barrierId, projectId: data.projectId, entityManager })
+ if (!barrierOpen) {
+ return { outcome: 'barrier-closed' }
+ }
+ const { error } = await tryCatch(() => childRunDispatchService(log).enqueueChildRun({ target, childRun, seedSteps }))
+ if (isNil(error)) {
+ return { outcome: 'enqueued' }
+ }
+ await childRunDispatchService(log).deleteChildRun({ id: childRun.id, projectId: data.projectId, entityManager })
+ await barrierService(log).releaseClaim({ signalId: claim.signalId, refId: childRun.id, projectId: data.projectId, entityManager })
+ return { outcome: 'failed', error }
+ })
+ switch (attempt.outcome) {
+ case 'enqueued':
+ await childRunDispatchService(log).reportChildRunStarted({ target, childRun })
+ return 'enqueued'
+ case 'barrier-closed':
+ await childRunDispatchService(log).cancelUnstartedChildRun({ target, childRun })
+ return 'barrier-closed'
+ case 'failed':
+ await settleIfDispatchGaveUp({ data, log })
+ throw attempt.error
+ }
+}
+
+async function settleIfDispatchGaveUp({ data, log }: JobParams): Promise<void> {
+ if (await distributedStore.exists(barrierSourceKey(data.barrierId))) {
+ return
+ }
+ const notDispatchedCount = await settleUnclaimedSignals({ data, log })
+ log.warn({ waitpoint: { id: data.barrierId }, notDispatchedCount }, '[fanOutDispatcher] A claim was released after dispatch gave up, marking it undispatched so the barrier can settle')
+}
+
+async function settleUnclaimedSignals({ data, log }: JobParams): Promise<number> {
+ const notDispatchedCount = await barrierService(log).markUnclaimedNotDispatched({ barrierId: data.barrierId, projectId: data.projectId })
+ await barrierQueue(log).enqueueEvaluation({ barrierId: data.barrierId, projectId: data.projectId })
+ return notDispatchedCount
+}
+
+const BATCH_STEP_TYPE = 'PROCESS_IN_BATCHES'
+
+const START_OF_BARRIER = -1
+
+const CLAIM_CHUNK_SIZE = 25
+
+const MAX_ENQUEUES_IN_FLIGHT = 5
+
+type DispatchOutcome = 'dispatched' | 'barrier-closed' | 'credits-exhausted'
+
+type EnqueueOutcome = 'enqueued' | 'barrier-closed'
+
+type EnqueueAttempt =
+ | { outcome: EnqueueOutcome }
+ | { outcome: 'failed', error: Error }
+
+type JobParams = {
+ data: BarrierJobData
+ log: FastifyBaseLogger
+}
+
+type HandleFanOutDispatchFailedParams = {
+ data: BarrierJobData
+ attemptsMade: number
+ attempts?: number
+ error: Error
+ log: FastifyBaseLogger
+}
+
+type FanOutDispatchGaveUpParams = {
+ attemptsMade: number
+ attempts?: number
+ error: Error
+}
+
+type DispatchChunksParams = {
+ data: BarrierJobData
+ source: BarrierFanOutPayload
+ target: ChildDispatchTarget
+ log: FastifyBaseLogger
+}
+
+type EnqueueChunkParams = DispatchChunksParams & {
+ claims: SignalClaim[]
+ childRuns: FlowRun[]
+}
+
+type EnqueueChildParams = {
+ data: BarrierJobData
+ claim: SignalClaim
+ childRun: FlowRun
+ seedSteps: Record<string, unknown>
+ target: ChildDispatchTarget
+ log: FastifyBaseLogger
+}
+
+export type GiveUpReason = 'exhausted' | 'unrecoverable'
diff --git a/packages/server/api/src/app/waitpoints/waitpoint-controller.ts b/packages/server/api/src/app/waitpoints/waitpoint-controller.ts
index b86db4d59a1..f6b643ef3ac 100644
--- a/packages/server/api/src/app/waitpoints/waitpoint-controller.ts
+++ b/packages/server/api/src/app/waitpoints/waitpoint-controller.ts
@@ -26,6 +26,7 @@ export const waitpointController: FastifyPluginAsyncZod = async (app) => {
httpRequestId: httpRequestId ?? undefined,
policy: barrier?.policy,
signalLabels: barrier?.signals?.map((signal) => signal.label ?? null),
+ fanOut: barrier?.fanOut,
})
return reply.status(StatusCodes.CREATED).send({
id: created.barrier.id,
@@ -61,6 +62,7 @@ async function buildBarrierState({ flowRunId, created }: BuildBarrierStateParams
const truncatedForSize = created.signals.length > MAX_INLINE_BARRIER_SIGNALS
return {
signalCount: created.signalCount,
+ batchSize: created.batchSize,
...(truncatedForSize ? { signalsTruncated: true } : {}),
...(inlineable ? { signals: await Promise.all(created.signals.map((signal) => toSignalLink({ flowRunId, signal }))) } : {}),
}
diff --git a/packages/server/api/src/app/waitpoints/waitpoint-service.ts b/packages/server/api/src/app/waitpoints/waitpoint-service.ts
index 6536da15d55..737d0e7ec49 100644
--- a/packages/server/api/src/app/waitpoints/waitpoint-service.ts
+++ b/packages/server/api/src/app/waitpoints/waitpoint-service.ts
@@ -6,9 +6,11 @@ import { EntityManager, FindOptionsWhere, In, Not } from 'typeorm'
import { z } from 'zod'
import { repoFactory } from '../core/db/repo-factory'
import { transaction } from '../core/db/transaction'
+import { distributedStore } from '../database/redis-connections'
import { flowRunRepo } from '../flows/flow-run/flow-run-service'
import { system } from '../helper/system/system'
import { AppSystemProp } from '../helper/system/system-props'
+import { barrierSourceKey } from './barrier-queue-factory'
import { WaitpointEntity } from './waitpoint-entity'
import { waitpointTimeoutJob } from './waitpoint-timeout-job'
import { CompleteParams, CompleteResult, CreateForPauseParams, CreateForPauseResult, FindPendingByVersionParams, HandleResumeSignalParams, Waitpoint, WaitpointStatus } from './waitpoint-types'
@@ -243,10 +245,13 @@ async function deleteWaitpointsWithTimers({ where, flowRunId, log }: DeleteWaitp
.createQueryBuilder()
.delete()
.where(where)
- .returning(['id', 'resumeDateTime'])
+ .returning(['id', 'resumeDateTime', 'type'])
.execute()
- const timed = DeletedWaitpointRows.parse(result.raw ?? []).filter((waitpoint) => !isNil(waitpoint.resumeDateTime))
+ const deleted = DeletedWaitpointRows.parse(result.raw ?? [])
+ const timed = deleted.filter((waitpoint) => !isNil(waitpoint.resumeDateTime))
await Promise.all(timed.map((waitpoint) => waitpointTimeoutJob.remove({ waitpointId: waitpoint.id, flowRunId, log })))
+ const barrierSourceKeys = deleted.filter((waitpoint) => waitpoint.type === PauseType.BARRIER).map((waitpoint) => barrierSourceKey(waitpoint.id))
+ await distributedStore.delete(barrierSourceKeys)
}
function stepNameOfWaitpointKey({ waitpointKey }: { waitpointKey: string }): string {
@@ -271,6 +276,7 @@ function clampWaitpointResumeDeadline({ requested, type, flowRunCreated, flowRun
const DeletedWaitpointRows = z.array(z.object({
id: z.string(),
resumeDateTime: z.coerce.date().nullable(),
+ type: z.enum(PauseType),
}))
type PausedDelivery = 'dispatched' | 'already-delivered' | 'unknown-waitpoint'
diff --git a/packages/server/api/test/integration/ce/flows/flow-run/barrier.test.ts b/packages/server/api/test/integration/ce/flows/flow-run/barrier.test.ts
index 49807ca5007..1dcad8365c4 100644
--- a/packages/server/api/test/integration/ce/flows/flow-run/barrier.test.ts
+++ b/packages/server/api/test/integration/ce/flows/flow-run/barrier.test.ts
@@ -1,26 +1,32 @@
-import { apId, isNil } from '@activepieces/core-utils'
+import { apId, assertNotNullOrUndefined, isNil } from '@activepieces/core-utils'
import { wideEvent } from '@activepieces/server-utils'
-import { BarrierSignalStatus, BarrierSummary, ErrorCode, FlowRunStatus, FlowVersionState, MAX_SIGNAL_REASON_LENGTH, PauseType, RunEnvironment } from '@activepieces/shared'
-import { Queue } from 'bullmq'
+import { BarrierSignalStatus, BarrierSummary, ErrorCode, FlowActionType, FlowRunStatus, FlowTrigger, FlowTriggerType, FlowVersionState, MAX_SIGNAL_REASON_LENGTH, PauseType, RunEnvironment } from '@activepieces/shared'
+import { Queue, UnrecoverableError } from 'bullmq'
import dayjs from 'dayjs'
import { FastifyInstance } from 'fastify'
import { StatusCodes } from 'http-status-codes'
+import { transaction } from '../../../../../src/app/core/db/transaction'
import { databaseConnection } from '../../../../../src/app/database/database-connection'
import { distributedStore, redisConnections } from '../../../../../src/app/database/redis-connections'
+import { childRunDispatchService } from '../../../../../src/app/flows/flow-run/child-run-dispatch-service'
import { systemJobIds } from '../../../../../src/app/helper/system-jobs/common'
import { systemJobsSchedule } from '../../../../../src/app/helper/system-jobs/system-job'
+import * as billingProviderModule from '../../../../../src/app/platform/billing-provider'
import { platformConfigurationService } from '../../../../../src/app/platform/platform-configuration.service'
-import { jobQueue } from '../../../../../src/app/workers/job-queue/job-queue'
import { barrierQueue } from '../../../../../src/app/waitpoints/barrier-queue'
import * as barrierQueueModule from '../../../../../src/app/waitpoints/barrier-queue'
-import { BarrierJobData } from '../../../../../src/app/waitpoints/barrier-queue-factory'
+import { BarrierJobData, BarrierJobName, barrierSourceKey } from '../../../../../src/app/waitpoints/barrier-queue-factory'
import { barrierService } from '../../../../../src/app/waitpoints/barrier-service'
+import * as barrierServiceModule from '../../../../../src/app/waitpoints/barrier-service'
+import { fanOutDispatchGaveUp, handleFanOutDispatch, handleFanOutDispatchFailed, handleFanOutDispatchGaveUp } from '../../../../../src/app/waitpoints/fan-out-dispatcher-job'
import { handleResumeDelayWaitpoint } from '../../../../../src/app/waitpoints/resume-delay-handler'
import { resumeService } from '../../../../../src/app/waitpoints/resume-service'
import { BARRIER_RECOVERY_CURSOR_KEY, DEADLINE_SWEEP_CURSOR_KEY, sweepOverdueDeadlines } from '../../../../../src/app/waitpoints/waitpoint-deadline-sweep'
import { waitpointService } from '../../../../../src/app/waitpoints/waitpoint-service'
import { waitpointTimeoutJob } from '../../../../../src/app/waitpoints/waitpoint-timeout-job'
import { Waitpoint, WaitpointStatus } from '../../../../../src/app/waitpoints/waitpoint-types'
+import * as jobQueueModule from '../../../../../src/app/workers/job-queue/job-queue'
+import { jobQueue } from '../../../../../src/app/workers/job-queue/job-queue'
import { db } from '../../../../helpers/db'
import { createMockFlow, createMockFlowRun, createMockFlowVersion } from '../../../../helpers/mocks'
import { createTestContext, TestContext } from '../../../../helpers/test-context'
@@ -29,6 +35,10 @@ import { setupTestEnvironment, teardownTestEnvironment } from '../../../../helpe
let app: FastifyInstance
let ctx: TestContext
+const FAN_OUT_ENTRY_STEP = 'step_1'
+
+const FAN_OUT_BATCH_STEP = 'fan_out'
+
beforeAll(async () => {
app = await setupTestEnvironment()
})
@@ -42,9 +52,40 @@ beforeEach(async () => {
})
async function createParentRun(status: FlowRunStatus = FlowRunStatus.PAUSED) {
+ return saveParentRun({ status })
+}
+
+async function createFanOutParentRun(status: FlowRunStatus = FlowRunStatus.PAUSED) {
+ const now = dayjs().toISOString()
+ return saveParentRun({
+ status,
+ trigger: {
+ type: FlowTriggerType.EMPTY,
+ name: 'trigger',
+ displayName: 'Trigger',
+ valid: true,
+ settings: {},
+ lastUpdatedDate: now,
+ nextAction: {
+ type: FlowActionType.CODE,
+ name: FAN_OUT_ENTRY_STEP,
+ displayName: 'Batch body',
+ valid: true,
+ skip: false,
+ lastUpdatedDate: now,
+ settings: {
+ sourceCode: { code: 'export const code = async () => true', packageJson: '{}' },
+ input: {},
+ },
+ },
+ },
+ })
+}
+
+async function saveParentRun({ status, trigger }: { status: FlowRunStatus, trigger?: FlowTrigger }) {
const flow = createMockFlow({ projectId: ctx.project.id })
await db.save('flow', flow)
- const flowVersion = createMockFlowVersion({ flowId: flow.id, state: FlowVersionState.LOCKED })
+ const flowVersion = createMockFlowVersion({ flowId: flow.id, state: FlowVersionState.LOCKED, trigger })
await db.save('flow_version', flowVersion)
const flowRun = createMockFlowRun({
projectId: ctx.project.id,
@@ -146,6 +187,105 @@ async function dropResumeJobs(flowRunId: string) {
await Promise.all(jobs.map((job) => job.remove()))
}
+async function giveUp(barrierId: string) {
+ return handleFanOutDispatchGaveUp({ data: { barrierId, projectId: ctx.project.id }, log: app.log })
+}
+
+async function failDispatch({ barrierId, attemptsMade }: { barrierId: string, attemptsMade: number }) {
+ return handleFanOutDispatchFailed({
+ data: { barrierId, projectId: ctx.project.id },
+ attemptsMade,
+ attempts: 5,
+ error: new Error('worker died'),
+ log: app.log,
+ })
+}
+
+async function createFanOutBarrier({ flowRunId, items, batchSize, stepName, batchStepName }: {
+ flowRunId: string
+ items: unknown[]
+ batchSize: number
+ stepName?: string
+ batchStepName?: string
+}) {
+ return barrierService(app.log).create({
+ flowRunId,
+ projectId: ctx.project.id,
+ platformId: ctx.platform.id,
+ stepName: stepName ?? FAN_OUT_BATCH_STEP,
+ version: 'V1',
+ fanOut: {
+ entryStepName: FAN_OUT_ENTRY_STEP,
+ batchStepName: batchStepName ?? FAN_OUT_BATCH_STEP,
+ batchSize,
+ items,
+ seedSteps: {},
+ },
+ })
+}
+
+async function withBarrierQueuePaused<T>(fn: () => Promise<T>): Promise<T> {
+ const queue = barrierQueue(app.log).get()
+ await queue.pause()
+ try {
+ await queue.drain(true)
+ return await fn()
+ }
+ finally {
+ await queue.resume()
+ }
+}
+
+async function listChildren(barrierId: string) {
+ return databaseConnection().getRepository('flow_run').findBy({ parentWaitpointId: barrierId })
+}
+
+async function dispatch(barrierId: string) {
+ return handleFanOutDispatch({ data: { barrierId, projectId: ctx.project.id }, log: app.log })
+}
+
+async function claimChunk({ barrierId, flowRunId, limit }: { barrierId: string, flowRunId: string, limit: number }) {
+ const target = await childRunDispatchService(app.log).prepareChildDispatch({ projectId: ctx.project.id, parentRunId: flowRunId, entryStepName: FAN_OUT_ENTRY_STEP })
+ return transaction(async (entityManager) => {
+ const claimResult = await barrierService(app.log).claimNextSignals({ barrierId, projectId: ctx.project.id, afterSequence: -1, limit, entityManager })
+ if (!claimResult.barrierOpen) {
+ return null
+ }
+ const childRuns = await childRunDispatchService(app.log).createChildRuns({
+ target,
+ parentWaitpointId: barrierId,
+ children: claimResult.claims.map((claim) => ({ id: claim.childRunId, dispatchIndex: claim.sequence })),
+ entityManager,
+ })
+ return { claims: claimResult.claims, childRuns }
+ })
+}
+
+function failNextChildEnqueue({ beforeFailure }: { beforeFailure?: () => Promise<unknown> }) {
+ const realJobQueue = jobQueueModule.jobQueue
+ const add = vi.fn(async (...args: Parameters<ReturnType<typeof realJobQueue>['add']>) => {
+ if (add.mock.calls.length === 1) {
+ await beforeFailure?.()
+ throw new Error('redis is down')
+ }
+ return realJobQueue(app.log).add(...args)
+ })
+ vi.spyOn(jobQueueModule, 'jobQueue').mockImplementation((log) => ({ ...realJobQueue(log), add }))
+}
+
+async function dispatchReadingChildPayloads(barrierId: string) {
+ const realJobQueue = jobQueueModule.jobQueue
+ const add = vi.fn(async (...args: Parameters<ReturnType<typeof realJobQueue>['add']>) => realJobQueue(app.log).add(...args))
+ vi.spyOn(jobQueueModule, 'jobQueue').mockImplementation((log) => ({ ...realJobQueue(log), add }))
+ try {
+ await dispatch(barrierId)
+ return add.mock.calls.map(([job]) => 'payload' in job.data ? job.data.payload : null)
+ }
+ finally {
+ vi.restoreAllMocks()
+ }
+}
+
async function waitFor(condition: () => Promise<boolean>): Promise<void> {
for (let attempt = 0; attempt < 100; attempt++) {
if (await condition()) {
@@ -1226,3 +1366,419 @@ describe('multi-approval confirm page', () => {
expect((await listSignals(signals[0].waitpointId))[0].status).toBe(BarrierSignalStatus.PENDING)
})
})
+
+describe('fan-out dispatch', () => {
+ it('creates exactly one child per signal even when two dispatchers run concurrently', async () => {
+ const { flowRun } = await createFanOutParentRun()
+ const { barrier, signalCount } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4, 5, 6], batchSize: 2 })
+ expect(signalCount).toBe(3)
+
+ await Promise.all([dispatch(barrier.id), dispatch(barrier.id)])
+ await waitFor(async () => (await listChildren(barrier.id)).length >= 3)
+ await dispatch(barrier.id)
+
+ const children = await listChildren(barrier.id)
+ expect(children).toHaveLength(3)
+ expect(children.map((child) => child.dispatchIndex).sort()).toEqual([0, 1, 2])
+ })
+
+ it('seeds each child under the batch step name, so a batch inside a loop still reads its items', async () => {
+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)
+ await withBarrierQueuePaused(async () => {
+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3], batchSize: 2, stepName: 'loop_1:3/batch_1', batchStepName: 'batch_1' })
+
+ const payloads = await dispatchReadingChildPayloads(barrier.id)
+
+ expect(payloads).toHaveLength(2)
+ expect(payloads).toEqual(expect.arrayContaining([
+ { type: 'inline', value: { batch_1: expect.objectContaining({ status: 'SUCCEEDED', output: { items: [1, 2] } }) } },
+ { type: 'inline', value: { batch_1: expect.objectContaining({ status: 'SUCCEEDED', output: { items: [3] } }) } },
+ ]))
+ })
+ })
+
+ it('stops without dispatching when the barrier was cancelled, and its signals are gone by cascade', async () => {
+ const { flowRun } = await createFanOutParentRun()
+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })
+
+ await waitpointService(app.log).deleteByFlowRunId({ flowRunId: flowRun.id, projectId: ctx.project.id })
+ expect(await listSignals(barrier.id)).toHaveLength(0)
+ const beforeCancel = (await listChildren(barrier.id)).length
+
+ await expect(dispatch(barrier.id)).resolves.toBeUndefined()
+
+ expect(await listChildren(barrier.id)).toHaveLength(beforeCancel)
+ })
+
+ it('deletes the stored source when a failed run discards its barrier, so a dispatch after it stops', async () => {
+ const { flowRun } = await createFanOutParentRun()
+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })
+ expect(await distributedStore.exists(barrierSourceKey(barrier.id))).toBe(true)
+
+ await waitpointService(app.log).deleteForFinishedRun({
+ flowRun: { ...flowRun, status: FlowRunStatus.FAILED, failedStep: { name: 'other_step', displayName: 'Other step', message: 'boom' } },
+ })
+
+ expect(await distributedStore.exists(barrierSourceKey(barrier.id))).toBe(false)
+ const beforeDispatch = (await listChildren(barrier.id)).length
+ await expect(dispatch(barrier.id)).resolves.toBeUndefined()
+ expect(await listChildren(barrier.id)).toHaveLength(beforeDispatch)
+ })
+
+ it('claims nothing once the barrier has released, so a timeout mid-dispatch adds no children', async () => {
+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)
+ await withBarrierQueuePaused(async () => {
+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })
+ await barrierService(app.log).release({ barrier, timedOut: true, releaseReason: 'timeout' })
+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.COMPLETED)
+
+ expect(await claimChunk({ barrierId: barrier.id, flowRunId: flowRun.id, limit: 25 })).toBeNull()
+ await expect(dispatch(barrier.id)).resolves.toBeUndefined()
+
+ expect(await listChildren(barrier.id)).toHaveLength(0)
+ })
+ })
+
+ it('lists every child claimed before a release in that release summary', async () => {
+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)
+ await withBarrierQueuePaused(async () => {
+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })
+ const chunk = await claimChunk({ barrierId: barrier.id, flowRunId: flowRun.id, limit: 2 })
+ assertNotNullOrUndefined(chunk, 'chunk')
+
+ await barrierService(app.log).release({ barrier, timedOut: true, releaseReason: 'timeout' })
+
+ const summary = await readSummary(barrier.id)
+ expect(summary.stillRunning).toBe(4)
+ const attributedRunIds = (summary.signals ?? []).map((signal) => signal.runId).filter((runId) => !isNil(runId))
+ expect(attributedRunIds.sort()).toEqual(chunk.childRuns.map((childRun) => childRun.id).sort())
+ await dispatch(barrier.id)
+ expect(await listChildren(barrier.id)).toHaveLength(2)
+ })
+ })
+
+ it('cancels a claimed child whose barrier released before its enqueue, so it never starts', async () => {
+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)
+ await withBarrierQueuePaused(async () => {
+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1], batchSize: 1 })
+ const realBarrierService = barrierServiceModule.barrierService
+ vi.spyOn(barrierServiceModule, 'barrierService').mockImplementation((log) => ({
+ ...realBarrierService(log),
+ lockForDispatch: async (params) => {
+ await realBarrierService(log).release({ barrier, timedOut: true, releaseReason: 'timeout' })
+ return realBarrierService(log).lockForDispatch(params)
+ },
+ }))
+ try {
+ await expect(dispatch(barrier.id)).resolves.toBeUndefined()
+ }
+ finally {
+ vi.restoreAllMocks()
+ }
+
+ expect(await readSummary(barrier.id)).toMatchObject({ total: 1, stillRunning: 1 })
+ await waitFor(async () => (await listChildren(barrier.id))[0]?.status === FlowRunStatus.CANCELED)
+ })
+ })
+
+ it('claims nothing for a barrier that no longer resolves', async () => {
+ const { flowRun } = await createFanOutParentRun()
+
+ expect(await claimChunk({ barrierId: apId(), flowRunId: flowRun.id, limit: 25 })).toBeNull()
+ })
+
+ it('refuses to resolve a dispatch target for a parent run outside the project', async () => {
+ await expect(childRunDispatchService(app.log).prepareChildDispatch({
+ projectId: ctx.project.id,
+ parentRunId: apId(),
+ entryStepName: FAN_OUT_ENTRY_STEP,
+ })).rejects.toThrow(ErrorCode.VALIDATION)
+ })
+
+ it('refuses to resolve a dispatch target for a step the flow version does not have', async () => {
+ const { flowRun } = await createFanOutParentRun()
+
+ await expect(childRunDispatchService(app.log).prepareChildDispatch({
+ projectId: ctx.project.id,
+ parentRunId: flowRun.id,
+ entryStepName: 'step_does_not_exist',
+ })).rejects.toThrow(ErrorCode.VALIDATION)
+ })
+
+ it('refuses to start a child run at the trigger, because the engine enters a run only at an action', async () => {
+ const { flowRun } = await createFanOutParentRun()
+
+ await expect(childRunDispatchService(app.log).prepareChildDispatch({
+ projectId: ctx.project.id,
+ parentRunId: flowRun.id,
+ entryStepName: 'trigger',
+ })).rejects.toThrow(ErrorCode.VALIDATION)
+ })
+
+ it('marks the rest undispatched when dispatch exhausts its attempts, so the barrier releases now', async () => {
+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)
+ await withBarrierQueuePaused(async () => {
+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })
+
+ await giveUp(barrier.id)
+ await releaseIfReady(barrier.id)
+
+ const summary = await readSummary(barrier.id)
+ expect(summary.notDispatched).toBe(4)
+ expect(summary.total).toBe(4)
+ })
+ })
+
+ it('queues the give-up as its own retried job once dispatch fails for good, and not mid-ladder', async () => {
+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)
+ await withBarrierQueuePaused(async () => {
+ const queue = barrierQueue(app.log).get()
+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2], batchSize: 1 })
+
+ await failDispatch({ barrierId: barrier.id, attemptsMade: 4 })
+ expect(await queue.getJob(`give-up-${barrier.id}`)).toBeUndefined()
+
+ await failDispatch({ barrierId: barrier.id, attemptsMade: 5 })
+ const giveUpJob = await queue.getJob(`give-up-${barrier.id}`)
+ expect(giveUpJob?.name).toBe(BarrierJobName.FAN_OUT_GIVE_UP)
+ expect(giveUpJob?.opts.attempts).toBe(5)
+ expect((await listSignals(barrier.id)).map((signal) => signal.status)).toEqual([BarrierSignalStatus.PENDING, BarrierSignalStatus.PENDING])
+ })
+ })
+
+ it('fails the give-up when settling the signals fails, so the queue retries it instead of leaving the parent waiting', async () => {
+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)
+ await withBarrierQueuePaused(async () => {
+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2], batchSize: 1 })
+ const realBarrierService = barrierServiceModule.barrierService
+ vi.spyOn(barrierServiceModule, 'barrierService').mockImplementation((log) => ({
+ ...realBarrierService(log),
+ markUnclaimedNotDispatched: async () => {
+ throw new Error('postgres is down')
+ },
+ }))
+ try {
+ await expect(giveUp(barrier.id)).rejects.toThrow('postgres is down')
+ }
+ finally {
+ vi.restoreAllMocks()
+ }
+
+ await giveUp(barrier.id)
+ await releaseIfReady(barrier.id)
+
+ expect(await readSummary(barrier.id)).toMatchObject({ total: 2, notDispatched: 2, stillRunning: 0 })
+ })
+ })
+
+ it('settles the unclaimed signals when the barrier worker runs a queued give-up', async () => {
+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)
+ const barrier = await withBarrierQueuePaused(async () => {
+ const created = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2], batchSize: 1 })
+ await barrierQueue(app.log).get().remove(`dispatch-${created.barrier.id}`)
+ return created.barrier
+ })
+
+ await barrierQueue(app.log).addFanOutGiveUp({ barrierId: barrier.id, projectId: ctx.project.id })
+ await waitFor(async () => (await readStatus(barrier.id)) !== WaitpointStatus.PENDING)
+
+ expect(await readSummary(barrier.id)).toMatchObject({ total: 2, notDispatched: 2, stillRunning: 0 })
+ expect(await distributedStore.exists(barrierSourceKey(barrier.id))).toBe(false)
+ })
+
+ it('spares the signals whose children are already live when dispatch gives up', async () => {
+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)
+ await withBarrierQueuePaused(async () => {
+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })
+ const chunk = await claimChunk({ barrierId: barrier.id, flowRunId: flowRun.id, limit: 2 })
+ assertNotNullOrUndefined(chunk, 'chunk')
+ const claimedRefIds = chunk.claims.map((claim) => claim.childRunId)
+
+ await giveUp(barrier.id)
+ await releaseIfReady(barrier.id)
+
+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.PENDING)
+ const afterGiveUp = await listSignals(barrier.id)
+ expect(afterGiveUp.filter((signal) => signal.status === BarrierSignalStatus.PENDING).map((signal) => signal.refId).sort()).toEqual([...claimedRefIds].sort())
+ expect(afterGiveUp.filter((signal) => signal.status === BarrierSignalStatus.NOT_DISPATCHED)).toHaveLength(2)
+ expect(await barrierService(app.log).hasUnclaimedSignals({ barrierId: barrier.id, projectId: ctx.project.id })).toBe(false)
+
+ await receiveSignal({ signalId: chunk.claims[0].signalId, status: BarrierSignalStatus.SUCCEEDED })
+ await releaseIfReady(barrier.id)
+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.PENDING)
+
+ await receiveSignal({ signalId: chunk.claims[1].signalId, status: BarrierSignalStatus.SUCCEEDED })
+ await releaseIfReady(barrier.id)
+
+ expect(await readSummary(barrier.id)).toMatchObject({ total: 4, succeeded: 2, notDispatched: 2, stillRunning: 0 })
+ })
+ })
+
+ it('settles a claim that a dispatcher releases after dispatch gave up, so the barrier does not wait for its deadline', async () => {
+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)
+ await withBarrierQueuePaused(async () => {
+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })
+ failNextChildEnqueue({ beforeFailure: () => giveUp(barrier.id) })
+ try {
+ await expect(dispatch(barrier.id)).rejects.toThrow('redis is down')
+ }
+ finally {
+ vi.restoreAllMocks()
+ }
+
+ const signals = await listSignals(barrier.id)
+ expect(signals.filter((signal) => signal.status === BarrierSignalStatus.NOT_DISPATCHED)).toHaveLength(1)
+ const live = signals.filter((signal) => signal.status === BarrierSignalStatus.PENDING)
+ expect(live).toHaveLength(3)
+ expect(await listChildren(barrier.id)).toHaveLength(3)
+
+ await Promise.all(live.map((signal) => receiveSignal({ signalId: signal.id, status: BarrierSignalStatus.SUCCEEDED })))
+ await releaseIfReady(barrier.id)
+
+ expect(await readSummary(barrier.id)).toMatchObject({ total: 4, succeeded: 3, notDispatched: 1, stillRunning: 0 })
+ })
+ })
+
+ it('leaves a claim that failed to enqueue for the next attempt while dispatch has not given up', async () => {
+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)
+ await withBarrierQueuePaused(async () => {
+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })
+ failNextChildEnqueue({})
+ try {
+ await expect(dispatch(barrier.id)).rejects.toThrow('redis is down')
+ }
+ finally {
+ vi.restoreAllMocks()
+ }
+ expect(await listChildren(barrier.id)).toHaveLength(3)
+ expect(await barrierService(app.log).hasUnclaimedSignals({ barrierId: barrier.id, projectId: ctx.project.id })).toBe(true)
+
+ await dispatch(barrier.id)
+
+ const children = await listChildren(barrier.id)
+ expect(children.map((child) => child.dispatchIndex).sort()).toEqual([0, 1, 2, 3])
+ })
+ })
+
+ it('dispatches nothing and settles the barrier when the platform has no run credits', async () => {
+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)
+ await withBarrierQueuePaused(async () => {
+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })
+ vi.spyOn(billingProviderModule, 'shouldBlockRunOnCredits').mockResolvedValue(true)
+ try {
+ await dispatch(barrier.id)
+ }
+ finally {
+ vi.restoreAllMocks()
+ }
+
+ expect(await listChildren(barrier.id)).toHaveLength(0)
+ await releaseIfReady(barrier.id)
+ expect(await readSummary(barrier.id)).toMatchObject({ total: 4, notDispatched: 4 })
+ })
+ })
+
+ it('starts the dispatch again when the engine retries a fan-out create that failed after its commit', async () => {
+ const { flowRun } = await createFanOutParentRun()
+ await withBarrierQueuePaused(async () => {
+ const queue = barrierQueue(app.log).get()
+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })
+ await queue.remove(`dispatch-${barrier.id}`)
+ await distributedStore.delete(barrierSourceKey(barrier.id))
+
+ const retried = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })
+
+ expect(retried.barrier.id).toBe(barrier.id)
+ expect(await distributedStore.exists(barrierSourceKey(barrier.id))).toBe(true)
+ expect(await queue.getJob(`dispatch-${barrier.id}`)).toBeDefined()
+ })
+ })
+
+ it('keeps a released fan-out barrier when the engine retries its create, so no batch runs twice', async () => {
+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)
+ await withBarrierQueuePaused(async () => {
+ const queue = barrierQueue(app.log).get()
+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3], batchSize: 1 })
+ await queue.remove(`dispatch-${barrier.id}`)
+ await dispatch(barrier.id)
+ const children = await listChildren(barrier.id)
+ for (const child of children) {
+ await barrierService(app.log).receiveSignal({ refId: child.id, projectId: ctx.project.id, status: BarrierSignalStatus.SUCCEEDED })
+ }
+ await releaseIfReady(barrier.id)
+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.COMPLETED)
+
+ const retried = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3], batchSize: 1 })
+ await dispatch(retried.barrier.id)
+
+ expect(retried.barrier.id).toBe(barrier.id)
+ expect(await databaseConnection().getRepository('flow_run').findBy({ parentRunId: flowRun.id })).toHaveLength(3)
+ expect(await queue.getJob(`dispatch-${barrier.id}`)).toBeUndefined()
+ expect(await distributedStore.exists(barrierSourceKey(barrier.id))).toBe(false)
+
+ await db.update('flow_run', flowRun.id, { status: FlowRunStatus.PAUSED })
+ await dropResumeJobs(flowRun.id)
+ await recoverUndeliveredResume(flowRun.id)
+
+ expect(await listResumeJobs(flowRun.id)).toHaveLength(1)
+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.CONSUMED)
+ })
+ })
+
+ it('keeps an open fan-out barrier when the engine retries its create, so its children are not dispatched again', async () => {
+ const { flowRun } = await createFanOutParentRun()
+ await withBarrierQueuePaused(async () => {
+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3], batchSize: 1 })
+ await dispatch(barrier.id)
+
+ const retried = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3], batchSize: 1 })
+ await dispatch(retried.barrier.id)
+
+ expect(retried.barrier.id).toBe(barrier.id)
+ expect(await listChildren(barrier.id)).toHaveLength(3)
+ })
+ })
+
+ it('replaces a consumed barrier when its step creates one again', async () => {
+ const { flowRun } = await createParentRun()
+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]'] })
+ await closeWithoutConsuming(barrier.id)
+ const closed = await barrierService(app.log).findById({ barrierId: barrier.id, projectId: ctx.project.id })
+ assertNotNullOrUndefined(closed, 'closed barrier')
+ await waitpointService(app.log).consume({ waitpoint: closed })
+
+ const recreated = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]'] })
+
+ expect(recreated.barrier.id).not.toBe(barrier.id)
+ expect(await readStatus(recreated.barrier.id)).toBe(WaitpointStatus.PENDING)
+ expect(await listSignals(recreated.barrier.id)).toEqual([expect.objectContaining({ status: BarrierSignalStatus.PENDING })])
+ expect(await db.findOneBy('waitpoint', { id: barrier.id })).toBeNull()
+ })
+
+ it('gives up on an unrecoverable failure and on the last attempt, but not mid-ladder', async () => {
+ expect(fanOutDispatchGaveUp({ attemptsMade: 1, attempts: 5, error: new Error('boom') })).toBeNull()
+ expect(fanOutDispatchGaveUp({ attemptsMade: 4, attempts: 5, error: new Error('boom') })).toBeNull()
+ expect(fanOutDispatchGaveUp({ attemptsMade: 5, attempts: 5, error: new Error('boom') })).toBe('exhausted')
+ expect(fanOutDispatchGaveUp({ attemptsMade: 1, attempts: 5, error: new UnrecoverableError('job stalled more than allowable limit') })).toBe('unrecoverable')
+ expect(fanOutDispatchGaveUp({ attemptsMade: 1, attempts: undefined, error: new Error('boom') })).toBe('exhausted')
+ })
+
+ it('clamps the batch size so a wide source cannot exceed the signal cap', async () => {
+ const { flowRun } = await createFanOutParentRun()
+ const previous = process.env.AP_MAX_BARRIER_SIGNALS
+ process.env.AP_MAX_BARRIER_SIGNALS = '3'
+ try {
+ const created = await createFanOutBarrier({ flowRunId: flowRun.id, items: Array.from({ length: 12 }, (_, index) => index), batchSize: 1 })
+ expect(created.signalCount).toBe(3)
+ expect(created.batchSize).toBe(4)
+ }
+ finally {
+ if (previous === undefined) {
+ delete process.env.AP_MAX_BARRIER_SIGNALS
+ }
+ else {
+ process.env.AP_MAX_BARRIER_SIGNALS = previous
+ }
+ }
+ })
+})
diff --git a/packages/server/engine/src/lib/handler/context/engine-constants.ts b/packages/server/engine/src/lib/handler/context/engine-constants.ts
index 729b99bd375..2a7fd58e6a4 100644
--- a/packages/server/engine/src/lib/handler/context/engine-constants.ts
+++ b/packages/server/engine/src/lib/handler/context/engine-constants.ts
@@ -27,6 +27,7 @@ type EngineConstantsParams = {
resumePayload?: ResumePayload
runEnvironment?: RunEnvironment
stepNameToTest?: string
+ entryStepName?: string
logsFileId?: string
timeoutInSeconds: number
platformId: PlatformId
@@ -68,6 +69,7 @@ export class EngineConstants {
public readonly resumePayload?: ResumePayload
public readonly runEnvironment?: RunEnvironment
public readonly stepNameToTest?: string
+ public readonly entryStepName?: string
public readonly logsFileId?: string
public readonly stepNames: string[] = []
public readonly actionRunMode: boolean
@@ -113,6 +115,7 @@ export class EngineConstants {
this.resumePayload = params.resumePayload
this.runEnvironment = params.runEnvironment
this.stepNameToTest = params.stepNameToTest
+ this.entryStepName = params.entryStepName
this.logsFileId = params.logsFileId
this.platformId = params.platformId
this.timeoutInSeconds = params.timeoutInSeconds
@@ -132,6 +135,7 @@ export class EngineConstants {
resumePayload: input.executionType === ExecutionType.RESUME ? input.resumePayload : undefined,
runEnvironment: input.runEnvironment,
stepNameToTest: input.stepNameToTest ?? undefined,
+ entryStepName: input.entryStepName,
logsFileId: input.logsFileId,
})
}
diff --git a/packages/server/engine/src/lib/helper/flow-run-progress-reporter.ts b/packages/server/engine/src/lib/helper/flow-run-progress-reporter.ts
index 2b5805d0060..ecad18a87c3 100644
--- a/packages/server/engine/src/lib/helper/flow-run-progress-reporter.ts
+++ b/packages/server/engine/src/lib/helper/flow-run-progress-reporter.ts
@@ -107,6 +107,7 @@ export const flowRunProgressReporter = {
executionState: {
steps: flowExecutorContext.steps,
tags: Array.from(flowExecutorContext.tags),
+ entryStepName: engineConstants.entryStepName,
},
})
const executionState = await zstdCompress(serialized)
diff --git a/packages/server/engine/src/lib/operations/flow.operation.ts b/packages/server/engine/src/lib/operations/flow.operation.ts
index 49846d41b17..c66cdf9273e 100644
--- a/packages/server/engine/src/lib/operations/flow.operation.ts
+++ b/packages/server/engine/src/lib/operations/flow.operation.ts
@@ -1,5 +1,6 @@
import { isNil, tryCatch } from '@activepieces/core-utils'
-import { EngineGenericError, EngineResponse, EngineResponseStatus, ExecuteFlowOperation, ExecuteTriggerResponse, ExecutionError, ExecutionErrorType, ExecutionState, ExecutionType, FlowActionType, FlowRunStatus, flowStructureUtil, GenericStepOutput, LoopStepOutput, ResumePayload, ResumeReason, StepOutput, StepOutputStatus, TriggerHookType, TriggerPayload } from '@activepieces/shared'
+import { EngineGenericError, EngineResponse, EngineResponseStatus, ExecuteFlowOperation, ExecuteTriggerResponse, ExecutionError, ExecutionErrorType, ExecutionState, ExecutionType, FlowActionType, FlowRunStatus, flowStructureUtil, GenericStepOutput, InvalidSeededStepsError, LoopStepOutput, ResumePayload, ResumeReason, StepOutput, StepOutputStatus, TriggerHookType, TriggerPayload } from '@activepieces/shared'
+import dayjs from 'dayjs'
import { engineFileApi } from '../api/engine-file-api'
import { EngineConstants, ResolvedBeginExecuteFlowOperation, ResolvedExecuteFlowOperation } from '../handler/context/engine-constants'
import { FlowExecutorContext } from '../handler/context/flow-execution-context'
@@ -119,6 +120,20 @@ const executeSingleStepOrFlow = async (input: ResolvedExecuteFlowOperation, cons
if (executionState.verdict.status !== FlowRunStatus.RUNNING) {
return executionState
}
+ if (!isNil(input.entryStepName)) {
+ if (input.executionType === ExecutionType.BEGIN) {
+ await flowRunProgressReporter.sendUpdate({
+ engineConstants: constants,
+ flowExecutorContext: executionState,
+ startTime: dayjs().toISOString(),
+ })
+ }
+ return flowExecutor.execute({
+ action: flowStructureUtil.getActionOrThrow(input.entryStepName, input.flowVersion.trigger),
+ executionState,
+ constants,
+ })
+ }
return flowExecutor.executeFromTrigger({
executionState,
constants,
@@ -158,6 +173,12 @@ async function buildFailedTriggerContext({ input, baseContext, error }: BuildFai
async function getFlowExecutionState(input: ResolvedExecuteFlowOperation, constants: EngineConstants, flowContext: FlowExecutorContext): Promise<FlowExecutorContext> {
if (input.executionType === ExecutionType.BEGIN) {
+ if (!isNil(input.entryStepName)) {
+ if (!isSeededSteps(input.triggerPayload)) {
+ throw new InvalidSeededStepsError(input.entryStepName)
+ }
+ return restoreSteps({ steps: input.triggerPayload, flowContext, keepFailedSteps: true })
+ }
const newPayload = await runOrReturnPayload(input, constants)
return flowContext.upsertStep(input.flowVersion.trigger.name,
GenericStepOutput.create({
@@ -166,17 +187,31 @@ async function getFlowExecutionState(input: ResolvedExecuteFlowOperation, consta
input: {},
}).setOutput(newPayload))
}
- flowContext = flowContext.addTags(input.executionState.tags)
- const isWaitpointResume = input.resumeReason === ResumeReason.WAITPOINT
- for (const [step, output] of Object.entries(input.executionState.steps)) {
- if (isStepRestorable({ status: output.status, isWaitpointResume })) {
- const newOutput = await insertSuccessStepsOrPausedRecursively({ stepOutput: output, isWaitpointResume })
+ return restoreSteps({
+ steps: input.executionState.steps,
+ flowContext: flowContext.addTags(input.executionState.tags),
+ keepFailedSteps: input.resumeReason === ResumeReason.WAITPOINT,
+ })
+}
+
+async function restoreSteps({ steps, flowContext, keepFailedSteps }: RestoreStepsParams): Promise<FlowExecutorContext> {
+ let restored = flowContext
+ for (const [step, output] of Object.entries(steps)) {
+ if (isStepRestorable({ status: output.status, isWaitpointResume: keepFailedSteps })) {
+ const newOutput = await insertSuccessStepsOrPausedRecursively({ stepOutput: output, isWaitpointResume: keepFailedSteps })
if (!isNil(newOutput)) {
- flowContext = await flowContext.upsertStep(step, newOutput)
+ restored = await restored.upsertStep(step, newOutput)
}
}
}
- return flowContext
+ return restored
+}
+
+function isSeededSteps(payload: unknown): payload is Record<string, StepOutput> {
+ if (payload === null || typeof payload !== 'object' || Array.isArray(payload)) {
+ return false
+ }
+ return Object.values(payload).every((step: unknown) => step !== null && typeof step === 'object' && 'type' in step && 'status' in step)
}
async function runOrReturnPayload(input: ResolvedBeginExecuteFlowOperation, constants: EngineConstants): Promise<TriggerPayload> {
@@ -236,6 +271,7 @@ async function resolveExecuteFlowOperation(operation: ExecuteFlowOperation): Pro
}
return {
...operation,
+ entryStepName: operation.entryStepName ?? executionState.entryStepName,
resumePayload: await resolveJobPayload({ payload: operation.resumePayload, apiUrl: operation.internalApiUrl, engineToken: operation.engineToken }) as ResumePayload,
executionState,
}
@@ -301,3 +337,9 @@ type InsertStepsParams = {
stepOutput: StepOutput
isWaitpointResume: boolean
}
+
+type RestoreStepsParams = {
+ steps: Record<string, StepOutput>
+ flowContext: FlowExecutorContext
+ keepFailedSteps: boolean
+}
diff --git a/packages/server/engine/test/operations/flow-operation-invariants.test.ts b/packages/server/engine/test/operations/flow-operation-invariants.test.ts
index 789c8717916..c6b2e91b6d0 100644
--- a/packages/server/engine/test/operations/flow-operation-invariants.test.ts
+++ b/packages/server/engine/test/operations/flow-operation-invariants.test.ts
@@ -1,3 +1,4 @@
+import { isNil } from '@activepieces/core-utils'
import {
ConnectionNotFoundError,
EngineFileNotFoundError,
@@ -14,6 +15,7 @@ import {
StreamStepProgress,
} from '@activepieces/shared'
import type { BeginExecuteFlowOperation, FlowAction, FlowVersion, ResumeExecuteFlowOperation } from '@activepieces/shared'
+import dayjs from 'dayjs'
import { describe, expect, it, vi } from 'vitest'
const { mockSendUpdate, mockBackup } = vi.hoisted(() => ({
@@ -154,6 +156,33 @@ function makeFlowVersionWithTwoApprovals(): FlowVersion {
}
}
+function makeFlowVersionWithContainerBody(): FlowVersion {
+ const loopAction = (name: string, items: unknown): FlowAction => ({
+ name,
+ displayName: name,
+ type: FlowActionType.LOOP_ON_ITEMS,
+ skip: false,
+ valid: true,
+ settings: { items },
+ })
+ const container: FlowAction = {
+ ...loopAction('step_1', []),
+ firstLoopAction: loopAction('step_2', [1, 2]),
+ nextAction: loopAction('step_3', []),
+ }
+ return {
+ ...makeFlowVersion(),
+ trigger: {
+ name: 'trigger_1',
+ valid: true,
+ displayName: 'Test Trigger',
+ type: FlowTriggerType.EMPTY,
+ settings: {},
+ nextAction: container,
+ },
+ }
+}
+
function makeResumeOperation(overrides?: Partial<ResumeExecuteFlowOperation>): ResumeExecuteFlowOperation {
return {
projectId: 'proj-1',
@@ -469,6 +498,124 @@ describe('flow operation invariants', () => {
expect(mockCreateWaitpoint).not.toHaveBeenCalled()
})
+
+ it('reports a start time for a run entered at a step, so a batch child is not left without one', async () => {
+ mockSendUpdate.mockClear()
+
+ await flowOperation.execute(makeBeginOperation({
+ flowVersion: makeFlowVersionWithContainerBody(),
+ entryStepName: 'step_2',
+ }))
+
+ const started = mockSendUpdate.mock.calls.filter((call) => !isNil(call[0].startTime))
+ expect(started).toHaveLength(1)
+ expect(dayjs(started[0][0].startTime).isValid()).toBe(true)
+ })
+
+ it('fails a run entered at a step with malformed seeded steps instead of throwing for a retry', async () => {
+ mockSendUpdate.mockClear()
+
+ const response = await flowOperation.execute(makeBeginOperation({
+ flowVersion: makeFlowVersionWithContainerBody(),
+ entryStepName: 'step_2',
+ triggerPayload: { type: 'inline', value: { step_1: 'not a step output' } },
+ }))
+
+ expect(response.status).toBe(EngineResponseStatus.OK)
+ const finalCtx = mockSendUpdate.mock.calls[mockSendUpdate.mock.calls.length - 1][0].flowExecutorContext
+ expect(finalCtx.verdict.status).toBe(FlowRunStatus.FAILED)
+ expect(finalCtx.verdict.failedStep.message).toEqual(expect.stringContaining('needs the steps it starts after'))
+ })
+
+ it('does not re-report a start time when a run entered at a step resumes', async () => {
+ mockSendUpdate.mockClear()
+ mockDownload.mockReset()
+ mockCreateWaitpoint.mockReset()
+ mockCreateWaitpoint.mockResolvedValue({
+ id: 'wp-after-batch',
+ resumeUrl: 'http://localhost:4200/api/v1/flow-runs/run-1/waitpoints/wp-after-batch',
+ })
+ mockDownload.mockResolvedValue(
+ new TextEncoder().encode(JSON.stringify({
+ executionState: {
+ steps: {
+ step_1: {
+ type: FlowActionType.LOOP_ON_ITEMS,
+ status: StepOutputStatus.SUCCEEDED,
+ input: {},
+ output: { item: null, index: 0, iterations: [] },
+ },
+ step_2: {
+ type: FlowActionType.LOOP_ON_ITEMS,
+ status: StepOutputStatus.PAUSED,
+ input: {},
+ output: { item: null, index: 0, iterations: [] },
+ },
+ },
+ tags: [],
+ entryStepName: 'step_2',
+ },
+ })),
+ )
+
+ await flowOperation.execute(makeResumeOperation({
+ flowVersion: makeFlowVersionWithContainerBody(),
+ resumePayload: {
+ type: 'inline',
+ value: { queryParams: { action: 'approve' }, body: {}, headers: {} },
+ },
+ }))
+
+ expect(mockSendUpdate.mock.calls.filter((call) => !isNil(call[0].startTime))).toHaveLength(0)
+ })
+
+ it('resumes a run entered mid-graph from its entry step instead of walking the flow from the trigger', async () => {
+ mockDownload.mockReset()
+ mockSendUpdate.mockClear()
+ mockCreateWaitpoint.mockReset()
+ mockCreateWaitpoint.mockResolvedValue({
+ id: 'wp-after-batch',
+ resumeUrl: 'http://localhost:4200/api/v1/flow-runs/run-1/waitpoints/wp-after-batch',
+ })
+
+ mockDownload.mockResolvedValue(
+ new TextEncoder().encode(JSON.stringify({
+ executionState: {
+ steps: {
+ step_1: {
+ type: FlowActionType.LOOP_ON_ITEMS,
+ status: StepOutputStatus.SUCCEEDED,
+ input: {},
+ output: { item: null, index: 0, iterations: [] },
+ },
+ step_2: {
+ type: FlowActionType.LOOP_ON_ITEMS,
+ status: StepOutputStatus.PAUSED,
+ input: {},
+ output: { item: null, index: 0, iterations: [] },
+ },
+ },
+ tags: [],
+ entryStepName: 'step_2',
+ },
+ })),
+ )
+
+ const operation: ResumeExecuteFlowOperation = {
+ ...makeResumeOperation(),
+ flowVersion: makeFlowVersionWithContainerBody(),
+ resumePayload: {
+ type: 'inline',
+ value: { queryParams: { action: 'approve' }, body: {}, headers: {} },
+ },
+ }
+
+ await flowOperation.execute(operation)
+
+ const finalCtx = mockSendUpdate.mock.calls[mockSendUpdate.mock.calls.length - 1][0].flowExecutorContext
+ expect(finalCtx.steps.step_2.output.iterations).toHaveLength(2)
+ expect(finalCtx.steps.step_3).toBeUndefined()
+ })
})
describe('BEGIN payload hydration', () => {
diff --git a/packages/server/worker/src/lib/execute/jobs/execute-flow.ts b/packages/server/worker/src/lib/execute/jobs/execute-flow.ts
index b82c98880ad..643a02d5d1f 100644
--- a/packages/server/worker/src/lib/execute/jobs/execute-flow.ts
+++ b/packages/server/worker/src/lib/execute/jobs/execute-flow.ts
@@ -161,6 +161,7 @@ function buildFlowOperation(
executionType: ExecutionType.BEGIN,
triggerPayload: data.payload,
executeTrigger: data.executeTrigger ?? false,
+ entryStepName: data.entryStepName,
sampleData: data.sampleData,
}
}