AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
feat(flow-runs): dispatch a barrier's children from a server-side queue job (#14890)
Othman Abu Ajamieh8b4cb913d0ec
diff --git a/brain/knowledge/decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md b/brain/knowledge/decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.mdindex df374ef9687..9d07ae8b9c2 100644--- a/brain/knowledge/decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md+++ b/brain/knowledge/decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md@@ -113,7 +113,7 @@ all, purely because child rows appear later than the dispatch that created them.them by design. `dispatchIndex` exists because attribution can legitimately be dropped while the run isstill a child: it is the "started mid-graph" flag retry refuses on, and the ordering key for listing onebarrier's children.-- **Accepted cost: ~30k row writes per 10 000-signal barrier** — insert, receive, delete — on top of the+- **Accepted cost: ~40k row writes per 10 000-signal barrier** — insert, claim, receive, delete — on top of thechildren themselves. Bounded, on a narrow table, and gone the moment the barrier resolves.- **A returning approver still sees a bare "already responded".** Signals die with the release, so there isnothing left to tell them what they decided. Decision 000009's gap stays open.diff --git a/brain/knowledge/decisions/000051-batch-children-are-dispatched-by-an-app-queue-job-not-the-engine.md b/brain/knowledge/decisions/000051-batch-children-are-dispatched-by-an-app-queue-job-not-the-engine.mdnew file mode 100644index 00000000000..9e04c055923--- /dev/null+++ b/brain/knowledge/decisions/000051-batch-children-are-dispatched-by-an-app-queue-job-not-the-engine.md@@ -0,0 +1,49 @@+---+status: accepted+---++# Batch children are dispatched by an app queue job, not by the engine++## Decision+A batch step makes one call, `POST /v1/waitpoints` with the barrier and its `fanOut` source, and pauses. The+server stores the source in `distributedStore` under `barrier_source:<barrierId>` and adds a `fan-out-dispatch`+job, carrying only the barrier id, to the barrier queue. The job runs in the app process and creates the children.++The job works in chunks. One transaction per chunk locks the barrier `FOR SHARE` while it is `PENDING`, takes the+next unclaimed signals `FOR UPDATE SKIP LOCKED`, claims them (`refId` = the new child run id, guarded by+`refId IS NULL AND status = 'PENDING'`) and inserts their `flow_run` rows. Each child is then enqueued in a short+transaction that holds the barrier `FOR SHARE` while it is still `PENDING`, with its run id as its job id. A child+whose barrier closed in between is cancelled instead.++When dispatch gives up for good, a `fan-out-give-up` job deletes the source key and then marks the **unclaimed**+signals `NOT_DISPATCHED`. Claimed signals keep their live children, and the barrier releases when those report.++## Context+The loop was first planned inside the parent's engine sandbox, which is killed at `FLOW_TIMEOUT_SECONDS`. A killed+loop has to work out what its previous attempt already sent, which needs a source digest, a complement of the+dispatched indices and a unique `(parentWaitpointId, dispatchIndex)` index. A queue job is redelivered rather than+killed, and it re-reads stored bytes instead of re-resolving the items.++## Why+- **The resume point is a claim, not a gap query.** Two dispatchers over one barrier do happen: the barrier worker+ runs on BullMQ defaults (`stalledInterval` 30 s, `maxStalledCount` 1), so a job that holds the event loop past+ 30 s is redelivered while the first copy still runs. The claim gives each signal at most one child.+- **Release and cancel wait for dispatch.** `closeBarrier` locks the barrier `FOR UPDATE`, and a cancel deletes+ it. Both wait for an open claim or enqueue, so nothing is claimed or enqueued after a release, and a cancel's+ child snapshot sees every child.+- **The step makes an ordinary HTTP call**, so there is no piece-framework version floor to forget.+- **The source is out of band** for the same reason `runs_metadata:<runId>` is: a multi-MB array in BullMQ job+ data is a large payload on a hot Redis queue.+- Rejected: keep the loop in the engine. It is bounded by the sandbox timeout and needs the resumable-complement+ machinery above.++## Consequences+- **The dispatcher holds a barrier-queue slot for the dispatch window**, about a minute at the 10 000-signal cap.+- **Nothing bounds the dispatch window except the barrier deadline**, because the parent pauses before its+ children exist.+- **Giving up is slow.** Five attempts with a `2^(n-1) x 8 min` backoff put about two hours before the give-up+ handler runs.+- **The batch size is clamped** so the signal count fits the platform cap+ (`platformConfigurationService.maxBarrierSignals`), and the clamped size is returned as `batchSize`.+- **A child claimed just before a release never starts.** It is cancelled at its enqueue, but the release summary+ still counts it as `stillRunning` with its run id, because its claim came first.diff --git a/brain/knowledge/flows-execution/index.md b/brain/knowledge/flows-execution/index.mdindex be49f28c4ea..ad430599afd 100644--- a/brain/knowledge/flows-execution/index.md+++ b/brain/knowledge/flows-execution/index.md@@ -49,7 +49,7 @@ Public read-only endpoints returning UI metadata for flows whose trigger is `@acA **Subflow** is a flow invoked by another flow rather than by its own external trigger — reached by a webhook POST to `/v1/webhooks/:flowId`, never a dedicated transport. Vocabulary from `@activepieces/piece-subflows`:- **Callable Flow** — the trigger that makes a flow callable; carries the parent's `data` payload and an optional `callbackUrl`. *Avoid:* child flow, nested flow, sub-workflow.- **Call Flow** — the action that invokes one subflow once, optionally waiting on a waitpoint for its `Respond` callback.-- **Subflow fan-out** — many calls dispatched from one parent step (e.g. one per CSV batch), fire-and-forget, no waiting per call. *Avoid:* scatter, broadcast.+- **Subflow fan-out** — many calls dispatched from one parent step (e.g. one per CSV batch), fire-and-forget, no waiting per call. Not the barrier's server-side fan-out, whose parent waits for every child. *Avoid:* scatter, broadcast.- **Batch** — the rows carried by one fan-out call: `{ batchIndex, headers, rows, extraData }`. *Avoid:* chunk, csv table, sub-table, shard.- Parent linkage is two headers (`x-parent-run-id`, `x-fail-parent-on-failure`); fan-out sets the latter false. Streaming bounds memory, not time — the step is still capped by `FLOW_TIMEOUT_SECONDS`.@@ -61,6 +61,7 @@ A **waitpoint** is a row recording something a run waits on — `DELAY` / `WEBHO- **Waitpoint key** — the path-qualified step identity the engine sends in `stepName`, e.g. `loop_1:3/approval`. It is only ever an identity key inside `waitpoint-service`. Gives each loop iteration its own row; without it a pause inside a loop fired on iteration 1 only.- **Consume** — what happens to a **waitpoint** once its resume has been enqueued: a barrier is tombstoned `CONSUMED` so the run stays provably barrier-owned until it ends, any other type is deleted. `waitpointService.consume`, the single owner of the rule. *Avoid:* "close the waitpoint".- **Signal** — one `waitpoint_signal` row per awaited thing on a barrier, created up front. Deleted on release, so a returning approver sees only "already responded".+- **Server-side fan-out** — one `POST /v1/waitpoints` hands a barrier its items, and a `fan-out-dispatch` job on the barrier queue creates one child run per signal. See [Server-side fan-out](server-side-fan-out.md).- Gotcha: a run can hold a COMPLETED leftover *and* an open PENDING pause at once, so any per-run waitpoint lookup must say which it wants — `findUndeliveredCompletedWaitpoint` declines only while a PENDING **barrier** is held, since delivering a resume never leaves a row at COMPLETED. An earlier form declined while *anything* was pending and hung every run holding a second waitpoint.- See [decision 000015](../decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md), [decision 000033](../decisions/000033-a-released-barrier-leaves-a-tombstone-until-the-run-ends.md) and the public [Waitpoints](https://www.activepieces.com/docs/install/architecture/waitpoints) page.@@ -84,6 +85,7 @@ Reusable flow/table blueprints. Types: OFFICIAL (Activepieces-curated, platformI- **Human Input** — forms, approvals, the resume confirmation page- **Subflows** — flow-calls-flow: Callable Flow, Call Flow, streaming fan-out- **Waitpoints** — how a paused run is parked; release vs resume, barriers, signals+- **Server-side fan-out** — the barrier's child dispatcher: chunks, claims, give-up, entry step- **Folders** — flow organization; the uncategorized sentinel- **Templates** — OFFICIAL / CUSTOM / SHARED blueprints- **Variables** — project-scoped values referenced from stepsdiff --git a/brain/knowledge/flows-execution/server-side-fan-out.md b/brain/knowledge/flows-execution/server-side-fan-out.mdnew file mode 100644index 00000000000..9c5f4a91a9d--- /dev/null+++ b/brain/knowledge/flows-execution/server-side-fan-out.md@@ -0,0 +1,75 @@+---+title: Server-side fan-out+icon: 🪓+---++# Server-side fan-out++The barrier's own child dispatcher. A batch step hands the server its items in one `POST /v1/waitpoints`, and a+`fan-out-dispatch` job on the barrier queue creates one child run per signal. The step that calls it+(`PROCESS_IN_BATCHES`) ships with the batch-step layer; until then only tests drive it. See decision+[000051](../decisions/000051-batch-children-are-dispatched-by-an-app-queue-job-not-the-engine.md) for why, and+[000015](../decisions/000015-fan-in-is-an-event-driven-waitpoint-barrier.md) for the barrier.++**Server-side fan-out** — one parent pause, N child runs created by the app. _Avoid_: "subflow fan-out", which is+fire-and-forget webhook calls with no barrier.+**Source** — `{ entryStepName, batchStepName, seedSteps, batches }` stored under `barrier_source:<barrierId>` with a 1-day TTL.+Deleted on release, on cancel and when dispatch gives up.+**Claim** — a signal is claimed when its `refId` holds its child run id. _Unclaimed_ means `PENDING` with no+`refId` (`unclaimedSignalWhere`).+**Chunk** — one transaction: the barrier `FOR SHARE`, the next 25 unclaimed signals `FOR UPDATE SKIP LOCKED`, the+claim, and the child rows.+**Entry step** — `entryStepName`, the action a child run starts at, with the parent's steps seeded as SUCCEEDED.+It is written into the child's execution-state log, so a resume re-enters it. It must be an action.+**Give-up** — the `fan-out-give-up` job (`give-up-<barrierId>`). The barrier worker's `failed` handler queues it on+exhausted attempts or an `UnrecoverableError`. It deletes the source, then marks the unclaimed signals `NOT_DISPATCHED`,+and it retries like any barrier job.++## Gotchas+- **The waitpoint key is not the batch step's name.** Inside a loop the barrier's `stepName` is the path key+ `loop_1:3/batch_1`, which keeps one barrier per iteration. The body reads its batch under the bare name, so the+ engine sends that name as `batchStepName` and the dispatcher seeds the batch under it.+- **The give-up cleanup runs as its own queue job, never inline in the `failed` handler.** Inline, a Postgres or+ Redis error there was logged and lost, and the parent stayed paused until its deadline. One gap is left: a process+ that dies between BullMQ's `moveToFailed` and that enqueue never queues the give-up.+- **Raw `UPDATE … RETURNING` through `repo.query()` returns `[rows, rowCount]` on node-postgres but a plain rows+ array on PGlite.** A `.length > 0` check on it is always true in production and only correct in tests. Read+ `RETURNING` rows through the query builder (`.returning().execute()`, then `raw`), as `claimNextSignals` does.+- **PGlite runs every transaction on one shared session, so row locks do nothing in the API tests.** The claim+ stays correct without them: the `UPDATE` keeps its `refId IS NULL AND status = 'PENDING'` guard, and the claims+ are the rows it returned, not the rows the `SELECT … FOR UPDATE` saw.+- **A stalled job is not a dead worker.** On BullMQ defaults the first stall redelivers the job to a second+ dispatcher while the first still runs. The second stall fails it as an `UnrecoverableError` on the next pickup,+ with `attemptsMade` unchanged, so `fanOutDispatchGaveUp` checks for both.+- **A claim released after the give-up must settle itself.** When an enqueue fails, the dispatcher deletes the+ child row and clears the claim so a later attempt can retry it. If the give-up sweep already ran, no attempt is+ left, so the dispatcher then checks the source key and, if it is gone, marks the signal `NOT_DISPATCHED`. The+ give-up handler deletes the key before its sweep, so one of the two always settles the signal.+- **A chunk settles before the job fails.** `enqueueChunk` uses `Promise.allSettled`, so every release lands before+ BullMQ fires `failed`. With `Promise.all` the sweep ran while the other tasks still claimed and released.+- **Never widen a sweep to "all `PENDING`".** A dispatched signal stays `PENDING` with a `refId` until its child+ ends, and a body that waits on an approval keeps it there for days. A status-only sweep marks live children+ `NOT_DISPATCHED`, the barrier releases, and their outcomes report into nothing.+- **A child claimed just before a release never starts.** Its enqueue needs the barrier `PENDING`, so the+ dispatcher cancels it instead (`cancelUnstartedChildRun`). The release summary still counts it as `stillRunning`+ with its run id, because its claim came first.+- **The engine retries `POST /v1/waitpoints` on a 5xx, also after the API committed.** `create` keys the retry on+ the existing row's status. `PENDING` is reused, and `create` stores the source and adds the dispatch job again;+ the fixed job id `dispatch-<barrierId>` and the claims make that repeat safe. `COMPLETED` is reused with no+ dispatch, because fast children can release the barrier before the retry lands, and its summary waits for the+ deferred resume. Only `CONSUMED` is replaced. Deleting a `COMPLETED` barrier gave it fresh signals and ran every+ batch twice.+- **Run credits are checked once per chunk.** With none left, the dispatcher marks the unclaimed signals+ `NOT_DISPATCHED`; children already claimed keep running.+- **Each child carries its own copy of `seedSteps`**, inline below `WEBHOOK_PAYLOAD_INLINE_THRESHOLD_KB` and as a+ file above it.+- **The parent's run timeline no longer measures dispatch cost.** Its first leg is one HTTP call. Measure the span+ of the children's `created` timestamps instead.+- **`barrier-queue-factory.ts` is the module graph's acyclic leaf.** `barrierSourceKey` and the job types live+ there so `waitpoint-service.ts` imports the key without the queue. Folding it into `barrier-queue.ts` adds an+ import cycle.++## Key files+- `packages/server/api/src/app/waitpoints/` — `handleFanOutDispatch`, `barrierService.claimNextSignals`+- `packages/server/api/src/app/flows/flow-run/` — `prepareChildDispatch`, `createChildRuns`, `enqueueChildRun`+- `packages/server/engine/src/lib/operations/` — `flowOperation`, which enters a run at `entryStepName`diff --git a/bun.lock b/bun.lockindex edeb7844494..d703a850322 100644--- a/bun.lock+++ b/bun.lock@@ -122,7 +122,7 @@},"packages/core/execution": {"name": "@activepieces/core-execution",- "version": "0.27.0",+ "version": "0.28.0","dependencies": {"@activepieces/core-piece-types": "workspace:*","@activepieces/core-utils": "workspace:*",diff --git a/docs/install/architecture/waitpoints.mdx b/docs/install/architecture/waitpoints.mdxindex 8fc46ba9dd1..720d7c80091 100644--- a/docs/install/architecture/waitpoints.mdx+++ b/docs/install/architecture/waitpoints.mdx@@ -104,7 +104,7 @@ Duplicate callbacks are absorbed by the uniqueness constraint; the engine never| Method | Path | Notes || --- | --- | --- |-| `POST` | `/v1/waitpoints` | Engine-only. Creates a `PENDING` waitpoint, returns its resume URL. For `BARRIER`, also creates its signals. |+| `POST` | `/v1/waitpoints` | Engine-only. Creates a `PENDING` waitpoint, returns its resume URL. For `BARRIER`, also creates the signals and starts the fan-out. || `ALL` | `/v1/flow-runs/:id/signals/:signalId/confirm` | Records one signal's outcome. `GET`/`HEAD` serve the page and never decide; only `POST` does. || `ALL` | `/v1/flow-runs/:id/waitpoints/:waitpointId` | Resume, async. || `ALL` | `/v1/flow-runs/:id/waitpoints/:waitpointId/sync` | Resume, sync. The HTTP response is whatever the flow produces after resuming. |diff --git a/packages/core/execution/package.json b/packages/core/execution/package.jsonindex b7ec7e97a07..b80b463c0cf 100644--- a/packages/core/execution/package.json+++ b/packages/core/execution/package.json@@ -1,6 +1,6 @@{"name": "@activepieces/core-execution",- "version": "0.27.0",+ "version": "0.28.0","type": "commonjs","main": "./dist/src/index.js","scripts": {diff --git a/packages/core/execution/src/lib/engine/engine-operation.ts b/packages/core/execution/src/lib/engine/engine-operation.tsindex 94d446108ef..42f3f26fe65 100644--- a/packages/core/execution/src/lib/engine/engine-operation.ts+++ b/packages/core/execution/src/lib/engine/engine-operation.ts@@ -119,6 +119,7 @@ type BaseExecuteFlowOperation<T extends ExecutionType> = BaseEngineOperation & {stepNameToTest: string | nullsampleData?: Record<string, unknown>logsFileId?: string+ entryStepName?: string}export enum StreamStepProgress {diff --git a/packages/core/execution/src/lib/engine/execution-errors.ts b/packages/core/execution/src/lib/engine/execution-errors.tsindex 50ca456f80f..789b7e4492f 100644--- a/packages/core/execution/src/lib/engine/execution-errors.ts+++ b/packages/core/execution/src/lib/engine/execution-errors.ts@@ -136,6 +136,12 @@ export class WaitpointRejectedError extends ExecutionError {}}+export class InvalidSeededStepsError extends ExecutionError {+ constructor(entryStepName: string, cause?: unknown) {+ super('InvalidSeededStepsError', formatMessage(`A run entered at step ${entryStepName} needs the steps it starts after as its payload`), ExecutionErrorType.USER, cause)+ }+}+export class EngineGenericError extends ExecutionError {constructor(name: string, message: string, cause?: unknown) {super(name, formatMessage(message), ExecutionErrorType.ENGINE, cause)diff --git a/packages/core/execution/src/lib/flow-run/execution/execution-journal.ts b/packages/core/execution/src/lib/flow-run/execution/execution-journal.tsindex c4b318b0dec..1def2660139 100644--- a/packages/core/execution/src/lib/flow-run/execution/execution-journal.ts+++ b/packages/core/execution/src/lib/flow-run/execution/execution-journal.ts@@ -145,8 +145,11 @@ export const executionJournal = {if (step.type !== FlowActionType.LOOP_ON_ITEMS) continueif (!step.output?.iterations) continue- for (const iteration of step.output.iterations) {- const nestedPath = this.getPathToStep(iteration, stepName, loopsIndexes, [...currentPath, [currentStepName, loopsIndexes[currentStepName]]])+ const iterations = step.output.iterations+ const iterationIndex = Math.max(Math.min(loopsIndexes[currentStepName] ?? 0, iterations.length - 1), 0)++ for (const iteration of iterations) {+ const nestedPath = this.getPathToStep(iteration, stepName, loopsIndexes, [...currentPath, [currentStepName, iterationIndex]])if (nestedPath) return nestedPath}}diff --git a/packages/core/execution/src/lib/flow-run/execution/execution-output.ts b/packages/core/execution/src/lib/flow-run/execution/execution-output.tsindex 4d756408ce6..fd26e38fa88 100755--- a/packages/core/execution/src/lib/flow-run/execution/execution-output.ts+++ b/packages/core/execution/src/lib/flow-run/execution/execution-output.ts@@ -10,11 +10,13 @@ export enum ExecutionType {export type ExecutionState = {steps: Record<string, StepOutput>tags: string[]+ entryStepName?: string}export const ExecutionState = z.object({steps: z.record(z.string(), z.unknown()),tags: z.array(z.string()),+ entryStepName: z.string().optional(),})export enum RunInternalErrorSource {diff --git a/packages/core/execution/src/lib/flow-run/waitpoint/index.ts b/packages/core/execution/src/lib/flow-run/waitpoint/index.tsindex c4a57199f6f..a08d71f58d5 100644--- a/packages/core/execution/src/lib/flow-run/waitpoint/index.ts+++ b/packages/core/execution/src/lib/flow-run/waitpoint/index.ts@@ -1,6 +1,7 @@import { isNil } from '@activepieces/core-utils'import { z } from 'zod'import { RespondResponse } from '../execution/flow-execution'+import { StepOutputStatus } from '../execution/step-output'export function shouldReleaseBarrier({ policy, sealed, counts }: ShouldReleaseBarrierParams): boolean {if (policy?.releaseOnFirstFailure === true && UNFAVOURABLE_SIGNAL_STATUSES.some((status) => countOf({ counts, status }) > 0)) {@@ -42,8 +43,18 @@ export const BarrierPolicy = z.object({})export type BarrierPolicy = z.infer<typeof BarrierPolicy>+export const BarrierFanOutSource = z.object({+ entryStepName: z.string(),+ batchStepName: z.string(),+ batchSize: z.number().int().positive(),+ items: z.array(z.unknown()),+ seedSteps: z.record(z.string(), z.looseObject({ type: z.string(), status: z.enum(StepOutputStatus) })),+})+export type BarrierFanOutSource = z.infer<typeof BarrierFanOutSource>+export const CreateBarrierRequest = z.object({policy: BarrierPolicy.optional(),+ fanOut: BarrierFanOutSource.optional(),signals: z.array(z.object({ label: z.string().optional() })).optional(),})export type CreateBarrierRequest = z.infer<typeof CreateBarrierRequest>@@ -64,6 +75,7 @@ export type CreateWaitpointRequest = z.infer<typeof CreateWaitpointRequest>export const BarrierCreatedState = z.object({signalCount: z.number().int().nonnegative(),+ batchSize: z.number().int().positive(),signals: z.array(z.object({label: z.string().nullable(),confirmUrl: z.string(),diff --git a/packages/core/execution/src/lib/workers/job-data.ts b/packages/core/execution/src/lib/workers/job-data.tsindex 76f2542f475..955d8ca00f6 100644--- a/packages/core/execution/src/lib/workers/job-data.ts+++ b/packages/core/execution/src/lib/workers/job-data.ts@@ -139,6 +139,7 @@ const ExecuteFlowJobDataCommon = z.object({export const BeginExecuteFlowJobData = ExecuteFlowJobDataCommon.extend({executionType: z.literal(ExecutionType.BEGIN),executeTrigger: z.boolean().optional(),+ entryStepName: z.string().optional(),})export type BeginExecuteFlowJobData = z.infer<typeof BeginExecuteFlowJobData>diff --git a/packages/core/execution/test/flow-run/execution/execution-journal.test.ts b/packages/core/execution/test/flow-run/execution/execution-journal.test.tsindex 26f293c8509..e2f8ff0d96c 100644--- a/packages/core/execution/test/flow-run/execution/execution-journal.test.ts+++ b/packages/core/execution/test/flow-run/execution/execution-journal.test.ts@@ -93,6 +93,16 @@ describe('executionJournal.getPathToStep', () => {const step5_1 = executionJournal.getPathToStep(steps, 'step5', { 'step2': 1, 'step4': 0 })expect(step5_1).toEqual([['step2', 1], ['step4', 0]])})++ it('should clamp a pinned loop index to the iterations that exist', () => {+ const steps: Record<string, StepOutput> = {+ loop: createLoopWithIterations([{ inner: createCodeStep() }, { inner: createCodeStep() }]),+ }++ expect(executionJournal.getPathToStep(steps, 'inner', { loop: 13 })).toEqual([['loop', 1]])+ expect(executionJournal.getPathToStep(steps, 'inner', {})).toEqual([['loop', 0]])+ expect(executionJournal.getPathToStep(steps, 'inner', { loop: -1 })).toEqual([['loop', 0]])+ })})describe('executionJournal.getStateAtPath', () => {diff --git a/packages/core/execution/test/flow-run/waitpoint/barrier-fan-out-source.test.ts b/packages/core/execution/test/flow-run/waitpoint/barrier-fan-out-source.test.tsnew file mode 100644index 00000000000..748bdf3391a--- /dev/null+++ b/packages/core/execution/test/flow-run/waitpoint/barrier-fan-out-source.test.ts@@ -0,0 +1,23 @@+import { describe, expect, it } from 'vitest'+import { StepOutputStatus } from '../../../src/lib/flow-run/execution/step-output'+import { BarrierFanOutSource } from '../../../src/lib/flow-run/waitpoint'++describe('BarrierFanOutSource', () => {+ const source = {+ entryStepName: 'step_2',+ batchStepName: 'batch_1',+ batchSize: 2,+ items: [1, 2, 3],+ }++ it('accepts seed steps that are step outputs and keeps their other fields', () => {+ const seedSteps = { trigger: { type: 'EMPTY', status: StepOutputStatus.SUCCEEDED, input: {}, output: { id: 1 } } }++ expect(BarrierFanOutSource.parse({ ...source, seedSteps }).seedSteps).toEqual(seedSteps)+ })++ it('refuses a seed step that is not a step output, so a bad seed fails at the barrier create instead of in every child', () => {+ expect(BarrierFanOutSource.safeParse({ ...source, seedSteps: { trigger: 'not a step output' } }).success).toBe(false)+ expect(BarrierFanOutSource.safeParse({ ...source, seedSteps: { trigger: { type: 'EMPTY', status: 'DONE' } } }).success).toBe(false)+ })+})diff --git a/packages/server/api/src/app/database/redis/distributed-store-factory.ts b/packages/server/api/src/app/database/redis/distributed-store-factory.tsindex 640f3fffd34..bbdbdfadbb4 100644--- a/packages/server/api/src/app/database/redis/distributed-store-factory.ts+++ b/packages/server/api/src/app/database/redis/distributed-store-factory.ts@@ -34,6 +34,11 @@ export const distributedStoreFactory = (getRedisClient: () => Promise<Redis>) =>return JSON.parse(value) as T},+ async exists(key: string): Promise<boolean> {+ const redisClient = await getRedisClient()+ return await redisClient.exists(key) === 1+ },+async putIfAbsent(key: string, value: unknown, ttlInSeconds: number): Promise<boolean> {const serializedValue = JSON.stringify(value)const redisClient = await getRedisClient()diff --git a/packages/server/api/src/app/flows/flow-run/child-run-dispatch-service.ts b/packages/server/api/src/app/flows/flow-run/child-run-dispatch-service.tsnew file mode 100644index 00000000000..dd9350dabb9--- /dev/null+++ b/packages/server/api/src/app/flows/flow-run/child-run-dispatch-service.ts@@ -0,0 +1,133 @@+import { ActivepiecesError, ErrorCode, FlowId, FlowRunId, FlowVersionId, isNil, omit, PlatformId, ProjectId, tryCatch } from '@activepieces/core-utils'+import { ExecutionType, FlowRun, FlowRunStatus, flowStructureUtil, RunEnvironment, StreamStepProgress } from '@activepieces/shared'+import { FastifyBaseLogger } from 'fastify'+import { EntityManager } from 'typeorm'+import { projectService } from '../../project/project-service'+import { flowVersionService } from '../flow-version/flow-version.service'+import { addToQueue, cancelSingleRun, flowRunRepo } from './flow-run-service'+import { flowRunSideEffects } from './flow-run-side-effects'++export const childRunDispatchService = (log: FastifyBaseLogger) => ({+ async prepareChildDispatch({ projectId, parentRunId, entryStepName }: PrepareChildDispatchParams): Promise<ChildDispatchTarget> {+ const parentRun = await flowRunRepo().findOneBy({ id: parentRunId, projectId })+ if (isNil(parentRun)) {+ throw new ActivepiecesError({+ code: ErrorCode.VALIDATION,+ params: { message: `The parent run ${parentRunId} does not exist in this project, so no child can be dispatched for it.` },+ })+ }+ const flowVersion = await flowVersionService(log).getOneOrThrow(parentRun.flowVersionId)+ const entryStep = flowStructureUtil.getStep(entryStepName, flowVersion.trigger)+ if (isNil(entryStep) || !flowStructureUtil.isAction(entryStep.type)) {+ throw new ActivepiecesError({+ code: ErrorCode.VALIDATION,+ params: { message: `The step ${entryStepName} is not an action in this flow version, so a child run cannot start from it.` },+ })+ }+ return {+ projectId,+ parentRunId,+ entryStepName,+ flowId: parentRun.flowId,+ flowVersionId: parentRun.flowVersionId,+ environment: parentRun.environment,+ platformId: await projectService(log).getPlatformId(projectId),+ }+ },++ async createChildRuns({ target, parentWaitpointId, children, entityManager }: CreateChildRunsParams): Promise<FlowRun[]> {+ const now = new Date().toISOString()+ const childRuns: FlowRun[] = children.map(({ id, dispatchIndex }) => ({+ id,+ projectId: target.projectId,+ flowId: target.flowId,+ flowVersionId: target.flowVersionId,+ environment: target.environment,+ parentRunId: target.parentRunId,+ parentWaitpointId,+ dispatchIndex,+ failParentOnFailure: false,+ status: FlowRunStatus.QUEUED,+ created: now,+ updated: now,+ tags: [],+ steps: {},+ }))+ if (childRuns.length > 0) {+ await flowRunRepo(entityManager).insert(childRuns.map((childRun) => omit(childRun, ['steps'])))+ }+ return childRuns+ },++ async enqueueChildRun({ target, childRun, seedSteps }: EnqueueChildRunParams): Promise<void> {+ await addToQueue({+ flowRun: childRun,+ platformId: target.platformId,+ payload: seedSteps,+ entryStepName: target.entryStepName,+ preferInlinePayload: true,+ executionType: ExecutionType.BEGIN,+ executeTrigger: false,+ workerHandlerId: undefined,+ httpRequestId: undefined,+ streamStepProgress: StreamStepProgress.NONE,+ }, log)+ },++ async reportChildRunStarted({ target, childRun }: ChildRunParams): Promise<void> {+ const logFields = { flowRun: { id: childRun.id, dispatchIndex: childRun.dispatchIndex }, project: { id: target.projectId }, waitpoint: { id: childRun.parentWaitpointId } }+ const { error } = await tryCatch(() => flowRunSideEffects(log).onStart({ flowRun: childRun, platformId: target.platformId }))+ if (error) {+ log.error({ ...logFields, error }, '[childRunDispatchService#reportChildRunStarted] Child run was enqueued but its start side effects failed')+ }+ log.info(logFields, '[childRunDispatchService#reportChildRunStarted] Child run dispatched')+ },++ async cancelUnstartedChildRun({ target, childRun }: ChildRunParams): Promise<void> {+ await cancelSingleRun(log, childRun, target.platformId)+ },++ async deleteChildRun({ id, projectId, entityManager }: DeleteChildRunParams): Promise<void> {+ await flowRunRepo(entityManager).delete({ id, projectId })+ },+})++type PrepareChildDispatchParams = {+ projectId: ProjectId+ parentRunId: FlowRunId+ entryStepName: string+}++type CreateChildRunsParams = {+ target: ChildDispatchTarget+ parentWaitpointId: string+ children: { id: FlowRunId, dispatchIndex: number }[]+ entityManager: EntityManager+}++type EnqueueChildRunParams = {+ target: ChildDispatchTarget+ childRun: FlowRun+ seedSteps: Record<string, unknown>+}++type ChildRunParams = {+ target: ChildDispatchTarget+ childRun: FlowRun+}++type DeleteChildRunParams = {+ id: FlowRunId+ projectId: ProjectId+ entityManager: EntityManager+}++export type ChildDispatchTarget = {+ projectId: ProjectId+ parentRunId: FlowRunId+ entryStepName: string+ flowId: FlowId+ flowVersionId: FlowVersionId+ environment: RunEnvironment+ platformId: PlatformId+}diff --git a/packages/server/api/src/app/flows/flow-run/flow-run-entity.ts b/packages/server/api/src/app/flows/flow-run/flow-run-entity.tsindex f4b0aa9f616..5262f4814b4 100644--- a/packages/server/api/src/app/flows/flow-run/flow-run-entity.ts+++ b/packages/server/api/src/app/flows/flow-run/flow-run-entity.ts@@ -12,7 +12,7 @@ import {BaseColumnSchemaPart,} from '../../database/database-common'-type FlowRunSchema = FlowRunWithDispatchIndex & {+type FlowRunSchema = FlowRun & {project: Projectflow: FlowflowVersion: FlowVersion@@ -217,7 +217,3 @@ export const FlowRunEntity = new EntitySchema<FlowRunSchema>({},},})--export type FlowRunWithDispatchIndex = FlowRun & {- dispatchIndex?: number | null-}diff --git a/packages/server/api/src/app/flows/flow-run/flow-run-service.ts b/packages/server/api/src/app/flows/flow-run/flow-run-service.tsindex 86ca41c6a0f..5d54475e915 100644--- a/packages/server/api/src/app/flows/flow-run/flow-run-service.ts+++ b/packages/server/api/src/app/flows/flow-run/flow-run-service.ts@@ -521,7 +521,7 @@ export const flowRunService = (log: FastifyBaseLogger) => ({})-async function cancelSingleRun(log: FastifyBaseLogger, flowRun: FlowRun, platformId: string): Promise<void> {+export async function cancelSingleRun(log: FastifyBaseLogger, flowRun: FlowRun, platformId: string): Promise<void> {await distributedLock(log).runExclusive({key: `runs_metadata_${flowRun.id}`,timeoutInSeconds: 30,@@ -645,7 +645,7 @@ export async function addToQueue(params: AddToQueueParams, log: FastifyBaseLoggeconst logsFileId = params.flowRun.logsFileId ?? apId()let jobPayload: JobPayload = { type: 'inline', value: null }- if (!isNil(params.payload) && isNil(params.workerHandlerId)) {+ if (!isNil(params.payload) && isNil(params.workerHandlerId) && !(params.preferInlinePayload ?? false)) {jobPayload = await payloadOffloader.offloadPayload(log, params.payload, params.flowRun.projectId, params.platformId)}else if (!isNil(params.payload)) {@@ -680,6 +680,7 @@ export async function addToQueue(params: AddToQueueParams, log: FastifyBaseLogge...commonJobData,executionType: ExecutionType.BEGIN,executeTrigger: params.executeTrigger,+ entryStepName: params.entryStepName,}const aParentIsBlockedOnThisRun = params.flowRun.failParentOnFailure && !isNil(params.flowRun.parentRunId)await jobQueue(log).add({@@ -872,10 +873,11 @@ type AddToQueueParamsCommon = {streamStepProgress: StreamStepProgresssampleData?: Record<string, unknown>jobId?: string+ preferInlinePayload?: boolean}export type AddToQueueParams = AddToQueueParamsCommon & (- | { executionType: ExecutionType.BEGIN, executeTrigger: boolean }+ | { executionType: ExecutionType.BEGIN, executeTrigger: boolean, entryStepName?: string }| { executionType: ExecutionType.RESUME, resumeReason: ResumeReason })diff --git a/packages/server/api/src/app/waitpoints/barrier-queue-factory.ts b/packages/server/api/src/app/waitpoints/barrier-queue-factory.tsindex d357b8d2f76..8c8b2e935b6 100644--- a/packages/server/api/src/app/waitpoints/barrier-queue-factory.ts+++ b/packages/server/api/src/app/waitpoints/barrier-queue-factory.ts@@ -3,6 +3,8 @@ import { Queue } from 'bullmq'import Redis from 'ioredis'import { QueueName } from '../workers/job'+export const barrierSourceKey = (barrierId: string): string => `barrier_source:${barrierId}`+export const evaluationDedupKey = (barrierId: string): string => `evaluate-${barrierId}`export const barrierQueueFactory = ({ createRedisConnection }: BarrierQueueFactoryParams) => {@@ -41,6 +43,18 @@ export const barrierQueueFactory = ({ createRedisConnection }: BarrierQueueFacto})},+ async addFanOutDispatch(params: BarrierJobData): Promise<void> {+ await requireQueue().add(BarrierJobName.FAN_OUT_DISPATCH, params, {+ jobId: `dispatch-${params.barrierId}`,+ })+ },++ async addFanOutGiveUp(params: BarrierJobData): Promise<void> {+ await requireQueue().add(BarrierJobName.FAN_OUT_GIVE_UP, params, {+ jobId: `give-up-${params.barrierId}`,+ })+ },+async clearEvaluationDedupKey(barrierId: string): Promise<void> {await requireQueue().removeDeduplicationKey(evaluationDedupKey(barrierId))},@@ -57,6 +71,8 @@ export const barrierQueueFactory = ({ createRedisConnection }: BarrierQueueFactoexport enum BarrierJobName {EVALUATE = 'evaluate-barrier',+ FAN_OUT_DISPATCH = 'fan-out-dispatch',+ FAN_OUT_GIVE_UP = 'fan-out-give-up',}export type BarrierJobData = {@@ -69,6 +85,13 @@ export type BarrierQueueConfig = {redisFailedJobRetentionMaxCount: number}+export type BarrierFanOutPayload = {+ entryStepName: string+ batchStepName: string+ seedSteps: Record<string, unknown>+ batches: unknown[][]+}+type BarrierQueueFactoryParams = {createRedisConnection: () => Promise<Redis>}diff --git a/packages/server/api/src/app/waitpoints/barrier-queue.ts b/packages/server/api/src/app/waitpoints/barrier-queue.tsindex 59354d1ddfe..2c4ad15f293 100644--- a/packages/server/api/src/app/waitpoints/barrier-queue.ts+++ b/packages/server/api/src/app/waitpoints/barrier-queue.ts@@ -1,12 +1,15 @@+import { isNil } from '@activepieces/core-utils'import { createLogger, wideEvent } from '@activepieces/server-utils'import { Job, Queue, Worker } from 'bullmq'import { FastifyBaseLogger } from 'fastify'import { redisConnections } from '../database/redis-connections'+import { exceptionHandler } from '../helper/exception-handler'import { system } from '../helper/system/system'import { AppSystemProp } from '../helper/system/system-props'import { QueueName } from '../workers/job'import { BarrierJobData, BarrierJobName, BarrierQueueConfig, barrierQueueFactory } from './barrier-queue-factory'import { barrierService } from './barrier-service'+import { handleFanOutDispatch, handleFanOutDispatchFailed, handleFanOutDispatchGaveUp } from './fan-out-dispatcher-job'let barrierWorker: Worker<BarrierJobData> | undefined = undefined@@ -43,6 +46,15 @@ export const barrierQueue = (log: FastifyBaseLogger) => ({autorun: true,},)+ barrierWorker.on('failed', (job, error) => {+ if (isNil(job) || job.name !== BarrierJobName.FAN_OUT_DISPATCH) {+ return+ }+ handleFanOutDispatchFailed({ data: job.data, attemptsMade: job.attemptsMade, attempts: job.opts.attempts, error, log }).catch((handlerError: unknown) => {+ exceptionHandler.handle(handlerError, log)+ })+ })+await barrierWorker.waitUntilReady()},@@ -50,6 +62,14 @@ export const barrierQueue = (log: FastifyBaseLogger) => ({await barrierQueueInstance.enqueueEvaluation(params)},+ async addFanOutDispatch(params: BarrierJobData): Promise<void> {+ await barrierQueueInstance.addFanOutDispatch(params)+ },++ async addFanOutGiveUp(params: BarrierJobData): Promise<void> {+ await barrierQueueInstance.addFanOutGiveUp(params)+ },+async clearEvaluationDedupKey(barrierId: string): Promise<void> {await barrierQueueInstance.clearEvaluationDedupKey(barrierId)},@@ -74,6 +94,12 @@ async function processBarrierJob({ job, log }: ProcessBarrierJobParams): Promiseawait barrierQueueInstance.clearEvaluationDedupKey(job.data.barrierId)await barrierService(log).releaseIfReady({ barrierId: job.data.barrierId, projectId: job.data.projectId })return+ case BarrierJobName.FAN_OUT_DISPATCH:+ await handleFanOutDispatch({ data: job.data, log })+ return+ case BarrierJobName.FAN_OUT_GIVE_UP:+ await handleFanOutDispatchGaveUp({ data: job.data, log })+ returndefault:log.warn({ job: { id: job.id, type: job.name } }, '[barrierQueue#worker] Unknown barrier job name, dropping it')}diff --git a/packages/server/api/src/app/waitpoints/barrier-service.ts b/packages/server/api/src/app/waitpoints/barrier-service.tsindex 5ed9cfabbbc..0285370425c 100644--- a/packages/server/api/src/app/waitpoints/barrier-service.ts+++ b/packages/server/api/src/app/waitpoints/barrier-service.ts@@ -1,17 +1,19 @@-import { apId, chunk, isNil, sanitizeObjectForPostgresql, tryCatch, unique } from '@activepieces/core-utils'-import { wideEvent } from '@activepieces/server-utils'+import { apId, chunk, isNil, sanitizeObjectForPostgresql, spreadIfDefined, tryCatch, unique } from '@activepieces/core-utils'+import { apDayjsDuration, wideEvent } from '@activepieces/server-utils'import { ActivepiecesError, BarrierPolicy, barrierReleasesOnLastPendingSignal, BarrierSignalCounts, BarrierSignalStatus, BarrierSummary, ErrorCode, MAX_INLINE_BARRIER_SIGNALS, PauseType, RespondResponse, shouldReleaseBarrier, WaitpointVersion } from '@activepieces/shared'import dayjs from 'dayjs'import { FastifyBaseLogger } from 'fastify'-import { EntityManager } from 'typeorm'+import { EntityManager, FindOptionsWhere, In, IsNull } from 'typeorm'import { z } from 'zod'import { repoFactory } from '../core/db/repo-factory'import { transaction } from '../core/db/transaction'+import { distributedStore } from '../database/redis-connections'import { flowRunRepo } from '../flows/flow-run/flow-run-service'import { system } from '../helper/system/system'import { AppSystemProp } from '../helper/system/system-props'import { platformConfigurationService } from '../platform/platform-configuration.service'import { barrierQueue } from './barrier-queue'+import { BarrierFanOutPayload, barrierSourceKey } from './barrier-queue-factory'import { resumeService } from './resume-service'import { WaitpointEntity } from './waitpoint-entity'import { WaitpointSignalEntity } from './waitpoint-signal-entity'@@ -23,15 +25,19 @@ const signalRepo = repoFactory(WaitpointSignalEntity)export const barrierService = (log: FastifyBaseLogger) => ({async create(params: CreateBarrierParams): Promise<CreateBarrierResult> {+ const maxSignals = await platformConfigurationService(log).maxBarrierSignals({ platformId: params.platformId })+ const batchSize = clampBatchSize({ itemCount: params.fanOut?.items.length ?? 0, requested: params.fanOut?.batchSize ?? 1, maxSignals })+ const batches = isNil(params.fanOut) ? [] : chunk(params.fanOut.items, batchSize)const labels = params.signalLabels ?? []- await assertSignalCountWithinLimit({ signalCount: labels.length, platformId: params.platformId, log })+ const signalCount = isNil(params.fanOut) ? labels.length : batches.length+ assertSignalCountWithinLimit({ signalCount, maxSignals })const flowRun = await flowRunRepo().findOneByOrFail({ id: params.flowRunId, projectId: params.projectId })const creation = await transaction(async (entityManager) => {const repo = waitpointRepo(entityManager)const existing = await repo.findOneBy({ flowRunId: params.flowRunId, projectId: params.projectId, stepName: params.stepName })- if (!isNil(existing) && existing.status === WaitpointStatus.PENDING) {- log.info({ flowRun: { id: params.flowRunId }, waitpoint: { id: existing.id } }, '[barrierService#create] Barrier already open for this step, reusing it')+ if (!isNil(existing) && existing.status !== WaitpointStatus.CONSUMED) {+ log.info({ flowRun: { id: params.flowRunId }, waitpoint: { id: existing.id, status: existing.status } }, '[barrierService#create] Barrier already exists for this step and its resume is not delivered, reusing it')return { inserted: false, barrier: existing, signals: await signalRepo(entityManager).findBy({ waitpointId: existing.id, projectId: params.projectId }) }}if (!isNil(existing)) {@@ -58,13 +64,21 @@ export const barrierService = (log: FastifyBaseLogger) => ({deadLetteredAt: null,}await repo.createQueryBuilder().insert().into('waitpoint').values(barrier).execute()- const signals = buildPendingSignals({ barrierId: barrier.id, projectId: params.projectId, labels })+ const signals = buildPendingSignals({ barrierId: barrier.id, projectId: params.projectId, signalCount, labels, fanOut: !isNil(params.fanOut) })for (const rows of chunk(signals, SIGNAL_INSERT_BATCH_SIZE)) {await signalRepo(entityManager).createQueryBuilder().insert().into('waitpoint_signal').values(rows).execute()}return { inserted: true, barrier, signals }})+ if (!isNil(params.fanOut) && creation.barrier.status === WaitpointStatus.PENDING) {+ await startFanOut({+ barrierId: creation.barrier.id,+ projectId: params.projectId,+ source: { entryStepName: params.fanOut.entryStepName, batchStepName: params.fanOut.batchStepName, seedSteps: params.fanOut.seedSteps, batches },+ log,+ })+ }if (creation.inserted) {if (!isNil(creation.barrier.resumeDateTime)) {await waitpointTimeoutJob.schedule({@@ -78,7 +92,7 @@ export const barrierService = (log: FastifyBaseLogger) => ({await barrierQueue(log).enqueueEvaluation({ barrierId: creation.barrier.id, projectId: params.projectId })}- return { barrier: creation.barrier, signals: creation.signals, signalCount: creation.signals.length }+ return { barrier: creation.barrier, signals: creation.signals, signalCount: creation.signals.length, batchSize }},async findById({ barrierId, projectId }: FindByIdParams): Promise<Waitpoint | null> {@@ -89,23 +103,70 @@ export const barrierService = (log: FastifyBaseLogger) => ({return signalRepo().findOneBy({ id: signalId, projectId })},+ async hasUnclaimedSignals({ barrierId, projectId }: BarrierScopeParams): Promise<boolean> {+ return signalRepo().existsBy(unclaimedSignalWhere({ barrierId, projectId }))+ },++ async claimNextSignals({ barrierId, projectId, afterSequence, limit, entityManager }: ClaimNextSignalsParams): Promise<SignalClaimResult> {+ const openBarrier = await lockOpenBarrier({ barrierId, projectId, entityManager })+ if (isNil(openBarrier)) {+ return { barrierOpen: false }+ }+ const candidates = ClaimCandidateRows.parse(await signalRepo(entityManager).query(+ 'SELECT "id", "sequence" FROM waitpoint_signal WHERE "waitpointId" = $1 AND "projectId" = $2 AND "status" = $3 AND "refId" IS NULL AND "sequence" > $4 ORDER BY "sequence" LIMIT $5 FOR UPDATE SKIP LOCKED',+ [barrierId, projectId, BarrierSignalStatus.PENDING, afterSequence, limit],+ ))+ if (candidates.length === 0) {+ return { barrierOpen: true, claims: [], lastSequence: null }+ }+ const childRunIds = candidates.map(() => apId())+ const claimed = await signalRepo(entityManager)+ .createQueryBuilder()+ .update()+ .set({ refId: () => `CASE "id" ${candidates.map((_, index) => `WHEN :signalId${index} THEN :childRunId${index}`).join(' ')} END` })+ .setParameters(Object.fromEntries(candidates.flatMap((candidate, index) => [[`signalId${index}`, candidate.id], [`childRunId${index}`, childRunIds[index]]])))+ .where({ id: In(candidates.map((candidate) => candidate.id)), projectId, status: BarrierSignalStatus.PENDING, refId: IsNull() })+ .returning(['id', 'sequence', 'refId'])+ .execute()+ const claims = ClaimedSignalRows.parse(claimed.raw ?? []).map((row) => ({ signalId: row.id, sequence: row.sequence, childRunId: row.refId }))+ return { barrierOpen: true, claims, lastSequence: candidates[candidates.length - 1].sequence }+ },++ async lockForDispatch({ barrierId, projectId, entityManager }: LockForDispatchParams): Promise<boolean> {+ return !isNil(await lockOpenBarrier({ barrierId, projectId, entityManager }))+ },++ async releaseClaim({ signalId, refId, projectId, entityManager }: ReleaseClaimParams): Promise<void> {+ await signalRepo(entityManager).update({ id: signalId, refId, projectId }, { refId: null })+ },++ async markUnclaimedNotDispatched({ barrierId, projectId }: BarrierScopeParams): Promise<number> {+ const result = await signalRepo().update(unclaimedSignalWhere({ barrierId, projectId }), { status: BarrierSignalStatus.NOT_DISPATCHED })+ return result.affected ?? 0+ },+async receiveSignal({ signalId, refId, projectId, status, result }: ReceiveSignalParams): Promise<boolean> {if (isNil(signalId) && isNil(refId)) {return false}const identity: SignalIdentity = isNil(signalId) ? { refId } : { id: signalId }const barrierIds = await updateSignalOutcome({ where: { ...identity, projectId }, status, result })+ if (barrierIds.length === 0) {+ log.warn({+ ...spreadIfDefined('flowRun', isNil(refId) ? undefined : { id: refId }),+ ...spreadIfDefined('signal', isNil(signalId) ? undefined : { id: signalId }),+ project: { id: projectId },+ outcome: status,+ }, '[barrierService#receiveSignal] No signal row matched this outcome, so it is being dropped; the barrier was released or the row was never claimed by this ref')+ return false+ }await Promise.all(barrierIds.map((barrierId) => barrierQueue(log).enqueueEvaluation({ barrierId, projectId })))- return barrierIds.length > 0+ return true},async recordDecision({ barrierId, signalId, projectId, status, result }: RecordDecisionParams): Promise<boolean> {const recorded = await transaction(async (entityManager) => {- const openBarrier = await waitpointRepo(entityManager)- .createQueryBuilder('waitpoint')- .setLock('pessimistic_read')- .where({ id: barrierId, projectId, type: PauseType.BARRIER, status: WaitpointStatus.PENDING })- .getOne()+ const openBarrier = await lockOpenBarrier({ barrierId, projectId, entityManager })if (isNil(openBarrier)) {return false}@@ -144,6 +205,7 @@ export const barrierService = (log: FastifyBaseLogger) => ({async release({ barrier, timedOut, releaseReason }: ReleaseParams): Promise<BarrierSummary | null> {const summary = await closeBarrier({ barrier, timedOut })+ await distributedStore.delete(barrierSourceKey(barrier.id))if (isNil(summary)) {log.info({ waitpoint: { id: barrier.id }, flowRun: { id: barrier.flowRunId } }, '[barrierService#release] Barrier was already closed by another release; leaving the resume to the release that closed it')return null@@ -180,9 +242,9 @@ async function updateSignalOutcome({ where, status, result, entityManager }: Updreturn unique(UpdatedSignalRows.parse(updateResult.raw ?? []).map((row) => row.waitpointId))}-function buildPendingSignals({ barrierId, projectId, labels }: BuildPendingSignalsParams): WaitpointSignal[] {+function buildPendingSignals({ barrierId, projectId, signalCount, labels, fanOut }: BuildPendingSignalsParams): WaitpointSignal[] {const now = new Date().toISOString()- return labels.map((label) => ({+ return Array.from({ length: signalCount }, (_, index) => ({id: apId(),created: now,updated: now,@@ -190,12 +252,29 @@ function buildPendingSignals({ barrierId, projectId, labels }: BuildPendingSignaprojectId,status: BarrierSignalStatus.PENDING,refId: null,- sequence: null,- label,+ sequence: fanOut ? index : null,+ label: labels[index] ?? null,result: null,}))}+function unclaimedSignalWhere({ barrierId, projectId }: BarrierScopeParams): FindOptionsWhere<WaitpointSignal> {+ return { waitpointId: barrierId, projectId, status: BarrierSignalStatus.PENDING, refId: IsNull() }+}++async function lockOpenBarrier({ barrierId, projectId, entityManager }: LockForDispatchParams): Promise<Waitpoint | null> {+ return waitpointRepo(entityManager)+ .createQueryBuilder('waitpoint')+ .setLock('pessimistic_read')+ .where({ id: barrierId, projectId, type: PauseType.BARRIER, status: WaitpointStatus.PENDING })+ .getOne()+}++async function startFanOut({ barrierId, projectId, source, log }: StartFanOutParams): Promise<void> {+ await distributedStore.put(barrierSourceKey(barrierId), source, SOURCE_TTL_SECONDS)+ await barrierQueue(log).addFanOutDispatch({ barrierId, projectId })+}+async function closeBarrier({ barrier, timedOut }: CloseBarrierParams): Promise<BarrierSummary | null> {return transaction(async (entityManager) => {const repo = waitpointRepo(entityManager)@@ -282,8 +361,12 @@ async function buildSummary({ barrierId, projectId, timedOut, entityManager }: B}}-async function assertSignalCountWithinLimit({ signalCount, platformId, log }: AssertSignalCountWithinLimitParams): Promise<void> {- const maxSignals = await platformConfigurationService(log).maxBarrierSignals({ platformId })+function clampBatchSize({ itemCount, requested, maxSignals }: ClampBatchSizeParams): number {+ const minimumBatchSize = Math.ceil(itemCount / maxSignals)+ return Math.max(requested, minimumBatchSize, 1)+}++function assertSignalCountWithinLimit({ signalCount, maxSignals }: AssertSignalCountWithinLimitParams): void {if (signalCount > maxSignals) {throw new ActivepiecesError({code: ErrorCode.VALIDATION,@@ -301,6 +384,12 @@ const SIGNAL_INSERT_BATCH_SIZE = 500const UpdatedSignalRows = z.array(z.object({ waitpointId: z.string() }))+const ClaimCandidateRows = z.array(z.object({ id: z.string(), sequence: z.number().int() }))++const ClaimedSignalRows = z.array(z.object({ id: z.string(), sequence: z.number().int(), refId: z.string() }))++const SOURCE_TTL_SECONDS = apDayjsDuration(1, 'day').asSeconds()+export type BarrierReleaseReason = 'predicate' | 'timeout'export type CreateBarrierParams = {@@ -314,12 +403,20 @@ export type CreateBarrierParams = {httpRequestId?: stringpolicy?: BarrierPolicysignalLabels?: (string | null)[]+ fanOut?: {+ entryStepName: string+ batchStepName: string+ batchSize: number+ items: unknown[]+ seedSteps: Record<string, unknown>+ }}export type CreateBarrierResult = {barrier: Waitpointsignals: WaitpointSignal[]signalCount: number+ batchSize: number}export type ReceiveSignalParams = {@@ -338,6 +435,16 @@ export type RecordDecisionParams = {result?: Record<string, unknown>}+export type SignalClaim = {+ signalId: string+ sequence: number+ childRunId: string+}++export type SignalClaimResult =+ | { barrierOpen: false }+ | { barrierOpen: true, claims: SignalClaim[], lastSequence: number | null }+type SignalIdentity = {id?: stringrefId?: string@@ -359,6 +466,35 @@ type FindSignalByIdParams = {projectId: string}+type BarrierScopeParams = {+ barrierId: string+ projectId: string+}++type ClaimNextSignalsParams = BarrierScopeParams & {+ afterSequence: number+ limit: number+ entityManager: EntityManager+}++type LockForDispatchParams = BarrierScopeParams & {+ entityManager: EntityManager+}++type ReleaseClaimParams = {+ signalId: string+ refId: string+ projectId: string+ entityManager?: EntityManager+}++type StartFanOutParams = {+ barrierId: string+ projectId: string+ source: BarrierFanOutPayload+ log: FastifyBaseLogger+}+type ReleaseIfReadyParams = {barrierId: stringprojectId: string@@ -403,13 +539,20 @@ type BuildSummaryParams = CountSignalsByStatusParams & {type BuildPendingSignalsParams = {barrierId: stringprojectId: string+ signalCount: numberlabels: (string | null)[]+ fanOut: boolean+}++type ClampBatchSizeParams = {+ itemCount: number+ requested: number+ maxSignals: number}type AssertSignalCountWithinLimitParams = {signalCount: number- platformId: string- log: FastifyBaseLogger+ maxSignals: number}type DefaultBarrierDeadlineParams = {diff --git a/packages/server/api/src/app/waitpoints/fan-out-dispatcher-job.ts b/packages/server/api/src/app/waitpoints/fan-out-dispatcher-job.tsnew file mode 100644index 00000000000..2d8094b5ffb--- /dev/null+++ b/packages/server/api/src/app/waitpoints/fan-out-dispatcher-job.ts@@ -0,0 +1,226 @@+import { isNil, tryCatch } from '@activepieces/core-utils'+import { FlowRun } from '@activepieces/shared'+import { UnrecoverableError } from 'bullmq'+import { FastifyBaseLogger } from 'fastify'+import pLimit from 'p-limit'+import { transaction } from '../core/db/transaction'+import { distributedStore } from '../database/redis-connections'+import { ChildDispatchTarget, childRunDispatchService } from '../flows/flow-run/child-run-dispatch-service'++import { shouldBlockRunOnCredits } from '../platform/billing-provider'+import { barrierQueue } from './barrier-queue'+import { BarrierFanOutPayload, BarrierJobData, barrierSourceKey } from './barrier-queue-factory'+import { barrierService, SignalClaim } from './barrier-service'+import { WaitpointStatus } from './waitpoint-types'++export async function handleFanOutDispatch({ data, log }: JobParams): Promise<void> {+ const barrier = await barrierService(log).findById({ barrierId: data.barrierId, projectId: data.projectId })+ if (isNil(barrier) || barrier.status !== WaitpointStatus.PENDING) {+ log.info({ waitpoint: { id: data.barrierId } }, '[fanOutDispatcher] Barrier is gone or already released, stopping without dispatching')+ return+ }++ const source = await distributedStore.get<BarrierFanOutPayload>(barrierSourceKey(data.barrierId))+ if (isNil(source)) {+ const notDispatchedCount = await settleUnclaimedSignals({ data, log })+ log.warn({ waitpoint: { id: data.barrierId }, notDispatchedCount }, '[fanOutDispatcher] The stored source is gone, marking the unclaimed signals undispatched so the barrier can settle')+ return+ }++ if (!await barrierService(log).hasUnclaimedSignals({ barrierId: data.barrierId, projectId: data.projectId })) {+ await barrierQueue(log).enqueueEvaluation({ barrierId: data.barrierId, projectId: data.projectId })+ return+ }++ const target = await childRunDispatchService(log).prepareChildDispatch({+ projectId: data.projectId,+ parentRunId: barrier.flowRunId,+ entryStepName: source.entryStepName,+ })+ const outcome = await dispatchChunks({ data, source, target, log })+ switch (outcome) {+ case 'barrier-closed':+ log.info({ waitpoint: { id: data.barrierId } }, '[fanOutDispatcher] The barrier was released or cancelled mid-fan-out, stopping without retrying')+ return+ case 'credits-exhausted': {+ const notDispatchedCount = await settleUnclaimedSignals({ data, log })+ log.warn({ waitpoint: { id: data.barrierId }, platform: { id: target.platformId }, notDispatchedCount }, '[fanOutDispatcher] The platform ran out of run credits, marking the unclaimed signals undispatched')+ return+ }+ case 'dispatched':+ await barrierQueue(log).enqueueEvaluation({ barrierId: data.barrierId, projectId: data.projectId })+ }+}++export function fanOutDispatchGaveUp({ attemptsMade, attempts, error }: FanOutDispatchGaveUpParams): GiveUpReason | null {+ if (error instanceof UnrecoverableError) {+ return 'unrecoverable'+ }+ if (attemptsMade >= (attempts ?? 1)) {+ return 'exhausted'+ }+ return null+}++export async function handleFanOutDispatchFailed({ data, attemptsMade, attempts, error, log }: HandleFanOutDispatchFailedParams): Promise<void> {+ const reason = fanOutDispatchGaveUp({ attemptsMade, attempts, error })+ if (isNil(reason)) {+ return+ }+ log.error({ error, waitpoint: { id: data.barrierId }, giveUpReason: reason }, '[fanOutDispatcher] Dispatch gave up, queueing the settlement of its unclaimed signals')+ await barrierQueue(log).addFanOutGiveUp(data)+}++export async function handleFanOutDispatchGaveUp({ data, log }: JobParams): Promise<void> {+ await distributedStore.delete(barrierSourceKey(data.barrierId))+ const notDispatchedCount = await settleUnclaimedSignals({ data, log })+ log.warn({ waitpoint: { id: data.barrierId }, notDispatchedCount }, '[fanOutDispatcher] Marked the unclaimed signals undispatched after dispatch gave up; signals already claimed keep their live children and release the barrier once those report')+}++async function dispatchChunks({ data, source, target, log }: DispatchChunksParams): Promise<DispatchOutcome> {+ let afterSequence: number | null = START_OF_BARRIER+ while (!isNil(afterSequence)) {+ if (await shouldBlockRunOnCredits({ platformId: target.platformId, environment: target.environment, log })) {+ return 'credits-exhausted'+ }+ const cursor: number = afterSequence+ const chunk = await transaction(async (entityManager) => {+ const claimResult = await barrierService(log).claimNextSignals({ barrierId: data.barrierId, projectId: data.projectId, afterSequence: cursor, limit: CLAIM_CHUNK_SIZE, entityManager })+ if (!claimResult.barrierOpen) {+ return null+ }+ const childRuns = await childRunDispatchService(log).createChildRuns({+ target,+ parentWaitpointId: data.barrierId,+ children: claimResult.claims.map((claim) => ({ id: claim.childRunId, dispatchIndex: claim.sequence })),+ entityManager,+ })+ return { claims: claimResult.claims, childRuns, lastSequence: claimResult.lastSequence }+ })+ if (isNil(chunk)) {+ return 'barrier-closed'+ }+ const outcomes = await enqueueChunk({ data, source, target, claims: chunk.claims, childRuns: chunk.childRuns, log })+ if (outcomes.includes('barrier-closed')) {+ return 'barrier-closed'+ }+ afterSequence = chunk.lastSequence ?? (cursor === START_OF_BARRIER ? null : START_OF_BARRIER)+ }+ return 'dispatched'+}++async function enqueueChunk({ data, source, target, claims, childRuns, log }: EnqueueChunkParams): Promise<EnqueueOutcome[]> {+ const limit = pLimit(MAX_ENQUEUES_IN_FLIGHT)+ const results = await Promise.allSettled(claims.map((claim, index) => limit(() => enqueueChild({+ data,+ claim,+ childRun: childRuns[index],+ seedSteps: { ...source.seedSteps, [source.batchStepName]: { type: BATCH_STEP_TYPE, status: 'SUCCEEDED', input: {}, output: { items: source.batches[claim.sequence] ?? [] } } },+ target,+ log,+ }))))+ const failure = results.find((result): result is PromiseRejectedResult => result.status === 'rejected')+ if (!isNil(failure)) {+ throw failure.reason+ }+ return results.flatMap((result) => result.status === 'fulfilled' ? [result.value] : [])+}++async function enqueueChild({ data, claim, childRun, seedSteps, target, log }: EnqueueChildParams): Promise<EnqueueOutcome> {+ const attempt = await transaction(async (entityManager): Promise<EnqueueAttempt> => {+ const barrierOpen = await barrierService(log).lockForDispatch({ barrierId: data.barrierId, projectId: data.projectId, entityManager })+ if (!barrierOpen) {+ return { outcome: 'barrier-closed' }+ }+ const { error } = await tryCatch(() => childRunDispatchService(log).enqueueChildRun({ target, childRun, seedSteps }))+ if (isNil(error)) {+ return { outcome: 'enqueued' }+ }+ await childRunDispatchService(log).deleteChildRun({ id: childRun.id, projectId: data.projectId, entityManager })+ await barrierService(log).releaseClaim({ signalId: claim.signalId, refId: childRun.id, projectId: data.projectId, entityManager })+ return { outcome: 'failed', error }+ })+ switch (attempt.outcome) {+ case 'enqueued':+ await childRunDispatchService(log).reportChildRunStarted({ target, childRun })+ return 'enqueued'+ case 'barrier-closed':+ await childRunDispatchService(log).cancelUnstartedChildRun({ target, childRun })+ return 'barrier-closed'+ case 'failed':+ await settleIfDispatchGaveUp({ data, log })+ throw attempt.error+ }+}++async function settleIfDispatchGaveUp({ data, log }: JobParams): Promise<void> {+ if (await distributedStore.exists(barrierSourceKey(data.barrierId))) {+ return+ }+ const notDispatchedCount = await settleUnclaimedSignals({ data, log })+ log.warn({ waitpoint: { id: data.barrierId }, notDispatchedCount }, '[fanOutDispatcher] A claim was released after dispatch gave up, marking it undispatched so the barrier can settle')+}++async function settleUnclaimedSignals({ data, log }: JobParams): Promise<number> {+ const notDispatchedCount = await barrierService(log).markUnclaimedNotDispatched({ barrierId: data.barrierId, projectId: data.projectId })+ await barrierQueue(log).enqueueEvaluation({ barrierId: data.barrierId, projectId: data.projectId })+ return notDispatchedCount+}++const BATCH_STEP_TYPE = 'PROCESS_IN_BATCHES'++const START_OF_BARRIER = -1++const CLAIM_CHUNK_SIZE = 25++const MAX_ENQUEUES_IN_FLIGHT = 5++type DispatchOutcome = 'dispatched' | 'barrier-closed' | 'credits-exhausted'++type EnqueueOutcome = 'enqueued' | 'barrier-closed'++type EnqueueAttempt =+ | { outcome: EnqueueOutcome }+ | { outcome: 'failed', error: Error }++type JobParams = {+ data: BarrierJobData+ log: FastifyBaseLogger+}++type HandleFanOutDispatchFailedParams = {+ data: BarrierJobData+ attemptsMade: number+ attempts?: number+ error: Error+ log: FastifyBaseLogger+}++type FanOutDispatchGaveUpParams = {+ attemptsMade: number+ attempts?: number+ error: Error+}++type DispatchChunksParams = {+ data: BarrierJobData+ source: BarrierFanOutPayload+ target: ChildDispatchTarget+ log: FastifyBaseLogger+}++type EnqueueChunkParams = DispatchChunksParams & {+ claims: SignalClaim[]+ childRuns: FlowRun[]+}++type EnqueueChildParams = {+ data: BarrierJobData+ claim: SignalClaim+ childRun: FlowRun+ seedSteps: Record<string, unknown>+ target: ChildDispatchTarget+ log: FastifyBaseLogger+}++export type GiveUpReason = 'exhausted' | 'unrecoverable'diff --git a/packages/server/api/src/app/waitpoints/waitpoint-controller.ts b/packages/server/api/src/app/waitpoints/waitpoint-controller.tsindex b86db4d59a1..f6b643ef3ac 100644--- a/packages/server/api/src/app/waitpoints/waitpoint-controller.ts+++ b/packages/server/api/src/app/waitpoints/waitpoint-controller.ts@@ -26,6 +26,7 @@ export const waitpointController: FastifyPluginAsyncZod = async (app) => {httpRequestId: httpRequestId ?? undefined,policy: barrier?.policy,signalLabels: barrier?.signals?.map((signal) => signal.label ?? null),+ fanOut: barrier?.fanOut,})return reply.status(StatusCodes.CREATED).send({id: created.barrier.id,@@ -61,6 +62,7 @@ async function buildBarrierState({ flowRunId, created }: BuildBarrierStateParamsconst truncatedForSize = created.signals.length > MAX_INLINE_BARRIER_SIGNALSreturn {signalCount: created.signalCount,+ batchSize: created.batchSize,...(truncatedForSize ? { signalsTruncated: true } : {}),...(inlineable ? { signals: await Promise.all(created.signals.map((signal) => toSignalLink({ flowRunId, signal }))) } : {}),}diff --git a/packages/server/api/src/app/waitpoints/waitpoint-service.ts b/packages/server/api/src/app/waitpoints/waitpoint-service.tsindex 6536da15d55..737d0e7ec49 100644--- a/packages/server/api/src/app/waitpoints/waitpoint-service.ts+++ b/packages/server/api/src/app/waitpoints/waitpoint-service.ts@@ -6,9 +6,11 @@ import { EntityManager, FindOptionsWhere, In, Not } from 'typeorm'import { z } from 'zod'import { repoFactory } from '../core/db/repo-factory'import { transaction } from '../core/db/transaction'+import { distributedStore } from '../database/redis-connections'import { flowRunRepo } from '../flows/flow-run/flow-run-service'import { system } from '../helper/system/system'import { AppSystemProp } from '../helper/system/system-props'+import { barrierSourceKey } from './barrier-queue-factory'import { WaitpointEntity } from './waitpoint-entity'import { waitpointTimeoutJob } from './waitpoint-timeout-job'import { CompleteParams, CompleteResult, CreateForPauseParams, CreateForPauseResult, FindPendingByVersionParams, HandleResumeSignalParams, Waitpoint, WaitpointStatus } from './waitpoint-types'@@ -243,10 +245,13 @@ async function deleteWaitpointsWithTimers({ where, flowRunId, log }: DeleteWaitp.createQueryBuilder().delete().where(where)- .returning(['id', 'resumeDateTime'])+ .returning(['id', 'resumeDateTime', 'type']).execute()- const timed = DeletedWaitpointRows.parse(result.raw ?? []).filter((waitpoint) => !isNil(waitpoint.resumeDateTime))+ const deleted = DeletedWaitpointRows.parse(result.raw ?? [])+ const timed = deleted.filter((waitpoint) => !isNil(waitpoint.resumeDateTime))await Promise.all(timed.map((waitpoint) => waitpointTimeoutJob.remove({ waitpointId: waitpoint.id, flowRunId, log })))+ const barrierSourceKeys = deleted.filter((waitpoint) => waitpoint.type === PauseType.BARRIER).map((waitpoint) => barrierSourceKey(waitpoint.id))+ await distributedStore.delete(barrierSourceKeys)}function stepNameOfWaitpointKey({ waitpointKey }: { waitpointKey: string }): string {@@ -271,6 +276,7 @@ function clampWaitpointResumeDeadline({ requested, type, flowRunCreated, flowRunconst DeletedWaitpointRows = z.array(z.object({id: z.string(),resumeDateTime: z.coerce.date().nullable(),+ type: z.enum(PauseType),}))type PausedDelivery = 'dispatched' | 'already-delivered' | 'unknown-waitpoint'diff --git a/packages/server/api/test/integration/ce/flows/flow-run/barrier.test.ts b/packages/server/api/test/integration/ce/flows/flow-run/barrier.test.tsindex 49807ca5007..1dcad8365c4 100644--- a/packages/server/api/test/integration/ce/flows/flow-run/barrier.test.ts+++ b/packages/server/api/test/integration/ce/flows/flow-run/barrier.test.ts@@ -1,26 +1,32 @@-import { apId, isNil } from '@activepieces/core-utils'+import { apId, assertNotNullOrUndefined, isNil } from '@activepieces/core-utils'import { wideEvent } from '@activepieces/server-utils'-import { BarrierSignalStatus, BarrierSummary, ErrorCode, FlowRunStatus, FlowVersionState, MAX_SIGNAL_REASON_LENGTH, PauseType, RunEnvironment } from '@activepieces/shared'-import { Queue } from 'bullmq'+import { BarrierSignalStatus, BarrierSummary, ErrorCode, FlowActionType, FlowRunStatus, FlowTrigger, FlowTriggerType, FlowVersionState, MAX_SIGNAL_REASON_LENGTH, PauseType, RunEnvironment } from '@activepieces/shared'+import { Queue, UnrecoverableError } from 'bullmq'import dayjs from 'dayjs'import { FastifyInstance } from 'fastify'import { StatusCodes } from 'http-status-codes'+import { transaction } from '../../../../../src/app/core/db/transaction'import { databaseConnection } from '../../../../../src/app/database/database-connection'import { distributedStore, redisConnections } from '../../../../../src/app/database/redis-connections'+import { childRunDispatchService } from '../../../../../src/app/flows/flow-run/child-run-dispatch-service'import { systemJobIds } from '../../../../../src/app/helper/system-jobs/common'import { systemJobsSchedule } from '../../../../../src/app/helper/system-jobs/system-job'+import * as billingProviderModule from '../../../../../src/app/platform/billing-provider'import { platformConfigurationService } from '../../../../../src/app/platform/platform-configuration.service'-import { jobQueue } from '../../../../../src/app/workers/job-queue/job-queue'import { barrierQueue } from '../../../../../src/app/waitpoints/barrier-queue'import * as barrierQueueModule from '../../../../../src/app/waitpoints/barrier-queue'-import { BarrierJobData } from '../../../../../src/app/waitpoints/barrier-queue-factory'+import { BarrierJobData, BarrierJobName, barrierSourceKey } from '../../../../../src/app/waitpoints/barrier-queue-factory'import { barrierService } from '../../../../../src/app/waitpoints/barrier-service'+import * as barrierServiceModule from '../../../../../src/app/waitpoints/barrier-service'+import { fanOutDispatchGaveUp, handleFanOutDispatch, handleFanOutDispatchFailed, handleFanOutDispatchGaveUp } from '../../../../../src/app/waitpoints/fan-out-dispatcher-job'import { handleResumeDelayWaitpoint } from '../../../../../src/app/waitpoints/resume-delay-handler'import { resumeService } from '../../../../../src/app/waitpoints/resume-service'import { BARRIER_RECOVERY_CURSOR_KEY, DEADLINE_SWEEP_CURSOR_KEY, sweepOverdueDeadlines } from '../../../../../src/app/waitpoints/waitpoint-deadline-sweep'import { waitpointService } from '../../../../../src/app/waitpoints/waitpoint-service'import { waitpointTimeoutJob } from '../../../../../src/app/waitpoints/waitpoint-timeout-job'import { Waitpoint, WaitpointStatus } from '../../../../../src/app/waitpoints/waitpoint-types'+import * as jobQueueModule from '../../../../../src/app/workers/job-queue/job-queue'+import { jobQueue } from '../../../../../src/app/workers/job-queue/job-queue'import { db } from '../../../../helpers/db'import { createMockFlow, createMockFlowRun, createMockFlowVersion } from '../../../../helpers/mocks'import { createTestContext, TestContext } from '../../../../helpers/test-context'@@ -29,6 +35,10 @@ import { setupTestEnvironment, teardownTestEnvironment } from '../../../../helpelet app: FastifyInstancelet ctx: TestContext+const FAN_OUT_ENTRY_STEP = 'step_1'++const FAN_OUT_BATCH_STEP = 'fan_out'+beforeAll(async () => {app = await setupTestEnvironment()})@@ -42,9 +52,40 @@ beforeEach(async () => {})async function createParentRun(status: FlowRunStatus = FlowRunStatus.PAUSED) {+ return saveParentRun({ status })+}++async function createFanOutParentRun(status: FlowRunStatus = FlowRunStatus.PAUSED) {+ const now = dayjs().toISOString()+ return saveParentRun({+ status,+ trigger: {+ type: FlowTriggerType.EMPTY,+ name: 'trigger',+ displayName: 'Trigger',+ valid: true,+ settings: {},+ lastUpdatedDate: now,+ nextAction: {+ type: FlowActionType.CODE,+ name: FAN_OUT_ENTRY_STEP,+ displayName: 'Batch body',+ valid: true,+ skip: false,+ lastUpdatedDate: now,+ settings: {+ sourceCode: { code: 'export const code = async () => true', packageJson: '{}' },+ input: {},+ },+ },+ },+ })+}++async function saveParentRun({ status, trigger }: { status: FlowRunStatus, trigger?: FlowTrigger }) {const flow = createMockFlow({ projectId: ctx.project.id })await db.save('flow', flow)- const flowVersion = createMockFlowVersion({ flowId: flow.id, state: FlowVersionState.LOCKED })+ const flowVersion = createMockFlowVersion({ flowId: flow.id, state: FlowVersionState.LOCKED, trigger })await db.save('flow_version', flowVersion)const flowRun = createMockFlowRun({projectId: ctx.project.id,@@ -146,6 +187,105 @@ async function dropResumeJobs(flowRunId: string) {await Promise.all(jobs.map((job) => job.remove()))}+async function giveUp(barrierId: string) {+ return handleFanOutDispatchGaveUp({ data: { barrierId, projectId: ctx.project.id }, log: app.log })+}++async function failDispatch({ barrierId, attemptsMade }: { barrierId: string, attemptsMade: number }) {+ return handleFanOutDispatchFailed({+ data: { barrierId, projectId: ctx.project.id },+ attemptsMade,+ attempts: 5,+ error: new Error('worker died'),+ log: app.log,+ })+}++async function createFanOutBarrier({ flowRunId, items, batchSize, stepName, batchStepName }: {+ flowRunId: string+ items: unknown[]+ batchSize: number+ stepName?: string+ batchStepName?: string+}) {+ return barrierService(app.log).create({+ flowRunId,+ projectId: ctx.project.id,+ platformId: ctx.platform.id,+ stepName: stepName ?? FAN_OUT_BATCH_STEP,+ version: 'V1',+ fanOut: {+ entryStepName: FAN_OUT_ENTRY_STEP,+ batchStepName: batchStepName ?? FAN_OUT_BATCH_STEP,+ batchSize,+ items,+ seedSteps: {},+ },+ })+}++async function withBarrierQueuePaused<T>(fn: () => Promise<T>): Promise<T> {+ const queue = barrierQueue(app.log).get()+ await queue.pause()+ try {+ await queue.drain(true)+ return await fn()+ }+ finally {+ await queue.resume()+ }+}++async function listChildren(barrierId: string) {+ return databaseConnection().getRepository('flow_run').findBy({ parentWaitpointId: barrierId })+}++async function dispatch(barrierId: string) {+ return handleFanOutDispatch({ data: { barrierId, projectId: ctx.project.id }, log: app.log })+}++async function claimChunk({ barrierId, flowRunId, limit }: { barrierId: string, flowRunId: string, limit: number }) {+ const target = await childRunDispatchService(app.log).prepareChildDispatch({ projectId: ctx.project.id, parentRunId: flowRunId, entryStepName: FAN_OUT_ENTRY_STEP })+ return transaction(async (entityManager) => {+ const claimResult = await barrierService(app.log).claimNextSignals({ barrierId, projectId: ctx.project.id, afterSequence: -1, limit, entityManager })+ if (!claimResult.barrierOpen) {+ return null+ }+ const childRuns = await childRunDispatchService(app.log).createChildRuns({+ target,+ parentWaitpointId: barrierId,+ children: claimResult.claims.map((claim) => ({ id: claim.childRunId, dispatchIndex: claim.sequence })),+ entityManager,+ })+ return { claims: claimResult.claims, childRuns }+ })+}++function failNextChildEnqueue({ beforeFailure }: { beforeFailure?: () => Promise<unknown> }) {+ const realJobQueue = jobQueueModule.jobQueue+ const add = vi.fn(async (...args: Parameters<ReturnType<typeof realJobQueue>['add']>) => {+ if (add.mock.calls.length === 1) {+ await beforeFailure?.()+ throw new Error('redis is down')+ }+ return realJobQueue(app.log).add(...args)+ })+ vi.spyOn(jobQueueModule, 'jobQueue').mockImplementation((log) => ({ ...realJobQueue(log), add }))+}++async function dispatchReadingChildPayloads(barrierId: string) {+ const realJobQueue = jobQueueModule.jobQueue+ const add = vi.fn(async (...args: Parameters<ReturnType<typeof realJobQueue>['add']>) => realJobQueue(app.log).add(...args))+ vi.spyOn(jobQueueModule, 'jobQueue').mockImplementation((log) => ({ ...realJobQueue(log), add }))+ try {+ await dispatch(barrierId)+ return add.mock.calls.map(([job]) => 'payload' in job.data ? job.data.payload : null)+ }+ finally {+ vi.restoreAllMocks()+ }+}+async function waitFor(condition: () => Promise<boolean>): Promise<void> {for (let attempt = 0; attempt < 100; attempt++) {if (await condition()) {@@ -1226,3 +1366,419 @@ describe('multi-approval confirm page', () => {expect((await listSignals(signals[0].waitpointId))[0].status).toBe(BarrierSignalStatus.PENDING)})})++describe('fan-out dispatch', () => {+ it('creates exactly one child per signal even when two dispatchers run concurrently', async () => {+ const { flowRun } = await createFanOutParentRun()+ const { barrier, signalCount } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4, 5, 6], batchSize: 2 })+ expect(signalCount).toBe(3)++ await Promise.all([dispatch(barrier.id), dispatch(barrier.id)])+ await waitFor(async () => (await listChildren(barrier.id)).length >= 3)+ await dispatch(barrier.id)++ const children = await listChildren(barrier.id)+ expect(children).toHaveLength(3)+ expect(children.map((child) => child.dispatchIndex).sort()).toEqual([0, 1, 2])+ })++ it('seeds each child under the batch step name, so a batch inside a loop still reads its items', async () => {+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)+ await withBarrierQueuePaused(async () => {+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3], batchSize: 2, stepName: 'loop_1:3/batch_1', batchStepName: 'batch_1' })++ const payloads = await dispatchReadingChildPayloads(barrier.id)++ expect(payloads).toHaveLength(2)+ expect(payloads).toEqual(expect.arrayContaining([+ { type: 'inline', value: { batch_1: expect.objectContaining({ status: 'SUCCEEDED', output: { items: [1, 2] } }) } },+ { type: 'inline', value: { batch_1: expect.objectContaining({ status: 'SUCCEEDED', output: { items: [3] } }) } },+ ]))+ })+ })++ it('stops without dispatching when the barrier was cancelled, and its signals are gone by cascade', async () => {+ const { flowRun } = await createFanOutParentRun()+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })++ await waitpointService(app.log).deleteByFlowRunId({ flowRunId: flowRun.id, projectId: ctx.project.id })+ expect(await listSignals(barrier.id)).toHaveLength(0)+ const beforeCancel = (await listChildren(barrier.id)).length++ await expect(dispatch(barrier.id)).resolves.toBeUndefined()++ expect(await listChildren(barrier.id)).toHaveLength(beforeCancel)+ })++ it('deletes the stored source when a failed run discards its barrier, so a dispatch after it stops', async () => {+ const { flowRun } = await createFanOutParentRun()+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })+ expect(await distributedStore.exists(barrierSourceKey(barrier.id))).toBe(true)++ await waitpointService(app.log).deleteForFinishedRun({+ flowRun: { ...flowRun, status: FlowRunStatus.FAILED, failedStep: { name: 'other_step', displayName: 'Other step', message: 'boom' } },+ })++ expect(await distributedStore.exists(barrierSourceKey(barrier.id))).toBe(false)+ const beforeDispatch = (await listChildren(barrier.id)).length+ await expect(dispatch(barrier.id)).resolves.toBeUndefined()+ expect(await listChildren(barrier.id)).toHaveLength(beforeDispatch)+ })++ it('claims nothing once the barrier has released, so a timeout mid-dispatch adds no children', async () => {+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)+ await withBarrierQueuePaused(async () => {+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })+ await barrierService(app.log).release({ barrier, timedOut: true, releaseReason: 'timeout' })+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.COMPLETED)++ expect(await claimChunk({ barrierId: barrier.id, flowRunId: flowRun.id, limit: 25 })).toBeNull()+ await expect(dispatch(barrier.id)).resolves.toBeUndefined()++ expect(await listChildren(barrier.id)).toHaveLength(0)+ })+ })++ it('lists every child claimed before a release in that release summary', async () => {+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)+ await withBarrierQueuePaused(async () => {+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })+ const chunk = await claimChunk({ barrierId: barrier.id, flowRunId: flowRun.id, limit: 2 })+ assertNotNullOrUndefined(chunk, 'chunk')++ await barrierService(app.log).release({ barrier, timedOut: true, releaseReason: 'timeout' })++ const summary = await readSummary(barrier.id)+ expect(summary.stillRunning).toBe(4)+ const attributedRunIds = (summary.signals ?? []).map((signal) => signal.runId).filter((runId) => !isNil(runId))+ expect(attributedRunIds.sort()).toEqual(chunk.childRuns.map((childRun) => childRun.id).sort())+ await dispatch(barrier.id)+ expect(await listChildren(barrier.id)).toHaveLength(2)+ })+ })++ it('cancels a claimed child whose barrier released before its enqueue, so it never starts', async () => {+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)+ await withBarrierQueuePaused(async () => {+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1], batchSize: 1 })+ const realBarrierService = barrierServiceModule.barrierService+ vi.spyOn(barrierServiceModule, 'barrierService').mockImplementation((log) => ({+ ...realBarrierService(log),+ lockForDispatch: async (params) => {+ await realBarrierService(log).release({ barrier, timedOut: true, releaseReason: 'timeout' })+ return realBarrierService(log).lockForDispatch(params)+ },+ }))+ try {+ await expect(dispatch(barrier.id)).resolves.toBeUndefined()+ }+ finally {+ vi.restoreAllMocks()+ }++ expect(await readSummary(barrier.id)).toMatchObject({ total: 1, stillRunning: 1 })+ await waitFor(async () => (await listChildren(barrier.id))[0]?.status === FlowRunStatus.CANCELED)+ })+ })++ it('claims nothing for a barrier that no longer resolves', async () => {+ const { flowRun } = await createFanOutParentRun()++ expect(await claimChunk({ barrierId: apId(), flowRunId: flowRun.id, limit: 25 })).toBeNull()+ })++ it('refuses to resolve a dispatch target for a parent run outside the project', async () => {+ await expect(childRunDispatchService(app.log).prepareChildDispatch({+ projectId: ctx.project.id,+ parentRunId: apId(),+ entryStepName: FAN_OUT_ENTRY_STEP,+ })).rejects.toThrow(ErrorCode.VALIDATION)+ })++ it('refuses to resolve a dispatch target for a step the flow version does not have', async () => {+ const { flowRun } = await createFanOutParentRun()++ await expect(childRunDispatchService(app.log).prepareChildDispatch({+ projectId: ctx.project.id,+ parentRunId: flowRun.id,+ entryStepName: 'step_does_not_exist',+ })).rejects.toThrow(ErrorCode.VALIDATION)+ })++ it('refuses to start a child run at the trigger, because the engine enters a run only at an action', async () => {+ const { flowRun } = await createFanOutParentRun()++ await expect(childRunDispatchService(app.log).prepareChildDispatch({+ projectId: ctx.project.id,+ parentRunId: flowRun.id,+ entryStepName: 'trigger',+ })).rejects.toThrow(ErrorCode.VALIDATION)+ })++ it('marks the rest undispatched when dispatch exhausts its attempts, so the barrier releases now', async () => {+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)+ await withBarrierQueuePaused(async () => {+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })++ await giveUp(barrier.id)+ await releaseIfReady(barrier.id)++ const summary = await readSummary(barrier.id)+ expect(summary.notDispatched).toBe(4)+ expect(summary.total).toBe(4)+ })+ })++ it('queues the give-up as its own retried job once dispatch fails for good, and not mid-ladder', async () => {+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)+ await withBarrierQueuePaused(async () => {+ const queue = barrierQueue(app.log).get()+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2], batchSize: 1 })++ await failDispatch({ barrierId: barrier.id, attemptsMade: 4 })+ expect(await queue.getJob(`give-up-${barrier.id}`)).toBeUndefined()++ await failDispatch({ barrierId: barrier.id, attemptsMade: 5 })+ const giveUpJob = await queue.getJob(`give-up-${barrier.id}`)+ expect(giveUpJob?.name).toBe(BarrierJobName.FAN_OUT_GIVE_UP)+ expect(giveUpJob?.opts.attempts).toBe(5)+ expect((await listSignals(barrier.id)).map((signal) => signal.status)).toEqual([BarrierSignalStatus.PENDING, BarrierSignalStatus.PENDING])+ })+ })++ it('fails the give-up when settling the signals fails, so the queue retries it instead of leaving the parent waiting', async () => {+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)+ await withBarrierQueuePaused(async () => {+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2], batchSize: 1 })+ const realBarrierService = barrierServiceModule.barrierService+ vi.spyOn(barrierServiceModule, 'barrierService').mockImplementation((log) => ({+ ...realBarrierService(log),+ markUnclaimedNotDispatched: async () => {+ throw new Error('postgres is down')+ },+ }))+ try {+ await expect(giveUp(barrier.id)).rejects.toThrow('postgres is down')+ }+ finally {+ vi.restoreAllMocks()+ }++ await giveUp(barrier.id)+ await releaseIfReady(barrier.id)++ expect(await readSummary(barrier.id)).toMatchObject({ total: 2, notDispatched: 2, stillRunning: 0 })+ })+ })++ it('settles the unclaimed signals when the barrier worker runs a queued give-up', async () => {+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)+ const barrier = await withBarrierQueuePaused(async () => {+ const created = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2], batchSize: 1 })+ await barrierQueue(app.log).get().remove(`dispatch-${created.barrier.id}`)+ return created.barrier+ })++ await barrierQueue(app.log).addFanOutGiveUp({ barrierId: barrier.id, projectId: ctx.project.id })+ await waitFor(async () => (await readStatus(barrier.id)) !== WaitpointStatus.PENDING)++ expect(await readSummary(barrier.id)).toMatchObject({ total: 2, notDispatched: 2, stillRunning: 0 })+ expect(await distributedStore.exists(barrierSourceKey(barrier.id))).toBe(false)+ })++ it('spares the signals whose children are already live when dispatch gives up', async () => {+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)+ await withBarrierQueuePaused(async () => {+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })+ const chunk = await claimChunk({ barrierId: barrier.id, flowRunId: flowRun.id, limit: 2 })+ assertNotNullOrUndefined(chunk, 'chunk')+ const claimedRefIds = chunk.claims.map((claim) => claim.childRunId)++ await giveUp(barrier.id)+ await releaseIfReady(barrier.id)++ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.PENDING)+ const afterGiveUp = await listSignals(barrier.id)+ expect(afterGiveUp.filter((signal) => signal.status === BarrierSignalStatus.PENDING).map((signal) => signal.refId).sort()).toEqual([...claimedRefIds].sort())+ expect(afterGiveUp.filter((signal) => signal.status === BarrierSignalStatus.NOT_DISPATCHED)).toHaveLength(2)+ expect(await barrierService(app.log).hasUnclaimedSignals({ barrierId: barrier.id, projectId: ctx.project.id })).toBe(false)++ await receiveSignal({ signalId: chunk.claims[0].signalId, status: BarrierSignalStatus.SUCCEEDED })+ await releaseIfReady(barrier.id)+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.PENDING)++ await receiveSignal({ signalId: chunk.claims[1].signalId, status: BarrierSignalStatus.SUCCEEDED })+ await releaseIfReady(barrier.id)++ expect(await readSummary(barrier.id)).toMatchObject({ total: 4, succeeded: 2, notDispatched: 2, stillRunning: 0 })+ })+ })++ it('settles a claim that a dispatcher releases after dispatch gave up, so the barrier does not wait for its deadline', async () => {+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)+ await withBarrierQueuePaused(async () => {+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })+ failNextChildEnqueue({ beforeFailure: () => giveUp(barrier.id) })+ try {+ await expect(dispatch(barrier.id)).rejects.toThrow('redis is down')+ }+ finally {+ vi.restoreAllMocks()+ }++ const signals = await listSignals(barrier.id)+ expect(signals.filter((signal) => signal.status === BarrierSignalStatus.NOT_DISPATCHED)).toHaveLength(1)+ const live = signals.filter((signal) => signal.status === BarrierSignalStatus.PENDING)+ expect(live).toHaveLength(3)+ expect(await listChildren(barrier.id)).toHaveLength(3)++ await Promise.all(live.map((signal) => receiveSignal({ signalId: signal.id, status: BarrierSignalStatus.SUCCEEDED })))+ await releaseIfReady(barrier.id)++ expect(await readSummary(barrier.id)).toMatchObject({ total: 4, succeeded: 3, notDispatched: 1, stillRunning: 0 })+ })+ })++ it('leaves a claim that failed to enqueue for the next attempt while dispatch has not given up', async () => {+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)+ await withBarrierQueuePaused(async () => {+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })+ failNextChildEnqueue({})+ try {+ await expect(dispatch(barrier.id)).rejects.toThrow('redis is down')+ }+ finally {+ vi.restoreAllMocks()+ }+ expect(await listChildren(barrier.id)).toHaveLength(3)+ expect(await barrierService(app.log).hasUnclaimedSignals({ barrierId: barrier.id, projectId: ctx.project.id })).toBe(true)++ await dispatch(barrier.id)++ const children = await listChildren(barrier.id)+ expect(children.map((child) => child.dispatchIndex).sort()).toEqual([0, 1, 2, 3])+ })+ })++ it('dispatches nothing and settles the barrier when the platform has no run credits', async () => {+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)+ await withBarrierQueuePaused(async () => {+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })+ vi.spyOn(billingProviderModule, 'shouldBlockRunOnCredits').mockResolvedValue(true)+ try {+ await dispatch(barrier.id)+ }+ finally {+ vi.restoreAllMocks()+ }++ expect(await listChildren(barrier.id)).toHaveLength(0)+ await releaseIfReady(barrier.id)+ expect(await readSummary(barrier.id)).toMatchObject({ total: 4, notDispatched: 4 })+ })+ })++ it('starts the dispatch again when the engine retries a fan-out create that failed after its commit', async () => {+ const { flowRun } = await createFanOutParentRun()+ await withBarrierQueuePaused(async () => {+ const queue = barrierQueue(app.log).get()+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })+ await queue.remove(`dispatch-${barrier.id}`)+ await distributedStore.delete(barrierSourceKey(barrier.id))++ const retried = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3, 4], batchSize: 1 })++ expect(retried.barrier.id).toBe(barrier.id)+ expect(await distributedStore.exists(barrierSourceKey(barrier.id))).toBe(true)+ expect(await queue.getJob(`dispatch-${barrier.id}`)).toBeDefined()+ })+ })++ it('keeps a released fan-out barrier when the engine retries its create, so no batch runs twice', async () => {+ const { flowRun } = await createFanOutParentRun(FlowRunStatus.RUNNING)+ await withBarrierQueuePaused(async () => {+ const queue = barrierQueue(app.log).get()+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3], batchSize: 1 })+ await queue.remove(`dispatch-${barrier.id}`)+ await dispatch(barrier.id)+ const children = await listChildren(barrier.id)+ for (const child of children) {+ await barrierService(app.log).receiveSignal({ refId: child.id, projectId: ctx.project.id, status: BarrierSignalStatus.SUCCEEDED })+ }+ await releaseIfReady(barrier.id)+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.COMPLETED)++ const retried = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3], batchSize: 1 })+ await dispatch(retried.barrier.id)++ expect(retried.barrier.id).toBe(barrier.id)+ expect(await databaseConnection().getRepository('flow_run').findBy({ parentRunId: flowRun.id })).toHaveLength(3)+ expect(await queue.getJob(`dispatch-${barrier.id}`)).toBeUndefined()+ expect(await distributedStore.exists(barrierSourceKey(barrier.id))).toBe(false)++ await db.update('flow_run', flowRun.id, { status: FlowRunStatus.PAUSED })+ await dropResumeJobs(flowRun.id)+ await recoverUndeliveredResume(flowRun.id)++ expect(await listResumeJobs(flowRun.id)).toHaveLength(1)+ expect(await readStatus(barrier.id)).toBe(WaitpointStatus.CONSUMED)+ })+ })++ it('keeps an open fan-out barrier when the engine retries its create, so its children are not dispatched again', async () => {+ const { flowRun } = await createFanOutParentRun()+ await withBarrierQueuePaused(async () => {+ const { barrier } = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3], batchSize: 1 })+ await dispatch(barrier.id)++ const retried = await createFanOutBarrier({ flowRunId: flowRun.id, items: [1, 2, 3], batchSize: 1 })+ await dispatch(retried.barrier.id)++ expect(retried.barrier.id).toBe(barrier.id)+ expect(await listChildren(barrier.id)).toHaveLength(3)+ })+ })++ it('replaces a consumed barrier when its step creates one again', async () => {+ const { flowRun } = await createParentRun()+ const { barrier } = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]'] })+ await closeWithoutConsuming(barrier.id)+ const closed = await barrierService(app.log).findById({ barrierId: barrier.id, projectId: ctx.project.id })+ assertNotNullOrUndefined(closed, 'closed barrier')+ await waitpointService(app.log).consume({ waitpoint: closed })++ const recreated = await createBarrier({ flowRunId: flowRun.id, signalLabels: ['[email protected]'] })++ expect(recreated.barrier.id).not.toBe(barrier.id)+ expect(await readStatus(recreated.barrier.id)).toBe(WaitpointStatus.PENDING)+ expect(await listSignals(recreated.barrier.id)).toEqual([expect.objectContaining({ status: BarrierSignalStatus.PENDING })])+ expect(await db.findOneBy('waitpoint', { id: barrier.id })).toBeNull()+ })++ it('gives up on an unrecoverable failure and on the last attempt, but not mid-ladder', async () => {+ expect(fanOutDispatchGaveUp({ attemptsMade: 1, attempts: 5, error: new Error('boom') })).toBeNull()+ expect(fanOutDispatchGaveUp({ attemptsMade: 4, attempts: 5, error: new Error('boom') })).toBeNull()+ expect(fanOutDispatchGaveUp({ attemptsMade: 5, attempts: 5, error: new Error('boom') })).toBe('exhausted')+ expect(fanOutDispatchGaveUp({ attemptsMade: 1, attempts: 5, error: new UnrecoverableError('job stalled more than allowable limit') })).toBe('unrecoverable')+ expect(fanOutDispatchGaveUp({ attemptsMade: 1, attempts: undefined, error: new Error('boom') })).toBe('exhausted')+ })++ it('clamps the batch size so a wide source cannot exceed the signal cap', async () => {+ const { flowRun } = await createFanOutParentRun()+ const previous = process.env.AP_MAX_BARRIER_SIGNALS+ process.env.AP_MAX_BARRIER_SIGNALS = '3'+ try {+ const created = await createFanOutBarrier({ flowRunId: flowRun.id, items: Array.from({ length: 12 }, (_, index) => index), batchSize: 1 })+ expect(created.signalCount).toBe(3)+ expect(created.batchSize).toBe(4)+ }+ finally {+ if (previous === undefined) {+ delete process.env.AP_MAX_BARRIER_SIGNALS+ }+ else {+ process.env.AP_MAX_BARRIER_SIGNALS = previous+ }+ }+ })+})diff --git a/packages/server/engine/src/lib/handler/context/engine-constants.ts b/packages/server/engine/src/lib/handler/context/engine-constants.tsindex 729b99bd375..2a7fd58e6a4 100644--- a/packages/server/engine/src/lib/handler/context/engine-constants.ts+++ b/packages/server/engine/src/lib/handler/context/engine-constants.ts@@ -27,6 +27,7 @@ type EngineConstantsParams = {resumePayload?: ResumePayloadrunEnvironment?: RunEnvironmentstepNameToTest?: string+ entryStepName?: stringlogsFileId?: stringtimeoutInSeconds: numberplatformId: PlatformId@@ -68,6 +69,7 @@ export class EngineConstants {public readonly resumePayload?: ResumePayloadpublic readonly runEnvironment?: RunEnvironmentpublic readonly stepNameToTest?: string+ public readonly entryStepName?: stringpublic readonly logsFileId?: stringpublic readonly stepNames: string[] = []public readonly actionRunMode: boolean@@ -113,6 +115,7 @@ export class EngineConstants {this.resumePayload = params.resumePayloadthis.runEnvironment = params.runEnvironmentthis.stepNameToTest = params.stepNameToTest+ this.entryStepName = params.entryStepNamethis.logsFileId = params.logsFileIdthis.platformId = params.platformIdthis.timeoutInSeconds = params.timeoutInSeconds@@ -132,6 +135,7 @@ export class EngineConstants {resumePayload: input.executionType === ExecutionType.RESUME ? input.resumePayload : undefined,runEnvironment: input.runEnvironment,stepNameToTest: input.stepNameToTest ?? undefined,+ entryStepName: input.entryStepName,logsFileId: input.logsFileId,})}diff --git a/packages/server/engine/src/lib/helper/flow-run-progress-reporter.ts b/packages/server/engine/src/lib/helper/flow-run-progress-reporter.tsindex 2b5805d0060..ecad18a87c3 100644--- a/packages/server/engine/src/lib/helper/flow-run-progress-reporter.ts+++ b/packages/server/engine/src/lib/helper/flow-run-progress-reporter.ts@@ -107,6 +107,7 @@ export const flowRunProgressReporter = {executionState: {steps: flowExecutorContext.steps,tags: Array.from(flowExecutorContext.tags),+ entryStepName: engineConstants.entryStepName,},})const executionState = await zstdCompress(serialized)diff --git a/packages/server/engine/src/lib/operations/flow.operation.ts b/packages/server/engine/src/lib/operations/flow.operation.tsindex 49846d41b17..c66cdf9273e 100644--- a/packages/server/engine/src/lib/operations/flow.operation.ts+++ b/packages/server/engine/src/lib/operations/flow.operation.ts@@ -1,5 +1,6 @@import { isNil, tryCatch } from '@activepieces/core-utils'-import { EngineGenericError, EngineResponse, EngineResponseStatus, ExecuteFlowOperation, ExecuteTriggerResponse, ExecutionError, ExecutionErrorType, ExecutionState, ExecutionType, FlowActionType, FlowRunStatus, flowStructureUtil, GenericStepOutput, LoopStepOutput, ResumePayload, ResumeReason, StepOutput, StepOutputStatus, TriggerHookType, TriggerPayload } from '@activepieces/shared'+import { EngineGenericError, EngineResponse, EngineResponseStatus, ExecuteFlowOperation, ExecuteTriggerResponse, ExecutionError, ExecutionErrorType, ExecutionState, ExecutionType, FlowActionType, FlowRunStatus, flowStructureUtil, GenericStepOutput, InvalidSeededStepsError, LoopStepOutput, ResumePayload, ResumeReason, StepOutput, StepOutputStatus, TriggerHookType, TriggerPayload } from '@activepieces/shared'+import dayjs from 'dayjs'import { engineFileApi } from '../api/engine-file-api'import { EngineConstants, ResolvedBeginExecuteFlowOperation, ResolvedExecuteFlowOperation } from '../handler/context/engine-constants'import { FlowExecutorContext } from '../handler/context/flow-execution-context'@@ -119,6 +120,20 @@ const executeSingleStepOrFlow = async (input: ResolvedExecuteFlowOperation, consif (executionState.verdict.status !== FlowRunStatus.RUNNING) {return executionState}+ if (!isNil(input.entryStepName)) {+ if (input.executionType === ExecutionType.BEGIN) {+ await flowRunProgressReporter.sendUpdate({+ engineConstants: constants,+ flowExecutorContext: executionState,+ startTime: dayjs().toISOString(),+ })+ }+ return flowExecutor.execute({+ action: flowStructureUtil.getActionOrThrow(input.entryStepName, input.flowVersion.trigger),+ executionState,+ constants,+ })+ }return flowExecutor.executeFromTrigger({executionState,constants,@@ -158,6 +173,12 @@ async function buildFailedTriggerContext({ input, baseContext, error }: BuildFaiasync function getFlowExecutionState(input: ResolvedExecuteFlowOperation, constants: EngineConstants, flowContext: FlowExecutorContext): Promise<FlowExecutorContext> {if (input.executionType === ExecutionType.BEGIN) {+ if (!isNil(input.entryStepName)) {+ if (!isSeededSteps(input.triggerPayload)) {+ throw new InvalidSeededStepsError(input.entryStepName)+ }+ return restoreSteps({ steps: input.triggerPayload, flowContext, keepFailedSteps: true })+ }const newPayload = await runOrReturnPayload(input, constants)return flowContext.upsertStep(input.flowVersion.trigger.name,GenericStepOutput.create({@@ -166,17 +187,31 @@ async function getFlowExecutionState(input: ResolvedExecuteFlowOperation, constainput: {},}).setOutput(newPayload))}- flowContext = flowContext.addTags(input.executionState.tags)- const isWaitpointResume = input.resumeReason === ResumeReason.WAITPOINT- for (const [step, output] of Object.entries(input.executionState.steps)) {- if (isStepRestorable({ status: output.status, isWaitpointResume })) {- const newOutput = await insertSuccessStepsOrPausedRecursively({ stepOutput: output, isWaitpointResume })+ return restoreSteps({+ steps: input.executionState.steps,+ flowContext: flowContext.addTags(input.executionState.tags),+ keepFailedSteps: input.resumeReason === ResumeReason.WAITPOINT,+ })+}++async function restoreSteps({ steps, flowContext, keepFailedSteps }: RestoreStepsParams): Promise<FlowExecutorContext> {+ let restored = flowContext+ for (const [step, output] of Object.entries(steps)) {+ if (isStepRestorable({ status: output.status, isWaitpointResume: keepFailedSteps })) {+ const newOutput = await insertSuccessStepsOrPausedRecursively({ stepOutput: output, isWaitpointResume: keepFailedSteps })if (!isNil(newOutput)) {- flowContext = await flowContext.upsertStep(step, newOutput)+ restored = await restored.upsertStep(step, newOutput)}}}- return flowContext+ return restored+}++function isSeededSteps(payload: unknown): payload is Record<string, StepOutput> {+ if (payload === null || typeof payload !== 'object' || Array.isArray(payload)) {+ return false+ }+ return Object.values(payload).every((step: unknown) => step !== null && typeof step === 'object' && 'type' in step && 'status' in step)}async function runOrReturnPayload(input: ResolvedBeginExecuteFlowOperation, constants: EngineConstants): Promise<TriggerPayload> {@@ -236,6 +271,7 @@ async function resolveExecuteFlowOperation(operation: ExecuteFlowOperation): Pro}return {...operation,+ entryStepName: operation.entryStepName ?? executionState.entryStepName,resumePayload: await resolveJobPayload({ payload: operation.resumePayload, apiUrl: operation.internalApiUrl, engineToken: operation.engineToken }) as ResumePayload,executionState,}@@ -301,3 +337,9 @@ type InsertStepsParams = {stepOutput: StepOutputisWaitpointResume: boolean}++type RestoreStepsParams = {+ steps: Record<string, StepOutput>+ flowContext: FlowExecutorContext+ keepFailedSteps: boolean+}diff --git a/packages/server/engine/test/operations/flow-operation-invariants.test.ts b/packages/server/engine/test/operations/flow-operation-invariants.test.tsindex 789c8717916..c6b2e91b6d0 100644--- a/packages/server/engine/test/operations/flow-operation-invariants.test.ts+++ b/packages/server/engine/test/operations/flow-operation-invariants.test.ts@@ -1,3 +1,4 @@+import { isNil } from '@activepieces/core-utils'import {ConnectionNotFoundError,EngineFileNotFoundError,@@ -14,6 +15,7 @@ import {StreamStepProgress,} from '@activepieces/shared'import type { BeginExecuteFlowOperation, FlowAction, FlowVersion, ResumeExecuteFlowOperation } from '@activepieces/shared'+import dayjs from 'dayjs'import { describe, expect, it, vi } from 'vitest'const { mockSendUpdate, mockBackup } = vi.hoisted(() => ({@@ -154,6 +156,33 @@ function makeFlowVersionWithTwoApprovals(): FlowVersion {}}+function makeFlowVersionWithContainerBody(): FlowVersion {+ const loopAction = (name: string, items: unknown): FlowAction => ({+ name,+ displayName: name,+ type: FlowActionType.LOOP_ON_ITEMS,+ skip: false,+ valid: true,+ settings: { items },+ })+ const container: FlowAction = {+ ...loopAction('step_1', []),+ firstLoopAction: loopAction('step_2', [1, 2]),+ nextAction: loopAction('step_3', []),+ }+ return {+ ...makeFlowVersion(),+ trigger: {+ name: 'trigger_1',+ valid: true,+ displayName: 'Test Trigger',+ type: FlowTriggerType.EMPTY,+ settings: {},+ nextAction: container,+ },+ }+}+function makeResumeOperation(overrides?: Partial<ResumeExecuteFlowOperation>): ResumeExecuteFlowOperation {return {projectId: 'proj-1',@@ -469,6 +498,124 @@ describe('flow operation invariants', () => {expect(mockCreateWaitpoint).not.toHaveBeenCalled()})++ it('reports a start time for a run entered at a step, so a batch child is not left without one', async () => {+ mockSendUpdate.mockClear()++ await flowOperation.execute(makeBeginOperation({+ flowVersion: makeFlowVersionWithContainerBody(),+ entryStepName: 'step_2',+ }))++ const started = mockSendUpdate.mock.calls.filter((call) => !isNil(call[0].startTime))+ expect(started).toHaveLength(1)+ expect(dayjs(started[0][0].startTime).isValid()).toBe(true)+ })++ it('fails a run entered at a step with malformed seeded steps instead of throwing for a retry', async () => {+ mockSendUpdate.mockClear()++ const response = await flowOperation.execute(makeBeginOperation({+ flowVersion: makeFlowVersionWithContainerBody(),+ entryStepName: 'step_2',+ triggerPayload: { type: 'inline', value: { step_1: 'not a step output' } },+ }))++ expect(response.status).toBe(EngineResponseStatus.OK)+ const finalCtx = mockSendUpdate.mock.calls[mockSendUpdate.mock.calls.length - 1][0].flowExecutorContext+ expect(finalCtx.verdict.status).toBe(FlowRunStatus.FAILED)+ expect(finalCtx.verdict.failedStep.message).toEqual(expect.stringContaining('needs the steps it starts after'))+ })++ it('does not re-report a start time when a run entered at a step resumes', async () => {+ mockSendUpdate.mockClear()+ mockDownload.mockReset()+ mockCreateWaitpoint.mockReset()+ mockCreateWaitpoint.mockResolvedValue({+ id: 'wp-after-batch',+ resumeUrl: 'http://localhost:4200/api/v1/flow-runs/run-1/waitpoints/wp-after-batch',+ })+ mockDownload.mockResolvedValue(+ new TextEncoder().encode(JSON.stringify({+ executionState: {+ steps: {+ step_1: {+ type: FlowActionType.LOOP_ON_ITEMS,+ status: StepOutputStatus.SUCCEEDED,+ input: {},+ output: { item: null, index: 0, iterations: [] },+ },+ step_2: {+ type: FlowActionType.LOOP_ON_ITEMS,+ status: StepOutputStatus.PAUSED,+ input: {},+ output: { item: null, index: 0, iterations: [] },+ },+ },+ tags: [],+ entryStepName: 'step_2',+ },+ })),+ )++ await flowOperation.execute(makeResumeOperation({+ flowVersion: makeFlowVersionWithContainerBody(),+ resumePayload: {+ type: 'inline',+ value: { queryParams: { action: 'approve' }, body: {}, headers: {} },+ },+ }))++ expect(mockSendUpdate.mock.calls.filter((call) => !isNil(call[0].startTime))).toHaveLength(0)+ })++ it('resumes a run entered mid-graph from its entry step instead of walking the flow from the trigger', async () => {+ mockDownload.mockReset()+ mockSendUpdate.mockClear()+ mockCreateWaitpoint.mockReset()+ mockCreateWaitpoint.mockResolvedValue({+ id: 'wp-after-batch',+ resumeUrl: 'http://localhost:4200/api/v1/flow-runs/run-1/waitpoints/wp-after-batch',+ })++ mockDownload.mockResolvedValue(+ new TextEncoder().encode(JSON.stringify({+ executionState: {+ steps: {+ step_1: {+ type: FlowActionType.LOOP_ON_ITEMS,+ status: StepOutputStatus.SUCCEEDED,+ input: {},+ output: { item: null, index: 0, iterations: [] },+ },+ step_2: {+ type: FlowActionType.LOOP_ON_ITEMS,+ status: StepOutputStatus.PAUSED,+ input: {},+ output: { item: null, index: 0, iterations: [] },+ },+ },+ tags: [],+ entryStepName: 'step_2',+ },+ })),+ )++ const operation: ResumeExecuteFlowOperation = {+ ...makeResumeOperation(),+ flowVersion: makeFlowVersionWithContainerBody(),+ resumePayload: {+ type: 'inline',+ value: { queryParams: { action: 'approve' }, body: {}, headers: {} },+ },+ }++ await flowOperation.execute(operation)++ const finalCtx = mockSendUpdate.mock.calls[mockSendUpdate.mock.calls.length - 1][0].flowExecutorContext+ expect(finalCtx.steps.step_2.output.iterations).toHaveLength(2)+ expect(finalCtx.steps.step_3).toBeUndefined()+ })})describe('BEGIN payload hydration', () => {diff --git a/packages/server/worker/src/lib/execute/jobs/execute-flow.ts b/packages/server/worker/src/lib/execute/jobs/execute-flow.tsindex b82c98880ad..643a02d5d1f 100644--- a/packages/server/worker/src/lib/execute/jobs/execute-flow.ts+++ b/packages/server/worker/src/lib/execute/jobs/execute-flow.ts@@ -161,6 +161,7 @@ function buildFlowOperation(executionType: ExecutionType.BEGIN,triggerPayload: data.payload,executeTrigger: data.executeTrigger ?? false,+ entryStepName: data.entryStepName,sampleData: data.sampleData,}}