AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
title: Server-side fan-out icon: 🪓
Server-side fan-out
The barrier's own child dispatcher. A batch step hands the server its items in one POST /v1/waitpoints, and a
fan-out-dispatch job on the barrier queue creates one child run per signal. The step that calls it
(PROCESS_IN_BATCHES) ships with the batch-step layer; until then only tests drive it. See decision
000051 for why, and
000015 for the barrier.
Server-side fan-out — one parent pause, N child runs created by the app. Avoid: "subflow fan-out", which is
fire-and-forget webhook calls with no barrier.
Source — { entryStepName, batchStepName, seedSteps, batches } stored under barrier_source:<barrierId> with a 1-day TTL.
Deleted on release, on cancel and when dispatch gives up.
Claim — a signal is claimed when its refId holds its child run id. Unclaimed means PENDING with no
refId (unclaimedSignalWhere).
Chunk — one transaction: the barrier FOR SHARE, the next 25 unclaimed signals FOR UPDATE SKIP LOCKED, the
claim, and the child rows.
Entry step — entryStepName, the action a child run starts at, with the parent's steps seeded as SUCCEEDED.
It is written into the child's execution-state log, so a resume re-enters it. It must be an action.
Give-up — the fan-out-give-up job (give-up-<barrierId>). The barrier worker's failed handler queues it on
exhausted attempts or an UnrecoverableError. It deletes the source, then marks the unclaimed signals NOT_DISPATCHED,
and it retries like any barrier job.
Gotchas
- The waitpoint key is not the batch step's name. Inside a loop the barrier's
stepNameis the path keyloop_1:3/batch_1, which keeps one barrier per iteration. The body reads its batch under the bare name, so the engine sends that name asbatchStepNameand the dispatcher seeds the batch under it. - The give-up cleanup runs as its own queue job, never inline in the
failedhandler. Inline, a Postgres or Redis error there was logged and lost, and the parent stayed paused until its deadline. One gap is left: a process that dies between BullMQ'smoveToFailedand that enqueue never queues the give-up. - Raw
UPDATE … RETURNINGthroughrepo.query()returns[rows, rowCount]on node-postgres but a plain rows array on PGlite. A.length > 0check on it is always true in production and only correct in tests. ReadRETURNINGrows through the query builder (.returning().execute(), thenraw), asclaimNextSignalsdoes. - PGlite runs every transaction on one shared session, so row locks do nothing in the API tests. The claim
stays correct without them: the
UPDATEkeeps itsrefId IS NULL AND status = 'PENDING'guard, and the claims are the rows it returned, not the rows theSELECT … FOR UPDATEsaw. - A stalled job is not a dead worker. On BullMQ defaults the first stall redelivers the job to a second
dispatcher while the first still runs. The second stall fails it as an
UnrecoverableErroron the next pickup, withattemptsMadeunchanged, sofanOutDispatchGaveUpchecks for both. - A claim released after the give-up must settle itself. When an enqueue fails, the dispatcher deletes the
child row and clears the claim so a later attempt can retry it. If the give-up sweep already ran, no attempt is
left, so the dispatcher then checks the source key and, if it is gone, marks the signal
NOT_DISPATCHED. The give-up handler deletes the key before its sweep, so one of the two always settles the signal. - A chunk settles before the job fails.
enqueueChunkusesPromise.allSettled, so every release lands before BullMQ firesfailed. WithPromise.allthe sweep ran while the other tasks still claimed and released. - Never widen a sweep to "all
PENDING". A dispatched signal staysPENDINGwith arefIduntil its child ends, and a body that waits on an approval keeps it there for days. A status-only sweep marks live childrenNOT_DISPATCHED, the barrier releases, and their outcomes report into nothing. - A child claimed just before a release never starts. Its enqueue needs the barrier
PENDING, so the dispatcher cancels it instead (cancelUnstartedChildRun). The release summary still counts it asstillRunningwith its run id, because its claim came first. - The engine retries
POST /v1/waitpointson a 5xx, also after the API committed.createkeys the retry on the existing row's status.PENDINGis reused, andcreatestores the source and adds the dispatch job again; the fixed job iddispatch-<barrierId>and the claims make that repeat safe.COMPLETEDis reused with no dispatch, because fast children can release the barrier before the retry lands, and its summary waits for the deferred resume. OnlyCONSUMEDis replaced. Deleting aCOMPLETEDbarrier gave it fresh signals and ran every batch twice. - Run credits are checked once per chunk. With none left, the dispatcher marks the unclaimed signals
NOT_DISPATCHED; children already claimed keep running. - Each child carries its own copy of
seedSteps, inline belowWEBHOOK_PAYLOAD_INLINE_THRESHOLD_KBand as a file above it. - The parent's run timeline no longer measures dispatch cost. Its first leg is one HTTP call. Measure the span
of the children's
createdtimestamps instead. barrier-queue-factory.tsis the module graph's acyclic leaf.barrierSourceKeyand the job types live there sowaitpoint-service.tsimports the key without the queue. Folding it intobarrier-queue.tsadds an import cycle.
Key files
packages/server/api/src/app/waitpoints/—handleFanOutDispatch,barrierService.claimNextSignalspackages/server/api/src/app/flows/flow-run/—prepareChildDispatch,createChildRuns,enqueueChildRunpackages/server/engine/src/lib/operations/—flowOperation, which enters a run atentryStepName