AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
fix(workers): jobs no longer get stuck after a worker reconnects (#16893)
Hazem Adelcddc8723214e
diff --git a/brain/knowledge/execution-runtime/workers.md b/brain/knowledge/execution-runtime/workers.mdindex 82bfc5ca36d..3d24793f0ec 100644--- a/brain/knowledge/execution-runtime/workers.md+++ b/brain/knowledge/execution-runtime/workers.md@@ -21,6 +21,7 @@ The deep `Resolver`/`Runtime` concurrency and bundle-caching model lives on the- `concurrency === 1` primes the sandbox to the full container RAM (cgroup-aware) via `primeFullContainerMemory()`.### Gotchas+- **A dispatcher waiter belongs to a socket, and a closed socket's waiter is skipped.** A poll parks a waiter for up to `WAITER_TIMEOUT_MS` (50s), which can outlive its socket. Before AI-152 a job that `releaseConnectionJobs` had just re-queued was often handed straight back to the dead worker's waiter. It then sat in `active`, recorded against a connection that had already been drained, until the stalled scan reclaimed it. `runLoop` now hands the job to the first waiter whose `socket.connected` is still true, and returns it to the queue when there isn't one.- **Piece bundles download over the worker's own `AP_FRONTEND_URL` (`internalApiUrl`), never the app's `PUBLIC_URL` — keep it that way.** Through 0.92.x the sandbox built `GET /v1/engine/pieces/bundle` from the `PUBLIC_URL` the app pushes over the socket, so a standalone worker ignored its own `AP_FRONTEND_URL=http://app` for this one hop. A `localhost` public URL (the docs' default for compose and `docker run -p 8080:80`) failed every uncached install with `fetch failed` (GIT-1838). A public URL that reached a *different* instance sharing the JWT secret returned 404, so custom-piece upload failed with `PIECE_BUNDLE_NOT_AVAILABLE` while the `PACKAGE_ARCHIVE` row sat intact in Postgres. Tell-tale: the bundle GET never shows up in the app's own request log. Fixed by GIT-1718.- **`AP_CONTAINER_TYPE=WORKER` moves every engine→app call onto `AP_FRONTEND_URL`.** In `WORKER_AND_APP` the engine reaches the app at `http://127.0.0.1:<port>/api/`; in `WORKER` it uses `AP_FRONTEND_URL + '/api/'` (`getApiUrl` in `worker/src/lib/config/configs.ts`), and the worker's Socket.IO connection follows it. If that URL is the public hostname, file uploads, run-log uploads, piece bundle downloads and the worker socket all cross the ingress/WAF/CDN, so WAF rules and websocket idle timeouts start failing runs that worked the day before. Point separated workers at the in-cluster app service instead. Found on Alan's fleet 2026-09-29, the day after being advised to switch to `WORKER`.- **The piece workspace pins `linker = "isolated"` in its own `bunfig.toml` — do not delete it, the engine's piece resolver depends on the layout bun chooses.** `pieceInstaller` generates a bun workspace at `<cache>/<version>/common` whose members are `pieces/<pieceName>-<pieceVersion>`, and the engine (`piece-loader.ts` → `resolveInstalledPieceEntry`) resolves a piece **only** at `pieces/<alias>/node_modules/<pieceName>`, which just the isolated linker produces. The workspace used to carry no bunfig and inherited whatever ancestor bunfig bun discovered above the cache mount (`/usr/src/app/bunfig.toml`); when bun instead uses the **hoisted** linker it writes the package as a real directory at the workspace root `node_modules/@activepieces/<name>` and leaves the member with no `node_modules` at all, so every run of that piece fails `PieceNotFoundError: Piece not found for package: <name>-<version>` while `bun install` still exits 0. It was silent and self-perpetuating — `markPiecesAsUsed` writes `ready` and `pieceCheckIfAlreadyInstalled` only looks for *a* `node_modules`, so the piece is reinstalled forever and never resolves. Sep 2026 production: 125/125 folders missing `node_modules` were exactly the 125 packages sitting hoisted at the workspace root; 7,400+ `EXECUTE_FLOW` jobs, 15+ platforms, ~60 official pieces, ~300/hr. Two things had to line up — bun 1.4.0 (Aug 31) changed the layout newly-installed members got, and removing the piece-upgrade Redis gate (#15236, deployed 2026-09-03 08:09 UTC) mass-upgraded every platform's flows to newer piece versions so almost every install was suddenly a *first-time* install. Diagnose on a worker with `for d in <cache>/v*/common/pieces/@activepieces/*/; do [ -d "$d/node_modules" ] || basename $d; done`. Fixed by pinning the linker plus a `LATEST_CACHE_VERSION` bump (v14 → v15) so the fleet rebuilds one clean workspace under the pinned layout rather than carrying a mixed one.diff --git a/packages/server/api/src/app/workers/job-queue/job-broker.ts b/packages/server/api/src/app/workers/job-queue/job-broker.tsindex fccbd820231..c0d42faa2a0 100644--- a/packages/server/api/src/app/workers/job-queue/job-broker.ts+++ b/packages/server/api/src/app/workers/job-queue/job-broker.ts@@ -2,6 +2,7 @@ import { isNil, tryCatch } from '@activepieces/core-utils'import { ConsumeJobRequest, ConsumeJobResponse, EngineResponseStatus, JobData } from '@activepieces/shared'import { Worker as BullMQWorker, Job, UnrecoverableError } from 'bullmq'import { FastifyBaseLogger } from 'fastify'+import { Socket } from 'socket.io'import { accessTokenManager } from '../../authentication/lib/access-token-manager'import { redisConnections } from '../../database/redis-connections'import { engineResponseWatcher } from '../engine-response-watcher'@@ -222,12 +223,12 @@ export const jobBroker = (log: FastifyBaseLogger) => ({log.info('[jobBroker] Job broker initialized')},- async poll(queueName: string = QueueName.WORKER_JOBS, connectionId?: string): Promise<ConsumeJobRequest | null> {+ async poll(queueName: string = QueueName.WORKER_JOBS, socket?: Pick<Socket, 'id' | 'connected'>): Promise<ConsumeJobRequest | null> {const worker = await ensureBullMQWorker(queueName, log)const dispatcher = ensureDispatcher(queueName, worker, log)- const job = await dispatcher.poll()- if (!isNil(job) && !isNil(connectionId)) {- jobAssignmentTracker.record({ connectionId, jobId: job.jobId, token: job.token, queueName: job.queueName })+ const job = await dispatcher.poll(socket)+ if (!isNil(job) && !isNil(socket)) {+ jobAssignmentTracker.record({ connectionId: socket.id, jobId: job.jobId, token: job.token, queueName: job.queueName })}return job},diff --git a/packages/server/api/src/app/workers/job-queue/queue-dispatcher.ts b/packages/server/api/src/app/workers/job-queue/queue-dispatcher.tsindex 229ec6bd151..25b0d4c9687 100644--- a/packages/server/api/src/app/workers/job-queue/queue-dispatcher.ts+++ b/packages/server/api/src/app/workers/job-queue/queue-dispatcher.ts@@ -2,6 +2,7 @@ import { isNil, tryCatch } from '@activepieces/core-utils'import { ConsumeJobRequest } from '@activepieces/shared'import { Worker as BullMQWorker } from 'bullmq'import { FastifyBaseLogger } from 'fastify'+import { Socket } from 'socket.io'const WAITER_TIMEOUT_MS = 50_000const ERROR_RETRY_DELAY_MS = 5_000@@ -17,7 +18,7 @@ function createQueueDispatcher(params: {const waiters: Waiter[] = []let loopRunning = false- async function poll(): Promise<ConsumeJobRequest | null> {+ async function poll(socket?: Pick<Socket, 'connected'>): Promise<ConsumeJobRequest | null> {return new Promise<ConsumeJobRequest | null>((resolve) => {const timer = setTimeout(() => {const idx = waiters.findIndex(w => w.resolve === resolve)@@ -27,7 +28,7 @@ function createQueueDispatcher(params: {resolve(null)}, WAITER_TIMEOUT_MS)- waiters.push({ resolve, timer })+ waiters.push({ resolve, timer, socket })startLoop()})}@@ -53,7 +54,7 @@ function createQueueDispatcher(params: {continue}- const waiter = waiters.shift()+ const waiter = takeConnectedWaiter()if (isNil(waiter)) {log.warn({ queueName, job: { id: job.jobId } }, '[QueueDispatcher] job dequeued but no waiter available, returning to queue')const { error: orphanError } = await tryCatch(() => onOrphanedJob(job.jobId, job.token, job.queueName, log))@@ -69,6 +70,16 @@ function createQueueDispatcher(params: {loopRunning = false}+ function takeConnectedWaiter(): Waiter | undefined {+ let waiter = waiters.shift()+ while (!isNil(waiter) && waiter.socket?.connected === false) {+ clearTimeout(waiter.timer)+ waiter.resolve(null)+ waiter = waiters.shift()+ }+ return waiter+ }+function close(): void {for (const waiter of waiters) {clearTimeout(waiter.timer)@@ -93,10 +104,11 @@ function sleep(ms: number): Promise<void> {type Waiter = {resolve: (value: ConsumeJobRequest | null) => voidtimer: ReturnType<typeof setTimeout>+ socket?: Pick<Socket, 'connected'>}export type QueueDispatcher = {- poll(): Promise<ConsumeJobRequest | null>+ poll(socket?: Pick<Socket, 'connected'>): Promise<ConsumeJobRequest | null>close(): voidwaiterCount(): number}diff --git a/packages/server/api/src/app/workers/machine/machine-controller.ts b/packages/server/api/src/app/workers/machine/machine-controller.tsindex a8a723eaa9f..29e908432f8 100644--- a/packages/server/api/src/app/workers/machine/machine-controller.ts+++ b/packages/server/api/src/app/workers/machine/machine-controller.ts@@ -31,7 +31,7 @@ export const workerMachineController: FastifyPluginAsyncZod = async (app) => {})const response = await machineService(app.log).onConnection(request, assignment)callback?.(response)- createRpcServer<WorkerToApiContract>(socket, createHandlers(app.log, assignment, socket.id), app.log)+ createRpcServer<WorkerToApiContract>(socket, createHandlers(app.log, assignment, socket), app.log)}})diff --git a/packages/server/api/src/app/workers/rpc/worker-rpc-service.ts b/packages/server/api/src/app/workers/rpc/worker-rpc-service.tsindex 81e319e1600..9dcbe6517c2 100644--- a/packages/server/api/src/app/workers/rpc/worker-rpc-service.ts+++ b/packages/server/api/src/app/workers/rpc/worker-rpc-service.ts@@ -2,6 +2,7 @@ import { assertNotNullOrUndefined, isNil, spreadIfDefined } from '@activepieces/import { apVersionUtil, onCallService, UNKNOWN_VERSION } from '@activepieces/server-utils'import { ExecutionType, FileCompression, FileLocation, FileType, FlowOperationType, FlowStatus, WebsocketClientEvent, WorkerToApiContract } from '@activepieces/shared'import { FastifyBaseLogger } from 'fastify'+import { Socket } from 'socket.io'import { aiRpcHandlers } from '../../ai/ai-rpc-handlers'import { websocketService } from '../../core/websockets.service'import { distributedStore, redisConnections } from '../../database/redis-connections'@@ -71,7 +72,7 @@ function workerVersionIsCompatible({ log, workerVersion, workerId, rpcMethod }:return false}-export function createHandlers(log: FastifyBaseLogger, assignment: WorkerGroupAssignment | null = null, connectionId?: string): WorkerToApiContract {+export function createHandlers(log: FastifyBaseLogger, assignment: WorkerGroupAssignment | null = null, socket?: Pick<Socket, 'id' | 'connected'>): WorkerToApiContract {return {async poll(input) {log.info({ worker: { id: input.workerId }, workerGroup: assignment ?? undefined }, '[workerRpc#poll] Poll request received')@@ -80,7 +81,7 @@ export function createHandlers(log: FastifyBaseLogger, assignment: WorkerGroupAsreturn null}const pollQueueName = workerGroupRouting.resolveQueueForWorker(assignment)- const job = await jobBroker(log).poll(pollQueueName, connectionId)+ const job = await jobBroker(log).poll(pollQueueName, socket)if (job) {log.info({ worker: { id: input.workerId }, job: { id: job.jobId, type: job.jobData.jobType } }, '[workerRpc#poll] Returning job to worker')}diff --git a/packages/server/api/test/unit/app/workers/job-queue/queue-dispatcher.test.ts b/packages/server/api/test/unit/app/workers/job-queue/queue-dispatcher.test.tsindex 5f3d0ab9b12..c35ade101bb 100644--- a/packages/server/api/test/unit/app/workers/job-queue/queue-dispatcher.test.ts+++ b/packages/server/api/test/unit/app/workers/job-queue/queue-dispatcher.test.ts@@ -96,6 +96,35 @@ describe('QueueDispatcher', () => {expect(result2).toEqual(job2)})+ it('skips a waiter whose worker disconnected and hands the job to a live one', async () => {+ const deadSocket = { connected: true }+ const deadPoll = dispatcher.poll(deadSocket)+ const livePoll = dispatcher.poll({ connected: true })+ await vi.advanceTimersByTimeAsync(0)++ deadSocket.connected = false+ const job = createFakeJob('requeued')+ pendingDequeues[0].resolve(job)+ await vi.advanceTimersByTimeAsync(0)++ expect(await deadPoll).toBeNull()+ expect(await livePoll).toEqual(job)+ expect(onOrphanedJobMock).not.toHaveBeenCalled()+ })++ it('returns the job to the queue when every waiter has disconnected', async () => {+ const deadSocket = { connected: true }+ const deadPoll = dispatcher.poll(deadSocket)+ await vi.advanceTimersByTimeAsync(0)++ deadSocket.connected = false+ pendingDequeues[0].resolve(createFakeJob('requeued'))+ await vi.advanceTimersByTimeAsync(0)++ expect(await deadPoll).toBeNull()+ expect(onOrphanedJobMock).toHaveBeenCalledWith('requeued', 'job-token', 'test-queue', mockLog)+ })+it('should only have one active getNextJob call at a time', async () => {dispatcher.poll()dispatcher.poll()