Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(workers): jobs no longer get stuck after a worker reconnects (#16893)

Hazem Adelcddc8723214e
diff --git a/brain/knowledge/execution-runtime/workers.md b/brain/knowledge/execution-runtime/workers.md
index 82bfc5ca36d..3d24793f0ec 100644
--- a/brain/knowledge/execution-runtime/workers.md
+++ b/brain/knowledge/execution-runtime/workers.md
@@ -21,6 +21,7 @@ The deep `Resolver`/`Runtime` concurrency and bundle-caching model lives on the
- `concurrency === 1` primes the sandbox to the full container RAM (cgroup-aware) via `primeFullContainerMemory()`.
### Gotchas
+- **A dispatcher waiter belongs to a socket, and a closed socket's waiter is skipped.** A poll parks a waiter for up to `WAITER_TIMEOUT_MS` (50s), which can outlive its socket. Before AI-152 a job that `releaseConnectionJobs` had just re-queued was often handed straight back to the dead worker's waiter. It then sat in `active`, recorded against a connection that had already been drained, until the stalled scan reclaimed it. `runLoop` now hands the job to the first waiter whose `socket.connected` is still true, and returns it to the queue when there isn't one.
- **Piece bundles download over the worker's own `AP_FRONTEND_URL` (`internalApiUrl`), never the app's `PUBLIC_URL` — keep it that way.** Through 0.92.x the sandbox built `GET /v1/engine/pieces/bundle` from the `PUBLIC_URL` the app pushes over the socket, so a standalone worker ignored its own `AP_FRONTEND_URL=http://app` for this one hop. A `localhost` public URL (the docs' default for compose and `docker run -p 8080:80`) failed every uncached install with `fetch failed` (GIT-1838). A public URL that reached a *different* instance sharing the JWT secret returned 404, so custom-piece upload failed with `PIECE_BUNDLE_NOT_AVAILABLE` while the `PACKAGE_ARCHIVE` row sat intact in Postgres. Tell-tale: the bundle GET never shows up in the app's own request log. Fixed by GIT-1718.
- **`AP_CONTAINER_TYPE=WORKER` moves every engine→app call onto `AP_FRONTEND_URL`.** In `WORKER_AND_APP` the engine reaches the app at `http://127.0.0.1:<port>/api/`; in `WORKER` it uses `AP_FRONTEND_URL + '/api/'` (`getApiUrl` in `worker/src/lib/config/configs.ts`), and the worker's Socket.IO connection follows it. If that URL is the public hostname, file uploads, run-log uploads, piece bundle downloads and the worker socket all cross the ingress/WAF/CDN, so WAF rules and websocket idle timeouts start failing runs that worked the day before. Point separated workers at the in-cluster app service instead. Found on Alan's fleet 2026-09-29, the day after being advised to switch to `WORKER`.
- **The piece workspace pins `linker = "isolated"` in its own `bunfig.toml` — do not delete it, the engine's piece resolver depends on the layout bun chooses.** `pieceInstaller` generates a bun workspace at `<cache>/<version>/common` whose members are `pieces/<pieceName>-<pieceVersion>`, and the engine (`piece-loader.ts` → `resolveInstalledPieceEntry`) resolves a piece **only** at `pieces/<alias>/node_modules/<pieceName>`, which just the isolated linker produces. The workspace used to carry no bunfig and inherited whatever ancestor bunfig bun discovered above the cache mount (`/usr/src/app/bunfig.toml`); when bun instead uses the **hoisted** linker it writes the package as a real directory at the workspace root `node_modules/@activepieces/<name>` and leaves the member with no `node_modules` at all, so every run of that piece fails `PieceNotFoundError: Piece not found for package: <name>-<version>` while `bun install` still exits 0. It was silent and self-perpetuating — `markPiecesAsUsed` writes `ready` and `pieceCheckIfAlreadyInstalled` only looks for *a* `node_modules`, so the piece is reinstalled forever and never resolves. Sep 2026 production: 125/125 folders missing `node_modules` were exactly the 125 packages sitting hoisted at the workspace root; 7,400+ `EXECUTE_FLOW` jobs, 15+ platforms, ~60 official pieces, ~300/hr. Two things had to line up — bun 1.4.0 (Aug 31) changed the layout newly-installed members got, and removing the piece-upgrade Redis gate (#15236, deployed 2026-09-03 08:09 UTC) mass-upgraded every platform's flows to newer piece versions so almost every install was suddenly a *first-time* install. Diagnose on a worker with `for d in <cache>/v*/common/pieces/@activepieces/*/; do [ -d "$d/node_modules" ] || basename $d; done`. Fixed by pinning the linker plus a `LATEST_CACHE_VERSION` bump (v14 → v15) so the fleet rebuilds one clean workspace under the pinned layout rather than carrying a mixed one.
diff --git a/packages/server/api/src/app/workers/job-queue/job-broker.ts b/packages/server/api/src/app/workers/job-queue/job-broker.ts
index fccbd820231..c0d42faa2a0 100644
--- a/packages/server/api/src/app/workers/job-queue/job-broker.ts
+++ b/packages/server/api/src/app/workers/job-queue/job-broker.ts
@@ -2,6 +2,7 @@ import { isNil, tryCatch } from '@activepieces/core-utils'
import { ConsumeJobRequest, ConsumeJobResponse, EngineResponseStatus, JobData } from '@activepieces/shared'
import { Worker as BullMQWorker, Job, UnrecoverableError } from 'bullmq'
import { FastifyBaseLogger } from 'fastify'
+import { Socket } from 'socket.io'
import { accessTokenManager } from '../../authentication/lib/access-token-manager'
import { redisConnections } from '../../database/redis-connections'
import { engineResponseWatcher } from '../engine-response-watcher'
@@ -222,12 +223,12 @@ export const jobBroker = (log: FastifyBaseLogger) => ({
log.info('[jobBroker] Job broker initialized')
},
- async poll(queueName: string = QueueName.WORKER_JOBS, connectionId?: string): Promise<ConsumeJobRequest | null> {
+ async poll(queueName: string = QueueName.WORKER_JOBS, socket?: Pick<Socket, 'id' | 'connected'>): Promise<ConsumeJobRequest | null> {
const worker = await ensureBullMQWorker(queueName, log)
const dispatcher = ensureDispatcher(queueName, worker, log)
- const job = await dispatcher.poll()
- if (!isNil(job) && !isNil(connectionId)) {
- jobAssignmentTracker.record({ connectionId, jobId: job.jobId, token: job.token, queueName: job.queueName })
+ const job = await dispatcher.poll(socket)
+ if (!isNil(job) && !isNil(socket)) {
+ jobAssignmentTracker.record({ connectionId: socket.id, jobId: job.jobId, token: job.token, queueName: job.queueName })
}
return job
},
diff --git a/packages/server/api/src/app/workers/job-queue/queue-dispatcher.ts b/packages/server/api/src/app/workers/job-queue/queue-dispatcher.ts
index 229ec6bd151..25b0d4c9687 100644
--- a/packages/server/api/src/app/workers/job-queue/queue-dispatcher.ts
+++ b/packages/server/api/src/app/workers/job-queue/queue-dispatcher.ts
@@ -2,6 +2,7 @@ import { isNil, tryCatch } from '@activepieces/core-utils'
import { ConsumeJobRequest } from '@activepieces/shared'
import { Worker as BullMQWorker } from 'bullmq'
import { FastifyBaseLogger } from 'fastify'
+import { Socket } from 'socket.io'
const WAITER_TIMEOUT_MS = 50_000
const ERROR_RETRY_DELAY_MS = 5_000
@@ -17,7 +18,7 @@ function createQueueDispatcher(params: {
const waiters: Waiter[] = []
let loopRunning = false
- async function poll(): Promise<ConsumeJobRequest | null> {
+ async function poll(socket?: Pick<Socket, 'connected'>): Promise<ConsumeJobRequest | null> {
return new Promise<ConsumeJobRequest | null>((resolve) => {
const timer = setTimeout(() => {
const idx = waiters.findIndex(w => w.resolve === resolve)
@@ -27,7 +28,7 @@ function createQueueDispatcher(params: {
resolve(null)
}, WAITER_TIMEOUT_MS)
- waiters.push({ resolve, timer })
+ waiters.push({ resolve, timer, socket })
startLoop()
})
}
@@ -53,7 +54,7 @@ function createQueueDispatcher(params: {
continue
}
- const waiter = waiters.shift()
+ const waiter = takeConnectedWaiter()
if (isNil(waiter)) {
log.warn({ queueName, job: { id: job.jobId } }, '[QueueDispatcher] job dequeued but no waiter available, returning to queue')
const { error: orphanError } = await tryCatch(() => onOrphanedJob(job.jobId, job.token, job.queueName, log))
@@ -69,6 +70,16 @@ function createQueueDispatcher(params: {
loopRunning = false
}
+ function takeConnectedWaiter(): Waiter | undefined {
+ let waiter = waiters.shift()
+ while (!isNil(waiter) && waiter.socket?.connected === false) {
+ clearTimeout(waiter.timer)
+ waiter.resolve(null)
+ waiter = waiters.shift()
+ }
+ return waiter
+ }
+
function close(): void {
for (const waiter of waiters) {
clearTimeout(waiter.timer)
@@ -93,10 +104,11 @@ function sleep(ms: number): Promise<void> {
type Waiter = {
resolve: (value: ConsumeJobRequest | null) => void
timer: ReturnType<typeof setTimeout>
+ socket?: Pick<Socket, 'connected'>
}
export type QueueDispatcher = {
- poll(): Promise<ConsumeJobRequest | null>
+ poll(socket?: Pick<Socket, 'connected'>): Promise<ConsumeJobRequest | null>
close(): void
waiterCount(): number
}
diff --git a/packages/server/api/src/app/workers/machine/machine-controller.ts b/packages/server/api/src/app/workers/machine/machine-controller.ts
index a8a723eaa9f..29e908432f8 100644
--- a/packages/server/api/src/app/workers/machine/machine-controller.ts
+++ b/packages/server/api/src/app/workers/machine/machine-controller.ts
@@ -31,7 +31,7 @@ export const workerMachineController: FastifyPluginAsyncZod = async (app) => {
})
const response = await machineService(app.log).onConnection(request, assignment)
callback?.(response)
- createRpcServer<WorkerToApiContract>(socket, createHandlers(app.log, assignment, socket.id), app.log)
+ createRpcServer<WorkerToApiContract>(socket, createHandlers(app.log, assignment, socket), app.log)
}
})
diff --git a/packages/server/api/src/app/workers/rpc/worker-rpc-service.ts b/packages/server/api/src/app/workers/rpc/worker-rpc-service.ts
index 81e319e1600..9dcbe6517c2 100644
--- a/packages/server/api/src/app/workers/rpc/worker-rpc-service.ts
+++ b/packages/server/api/src/app/workers/rpc/worker-rpc-service.ts
@@ -2,6 +2,7 @@ import { assertNotNullOrUndefined, isNil, spreadIfDefined } from '@activepieces/
import { apVersionUtil, onCallService, UNKNOWN_VERSION } from '@activepieces/server-utils'
import { ExecutionType, FileCompression, FileLocation, FileType, FlowOperationType, FlowStatus, WebsocketClientEvent, WorkerToApiContract } from '@activepieces/shared'
import { FastifyBaseLogger } from 'fastify'
+import { Socket } from 'socket.io'
import { aiRpcHandlers } from '../../ai/ai-rpc-handlers'
import { websocketService } from '../../core/websockets.service'
import { distributedStore, redisConnections } from '../../database/redis-connections'
@@ -71,7 +72,7 @@ function workerVersionIsCompatible({ log, workerVersion, workerId, rpcMethod }:
return false
}
-export function createHandlers(log: FastifyBaseLogger, assignment: WorkerGroupAssignment | null = null, connectionId?: string): WorkerToApiContract {
+export function createHandlers(log: FastifyBaseLogger, assignment: WorkerGroupAssignment | null = null, socket?: Pick<Socket, 'id' | 'connected'>): WorkerToApiContract {
return {
async poll(input) {
log.info({ worker: { id: input.workerId }, workerGroup: assignment ?? undefined }, '[workerRpc#poll] Poll request received')
@@ -80,7 +81,7 @@ export function createHandlers(log: FastifyBaseLogger, assignment: WorkerGroupAs
return null
}
const pollQueueName = workerGroupRouting.resolveQueueForWorker(assignment)
- const job = await jobBroker(log).poll(pollQueueName, connectionId)
+ const job = await jobBroker(log).poll(pollQueueName, socket)
if (job) {
log.info({ worker: { id: input.workerId }, job: { id: job.jobId, type: job.jobData.jobType } }, '[workerRpc#poll] Returning job to worker')
}
diff --git a/packages/server/api/test/unit/app/workers/job-queue/queue-dispatcher.test.ts b/packages/server/api/test/unit/app/workers/job-queue/queue-dispatcher.test.ts
index 5f3d0ab9b12..c35ade101bb 100644
--- a/packages/server/api/test/unit/app/workers/job-queue/queue-dispatcher.test.ts
+++ b/packages/server/api/test/unit/app/workers/job-queue/queue-dispatcher.test.ts
@@ -96,6 +96,35 @@ describe('QueueDispatcher', () => {
expect(result2).toEqual(job2)
})
+ it('skips a waiter whose worker disconnected and hands the job to a live one', async () => {
+ const deadSocket = { connected: true }
+ const deadPoll = dispatcher.poll(deadSocket)
+ const livePoll = dispatcher.poll({ connected: true })
+ await vi.advanceTimersByTimeAsync(0)
+
+ deadSocket.connected = false
+ const job = createFakeJob('requeued')
+ pendingDequeues[0].resolve(job)
+ await vi.advanceTimersByTimeAsync(0)
+
+ expect(await deadPoll).toBeNull()
+ expect(await livePoll).toEqual(job)
+ expect(onOrphanedJobMock).not.toHaveBeenCalled()
+ })
+
+ it('returns the job to the queue when every waiter has disconnected', async () => {
+ const deadSocket = { connected: true }
+ const deadPoll = dispatcher.poll(deadSocket)
+ await vi.advanceTimersByTimeAsync(0)
+
+ deadSocket.connected = false
+ pendingDequeues[0].resolve(createFakeJob('requeued'))
+ await vi.advanceTimersByTimeAsync(0)
+
+ expect(await deadPoll).toBeNull()
+ expect(onOrphanedJobMock).toHaveBeenCalledWith('requeued', 'job-token', 'test-queue', mockLog)
+ })
+
it('should only have one active getNextJob call at a time', async () => {
dispatcher.poll()
dispatcher.poll()