Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

feat(workers): emit structured job.failed log for queue alerting (#15541)

Amr Elmohamadydbc13dc067cf
diff --git a/brain/knowledge/execution-runtime/workers.md b/brain/knowledge/execution-runtime/workers.md
index d31d32b7247..4247e53dfb3 100644
--- a/brain/knowledge/execution-runtime/workers.md
+++ b/brain/knowledge/execution-runtime/workers.md
@@ -77,7 +77,7 @@ Whole-fleet throughput can collapse while every worker is healthy and idle, beca
**Diagnostic signature** (how it was found 2026-07-25/26), all from ClickHouse `default.otel_logs`: worker fleet constant (`uniqExact(LogAttributes['host'])` on `event='system.snapshot'`), per-job latency flat or improving (`avg(executionMs)` over `job.execute`) so workers aren't the bottleneck, and app-side dequeues falling on the same curve as executions. The decisive one is **uniform stretch**: for `EXECUTE_POLLING`, group by hour and compute `count() / uniqExact(flowVersion.id)`. Flat `distinctTriggers` with decaying fires-per-trigger (13,050 triggers steady while each went 15.2/hr → 1.05/hr over 30h) means a shared dispatcher throttling all of them equally; triggers actually dying looks the opposite — `distinctTriggers` drops, fires-per-trigger holds. Rule out diurnal against the same hours a week earlier first.
-Three latent hazards in the same file: `await dequeue(...)` has **no timeout**, so a hang on Redis leaves `loopRunning` true forever and every later `poll()` short-circuits, parks for `WAITER_TIMEOUT_MS` (50s) and returns null — that queue is dead for the life of the process and only a restart clears it. `tryDequeue` **recurses** once per skipped job (deferred-failure, invalid-schema, DISCARD, delay paths), so discarded zombie repeats burn dispatcher budget without producing executions (520k dequeues vs 270k executions) and a burst drives unbounded async recursion. And `concurrency: 500` on the BullMQWorker is **inert** — `autorun: false` means BullMQ's internal run loop never starts, so it reads as 500-way parallel dispatch while being 1-way.
+Three latent hazards in the same file: `await dequeue(...)` has **no timeout**, so a hang on Redis leaves `loopRunning` true forever and every later `poll()` short-circuits, parks for `WAITER_TIMEOUT_MS` (50s) and returns null — that queue is dead for the life of the process and only a restart clears it. `tryDequeue` **recurses** once per skipped job (deferred-failure, invalid-schema, DISCARD, delay paths), so discarded zombie repeats burn dispatcher budget without producing executions (520k dequeues vs 270k executions) and a burst drives unbounded async recursion. And `concurrency: 500` on the BullMQWorker is **inert** — `autorun: false` means BullMQ's internal run loop never starts, so it reads as 500-way parallel dispatch while being 1-way. Same root cause bites `Worker.on('failed')`: it emits from that same off run loop, so on `workerJobs`/`platform-*`/`project-*` the local handler never fires. Terminal failures reach observers only through the explicit `job.moveToFailed(...)` sites in `job-broker.ts` (deferred failure, invalid schema, engine INTERNAL_ERROR) or through `QueueEvents` off Redis' event stream — there is no third path. `runsMetadata` and `system-job-queue` use `autorun: true` and their `Worker.on('failed')` handlers fire normally.
#### A worker can wedge mid-poll-loop and still report healthy
diff --git a/packages/server/api/src/app/flows/flow-run/flow-runs-queue.ts b/packages/server/api/src/app/flows/flow-run/flow-runs-queue.ts
index 738f73d6ebc..2759eddc806 100644
--- a/packages/server/api/src/app/flows/flow-run/flow-runs-queue.ts
+++ b/packages/server/api/src/app/flows/flow-run/flow-runs-queue.ts
@@ -12,6 +12,7 @@ import { resumeService } from '../../waitpoints/resume-service'
import { waitpointService } from '../../waitpoints/waitpoint-service'
import { WaitpointStatus } from '../../waitpoints/waitpoint-types'
import { legacyRedisMetadataKey, QueueName, redisMetadataKey, runsMetadataDeduplicationId, RunsMetadataJobData, RunsMetadataQueueConfig, runsMetadataQueueFactory, RunsMetadataUpsertData } from '../../workers/job'
+import { jobFailureLogger } from '../../workers/job-queue/job-failure-logger'
import { flowService } from '../flow/flow.service'
import { flowRunRepo } from './flow-run-service'
import { flowRunSideEffects } from './flow-run-side-effects'
@@ -169,6 +170,20 @@ export const runsMetadataQueue = (log: FastifyBaseLogger) => ({
},
)
+ runsMetadataWorker.on('failed', (job, err) => {
+ const attemptsUsed = job?.attemptsMade ?? 0
+ const maxAttempts = job?.opts?.attempts ?? 1
+ if (attemptsUsed >= maxAttempts) {
+ jobFailureLogger.logJobFailed({
+ queueName: QueueName.RUNS_METADATA,
+ jobId: job?.id,
+ jobType: job?.name ?? 'update-run-metadata',
+ error: err,
+ log,
+ })
+ }
+ })
+
await runsMetadataWorker.waitUntilReady()
},
diff --git a/packages/server/api/src/app/helper/system-jobs/system-job.ts b/packages/server/api/src/app/helper/system-jobs/system-job.ts
index 239ee33ddec..7635bc426f9 100644
--- a/packages/server/api/src/app/helper/system-jobs/system-job.ts
+++ b/packages/server/api/src/app/helper/system-jobs/system-job.ts
@@ -4,6 +4,7 @@ import { Job, JobsOptions, Queue, Worker } from 'bullmq'
import { Dayjs } from 'dayjs'
import { FastifyBaseLogger } from 'fastify'
import { redisConnections } from '../../database/redis-connections'
+import { jobFailureLogger } from '../../workers/job-queue/job-failure-logger'
import { exceptionHandler } from '../exception-handler'
import { SystemJobData, SystemJobName, SystemJobSchedule } from './common'
import { systemJobHandlers } from './job-handlers'
@@ -58,8 +59,15 @@ export const systemJobsSchedule = (log: FastifyBaseLogger): SystemJobSchedule =>
systemJobWorker.on('failed', (job, err) => {
const attemptsUsed = job?.attemptsMade ?? 0
- const maxAttempts = job?.opts?.attempts ?? Infinity
+ const maxAttempts = job?.opts?.attempts ?? 1
if (attemptsUsed >= maxAttempts) {
+ jobFailureLogger.logJobFailed({
+ queueName: SYSTEM_JOB_QUEUE,
+ jobId: job?.id,
+ jobType: job?.name ?? 'unknown',
+ error: err,
+ log,
+ })
exceptionHandler.handle(err, log)
}
})
diff --git a/packages/server/api/src/app/workers/job-queue/job-broker.ts b/packages/server/api/src/app/workers/job-queue/job-broker.ts
index 923d67eecda..9c5aafbc1af 100644
--- a/packages/server/api/src/app/workers/job-queue/job-broker.ts
+++ b/packages/server/api/src/app/workers/job-queue/job-broker.ts
@@ -10,6 +10,7 @@ import { jobMigrations } from '../migrations/job-data-migrations'
import { rateLimiterInterceptor } from './interceptors/rate-limiter-interceptor'
import { zombiePollingInterceptor } from './interceptors/zombie-polling-interceptor'
import { jobAssignmentTracker } from './job-assignment-tracker'
+import { jobFailureLogger } from './job-failure-logger'
import { InterceptorVerdict, JobInterceptor } from './job-interceptor'
import { callerWaitingForResponse } from './job-queue'
import { createQueueDispatcher, QueueDispatcher } from './queue-dispatcher'
@@ -92,13 +93,17 @@ async function tryDequeue(worker: BullMQWorker, queueName: string, log: FastifyB
{ queueName, job: { id: job.id }, jobName: job.name, deferredFailure: job.deferredFailure },
'[jobBroker#tryDequeue] Failing job with deferred failure (BullMQ stalled limit exceeded)',
)
- const { error: failError } = await tryCatch(() => job.moveToFailed(new UnrecoverableError(job.deferredFailure), token, false))
+ const deferredError = new UnrecoverableError(job.deferredFailure)
+ const { error: failError } = await tryCatch(() => job.moveToFailed(deferredError, token, false))
if (failError) {
log.error(
{ queueName, job: { id: job.id }, error: String(failError) },
'[jobBroker#tryDequeue] Failed to fail deferred-failure job',
)
}
+ else {
+ jobFailureLogger.logJobFailed({ queueName, jobId: job.id, jobType: job.name, error: deferredError, log })
+ }
return tryDequeue(worker, queueName, log)
}
@@ -120,10 +125,14 @@ async function tryDequeue(worker: BullMQWorker, queueName: string, log: FastifyB
{ queueName, job: { id: jobId, type: migratedData.jobType }, schemaVersion: migratedData.schemaVersion, issues: parseResult.error.issues },
'[jobBroker#tryDequeue] Failing job with invalid schema as unrecoverable',
)
- const { error: failError } = await tryCatch(() => job.moveToFailed(new UnrecoverableError(reason), token, false))
+ const schemaError = new UnrecoverableError(reason)
+ const { error: failError } = await tryCatch(() => job.moveToFailed(schemaError, token, false))
if (failError) {
log.error({ queueName, job: { id: jobId }, error: String(failError) }, '[jobBroker#tryDequeue] Failed to fail invalid-schema job')
}
+ else {
+ jobFailureLogger.logJobFailed({ queueName, jobId, jobType: migratedData.jobType, error: schemaError, log })
+ }
return tryDequeue(worker, queueName, log)
}
@@ -263,7 +272,19 @@ export const jobBroker = (log: FastifyBaseLogger) => ({
await job.moveToCompleted({ response: undefined }, input.token, false)
return
}
- await job.moveToFailed(new Error(buildFailedReason(input.errorMessage ?? 'Internal error', input.logs)), input.token)
+ const engineErrorMessage = input.errorMessage ?? 'Internal error'
+ const engineError = new Error(buildFailedReason(engineErrorMessage, input.logs))
+ await job.moveToFailed(engineError, input.token)
+ if ((job.attemptsMade ?? 0) >= (job.opts?.attempts ?? 1)) {
+ jobFailureLogger.logJobFailed({
+ queueName: input.queueName,
+ jobId: input.jobId,
+ jobType: jobData.jobType,
+ error: engineError,
+ log,
+ signatureOverride: jobFailureLogger.signatureFromMessage(engineErrorMessage, 'EngineError'),
+ })
+ }
return
}
diff --git a/packages/server/api/src/app/workers/job-queue/job-failure-logger.ts b/packages/server/api/src/app/workers/job-queue/job-failure-logger.ts
new file mode 100644
index 00000000000..72ae72264f0
--- /dev/null
+++ b/packages/server/api/src/app/workers/job-queue/job-failure-logger.ts
@@ -0,0 +1,55 @@
+import path from 'path'
+import { FastifyBaseLogger } from 'fastify'
+
+const STACK_FRAME_RE = /at\s+(?:.+?\s+\()?([^\s()]+):(\d+):\d+\)?/
+
+export const jobFailureLogger = {
+ signatureOf(error: unknown): string {
+ const errorName = readErrorName(error)
+ const frame = readTopStackFrame(error)
+ return frame ? `${errorName}@${frame}` : errorName
+ },
+ signatureFromMessage(message: string, sourceLabel: string): string {
+ const firstLine = message.split('\n', 1)[0].trim()
+ const colonIdx = firstLine.indexOf(':')
+ const head = colonIdx > 0 ? firstLine.slice(0, colonIdx) : firstLine
+ const clean = head.replace(/\s+/g, ' ').slice(0, 80)
+ return `${sourceLabel}@${clean || 'unknown'}`
+ },
+ logJobFailed({ queueName, jobId, jobType, error, log, signatureOverride }: LogJobFailedParams): void {
+ const errorSignature = signatureOverride ?? jobFailureLogger.signatureOf(error)
+ log.error({
+ queue: { name: queueName },
+ job: { id: jobId ?? '', type: jobType },
+ error,
+ errorSignature,
+ }, 'job.failed')
+ },
+}
+
+function readErrorName(error: unknown): string {
+ if (error instanceof Error) {
+ const declared = error.name && error.name !== 'Error' ? error.name : undefined
+ return declared ?? error.constructor?.name ?? 'Error'
+ }
+ return typeof error
+}
+
+function readTopStackFrame(error: unknown): string | null {
+ if (!(error instanceof Error) || !error.stack) return null
+ for (const line of error.stack.split('\n')) {
+ const match = STACK_FRAME_RE.exec(line)
+ if (!match) continue
+ return `${path.basename(match[1])}:${match[2]}`
+ }
+ return null
+}
+
+type LogJobFailedParams = {
+ queueName: string
+ jobId: string | undefined
+ jobType: string
+ error: unknown
+ log: FastifyBaseLogger
+ signatureOverride?: string
+}
diff --git a/packages/server/api/test/unit/app/workers/job-queue/job-failure-logger.test.ts b/packages/server/api/test/unit/app/workers/job-queue/job-failure-logger.test.ts
new file mode 100644
index 00000000000..bb1b2996d38
--- /dev/null
+++ b/packages/server/api/test/unit/app/workers/job-queue/job-failure-logger.test.ts
@@ -0,0 +1,55 @@
+import { describe, expect, it } from 'vitest'
+import { jobFailureLogger } from '../../../../../src/app/workers/job-queue/job-failure-logger'
+
+describe('jobFailureLogger.signatureOf', () => {
+ it('groups two failures thrown at the same site under the same signature', () => {
+ function throwAtFixedLine(): never {
+ throw new Error('same call site')
+ }
+ let sigA = ''
+ let sigB = ''
+ try { throwAtFixedLine() }
+ catch (e) { sigA = jobFailureLogger.signatureOf(e) }
+ try { throwAtFixedLine() }
+ catch (e) { sigB = jobFailureLogger.signatureOf(e) }
+ expect(sigA).toBe(sigB)
+ expect(sigA).toMatch(/^Error@job-failure-logger\.test\.ts:\d+$/)
+ })
+
+ it('preserves the subclass name for a custom Error', () => {
+ class BoomError extends Error {
+ constructor() {
+ super('boom')
+ this.name = 'BoomError'
+ }
+ }
+ try { throw new BoomError() }
+ catch (e) {
+ expect(jobFailureLogger.signatureOf(e)).toMatch(/^BoomError@/)
+ }
+ })
+
+ it('returns just the typeof for non-Error thrown values', () => {
+ expect(jobFailureLogger.signatureOf('nope')).toBe('string')
+ expect(jobFailureLogger.signatureOf({ a: 1 })).toBe('object')
+ expect(jobFailureLogger.signatureOf(undefined)).toBe('undefined')
+ })
+})
+
+describe('jobFailureLogger.signatureFromMessage', () => {
+ it('groups by the leading token before the first colon', () => {
+ expect(jobFailureLogger.signatureFromMessage('ConnectionNotFoundError: connection abc not found', 'EngineError'))
+ .toBe('EngineError@ConnectionNotFoundError')
+ expect(jobFailureLogger.signatureFromMessage('ConnectionNotFoundError: connection xyz missing', 'EngineError'))
+ .toBe('EngineError@ConnectionNotFoundError')
+ })
+
+ it('uses the first line when no colon is present, truncated to 80 chars', () => {
+ expect(jobFailureLogger.signatureFromMessage('Sandbox process exited with code 137', 'EngineError'))
+ .toBe('EngineError@Sandbox process exited with code 137')
+ })
+
+ it('handles empty message with unknown fallback', () => {
+ expect(jobFailureLogger.signatureFromMessage('', 'EngineError')).toBe('EngineError@unknown')
+ })
+})