Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(agents): an AI agent step no longer fails when its worker briefly disconnects (#16896)

ziyadev6c1a97299d89
diff --git a/brain/knowledge/execution-runtime/workers.md b/brain/knowledge/execution-runtime/workers.md
index 3d24793f0ec..f2d83ea969c 100644
--- a/brain/knowledge/execution-runtime/workers.md
+++ b/brain/knowledge/execution-runtime/workers.md
@@ -21,6 +21,7 @@ The deep `Resolver`/`Runtime` concurrency and bundle-caching model lives on the
- `concurrency === 1` primes the sandbox to the full container RAM (cgroup-aware) via `primeFullContainerMemory()`.
### Gotchas
+- **An RPC sent during a reconnect is lost.** socket.io-client flushes buffered emits in `onconnect`, before the worker's `connect` handler runs, and the app only attaches its RPC server after `FETCH_WORKER_SETTINGS` and `onConnection` resolve. So anything sent in that window reaches a socket with no listener. An acked call lands only on retry after the 60s timeout (a flow step's `resumeFlowStep` in the AI-156 repro), and an `rpc-notify` is lost for good. Tracked in ENG-1387.
- **A dispatcher waiter belongs to a socket, and a closed socket's waiter is skipped.** A poll parks a waiter for up to `WAITER_TIMEOUT_MS` (50s), which can outlive its socket. Before AI-152 a job that `releaseConnectionJobs` had just re-queued was often handed straight back to the dead worker's waiter. It then sat in `active`, recorded against a connection that had already been drained, until the stalled scan reclaimed it. `runLoop` now hands the job to the first waiter whose `socket.connected` is still true, and returns it to the queue when there isn't one.
- **Piece bundles download over the worker's own `AP_FRONTEND_URL` (`internalApiUrl`), never the app's `PUBLIC_URL` — keep it that way.** Through 0.92.x the sandbox built `GET /v1/engine/pieces/bundle` from the `PUBLIC_URL` the app pushes over the socket, so a standalone worker ignored its own `AP_FRONTEND_URL=http://app` for this one hop. A `localhost` public URL (the docs' default for compose and `docker run -p 8080:80`) failed every uncached install with `fetch failed` (GIT-1838). A public URL that reached a *different* instance sharing the JWT secret returned 404, so custom-piece upload failed with `PIECE_BUNDLE_NOT_AVAILABLE` while the `PACKAGE_ARCHIVE` row sat intact in Postgres. Tell-tale: the bundle GET never shows up in the app's own request log. Fixed by GIT-1718.
- **`AP_CONTAINER_TYPE=WORKER` moves every engine→app call onto `AP_FRONTEND_URL`.** In `WORKER_AND_APP` the engine reaches the app at `http://127.0.0.1:<port>/api/`; in `WORKER` it uses `AP_FRONTEND_URL + '/api/'` (`getApiUrl` in `worker/src/lib/config/configs.ts`), and the worker's Socket.IO connection follows it. If that URL is the public hostname, file uploads, run-log uploads, piece bundle downloads and the worker socket all cross the ingress/WAF/CDN, so WAF rules and websocket idle timeouts start failing runs that worked the day before. Point separated workers at the in-cluster app service instead. Found on Alan's fleet 2026-09-29, the day after being advised to switch to `WORKER`.
diff --git a/brain/knowledge/flows-execution/chat.md b/brain/knowledge/flows-execution/chat.md
index 245eef44e16..8336a5194ea 100644
--- a/brain/knowledge/flows-execution/chat.md
+++ b/brain/knowledge/flows-execution/chat.md
@@ -42,6 +42,7 @@ A turn is kept alive / reclaimed by three separate mechanisms in `execute-agent-
- **A turn limit only works as a stop condition.** One `streamText` call runs every step of a turn, so a check after it returns can't stop a runaway. The token ceiling used to be checked only there, and a turn on 24 Oct 2026 ran 181 steps and 50.8M input tokens (AI-86). Put every per-turn bound in `loopStopCondition`. Test mocks must report usage in the V3 shape (`inputTokens: { total, ... }`); flat numbers read as 0 tokens.
- **The streaming lock is per run, not per conversation.** A re-send can claim `activeRunId` before the first job reaches `getAgentConfig`. `agentHelpers.acquireStreamingLock` only lets the owning run set STREAMING; an older run gets `AGENT_RUN_SUPERSEDED` and the worker exits quietly, with no ERROR save and no events. A lock that ignored `runId` let the old run stream unowned while the new one was rejected as busy and marked ERROR, which is how users saw nothing after tapping a starter card twice.
- **A redelivered run is a duplicate, not busy.** A worker socket blip makes the app requeue a chat job whose worker is still running it (`releaseConnectionJobs`). The second delivery finds its own `activeRunId` STREAMING, so `acquireStreamingLock` returns `duplicate` and it exits quietly like a superseded run. When this came back as `busy` it saved ERROR, and users saw "An agent is already running for this conversation" in the middle of a working turn (prod, 2026-10-13). If the first worker really died, the 90s `CHAT_STALE_SWEEP` frees the conversation. This only works when the conversation carries the run's `activeRunId`: chat claims it in the controller, and a flow-step run claims it when `getAgentConfig` creates the conversation. Without that claim the redelivery read as `busy` and failed the job (prod, 2026-10-21). A copy that arrives after its run already finished finds the row IDLE, so the lock alone would let the agent run again and its tools act twice (AI-153). `acquireStreamingLock` therefore also sets a Redis marker per conversation and run (`agent-run-started:<conversationId>:<runId>`, 24h) with `putIfAbsent`, and any later delivery of that run is a `duplicate`. That's safe because every enqueue mints a fresh `runId` and agent jobs run with `attempts: 1`. If the first worker really died, the redelivery is still dropped, so `CHAT_STALE_SWEEP` fails the step itself: a flow-step conversation keeps its `waitpointId`, and the sweep resumes that waitpoint with a `failure` result, so the flow doesn't sit paused until the 3h step timeout. It never re-runs the agent, because its tools may already have acted.
+- **A flow-step run that loses its `getAgentConfig` response exits quietly, and the redelivered copy finishes it.** A socket drop mid-RPC surfaces on the worker as `RpcTimeoutError.disconnected`. The API handler doesn't stop when the socket drops: it still takes the streaming lock, so the copy used to get `duplicate` and the step failed, either right away as "provider temporarily unavailable" or 90s later through `CHAT_STALE_SWEEP`. Now the worker exits quietly on a disconnect, and `getAgentConfig` in `worker-rpc-service.ts` calls `agentHelpers.handLockToRedelivery` when the worker socket is gone by the time it replies. That writes a one-time `agent-run-handed-off` token (60s), and the next delivery of that run consumes it in `acquireStreamingLock` instead of being dropped (`'handed-off'`), so the copy runs the step (AI-156). The first delivery already saved the user message, so a handed-off copy re-reads the history after the lock and does not append it again. The hand-off never touches the row: setting it IDLE would hide the run from `CHAT_STALE_SWEEP` when the copy was already turned away, and the flow would sit paused for 3h. The token's TTL is shorter than the sweep window, so a late copy can't revive a step the sweep already failed. A plain timeout on a live socket is never redelivered, so it still fails the step. Chat still reports the error, because its buffered emits would mark ERROR over a copy that is streaming.
- **A failed turn used to leave status ERROR with an empty reply.** The error text only travelled as a websocket `ERROR` event, so a reload, another tab, or a user who was away saw a dead conversation (and often no user message either when the job died inside `getAgentConfig`, before it persists the turn). The `execute-agent-run.ts` catch now sends `failure: { message, userMessage? }` on the empty `saveAgentMessages` call and `conversation-rpc.ts` appends it to `uiMessages` as an assistant text reply; `use-chat.ts` shows the Retry banner when a loaded conversation is ERROR. Stale STREAMING to IDLE recovery, a worker that never picks the job up, and `assertAgentMessageRateLimitNotExceeded` still leave no reply of their own.
- **A built flow ends with a "Turn it on?" card, never an auto-publish**: an audit found 40% of built-but-unpublished flows ended on "open it to review", some claiming LIVE. Chat never publishes on its own; only a yes to the card calls `ap_lock_and_publish` (`build_flow.md` "Turn it on?"). Pinned by the `build-then-ask-turn-it-on` fixture (`askedToTurnItOn`, `noLiveClaimWithoutPublish`). The live-claim check is a regex plus "was `ap_lock_and_publish` called", so it cannot see whether publish succeeded.
- **Server-managed connections**: the LLM never sees connection externalIds; `ap_execute_action` auto-fills them from a Redis store.
diff --git a/packages/core/execution/src/lib/engine/rpc.ts b/packages/core/execution/src/lib/engine/rpc.ts
index d3eed21863d..c884a24164c 100644
--- a/packages/core/execution/src/lib/engine/rpc.ts
+++ b/packages/core/execution/src/lib/engine/rpc.ts
@@ -3,6 +3,7 @@ import { ActivepiecesError, isObject, spreadIfNotUndefined, toError } from '@act
const RPC_EVENT = 'rpc'
const NOTIFY_EVENT = 'rpc-notify'
const AP_ERROR_PROP = 'apError'
+const SOCKET_DISCONNECTED_MESSAGE = 'socket has been disconnected'
// eslint-disable-next-line @typescript-eslint/no-explicit-any
type Contract = Record<string, (input: any) => any>
@@ -114,10 +115,12 @@ function isRpcErrorEnvelope(value: unknown): value is { __rpcError: string, __rp
export class RpcTimeoutError extends Error {
readonly method: string
readonly timeoutMs: number
+ readonly disconnected: boolean
constructor({ method, timeoutMs, cause }: { method: string, timeoutMs: number, cause: unknown }) {
super(`RPC [${method}] failed (timeout: ${timeoutMs}ms): ${toError(cause).message}`)
this.method = method
this.timeoutMs = timeoutMs
+ this.disconnected = toError(cause).message === SOCKET_DISCONNECTED_MESSAGE
}
}
diff --git a/packages/server/api/src/app/database/redis/distributed-store-factory.ts b/packages/server/api/src/app/database/redis/distributed-store-factory.ts
index bbdbdfadbb4..767430931c0 100644
--- a/packages/server/api/src/app/database/redis/distributed-store-factory.ts
+++ b/packages/server/api/src/app/database/redis/distributed-store-factory.ts
@@ -55,11 +55,11 @@ export const distributedStoreFactory = (getRedisClient: () => Promise<Redis>) =>
return true
},
- async delete(keys: string | string[]): Promise<void> {
+ async delete(keys: string | string[]): Promise<number> {
const keysArray = Array.isArray(keys) ? keys : [keys]
- if (keysArray.length === 0) return
+ if (keysArray.length === 0) return 0
const redisClient = await getRedisClient()
- await redisClient.del(...keysArray)
+ return redisClient.del(...keysArray)
},
async putBoolean(key: string, value: boolean): Promise<void> {
diff --git a/packages/server/api/src/app/ee/agent/agent-helpers.ts b/packages/server/api/src/app/ee/agent/agent-helpers.ts
index 6b5fa9dffff..84dcae028a6 100644
--- a/packages/server/api/src/app/ee/agent/agent-helpers.ts
+++ b/packages/server/api/src/app/ee/agent/agent-helpers.ts
@@ -28,6 +28,7 @@ import { UserMemoryEntity } from './user-memory-entity'
const STREAMING_STALENESS_TIMEOUT_MS = 90 * 1_000
const STARTED_RUN_MARKER_TTL_SECONDS = 24 * 60 * 60
+const HANDED_OFF_RUN_TTL_SECONDS = 60
// Interactive-eval conversations carry this id prefix (within the 21-char id column) so both the
// eval endpoints and the regular chat path can tell them apart from real user conversations.
@@ -433,7 +434,8 @@ async function acquireStreamingLock({ conversationId, runId }: { conversationId:
if (!isNil(runId)) {
const firstDelivery = await distributedStore.putIfAbsent(`agent-run-started:${conversationId}:${runId}`, true, STARTED_RUN_MARKER_TTL_SECONDS)
if (!firstDelivery) {
- return 'duplicate'
+ const handedToThisDelivery = await distributedStore.delete(handedOffRunKey({ conversationId, runId })) > 0
+ return handedToThisDelivery ? 'handed-off' : 'duplicate'
}
}
const builder = conversationRepo()
@@ -457,6 +459,14 @@ async function acquireStreamingLock({ conversationId, runId }: { conversationId:
return current.activeRunId === runId ? 'duplicate' : 'superseded'
}
+async function handLockToRedelivery({ conversationId, runId }: { conversationId: string, runId: string }): Promise<void> {
+ await distributedStore.put(handedOffRunKey({ conversationId, runId }), true, HANDED_OFF_RUN_TTL_SECONDS)
+}
+
+function handedOffRunKey({ conversationId, runId }: { conversationId: string, runId: string }): string {
+ return `agent-run-handed-off:${conversationId}:${runId}`
+}
+
async function resolveFlowTools({ projectId, tools, log }: { projectId: string, tools: AgentTool[], log: FastifyBaseLogger }): Promise<ResolvedAgentFlowTool[]> {
const flowToolRequests = tools.filter((tool): tool is AgentFlowTool => tool.type === AgentToolType.FLOW)
if (flowToolRequests.length === 0) {
@@ -519,6 +529,7 @@ export const agentHelpers = {
jobFieldsFromConfig,
claimConversationForRun,
acquireStreamingLock,
+ handLockToRedelivery,
resolveFlowTools,
agentsSurfaceAvailable,
assertAgentsSurfaceAvailable,
@@ -557,7 +568,7 @@ type SweptConversation = z.infer<typeof SweptConversation>
type AgentJobConfigFields = Pick<ExecuteAgentRunJobData, 'tools' | 'structuredOutput' | 'maxSteps' | 'modelName' | 'provider' | 'providerConfigId' | 'modelTierId' | 'promptOverride'>
-export type StreamingLockResult = 'acquired' | 'busy' | 'superseded' | 'duplicate'
+export type StreamingLockResult = 'acquired' | 'handed-off' | 'busy' | 'superseded' | 'duplicate'
export type PlatformWorkModel = {
config: GetProviderConfigResponse
diff --git a/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts b/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts
index a2227932645..1c2d2ee9594 100644
--- a/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts
+++ b/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts
@@ -238,10 +238,12 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({
? { ...agentPrompt.guides, ...promptOverride.guides }
: agentPrompt.guides
- const previousMessages = conversation.messages as ModelMessage[]
+ const resumedHistory = lock === 'handed-off' ? await agentHelpers.conversationRepo().findOneByOrFail({ id: conversationId, platformId }) : null
+ const history = resumedHistory ?? conversation
+ const previousMessages = history.messages as ModelMessage[]
const onboardingNote = agentSurfaceNotes.onboardingNote({ source: conversation.source, ...spreadIfDefined('messageSource', input.messageSource) })
const newUserMessage: ModelMessage = { role: 'user' as const, content: isNil(onboardingNote) ? userContent : withLeadingNote({ content: userContent, note: onboardingNote }) }
- const allMessages = [...previousMessages, newUserMessage]
+ const allMessages = isNil(resumedHistory) ? [...previousMessages, newUserMessage] : previousMessages
const fastModelSeesMessages = conversation.source !== AgentRunSource.CHAT
const modelsSeeingMessages = isNil(tierRun)
? [resolvedModelId, ...(fastModelSeesMessages ? [fastModelId] : [])].map((modelId) => ({ provider: providerConfig.provider, modelId }))
@@ -252,11 +254,10 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({
models: modelsSeeingMessages.map((model) => catalog.lookup(model)),
})
- const previousUiMessages = (conversation.uiMessages ?? []) as PersistedAgentMessage[]
- const uiMessagesWithUser: PersistedAgentMessage[] = [
- ...previousUiMessages,
- { role: PersistedAgentRole.USER, parts: [{ type: PersistedAgentPartType.TEXT, text: userMessage }] },
- ]
+ const previousUiMessages = (history.uiMessages ?? []) as PersistedAgentMessage[]
+ const uiMessagesWithUser: PersistedAgentMessage[] = isNil(resumedHistory)
+ ? [...previousUiMessages, { role: PersistedAgentRole.USER, parts: [{ type: PersistedAgentPartType.TEXT, text: userMessage }] }]
+ : previousUiMessages
await agentHelpers.conversationRepo().update(conversationId, {
messages: allMessages,
uiMessages: JSON.parse(JSON.stringify(uiMessagesWithUser)),
diff --git a/packages/server/api/src/app/workers/rpc/worker-rpc-service.ts b/packages/server/api/src/app/workers/rpc/worker-rpc-service.ts
index b4b1b683ed0..574ef6baae4 100644
--- a/packages/server/api/src/app/workers/rpc/worker-rpc-service.ts
+++ b/packages/server/api/src/app/workers/rpc/worker-rpc-service.ts
@@ -1,11 +1,12 @@
import { assertNotNullOrUndefined, isNil, spreadIfDefined } from '@activepieces/core-utils'
import { apVersionUtil, onCallService, UNKNOWN_VERSION } from '@activepieces/server-utils'
-import { ExecutionType, FileCompression, FileLocation, FileType, FlowOperationType, FlowStatus, WebsocketClientEvent, WorkerToApiContract } from '@activepieces/shared'
+import { AgentRunSource, ExecutionType, FileCompression, FileLocation, FileType, FlowOperationType, FlowStatus, WebsocketClientEvent, WorkerToApiContract } from '@activepieces/shared'
import { FastifyBaseLogger } from 'fastify'
import { Socket } from 'socket.io'
import { aiRpcHandlers } from '../../ai/ai-rpc-handlers'
import { websocketService } from '../../core/websockets.service'
import { distributedStore, redisConnections } from '../../database/redis-connections'
+import { agentHelpers } from '../../ee/agent/agent-helpers'
import { agentRpcHandlers } from '../../ee/agent/agent-rpc-handlers'
import { chatPersonalizationService } from '../../ee/agent/personalization/chat-personalization-service'
import { eventDestinationRpcHandlers } from '../../event-destinations/event-destinations-rpc-handlers'
@@ -344,7 +345,14 @@ export function createHandlers(log: FastifyBaseLogger, assignment: WorkerGroupAs
},
async getAgentConfig(input) {
- return agentRpcHandlers(agentRpcLog(log, input)).getAgentConfig(input)
+ const config = await agentRpcHandlers(agentRpcLog(log, input)).getAgentConfig(input)
+ const { conversationId, runId, source } = input
+ const flowStepReplyUndeliverable = socket?.connected === false && source === AgentRunSource.FLOW_STEP && !isNil(runId)
+ if (flowStepReplyUndeliverable) {
+ log.warn({ conversation: { id: conversationId }, run: { id: runId } }, '[workerRpc#getAgentConfig] Worker disconnected before the config reply, handing the lock to the redelivered copy')
+ await agentHelpers.handLockToRedelivery({ conversationId, runId })
+ }
+ return config
},
async saveAgentMessages(input) {
diff --git a/packages/server/api/test/integration/ce/agent/agent-run-worker-handoff.test.ts b/packages/server/api/test/integration/ce/agent/agent-run-worker-handoff.test.ts
index bdb7e4d4aa3..c6872a196cb 100644
--- a/packages/server/api/test/integration/ce/agent/agent-run-worker-handoff.test.ts
+++ b/packages/server/api/test/integration/ce/agent/agent-run-worker-handoff.test.ts
@@ -4,6 +4,7 @@ import { FastifyInstance } from 'fastify'
import { StatusCodes } from 'http-status-codes'
import { afterAll, beforeAll, describe, expect, it } from 'vitest'
import { accessTokenManager } from '../../../../src/app/authentication/lib/access-token-manager'
+import { agentHelpers } from '../../../../src/app/ee/agent/agent-helpers'
import { agentRpcHandlers } from '../../../../src/app/ee/agent/agent-rpc-handlers'
import { db } from '../../../helpers/db'
import { mockAndSaveAIProvider } from '../../../helpers/mocks'
@@ -111,4 +112,35 @@ describe('community edition: the worker picking up an agent step', () => {
error: { code: ErrorCode.AGENT_RUN_SUPERSEDED },
})
})
+
+ it('gives the copy that takes over a lost config the instruction once, not twice', async () => {
+ const ctx = await createTestContext(app)
+ await mockAndSaveAIProvider({ platformId: ctx.platform.id, provider: AIProviderName.OPENAI })
+ const conversationId = await startFlowStepRun(ctx, 'ce-handoff-resume')
+ const delivery = {
+ conversationId,
+ runId: apId(),
+ platformId: ctx.platform.id,
+ userId: ctx.user.id,
+ projectId: ctx.project.id,
+ userMessage: 'summarise the thread',
+ modelName: 'gpt-4o',
+ provider: AIProviderName.OPENAI,
+ source: AgentRunSource.FLOW_STEP,
+ waitpointId: apId(),
+ }
+ await agentRpcHandlers(app.log).getAgentConfig(delivery)
+ await agentHelpers.handLockToRedelivery({ conversationId, runId: delivery.runId })
+
+ const config = await agentRpcHandlers(app.log).getAgentConfig(delivery)
+
+ const stored = await db.findOneByOrFail<{ messages: { role: string }[], uiMessages: { role: string }[] }>('agent_conversation', { id: conversationId })
+ expect(userTurns({ messages: config.previousUiMessages })).toBe(1)
+ expect(userTurns({ messages: stored.messages })).toBe(1)
+ expect(userTurns({ messages: stored.uiMessages })).toBe(1)
+ })
})
+
+function userTurns({ messages }: { messages: { role: string }[] }): number {
+ return messages.filter((message) => message.role === 'user').length
+}
diff --git a/packages/server/api/test/integration/cloud/agent/agent-resend-race.test.ts b/packages/server/api/test/integration/cloud/agent/agent-resend-race.test.ts
index b17c0b7f43b..480e7c5be8b 100644
--- a/packages/server/api/test/integration/cloud/agent/agent-resend-race.test.ts
+++ b/packages/server/api/test/integration/cloud/agent/agent-resend-race.test.ts
@@ -48,6 +48,34 @@ describe('re-sending a message while the first run has not started yet', () => {
expect(stored.activeRunId).toBe('run-1')
})
+ it('lets the redelivered copy take a lock handed to it once, when the first delivery never received its config', async () => {
+ const ctx = await createTestContext(app, { plan: { chatEnabled: true } })
+ const created = await ctx.post('/v1/agents/conversations', { title: 'Undelivered config' })
+ const conversationId = created.json().id
+ await agentHelpers.claimConversationForRun({ conversationId, runId: 'run-1' })
+ expect(await agentHelpers.acquireStreamingLock({ conversationId, runId: 'run-1' })).toBe('acquired')
+
+ await agentHelpers.handLockToRedelivery({ conversationId, runId: 'run-1' })
+
+ expect(await agentHelpers.acquireStreamingLock({ conversationId, runId: 'run-1' })).toBe('handed-off')
+ expect(await agentHelpers.acquireStreamingLock({ conversationId, runId: 'run-1' })).toBe('duplicate')
+ })
+
+ it('leaves the run streaming for the stale sweep when the copy is turned away before the hand-off', async () => {
+ const ctx = await createTestContext(app, { plan: { chatEnabled: true } })
+ const created = await ctx.post('/v1/agents/conversations', { title: 'Early copy' })
+ const conversationId = created.json().id
+ await agentHelpers.claimConversationForRun({ conversationId, runId: 'run-1' })
+ expect(await agentHelpers.acquireStreamingLock({ conversationId, runId: 'run-1' })).toBe('acquired')
+ expect(await agentHelpers.acquireStreamingLock({ conversationId, runId: 'run-1' })).toBe('duplicate')
+
+ await agentHelpers.handLockToRedelivery({ conversationId, runId: 'run-1' })
+
+ const stored = await db.findOneByOrFail<{ status: string, activeRunId: string }>('agent_conversation', { id: conversationId })
+ expect(stored.status).toBe(AgentConversationStatus.STREAMING)
+ expect(stored.activeRunId).toBe('run-1')
+ })
+
it('drops a copy delivered after its run finished, so the agent and its tools do not run twice', async () => {
const ctx = await createTestContext(app, { plan: { chatEnabled: true } })
const created = await ctx.post('/v1/agents/conversations', { title: 'Late copy' })
diff --git a/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts b/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts
index e4f793a302a..b860b37f6ff 100644
--- a/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts
+++ b/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts
@@ -1,6 +1,6 @@
import { ActivepiecesAiBilling, ActivepiecesAiConsumerSource, AiCallContext, AIProviderName, AiUsageEnvironment, ErrorCode, formatPieceError, isNil, isObject, spreadIfDefined, tryCatch, tryCatchSync } from '@activepieces/core-utils'
import { aiUtils, wideEvent } from '@activepieces/server-utils'
-import { AGENT_RUN_USAGE_ORIGIN, AgentConfigResponse, AgentEvent, AgentEventType, AgentKnowledgeBaseTool, AgentMcpTool, AgentModelCandidate, AgentOutputField, AgentPhase, AgentPieceTool, AgentResult, AgentRunSource, AgentTool, agentToolSkills, AgentToolType, AiProviderCredentials, apErrorOf, chatBilling, EngineResponseStatus, ExecuteAgentRunJobData, MAX_AGENT_TURN_WALL_CLOCK_MS, PersistedAgentMessage, PersistedAgentMessageSchema, PersistedAgentPart, PersistedAgentPartType, PersistedAgentRole, ResolvedAgentFlowTool, SaveAgentMessagesRequest, SkillSurface, WorkerJobType } from '@activepieces/shared'
+import { AGENT_RUN_USAGE_ORIGIN, AgentConfigResponse, AgentEvent, AgentEventType, AgentKnowledgeBaseTool, AgentMcpTool, AgentModelCandidate, AgentOutputField, AgentPhase, AgentPieceTool, AgentResult, AgentRunSource, AgentTool, agentToolSkills, AgentToolType, AiProviderCredentials, apErrorOf, chatBilling, EngineResponseStatus, ExecuteAgentRunJobData, MAX_AGENT_TURN_WALL_CLOCK_MS, PersistedAgentMessage, PersistedAgentMessageSchema, PersistedAgentPart, PersistedAgentPartType, PersistedAgentRole, ResolvedAgentFlowTool, RpcTimeoutError, SaveAgentMessagesRequest, SkillSurface, WorkerJobType } from '@activepieces/shared'
import { createUIMessageStream, LanguageModel, ModelMessage, streamText, ToolSet, toUIMessageStream } from 'ai'
import { FireAndForgetJobResult, JobContext, JobHandler, JobResultKind } from '../../../types'
import { toResolvedAiFile } from '../../ai/ai-files'
@@ -461,6 +461,11 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForge
log.info({ conversation: { id: conversationId }, run: { id: runId } }, '[executeAgentRun] Run superseded by a newer message, exiting quietly')
return { kind: JobResultKind.FIRE_AND_FORGET, status: EngineResponseStatus.OK }
}
+ const configLostToDisconnect = source === AgentRunSource.FLOW_STEP && !configLoaded && err instanceof RpcTimeoutError && err.disconnected
+ if (configLostToDisconnect) {
+ log.warn({ error: err, conversation: { id: conversationId }, run: { id: runId } }, '[executeAgentRun] Config response lost in transit, leaving the step to the redelivered copy or the stale sweep')
+ return { kind: JobResultKind.FIRE_AND_FORGET, status: EngineResponseStatus.OK }
+ }
const errorClass = classifyAgentRunError({ error: err, provider: runProvider })
log[errorClass === 'internal' ? 'error' : 'warn']({ error: err, conversation: { id: conversationId }, provider: runProvider, model: { id: runModelId }, agentRun: { errorClass, source } }, '[executeAgentRun] Agent job failed')
const errorMessage = apErrorOf(err)?.message ?? formatPieceError(err).message
diff --git a/packages/server/worker/test/lib/execute/jobs/ee/agent/execute-agent-run-config-failure.test.ts b/packages/server/worker/test/lib/execute/jobs/ee/agent/execute-agent-run-config-failure.test.ts
index 4f1196a03c2..283e66197be 100644
--- a/packages/server/worker/test/lib/execute/jobs/ee/agent/execute-agent-run-config-failure.test.ts
+++ b/packages/server/worker/test/lib/execute/jobs/ee/agent/execute-agent-run-config-failure.test.ts
@@ -1,5 +1,5 @@
import { ActivepiecesError, ErrorCode } from '@activepieces/core-utils'
-import { AgentEvent, AgentEventType, AgentRunSource, AI_PROVIDER_ENTITY_TYPES, EngineResponseStatus, ExecuteAgentRunJobData, LATEST_JOB_DATA_SCHEMA_VERSION, WorkerJobType } from '@activepieces/shared'
+import { AgentEvent, AgentEventType, AgentRunSource, AI_PROVIDER_ENTITY_TYPES, EngineResponseStatus, ExecuteAgentRunJobData, LATEST_JOB_DATA_SCHEMA_VERSION, RpcTimeoutError, WorkerJobType } from '@activepieces/shared'
import { APICallError, RetryError } from 'ai'
import { describe, expect, it } from 'vitest'
import { executeAgentRunJob } from '../../../../../../src/lib/execute/jobs/ee/agent/execute-agent-run'
@@ -92,6 +92,33 @@ describe('executeAgentRunJob — a config failure must not swallow the turn', ()
expect(events).toHaveLength(0)
})
+ it('leaves a flow step to its redelivered copy when the config response is lost to a socket drop, so the step is not failed as a provider outage', async () => {
+ const { ctx, resumed, events } = buildContext(new RpcTimeoutError({ method: 'getAgentConfig', timeoutMs: 60_000, cause: new Error('socket has been disconnected') }))
+
+ const result = await executeAgentRunJob.execute(ctx, buildJobData({ source: AgentRunSource.FLOW_STEP, flowRunId: 'flow-run-1', waitpointId: 'waitpoint-1' }))
+
+ expect(result.status).toBe(EngineResponseStatus.OK)
+ expect(resumed).toHaveLength(0)
+ expect(events).toHaveLength(0)
+ })
+
+ it('fails a flow step whose config request timed out on a live socket, since nothing redelivers it', async () => {
+ const { ctx, resumed } = buildContext(new RpcTimeoutError({ method: 'getAgentConfig', timeoutMs: 60_000, cause: new Error('operation has timed out') }))
+
+ await executeAgentRunJob.execute(ctx, buildJobData({ source: AgentRunSource.FLOW_STEP, flowRunId: 'flow-run-1', waitpointId: 'waitpoint-1' })).catch(() => undefined)
+
+ expect(resumed).toHaveLength(1)
+ expect(resumed[0].waitpointId).toBe('waitpoint-1')
+ })
+
+ it('still tells the chat client the turn failed when its config response is lost, so it is not left streaming', async () => {
+ const { ctx, events } = buildContext(new RpcTimeoutError({ method: 'getAgentConfig', timeoutMs: 60_000, cause: new Error('socket has been disconnected') }))
+
+ await executeAgentRunJob.execute(ctx, buildJobData({ source: AgentRunSource.CHAT })).catch(() => undefined)
+
+ expect(events.map((event) => event.type)).toEqual([AgentEventType.ERROR, AgentEventType.FINISHED])
+ })
+
it('fails a flow step on a model our credits do not serve with the reason alone, as a user failure', async () => {
const reason = 'The model "openai/gpt-4o" is not available on Activepieces AI credits. Available models: anthropic/claude-sonnet-4.6'
const { ctx, resumed } = buildContext(Object.assign(new Error(`RPC [getAgentConfig] handler threw: ENTITY_NOT_FOUND: ${reason}`), {