AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
fix(agents): an AI agent step no longer fails when its worker briefly disconnects (#16896)
ziyadev6c1a97299d89
diff --git a/brain/knowledge/execution-runtime/workers.md b/brain/knowledge/execution-runtime/workers.mdindex 3d24793f0ec..f2d83ea969c 100644--- a/brain/knowledge/execution-runtime/workers.md+++ b/brain/knowledge/execution-runtime/workers.md@@ -21,6 +21,7 @@ The deep `Resolver`/`Runtime` concurrency and bundle-caching model lives on the- `concurrency === 1` primes the sandbox to the full container RAM (cgroup-aware) via `primeFullContainerMemory()`.### Gotchas+- **An RPC sent during a reconnect is lost.** socket.io-client flushes buffered emits in `onconnect`, before the worker's `connect` handler runs, and the app only attaches its RPC server after `FETCH_WORKER_SETTINGS` and `onConnection` resolve. So anything sent in that window reaches a socket with no listener. An acked call lands only on retry after the 60s timeout (a flow step's `resumeFlowStep` in the AI-156 repro), and an `rpc-notify` is lost for good. Tracked in ENG-1387.- **A dispatcher waiter belongs to a socket, and a closed socket's waiter is skipped.** A poll parks a waiter for up to `WAITER_TIMEOUT_MS` (50s), which can outlive its socket. Before AI-152 a job that `releaseConnectionJobs` had just re-queued was often handed straight back to the dead worker's waiter. It then sat in `active`, recorded against a connection that had already been drained, until the stalled scan reclaimed it. `runLoop` now hands the job to the first waiter whose `socket.connected` is still true, and returns it to the queue when there isn't one.- **Piece bundles download over the worker's own `AP_FRONTEND_URL` (`internalApiUrl`), never the app's `PUBLIC_URL` — keep it that way.** Through 0.92.x the sandbox built `GET /v1/engine/pieces/bundle` from the `PUBLIC_URL` the app pushes over the socket, so a standalone worker ignored its own `AP_FRONTEND_URL=http://app` for this one hop. A `localhost` public URL (the docs' default for compose and `docker run -p 8080:80`) failed every uncached install with `fetch failed` (GIT-1838). A public URL that reached a *different* instance sharing the JWT secret returned 404, so custom-piece upload failed with `PIECE_BUNDLE_NOT_AVAILABLE` while the `PACKAGE_ARCHIVE` row sat intact in Postgres. Tell-tale: the bundle GET never shows up in the app's own request log. Fixed by GIT-1718.- **`AP_CONTAINER_TYPE=WORKER` moves every engine→app call onto `AP_FRONTEND_URL`.** In `WORKER_AND_APP` the engine reaches the app at `http://127.0.0.1:<port>/api/`; in `WORKER` it uses `AP_FRONTEND_URL + '/api/'` (`getApiUrl` in `worker/src/lib/config/configs.ts`), and the worker's Socket.IO connection follows it. If that URL is the public hostname, file uploads, run-log uploads, piece bundle downloads and the worker socket all cross the ingress/WAF/CDN, so WAF rules and websocket idle timeouts start failing runs that worked the day before. Point separated workers at the in-cluster app service instead. Found on Alan's fleet 2026-09-29, the day after being advised to switch to `WORKER`.diff --git a/brain/knowledge/flows-execution/chat.md b/brain/knowledge/flows-execution/chat.mdindex 245eef44e16..8336a5194ea 100644--- a/brain/knowledge/flows-execution/chat.md+++ b/brain/knowledge/flows-execution/chat.md@@ -42,6 +42,7 @@ A turn is kept alive / reclaimed by three separate mechanisms in `execute-agent-- **A turn limit only works as a stop condition.** One `streamText` call runs every step of a turn, so a check after it returns can't stop a runaway. The token ceiling used to be checked only there, and a turn on 24 Oct 2026 ran 181 steps and 50.8M input tokens (AI-86). Put every per-turn bound in `loopStopCondition`. Test mocks must report usage in the V3 shape (`inputTokens: { total, ... }`); flat numbers read as 0 tokens.- **The streaming lock is per run, not per conversation.** A re-send can claim `activeRunId` before the first job reaches `getAgentConfig`. `agentHelpers.acquireStreamingLock` only lets the owning run set STREAMING; an older run gets `AGENT_RUN_SUPERSEDED` and the worker exits quietly, with no ERROR save and no events. A lock that ignored `runId` let the old run stream unowned while the new one was rejected as busy and marked ERROR, which is how users saw nothing after tapping a starter card twice.- **A redelivered run is a duplicate, not busy.** A worker socket blip makes the app requeue a chat job whose worker is still running it (`releaseConnectionJobs`). The second delivery finds its own `activeRunId` STREAMING, so `acquireStreamingLock` returns `duplicate` and it exits quietly like a superseded run. When this came back as `busy` it saved ERROR, and users saw "An agent is already running for this conversation" in the middle of a working turn (prod, 2026-10-13). If the first worker really died, the 90s `CHAT_STALE_SWEEP` frees the conversation. This only works when the conversation carries the run's `activeRunId`: chat claims it in the controller, and a flow-step run claims it when `getAgentConfig` creates the conversation. Without that claim the redelivery read as `busy` and failed the job (prod, 2026-10-21). A copy that arrives after its run already finished finds the row IDLE, so the lock alone would let the agent run again and its tools act twice (AI-153). `acquireStreamingLock` therefore also sets a Redis marker per conversation and run (`agent-run-started:<conversationId>:<runId>`, 24h) with `putIfAbsent`, and any later delivery of that run is a `duplicate`. That's safe because every enqueue mints a fresh `runId` and agent jobs run with `attempts: 1`. If the first worker really died, the redelivery is still dropped, so `CHAT_STALE_SWEEP` fails the step itself: a flow-step conversation keeps its `waitpointId`, and the sweep resumes that waitpoint with a `failure` result, so the flow doesn't sit paused until the 3h step timeout. It never re-runs the agent, because its tools may already have acted.+- **A flow-step run that loses its `getAgentConfig` response exits quietly, and the redelivered copy finishes it.** A socket drop mid-RPC surfaces on the worker as `RpcTimeoutError.disconnected`. The API handler doesn't stop when the socket drops: it still takes the streaming lock, so the copy used to get `duplicate` and the step failed, either right away as "provider temporarily unavailable" or 90s later through `CHAT_STALE_SWEEP`. Now the worker exits quietly on a disconnect, and `getAgentConfig` in `worker-rpc-service.ts` calls `agentHelpers.handLockToRedelivery` when the worker socket is gone by the time it replies. That writes a one-time `agent-run-handed-off` token (60s), and the next delivery of that run consumes it in `acquireStreamingLock` instead of being dropped (`'handed-off'`), so the copy runs the step (AI-156). The first delivery already saved the user message, so a handed-off copy re-reads the history after the lock and does not append it again. The hand-off never touches the row: setting it IDLE would hide the run from `CHAT_STALE_SWEEP` when the copy was already turned away, and the flow would sit paused for 3h. The token's TTL is shorter than the sweep window, so a late copy can't revive a step the sweep already failed. A plain timeout on a live socket is never redelivered, so it still fails the step. Chat still reports the error, because its buffered emits would mark ERROR over a copy that is streaming.- **A failed turn used to leave status ERROR with an empty reply.** The error text only travelled as a websocket `ERROR` event, so a reload, another tab, or a user who was away saw a dead conversation (and often no user message either when the job died inside `getAgentConfig`, before it persists the turn). The `execute-agent-run.ts` catch now sends `failure: { message, userMessage? }` on the empty `saveAgentMessages` call and `conversation-rpc.ts` appends it to `uiMessages` as an assistant text reply; `use-chat.ts` shows the Retry banner when a loaded conversation is ERROR. Stale STREAMING to IDLE recovery, a worker that never picks the job up, and `assertAgentMessageRateLimitNotExceeded` still leave no reply of their own.- **A built flow ends with a "Turn it on?" card, never an auto-publish**: an audit found 40% of built-but-unpublished flows ended on "open it to review", some claiming LIVE. Chat never publishes on its own; only a yes to the card calls `ap_lock_and_publish` (`build_flow.md` "Turn it on?"). Pinned by the `build-then-ask-turn-it-on` fixture (`askedToTurnItOn`, `noLiveClaimWithoutPublish`). The live-claim check is a regex plus "was `ap_lock_and_publish` called", so it cannot see whether publish succeeded.- **Server-managed connections**: the LLM never sees connection externalIds; `ap_execute_action` auto-fills them from a Redis store.diff --git a/packages/core/execution/src/lib/engine/rpc.ts b/packages/core/execution/src/lib/engine/rpc.tsindex d3eed21863d..c884a24164c 100644--- a/packages/core/execution/src/lib/engine/rpc.ts+++ b/packages/core/execution/src/lib/engine/rpc.ts@@ -3,6 +3,7 @@ import { ActivepiecesError, isObject, spreadIfNotUndefined, toError } from '@actconst RPC_EVENT = 'rpc'const NOTIFY_EVENT = 'rpc-notify'const AP_ERROR_PROP = 'apError'+const SOCKET_DISCONNECTED_MESSAGE = 'socket has been disconnected'// eslint-disable-next-line @typescript-eslint/no-explicit-anytype Contract = Record<string, (input: any) => any>@@ -114,10 +115,12 @@ function isRpcErrorEnvelope(value: unknown): value is { __rpcError: string, __rpexport class RpcTimeoutError extends Error {readonly method: stringreadonly timeoutMs: number+ readonly disconnected: booleanconstructor({ method, timeoutMs, cause }: { method: string, timeoutMs: number, cause: unknown }) {super(`RPC [${method}] failed (timeout: ${timeoutMs}ms): ${toError(cause).message}`)this.method = methodthis.timeoutMs = timeoutMs+ this.disconnected = toError(cause).message === SOCKET_DISCONNECTED_MESSAGE}}diff --git a/packages/server/api/src/app/database/redis/distributed-store-factory.ts b/packages/server/api/src/app/database/redis/distributed-store-factory.tsindex bbdbdfadbb4..767430931c0 100644--- a/packages/server/api/src/app/database/redis/distributed-store-factory.ts+++ b/packages/server/api/src/app/database/redis/distributed-store-factory.ts@@ -55,11 +55,11 @@ export const distributedStoreFactory = (getRedisClient: () => Promise<Redis>) =>return true},- async delete(keys: string | string[]): Promise<void> {+ async delete(keys: string | string[]): Promise<number> {const keysArray = Array.isArray(keys) ? keys : [keys]- if (keysArray.length === 0) return+ if (keysArray.length === 0) return 0const redisClient = await getRedisClient()- await redisClient.del(...keysArray)+ return redisClient.del(...keysArray)},async putBoolean(key: string, value: boolean): Promise<void> {diff --git a/packages/server/api/src/app/ee/agent/agent-helpers.ts b/packages/server/api/src/app/ee/agent/agent-helpers.tsindex 6b5fa9dffff..84dcae028a6 100644--- a/packages/server/api/src/app/ee/agent/agent-helpers.ts+++ b/packages/server/api/src/app/ee/agent/agent-helpers.ts@@ -28,6 +28,7 @@ import { UserMemoryEntity } from './user-memory-entity'const STREAMING_STALENESS_TIMEOUT_MS = 90 * 1_000const STARTED_RUN_MARKER_TTL_SECONDS = 24 * 60 * 60+const HANDED_OFF_RUN_TTL_SECONDS = 60// Interactive-eval conversations carry this id prefix (within the 21-char id column) so both the// eval endpoints and the regular chat path can tell them apart from real user conversations.@@ -433,7 +434,8 @@ async function acquireStreamingLock({ conversationId, runId }: { conversationId:if (!isNil(runId)) {const firstDelivery = await distributedStore.putIfAbsent(`agent-run-started:${conversationId}:${runId}`, true, STARTED_RUN_MARKER_TTL_SECONDS)if (!firstDelivery) {- return 'duplicate'+ const handedToThisDelivery = await distributedStore.delete(handedOffRunKey({ conversationId, runId })) > 0+ return handedToThisDelivery ? 'handed-off' : 'duplicate'}}const builder = conversationRepo()@@ -457,6 +459,14 @@ async function acquireStreamingLock({ conversationId, runId }: { conversationId:return current.activeRunId === runId ? 'duplicate' : 'superseded'}+async function handLockToRedelivery({ conversationId, runId }: { conversationId: string, runId: string }): Promise<void> {+ await distributedStore.put(handedOffRunKey({ conversationId, runId }), true, HANDED_OFF_RUN_TTL_SECONDS)+}++function handedOffRunKey({ conversationId, runId }: { conversationId: string, runId: string }): string {+ return `agent-run-handed-off:${conversationId}:${runId}`+}+async function resolveFlowTools({ projectId, tools, log }: { projectId: string, tools: AgentTool[], log: FastifyBaseLogger }): Promise<ResolvedAgentFlowTool[]> {const flowToolRequests = tools.filter((tool): tool is AgentFlowTool => tool.type === AgentToolType.FLOW)if (flowToolRequests.length === 0) {@@ -519,6 +529,7 @@ export const agentHelpers = {jobFieldsFromConfig,claimConversationForRun,acquireStreamingLock,+ handLockToRedelivery,resolveFlowTools,agentsSurfaceAvailable,assertAgentsSurfaceAvailable,@@ -557,7 +568,7 @@ type SweptConversation = z.infer<typeof SweptConversation>type AgentJobConfigFields = Pick<ExecuteAgentRunJobData, 'tools' | 'structuredOutput' | 'maxSteps' | 'modelName' | 'provider' | 'providerConfigId' | 'modelTierId' | 'promptOverride'>-export type StreamingLockResult = 'acquired' | 'busy' | 'superseded' | 'duplicate'+export type StreamingLockResult = 'acquired' | 'handed-off' | 'busy' | 'superseded' | 'duplicate'export type PlatformWorkModel = {config: GetProviderConfigResponsediff --git a/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts b/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.tsindex a2227932645..1c2d2ee9594 100644--- a/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts+++ b/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts@@ -238,10 +238,12 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({? { ...agentPrompt.guides, ...promptOverride.guides }: agentPrompt.guides- const previousMessages = conversation.messages as ModelMessage[]+ const resumedHistory = lock === 'handed-off' ? await agentHelpers.conversationRepo().findOneByOrFail({ id: conversationId, platformId }) : null+ const history = resumedHistory ?? conversation+ const previousMessages = history.messages as ModelMessage[]const onboardingNote = agentSurfaceNotes.onboardingNote({ source: conversation.source, ...spreadIfDefined('messageSource', input.messageSource) })const newUserMessage: ModelMessage = { role: 'user' as const, content: isNil(onboardingNote) ? userContent : withLeadingNote({ content: userContent, note: onboardingNote }) }- const allMessages = [...previousMessages, newUserMessage]+ const allMessages = isNil(resumedHistory) ? [...previousMessages, newUserMessage] : previousMessagesconst fastModelSeesMessages = conversation.source !== AgentRunSource.CHATconst modelsSeeingMessages = isNil(tierRun)? [resolvedModelId, ...(fastModelSeesMessages ? [fastModelId] : [])].map((modelId) => ({ provider: providerConfig.provider, modelId }))@@ -252,11 +254,10 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({models: modelsSeeingMessages.map((model) => catalog.lookup(model)),})- const previousUiMessages = (conversation.uiMessages ?? []) as PersistedAgentMessage[]- const uiMessagesWithUser: PersistedAgentMessage[] = [- ...previousUiMessages,- { role: PersistedAgentRole.USER, parts: [{ type: PersistedAgentPartType.TEXT, text: userMessage }] },- ]+ const previousUiMessages = (history.uiMessages ?? []) as PersistedAgentMessage[]+ const uiMessagesWithUser: PersistedAgentMessage[] = isNil(resumedHistory)+ ? [...previousUiMessages, { role: PersistedAgentRole.USER, parts: [{ type: PersistedAgentPartType.TEXT, text: userMessage }] }]+ : previousUiMessagesawait agentHelpers.conversationRepo().update(conversationId, {messages: allMessages,uiMessages: JSON.parse(JSON.stringify(uiMessagesWithUser)),diff --git a/packages/server/api/src/app/workers/rpc/worker-rpc-service.ts b/packages/server/api/src/app/workers/rpc/worker-rpc-service.tsindex b4b1b683ed0..574ef6baae4 100644--- a/packages/server/api/src/app/workers/rpc/worker-rpc-service.ts+++ b/packages/server/api/src/app/workers/rpc/worker-rpc-service.ts@@ -1,11 +1,12 @@import { assertNotNullOrUndefined, isNil, spreadIfDefined } from '@activepieces/core-utils'import { apVersionUtil, onCallService, UNKNOWN_VERSION } from '@activepieces/server-utils'-import { ExecutionType, FileCompression, FileLocation, FileType, FlowOperationType, FlowStatus, WebsocketClientEvent, WorkerToApiContract } from '@activepieces/shared'+import { AgentRunSource, ExecutionType, FileCompression, FileLocation, FileType, FlowOperationType, FlowStatus, WebsocketClientEvent, WorkerToApiContract } from '@activepieces/shared'import { FastifyBaseLogger } from 'fastify'import { Socket } from 'socket.io'import { aiRpcHandlers } from '../../ai/ai-rpc-handlers'import { websocketService } from '../../core/websockets.service'import { distributedStore, redisConnections } from '../../database/redis-connections'+import { agentHelpers } from '../../ee/agent/agent-helpers'import { agentRpcHandlers } from '../../ee/agent/agent-rpc-handlers'import { chatPersonalizationService } from '../../ee/agent/personalization/chat-personalization-service'import { eventDestinationRpcHandlers } from '../../event-destinations/event-destinations-rpc-handlers'@@ -344,7 +345,14 @@ export function createHandlers(log: FastifyBaseLogger, assignment: WorkerGroupAs},async getAgentConfig(input) {- return agentRpcHandlers(agentRpcLog(log, input)).getAgentConfig(input)+ const config = await agentRpcHandlers(agentRpcLog(log, input)).getAgentConfig(input)+ const { conversationId, runId, source } = input+ const flowStepReplyUndeliverable = socket?.connected === false && source === AgentRunSource.FLOW_STEP && !isNil(runId)+ if (flowStepReplyUndeliverable) {+ log.warn({ conversation: { id: conversationId }, run: { id: runId } }, '[workerRpc#getAgentConfig] Worker disconnected before the config reply, handing the lock to the redelivered copy')+ await agentHelpers.handLockToRedelivery({ conversationId, runId })+ }+ return config},async saveAgentMessages(input) {diff --git a/packages/server/api/test/integration/ce/agent/agent-run-worker-handoff.test.ts b/packages/server/api/test/integration/ce/agent/agent-run-worker-handoff.test.tsindex bdb7e4d4aa3..c6872a196cb 100644--- a/packages/server/api/test/integration/ce/agent/agent-run-worker-handoff.test.ts+++ b/packages/server/api/test/integration/ce/agent/agent-run-worker-handoff.test.ts@@ -4,6 +4,7 @@ import { FastifyInstance } from 'fastify'import { StatusCodes } from 'http-status-codes'import { afterAll, beforeAll, describe, expect, it } from 'vitest'import { accessTokenManager } from '../../../../src/app/authentication/lib/access-token-manager'+import { agentHelpers } from '../../../../src/app/ee/agent/agent-helpers'import { agentRpcHandlers } from '../../../../src/app/ee/agent/agent-rpc-handlers'import { db } from '../../../helpers/db'import { mockAndSaveAIProvider } from '../../../helpers/mocks'@@ -111,4 +112,35 @@ describe('community edition: the worker picking up an agent step', () => {error: { code: ErrorCode.AGENT_RUN_SUPERSEDED },})})++ it('gives the copy that takes over a lost config the instruction once, not twice', async () => {+ const ctx = await createTestContext(app)+ await mockAndSaveAIProvider({ platformId: ctx.platform.id, provider: AIProviderName.OPENAI })+ const conversationId = await startFlowStepRun(ctx, 'ce-handoff-resume')+ const delivery = {+ conversationId,+ runId: apId(),+ platformId: ctx.platform.id,+ userId: ctx.user.id,+ projectId: ctx.project.id,+ userMessage: 'summarise the thread',+ modelName: 'gpt-4o',+ provider: AIProviderName.OPENAI,+ source: AgentRunSource.FLOW_STEP,+ waitpointId: apId(),+ }+ await agentRpcHandlers(app.log).getAgentConfig(delivery)+ await agentHelpers.handLockToRedelivery({ conversationId, runId: delivery.runId })++ const config = await agentRpcHandlers(app.log).getAgentConfig(delivery)++ const stored = await db.findOneByOrFail<{ messages: { role: string }[], uiMessages: { role: string }[] }>('agent_conversation', { id: conversationId })+ expect(userTurns({ messages: config.previousUiMessages })).toBe(1)+ expect(userTurns({ messages: stored.messages })).toBe(1)+ expect(userTurns({ messages: stored.uiMessages })).toBe(1)+ })})++function userTurns({ messages }: { messages: { role: string }[] }): number {+ return messages.filter((message) => message.role === 'user').length+}diff --git a/packages/server/api/test/integration/cloud/agent/agent-resend-race.test.ts b/packages/server/api/test/integration/cloud/agent/agent-resend-race.test.tsindex b17c0b7f43b..480e7c5be8b 100644--- a/packages/server/api/test/integration/cloud/agent/agent-resend-race.test.ts+++ b/packages/server/api/test/integration/cloud/agent/agent-resend-race.test.ts@@ -48,6 +48,34 @@ describe('re-sending a message while the first run has not started yet', () => {expect(stored.activeRunId).toBe('run-1')})+ it('lets the redelivered copy take a lock handed to it once, when the first delivery never received its config', async () => {+ const ctx = await createTestContext(app, { plan: { chatEnabled: true } })+ const created = await ctx.post('/v1/agents/conversations', { title: 'Undelivered config' })+ const conversationId = created.json().id+ await agentHelpers.claimConversationForRun({ conversationId, runId: 'run-1' })+ expect(await agentHelpers.acquireStreamingLock({ conversationId, runId: 'run-1' })).toBe('acquired')++ await agentHelpers.handLockToRedelivery({ conversationId, runId: 'run-1' })++ expect(await agentHelpers.acquireStreamingLock({ conversationId, runId: 'run-1' })).toBe('handed-off')+ expect(await agentHelpers.acquireStreamingLock({ conversationId, runId: 'run-1' })).toBe('duplicate')+ })++ it('leaves the run streaming for the stale sweep when the copy is turned away before the hand-off', async () => {+ const ctx = await createTestContext(app, { plan: { chatEnabled: true } })+ const created = await ctx.post('/v1/agents/conversations', { title: 'Early copy' })+ const conversationId = created.json().id+ await agentHelpers.claimConversationForRun({ conversationId, runId: 'run-1' })+ expect(await agentHelpers.acquireStreamingLock({ conversationId, runId: 'run-1' })).toBe('acquired')+ expect(await agentHelpers.acquireStreamingLock({ conversationId, runId: 'run-1' })).toBe('duplicate')++ await agentHelpers.handLockToRedelivery({ conversationId, runId: 'run-1' })++ const stored = await db.findOneByOrFail<{ status: string, activeRunId: string }>('agent_conversation', { id: conversationId })+ expect(stored.status).toBe(AgentConversationStatus.STREAMING)+ expect(stored.activeRunId).toBe('run-1')+ })+it('drops a copy delivered after its run finished, so the agent and its tools do not run twice', async () => {const ctx = await createTestContext(app, { plan: { chatEnabled: true } })const created = await ctx.post('/v1/agents/conversations', { title: 'Late copy' })diff --git a/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts b/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.tsindex e4f793a302a..b860b37f6ff 100644--- a/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts+++ b/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts@@ -1,6 +1,6 @@import { ActivepiecesAiBilling, ActivepiecesAiConsumerSource, AiCallContext, AIProviderName, AiUsageEnvironment, ErrorCode, formatPieceError, isNil, isObject, spreadIfDefined, tryCatch, tryCatchSync } from '@activepieces/core-utils'import { aiUtils, wideEvent } from '@activepieces/server-utils'-import { AGENT_RUN_USAGE_ORIGIN, AgentConfigResponse, AgentEvent, AgentEventType, AgentKnowledgeBaseTool, AgentMcpTool, AgentModelCandidate, AgentOutputField, AgentPhase, AgentPieceTool, AgentResult, AgentRunSource, AgentTool, agentToolSkills, AgentToolType, AiProviderCredentials, apErrorOf, chatBilling, EngineResponseStatus, ExecuteAgentRunJobData, MAX_AGENT_TURN_WALL_CLOCK_MS, PersistedAgentMessage, PersistedAgentMessageSchema, PersistedAgentPart, PersistedAgentPartType, PersistedAgentRole, ResolvedAgentFlowTool, SaveAgentMessagesRequest, SkillSurface, WorkerJobType } from '@activepieces/shared'+import { AGENT_RUN_USAGE_ORIGIN, AgentConfigResponse, AgentEvent, AgentEventType, AgentKnowledgeBaseTool, AgentMcpTool, AgentModelCandidate, AgentOutputField, AgentPhase, AgentPieceTool, AgentResult, AgentRunSource, AgentTool, agentToolSkills, AgentToolType, AiProviderCredentials, apErrorOf, chatBilling, EngineResponseStatus, ExecuteAgentRunJobData, MAX_AGENT_TURN_WALL_CLOCK_MS, PersistedAgentMessage, PersistedAgentMessageSchema, PersistedAgentPart, PersistedAgentPartType, PersistedAgentRole, ResolvedAgentFlowTool, RpcTimeoutError, SaveAgentMessagesRequest, SkillSurface, WorkerJobType } from '@activepieces/shared'import { createUIMessageStream, LanguageModel, ModelMessage, streamText, ToolSet, toUIMessageStream } from 'ai'import { FireAndForgetJobResult, JobContext, JobHandler, JobResultKind } from '../../../types'import { toResolvedAiFile } from '../../ai/ai-files'@@ -461,6 +461,11 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForgelog.info({ conversation: { id: conversationId }, run: { id: runId } }, '[executeAgentRun] Run superseded by a newer message, exiting quietly')return { kind: JobResultKind.FIRE_AND_FORGET, status: EngineResponseStatus.OK }}+ const configLostToDisconnect = source === AgentRunSource.FLOW_STEP && !configLoaded && err instanceof RpcTimeoutError && err.disconnected+ if (configLostToDisconnect) {+ log.warn({ error: err, conversation: { id: conversationId }, run: { id: runId } }, '[executeAgentRun] Config response lost in transit, leaving the step to the redelivered copy or the stale sweep')+ return { kind: JobResultKind.FIRE_AND_FORGET, status: EngineResponseStatus.OK }+ }const errorClass = classifyAgentRunError({ error: err, provider: runProvider })log[errorClass === 'internal' ? 'error' : 'warn']({ error: err, conversation: { id: conversationId }, provider: runProvider, model: { id: runModelId }, agentRun: { errorClass, source } }, '[executeAgentRun] Agent job failed')const errorMessage = apErrorOf(err)?.message ?? formatPieceError(err).messagediff --git a/packages/server/worker/test/lib/execute/jobs/ee/agent/execute-agent-run-config-failure.test.ts b/packages/server/worker/test/lib/execute/jobs/ee/agent/execute-agent-run-config-failure.test.tsindex 4f1196a03c2..283e66197be 100644--- a/packages/server/worker/test/lib/execute/jobs/ee/agent/execute-agent-run-config-failure.test.ts+++ b/packages/server/worker/test/lib/execute/jobs/ee/agent/execute-agent-run-config-failure.test.ts@@ -1,5 +1,5 @@import { ActivepiecesError, ErrorCode } from '@activepieces/core-utils'-import { AgentEvent, AgentEventType, AgentRunSource, AI_PROVIDER_ENTITY_TYPES, EngineResponseStatus, ExecuteAgentRunJobData, LATEST_JOB_DATA_SCHEMA_VERSION, WorkerJobType } from '@activepieces/shared'+import { AgentEvent, AgentEventType, AgentRunSource, AI_PROVIDER_ENTITY_TYPES, EngineResponseStatus, ExecuteAgentRunJobData, LATEST_JOB_DATA_SCHEMA_VERSION, RpcTimeoutError, WorkerJobType } from '@activepieces/shared'import { APICallError, RetryError } from 'ai'import { describe, expect, it } from 'vitest'import { executeAgentRunJob } from '../../../../../../src/lib/execute/jobs/ee/agent/execute-agent-run'@@ -92,6 +92,33 @@ describe('executeAgentRunJob — a config failure must not swallow the turn', ()expect(events).toHaveLength(0)})+ it('leaves a flow step to its redelivered copy when the config response is lost to a socket drop, so the step is not failed as a provider outage', async () => {+ const { ctx, resumed, events } = buildContext(new RpcTimeoutError({ method: 'getAgentConfig', timeoutMs: 60_000, cause: new Error('socket has been disconnected') }))++ const result = await executeAgentRunJob.execute(ctx, buildJobData({ source: AgentRunSource.FLOW_STEP, flowRunId: 'flow-run-1', waitpointId: 'waitpoint-1' }))++ expect(result.status).toBe(EngineResponseStatus.OK)+ expect(resumed).toHaveLength(0)+ expect(events).toHaveLength(0)+ })++ it('fails a flow step whose config request timed out on a live socket, since nothing redelivers it', async () => {+ const { ctx, resumed } = buildContext(new RpcTimeoutError({ method: 'getAgentConfig', timeoutMs: 60_000, cause: new Error('operation has timed out') }))++ await executeAgentRunJob.execute(ctx, buildJobData({ source: AgentRunSource.FLOW_STEP, flowRunId: 'flow-run-1', waitpointId: 'waitpoint-1' })).catch(() => undefined)++ expect(resumed).toHaveLength(1)+ expect(resumed[0].waitpointId).toBe('waitpoint-1')+ })++ it('still tells the chat client the turn failed when its config response is lost, so it is not left streaming', async () => {+ const { ctx, events } = buildContext(new RpcTimeoutError({ method: 'getAgentConfig', timeoutMs: 60_000, cause: new Error('socket has been disconnected') }))++ await executeAgentRunJob.execute(ctx, buildJobData({ source: AgentRunSource.CHAT })).catch(() => undefined)++ expect(events.map((event) => event.type)).toEqual([AgentEventType.ERROR, AgentEventType.FINISHED])+ })+it('fails a flow step on a model our credits do not serve with the reason alone, as a user failure', async () => {const reason = 'The model "openai/gpt-4o" is not available on Activepieces AI credits. Available models: anthropic/claude-sonnet-4.6'const { ctx, resumed } = buildContext(Object.assign(new Error(`RPC [getAgentConfig] handler threw: ENTITY_NOT_FOUND: ${reason}`), {