AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
fix(agents): AI agent steps no longer fail after a worker reconnects (#16880)
Hazem Adeldbf46098ca6e
diff --git a/brain/knowledge/flows-execution/chat.md b/brain/knowledge/flows-execution/chat.mdindex 2aa23d97ce1..4ed667ee819 100644--- a/brain/knowledge/flows-execution/chat.md+++ b/brain/knowledge/flows-execution/chat.md@@ -40,7 +40,7 @@ A turn is kept alive / reclaimed by three separate mechanisms in `execute-agent-- **Chat has no fixed step cap.** A chat turn is bounded by credits (checked after every step), the runaway-token ceiling (`RUNAWAY_TURN_CONTEXT_MULTIPLE` × the context window), the identical-failure guard and the 2-hour `MAX_AGENT_TURN_WALL_CLOCK_MS`, so it finishes long jobs. A saved agent's author-set `maxSteps` is still honored on its last step: an agent with structured output is forced to call `TASK_COMPLETION_TOOL_NAME` so the flow gets a result, and any other agent runs with `toolChoice: 'none'` plus a wrap-up note, so it replies with what's done and what's left. Never a silent stop or an error.- **A turn limit only works as a stop condition.** One `streamText` call runs every step of a turn, so a check after it returns can't stop a runaway. The token ceiling used to be checked only there, and a turn on 24 Oct 2026 ran 181 steps and 50.8M input tokens (AI-86). Put every per-turn bound in `loopStopCondition`. Test mocks must report usage in the V3 shape (`inputTokens: { total, ... }`); flat numbers read as 0 tokens.- **The streaming lock is per run, not per conversation.** A re-send can claim `activeRunId` before the first job reaches `getAgentConfig`. `agentHelpers.acquireStreamingLock` only lets the owning run set STREAMING; an older run gets `AGENT_RUN_SUPERSEDED` and the worker exits quietly, with no ERROR save and no events. A lock that ignored `runId` let the old run stream unowned while the new one was rejected as busy and marked ERROR, which is how users saw nothing after tapping a starter card twice.-- **A redelivered run is a duplicate, not busy.** A worker socket blip makes the app requeue a chat job whose worker is still running it (`releaseConnectionJobs`). The second delivery finds its own `activeRunId` STREAMING, so `acquireStreamingLock` returns `duplicate` and it exits quietly like a superseded run. When this came back as `busy` it saved ERROR, and users saw "An agent is already running for this conversation" in the middle of a working turn (prod, 2026-10-13). If the first worker really died, the 90s `CHAT_STALE_SWEEP` frees the conversation.+- **A redelivered run is a duplicate, not busy.** A worker socket blip makes the app requeue a chat job whose worker is still running it (`releaseConnectionJobs`). The second delivery finds its own `activeRunId` STREAMING, so `acquireStreamingLock` returns `duplicate` and it exits quietly like a superseded run. When this came back as `busy` it saved ERROR, and users saw "An agent is already running for this conversation" in the middle of a working turn (prod, 2026-10-13). If the first worker really died, the 90s `CHAT_STALE_SWEEP` frees the conversation. This only works when the conversation carries the run's `activeRunId`: chat claims it in the controller, and a flow-step run claims it when `getAgentConfig` creates the conversation. Without that claim the redelivery read as `busy` and failed the job (prod, 2026-10-21). If the first worker really died, the redelivery is still dropped, so `CHAT_STALE_SWEEP` fails the step itself: a flow-step conversation keeps its `waitpointId`, and the sweep resumes that waitpoint with a `failure` result, so the flow doesn't sit paused until the 3h step timeout. It never re-runs the agent, because its tools may already have acted.- **A failed turn used to leave status ERROR with an empty reply.** The error text only travelled as a websocket `ERROR` event, so a reload, another tab, or a user who was away saw a dead conversation (and often no user message either when the job died inside `getAgentConfig`, before it persists the turn). The `execute-agent-run.ts` catch now sends `failure: { message, userMessage? }` on the empty `saveAgentMessages` call and `conversation-rpc.ts` appends it to `uiMessages` as an assistant text reply; `use-chat.ts` shows the Retry banner when a loaded conversation is ERROR. Stale STREAMING to IDLE recovery, a worker that never picks the job up, and `assertAgentMessageRateLimitNotExceeded` still leave no reply of their own.- **A built flow ends with a "Turn it on?" card, never an auto-publish**: an audit found 40% of built-but-unpublished flows ended on "open it to review", some claiming LIVE. Chat never publishes on its own; only a yes to the card calls `ap_lock_and_publish` (`build_flow.md` "Turn it on?"). Pinned by the `build-then-ask-turn-it-on` fixture (`askedToTurnItOn`, `noLiveClaimWithoutPublish`). The live-claim check is a regex plus "was `ap_lock_and_publish` called", so it cannot see whether publish succeeded.- **Server-managed connections**: the LLM never sees connection externalIds; `ap_execute_action` auto-fills them from a Redis store.diff --git a/bun.lock b/bun.lockindex d600caedfd7..05e37b4e221 100644--- a/bun.lock+++ b/bun.lock@@ -122,7 +122,7 @@},"packages/core/execution": {"name": "@activepieces/core-execution",- "version": "0.28.0",+ "version": "0.28.1","dependencies": {"@activepieces/core-piece-types": "workspace:*","@activepieces/core-utils": "workspace:*",@@ -167,7 +167,7 @@},"packages/core/shared": {"name": "@activepieces/shared",- "version": "0.208.0",+ "version": "0.208.1","dependencies": {"@activepieces/core-execution": "workspace:*","@activepieces/core-formula": "workspace:*",diff --git a/packages/core/execution/package.json b/packages/core/execution/package.jsonindex b80b463c0cf..1ecfb5bc063 100644--- a/packages/core/execution/package.json+++ b/packages/core/execution/package.json@@ -1,6 +1,6 @@{"name": "@activepieces/core-execution",- "version": "0.28.0",+ "version": "0.28.1","type": "commonjs","main": "./dist/src/index.js","scripts": {diff --git a/packages/core/execution/src/lib/workers/worker-contract.ts b/packages/core/execution/src/lib/workers/worker-contract.tsindex 702ec9630d6..ceba0d8d7ae 100644--- a/packages/core/execution/src/lib/workers/worker-contract.ts+++ b/packages/core/execution/src/lib/workers/worker-contract.ts@@ -137,6 +137,7 @@ export type GetAgentConfigRequest = {messageSource?: 'onboarding'agentId?: stringflowRunId?: string+ waitpointId?: stringprojectId?: string | nulluserMessage: stringmodelName: string | nulldiff --git a/packages/core/shared/package.json b/packages/core/shared/package.jsonindex a6ce6d9e9a3..a61e76d1e5d 100644--- a/packages/core/shared/package.json+++ b/packages/core/shared/package.json@@ -1,6 +1,6 @@{"name": "@activepieces/shared",- "version": "0.208.0",+ "version": "0.208.1","type": "commonjs","sideEffects": false,"main": "./dist/src/index.js",diff --git a/packages/core/shared/src/lib/ee/agent/index.ts b/packages/core/shared/src/lib/ee/agent/index.tsindex 76d7ab95c85..4d21b34dab3 100644--- a/packages/core/shared/src/lib/ee/agent/index.ts+++ b/packages/core/shared/src/lib/ee/agent/index.ts@@ -201,6 +201,7 @@ export const AgentConversation = z.object({userId: z.string(),agentId: Nullable(z.string()),flowRunId: Nullable(z.string()),+ waitpointId: Nullable(z.string()),aiCredits: Nullable(z.number().int()),source: z.enum(AgentRunSource),title: Nullable(z.string()),diff --git a/packages/server/api/src/app/database/migration/postgres/1876000000000-AddAgentConversationWaitpointId.ts b/packages/server/api/src/app/database/migration/postgres/1876000000000-AddAgentConversationWaitpointId.tsnew file mode 100644index 00000000000..284b17c88d0--- /dev/null+++ b/packages/server/api/src/app/database/migration/postgres/1876000000000-AddAgentConversationWaitpointId.ts@@ -0,0 +1,22 @@+import { QueryRunner } from 'typeorm'+import { Migration } from '../../migration'++export class AddAgentConversationWaitpointId1876000000000 implements Migration {+ name = 'AddAgentConversationWaitpointId1876000000000'+ breaking = false+ release = '0.93.0'++ public async up(queryRunner: QueryRunner): Promise<void> {+ await queryRunner.query(`+ ALTER TABLE "agent_conversation"+ ADD COLUMN IF NOT EXISTS "waitpointId" character varying(21)+ `)+ }++ public async down(queryRunner: QueryRunner): Promise<void> {+ await queryRunner.query(`+ ALTER TABLE "agent_conversation"+ DROP COLUMN IF EXISTS "waitpointId"+ `)+ }+}diff --git a/packages/server/api/src/app/database/postgres-connection.ts b/packages/server/api/src/app/database/postgres-connection.tsindex aa073a6b41d..59a6101b845 100644--- a/packages/server/api/src/app/database/postgres-connection.ts+++ b/packages/server/api/src/app/database/postgres-connection.ts@@ -462,6 +462,7 @@ import { DropChatRolloutUserTable1870000000000 } from './migration/postgres/1870import { AddBarrierChildAttribution1871000000000 } from './migration/postgres/1871000000000-AddBarrierChildAttribution'import { AddPieceTranslationTable1873000000000 } from './migration/postgres/1873000000000-AddPieceTranslationTable'import { AddDefaultChatModelToPlatformConfiguration1875000000000 } from './migration/postgres/1875000000000-AddDefaultChatModelToPlatformConfiguration'+import { AddAgentConversationWaitpointId1876000000000 } from './migration/postgres/1876000000000-AddAgentConversationWaitpointId'import { AddAgentConversationProviderConfigIdIndex1869000000000 } from './migration/postgres/background/1869000000000-AddAgentConversationProviderConfigIdIndex'import { ReshapeFlowRunListIndexesForBarrierChildren1871000000000 } from './migration/postgres/background/1871000000000-ReshapeFlowRunListIndexesForBarrierChildren'import { BackfillPieceTranslation1874000000000 } from './migration/postgres/background/1874000000000-BackfillPieceTranslation'@@ -949,6 +950,7 @@ export const getMigrations = (): (new () => Migration)[] => {AddBarrierChildAttribution1871000000000,AddPieceTranslationTable1873000000000,AddDefaultChatModelToPlatformConfiguration1875000000000,+ AddAgentConversationWaitpointId1876000000000,]return migrations}diff --git a/packages/server/api/src/app/ee/agent/agent-conversation-entity.ts b/packages/server/api/src/app/ee/agent/agent-conversation-entity.tsindex ee5f5eac237..976b863a031 100644--- a/packages/server/api/src/app/ee/agent/agent-conversation-entity.ts+++ b/packages/server/api/src/app/ee/agent/agent-conversation-entity.ts@@ -29,6 +29,10 @@ export const AgentConversationEntity = new EntitySchema<AgentConversationWithRel...ApIdSchema,nullable: true,},+ waitpointId: {+ ...ApIdSchema,+ nullable: true,+ },aiCredits: {type: Number,nullable: true,diff --git a/packages/server/api/src/app/ee/agent/agent-helpers.ts b/packages/server/api/src/app/ee/agent/agent-helpers.tsindex 4a639a4a7dd..437d5640b6c 100644--- a/packages/server/api/src/app/ee/agent/agent-helpers.ts+++ b/packages/server/api/src/app/ee/agent/agent-helpers.ts@@ -1,8 +1,9 @@import { setTimeout as delay } from 'node:timers/promises'import { ExecuteAgentRunJobData } from '@activepieces/core-execution'+import { AGENT_STEP_TIMEOUT_MS, AgentResult, AgentTaskStatus } from '@activepieces/core-piece-types'import { ActivepiecesError, AIProviderName, apId, assertNotNullOrUndefined, ErrorCode, isNil, spreadIfDefined, tryCatch, tryCatchSync, unique } from '@activepieces/core-utils'import { aiUtils, modelTierCatalog, ModelTierSurface } from '@activepieces/server-utils'-import { AgentConfig, AgentConversation, AgentConversationStatus, AgentFlowTool, AgentTool, AgentToolType, AI_PROVIDER_CAPABILITIES, AI_PROVIDER_ENTITY_TYPES, AIProviderModelType, FlowVersionState, GetAgentMemoryResponse, GetProviderConfigResponse, Project, ProjectType, ResolvedAgentFlowTool, UserMemory } from '@activepieces/shared'+import { AgentConfig, AgentConversation, AgentConversationStatus, AgentFlowTool, AgentRunSource, AgentTool, AgentToolType, AI_PROVIDER_CAPABILITIES, AI_PROVIDER_ENTITY_TYPES, AIProviderModelType, FlowVersionState, GetAgentMemoryResponse, GetProviderConfigResponse, Project, ProjectType, ResolvedAgentFlowTool, UserMemory } from '@activepieces/shared'import { LanguageModel } from 'ai'import { FastifyBaseLogger } from 'fastify'import { Repository } from 'typeorm'@@ -21,6 +22,7 @@ import { platformPlanService } from '../platform/platform-plan/platform-plan.serimport { AgentConversationEntity, AgentConversationWithRelations } from './agent-conversation-entity'import { agentModelResolution } from './agent-model-resolution'import { agentModelTier } from './agent-model-tier'+import { flowStepRpc } from './rpc/flow-step-rpc'import { UserMemoryEntity } from './user-memory-entity'const STREAMING_STALENESS_TIMEOUT_MS = 90 * 1_000@@ -288,12 +290,33 @@ async function recoverAllStaleStreamingConversations({ log }: { log: FastifyBase.where('status = :streaming', { streaming: AgentConversationStatus.STREAMING }).andWhere('updated < NOW() - make_interval(secs => :staleSecs)', { staleSecs: STREAMING_STALENESS_TIMEOUT_MS / 1_000 }).andWhere('id NOT LIKE :evalPrefix', { evalPrefix: `${EVAL_CONVERSATION_ID_PREFIX}%` })+ .returning(['id', 'source', 'flowRunId', 'waitpointId']).execute()- const recovered = result.affected ?? 0- if (recovered > 0) {- log.warn({ recovered }, '[agentHelpers] Swept stale STREAMING conversations to IDLE')+ const swept = SweptConversation.array().parse(result.raw)+ if (swept.length > 0) {+ log.warn({ recovered: swept.length }, '[agentHelpers] Swept stale STREAMING conversations to IDLE')+ }+ await Promise.all(swept.map((conversation) => failAbandonedFlowStep({ conversation, log })))+ return { recovered: swept.length }+}++async function failAbandonedFlowStep({ conversation, log }: { conversation: SweptConversation, log: FastifyBaseLogger }): Promise<void> {+ const { id, source, flowRunId, waitpointId } = conversation+ const holdsAPausedFlow = source === AgentRunSource.FLOW_STEP && !isNil(flowRunId) && !isNil(waitpointId)+ if (!holdsAPausedFlow) {+ return+ }+ const { error } = await tryCatch(() => flowStepRpc(log).resumeFlowStep({ conversationId: id, flowRunId, waitpointId, output: ABANDONED_FLOW_STEP_RESULT }))+ if (!isNil(error)) {+ log.error({ error, conversation: { id }, flowRun: { id: flowRunId }, waitpoint: { id: waitpointId } }, '[agentHelpers] Could not release the flow step of an abandoned agent run, the next sweep retries it until the step times out')+ await conversationRepo()+ .createQueryBuilder()+ .update()+ .set({ status: AgentConversationStatus.STREAMING })+ .where('id = :id', { id })+ .andWhere('created > NOW() - make_interval(secs => :stepTimeoutSecs)', { stepTimeoutSecs: AGENT_STEP_TIMEOUT_MS / 1_000 })+ .execute()}- return { recovered }}async function incrementAndCheckLimit({ key, limit, ttlSeconds }: { key: string, limit: number, ttlSeconds: number }): Promise<{ allowed: boolean, count: number }> {@@ -507,6 +530,11 @@ export const agentHelpers = {saveUserMemory,}+const ABANDONED_FLOW_STEP_RESULT: AgentResult = { prompt: '', steps: [], status: AgentTaskStatus.FAILED, failure: 'The agent stopped responding before it finished. Run the flow again to retry this step.' }++const SweptConversation = AgentConversation.pick({ id: true, source: true, flowRunId: true, waitpointId: true })+type SweptConversation = z.infer<typeof SweptConversation>+type AgentJobConfigFields = Pick<ExecuteAgentRunJobData, 'tools' | 'structuredOutput' | 'maxSteps' | 'modelName' | 'provider' | 'providerConfigId' | 'modelTierId' | 'promptOverride'>export type StreamingLockResult = 'acquired' | 'busy' | 'superseded' | 'duplicate'diff --git a/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts b/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.tsindex bbb06d8d2ea..17216651a5b 100644--- a/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts+++ b/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts@@ -29,7 +29,7 @@ import { CONNECTION_INVENTORY_LIMIT, loadOrStartConversation } from './rpc-shareexport const agentConfigRpc = (log: FastifyBaseLogger) => ({async getAgentConfig(input: GetAgentConfigRequest): Promise<AgentConfigResponse> {- const { conversationId, platformId, userId, userMessage, modelName, files, promptOverride, dryRun, discoveryOnly, source: requestedSource, projectId: requestedProjectId, agentId: linkedAgentId, flowRunId } = input+ const { conversationId, platformId, userId, userMessage, modelName, files, promptOverride, dryRun, discoveryOnly, source: requestedSource, projectId: requestedProjectId, agentId: linkedAgentId, flowRunId, waitpointId } = input// A flow-step run gets none of the owner's chat context, so it is not fetched. Reading it// anyway meant an owner without an MCP token or a user record failed the run outright.@@ -40,7 +40,7 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({const carriesChatContext = requestedSource !== AgentRunSource.FLOW_STEP && requestedSource !== AgentRunSource.AGENT && !isBuilderconst [conversation, userProjects, enabledAiTools, providerChoices] = await Promise.all([- loadOrStartConversation({ conversationId, platformId, userId, source: requestedSource, projectId: requestedProjectId, modelName, agentId: linkedAgentId, flowRunId }),+ loadOrStartConversation({ conversationId, runId: input.runId, platformId, userId, source: requestedSource, projectId: requestedProjectId, modelName, agentId: linkedAgentId, flowRunId, waitpointId }),agentHelpers.getUserProjects({ platformId, userId, log }),aiToolConfigService(log).getEnabledTools({ platformId }),aiToolConfigService(log).getProviderChoices({ platformId }),diff --git a/packages/server/api/src/app/ee/agent/rpc/rpc-shared.ts b/packages/server/api/src/app/ee/agent/rpc/rpc-shared.tsindex 904da56b1ee..be9eb2922d4 100644--- a/packages/server/api/src/app/ee/agent/rpc/rpc-shared.ts+++ b/packages/server/api/src/app/ee/agent/rpc/rpc-shared.ts@@ -74,8 +74,9 @@ export async function configuredToolConversationOrThrow({ conversationId }: { co}}-export async function loadOrStartConversation({ conversationId, platformId, userId, source, projectId, modelName, agentId, flowRunId }: {+export async function loadOrStartConversation({ conversationId, runId, platformId, userId, source, projectId, modelName, agentId, flowRunId, waitpointId }: {conversationId: string+ runId?: stringplatformId: stringuserId: stringsource?: AgentRunSource@@ -83,12 +84,13 @@ export async function loadOrStartConversation({ conversationId, platformId, usermodelName?: string | nullagentId?: stringflowRunId?: string+ waitpointId?: string}): Promise<AgentConversation> {if (source !== AgentRunSource.FLOW_STEP) {return agentHelpers.getConversationOrThrow({ id: conversationId, platformId, userId })}const conversation = await agentHelpers.conversationRepo().findOneBy({ id: conversationId })- ?? await insertFlowStepConversation({ conversationId, platformId, userId, projectId, modelName, agentId, flowRunId })+ ?? await insertFlowStepConversation({ conversationId, runId, platformId, userId, projectId, modelName, agentId, flowRunId, waitpointId })const ownedByThisRun = conversation.platformId === platformId && conversation.userId === userId && conversation.source === AgentRunSource.FLOW_STEPif (!ownedByThisRun) {throw new ActivepiecesError({ code: ErrorCode.AUTHORIZATION, params: { message: 'That conversation belongs to someone else' } })@@ -283,14 +285,16 @@ export function recordAgentFlowToolUse({ run, conversationId, flow, tool, outcom})}-async function insertFlowStepConversation({ conversationId, platformId, userId, projectId, modelName, agentId, flowRunId }: {+async function insertFlowStepConversation({ conversationId, runId, platformId, userId, projectId, modelName, agentId, flowRunId, waitpointId }: {conversationId: string+ runId?: stringplatformId: stringuserId: stringprojectId?: string | nullmodelName?: string | nullagentId?: stringflowRunId?: string+ waitpointId?: string}): Promise<AgentConversation> {await agentHelpers.conversationRepo().createQueryBuilder().insert().values({id: conversationId,@@ -300,10 +304,12 @@ async function insertFlowStepConversation({ conversationId, platformId, userId,source: AgentRunSource.FLOW_STEP,agentId: agentId ?? null,flowRunId: flowRunId ?? null,+ waitpointId: waitpointId ?? null,title: null,modelName: modelName ?? null,messages: [],status: AgentConversationStatus.IDLE,+ activeRunId: runId ?? null,}).orIgnore().execute()return agentHelpers.conversationRepo().findOneByOrFail({ id: conversationId })}diff --git a/packages/server/api/test/integration/ce/agent/agent-run-worker-handoff.test.ts b/packages/server/api/test/integration/ce/agent/agent-run-worker-handoff.test.tsindex 8521c22db82..ec4bd1be7d3 100644--- a/packages/server/api/test/integration/ce/agent/agent-run-worker-handoff.test.ts+++ b/packages/server/api/test/integration/ce/agent/agent-run-worker-handoff.test.ts@@ -1,10 +1,11 @@-import { AIProviderName, apId } from '@activepieces/core-utils'+import { AIProviderName, apId, ErrorCode } from '@activepieces/core-utils'import { AgentRunSource } from '@activepieces/shared'import { FastifyInstance } from 'fastify'import { StatusCodes } from 'http-status-codes'import { afterAll, beforeAll, describe, expect, it } from 'vitest'-import { agentRpcHandlers } from '../../../../src/app/ee/agent/agent-rpc-handlers'import { accessTokenManager } from '../../../../src/app/authentication/lib/access-token-manager'+import { agentRpcHandlers } from '../../../../src/app/ee/agent/agent-rpc-handlers'+import { db } from '../../../helpers/db'import { mockAndSaveAIProvider } from '../../../helpers/mocks'import { createTestContext, TestContext } from '../../../helpers/test-context'import { setupTestEnvironment, teardownTestEnvironment } from '../../../helpers/test-setup'@@ -83,4 +84,30 @@ describe('community edition: the worker picking up an agent step', () => {expect(config.agentsAvailable).toBe(false)})++ it('drops a redelivered copy of a running step as superseded, so it is not failed as a concurrent run', async () => {+ const ctx = await createTestContext(app)+ await mockAndSaveAIProvider({ platformId: ctx.platform.id, provider: AIProviderName.OPENAI })+ const conversationId = await startFlowStepRun(ctx, 'ce-handoff-redelivery')+ const delivery = {+ conversationId,+ runId: apId(),+ platformId: ctx.platform.id,+ userId: ctx.user.id,+ projectId: ctx.project.id,+ userMessage: 'summarise the thread',+ modelName: 'gpt-4o',+ provider: AIProviderName.OPENAI,+ source: AgentRunSource.FLOW_STEP,+ waitpointId: apId(),+ }++ await agentRpcHandlers(app.log).getAgentConfig(delivery)+ const conversation = await db.findOneByOrFail<{ activeRunId: string, waitpointId: string }>('agent_conversation', { id: conversationId })+ expect(conversation).toMatchObject({ activeRunId: delivery.runId, waitpointId: delivery.waitpointId })++ await expect(agentRpcHandlers(app.log).getAgentConfig(delivery)).rejects.toMatchObject({+ error: { code: ErrorCode.AGENT_RUN_SUPERSEDED },+ })+ })})diff --git a/packages/server/api/test/integration/cloud/agent/agent-stuck-streaming.test.ts b/packages/server/api/test/integration/cloud/agent/agent-stuck-streaming.test.tsindex b01fc3a91c0..dee1c9f6627 100644--- a/packages/server/api/test/integration/cloud/agent/agent-stuck-streaming.test.ts+++ b/packages/server/api/test/integration/cloud/agent/agent-stuck-streaming.test.ts@@ -1,10 +1,13 @@-import { apId, AgentConversationStatus } from '@activepieces/shared'+import { AgentConversationStatus, AgentRunSource, apId, FlowRunStatus, FlowVersionState } from '@activepieces/shared'import { FastifyInstance } from 'fastify'import { StatusCodes } from 'http-status-codes'-import { afterAll, beforeAll, describe, expect, it } from 'vitest'+import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest'import { agentHelpers } from '../../../../src/app/ee/agent/agent-helpers'+import { payloadOffloader } from '../../../../src/app/workers/payload-offloader'+import { WaitpointStatus } from '../../../../src/app/waitpoints/waitpoint-types'import { db } from '../../../helpers/db'-import { createTestContext } from '../../../helpers/test-context'+import { createMockFlow, createMockFlowRun, createMockFlowVersion } from '../../../helpers/mocks'+import { createTestContext, TestContext } from '../../../helpers/test-context'import { setupTestEnvironment, teardownTestEnvironment } from '../../../helpers/test-setup'let app: FastifyInstance@@ -189,4 +192,98 @@ describe('CHAT_STALE_SWEEP proactive recovery', () => {const row = await db.findOneByOrFail<{ status: string }>('agent_conversation', { id: evalId })expect(row.status).toBe(AgentConversationStatus.STREAMING)})++ it('releases the paused flow of a flow-step run whose worker died, so the flow does not wait out the step timeout', async () => {+ const ctx = await createTestContext(app, { plan: { chatEnabled: true } })+ const abandoned = await pausedFlowStepRun({ ctx, lastHeartbeat: new Date(Date.now() - 5 * 60 * 1_000) })+ const running = await pausedFlowStepRun({ ctx, lastHeartbeat: new Date(Date.now() - 30 * 1_000) })++ await agentHelpers.recoverAllStaleStreamingConversations({ log: app.log })+ await agentHelpers.recoverAllStaleStreamingConversations({ log: app.log })++ expect(await db.findOneBy('waitpoint', { id: abandoned.waitpointId })).toBeNull()+ expect(await db.findOneBy('waitpoint', { id: running.waitpointId })).not.toBeNull()+ })++ it('resumes the flow with a failure that says the agent stopped, so the step fails with a reason', async () => {+ const ctx = await createTestContext(app, { plan: { chatEnabled: true } })+ await pausedFlowStepRun({ ctx, lastHeartbeat: new Date(Date.now() - 5 * 60 * 1_000) })+ const offload = vi.spyOn(payloadOffloader, 'offloadPayload')++ await agentHelpers.recoverAllStaleStreamingConversations({ log: app.log })++ const resumedWith = offload.mock.calls.map(([, payload]) => payload)+ offload.mockRestore()+ expect(resumedWith).toContainEqual(expect.objectContaining({ body: expect.objectContaining({ failure: expect.stringContaining('The agent stopped responding before it finished') }) }))+ })++ it('never resumes a flow outside the conversation\'s own project, even if the row points at one', async () => {+ const victim = await createTestContext(app, { plan: { chatEnabled: true } })+ const attacker = await createTestContext(app, { plan: { chatEnabled: true } })+ const foreign = await pausedFlowStepRun({ ctx: victim, lastHeartbeat: new Date(Date.now() - 5 * 60 * 1_000), conversationProjectId: attacker.project.id })++ await agentHelpers.recoverAllStaleStreamingConversations({ log: app.log })++ expect(await db.findOneBy('waitpoint', { id: foreign.waitpointId })).not.toBeNull()+ })++ it('frees a flow-step row written by an older worker without a waitpoint, and leaves the step to its own timeout', async () => {+ const ctx = await createTestContext(app, { plan: { chatEnabled: true } })+ const fiveMinutesAgo = new Date(Date.now() - 5 * 60 * 1_000)+ const legacy = await pausedFlowStepRun({ ctx, lastHeartbeat: fiveMinutesAgo })+ await db.update('agent_conversation', legacy.conversationId, { waitpointId: null, updated: fiveMinutesAgo.toISOString() })++ await agentHelpers.recoverAllStaleStreamingConversations({ log: app.log })++ const row = await db.findOneByOrFail<{ status: string }>('agent_conversation', { id: legacy.conversationId })+ expect(row.status).toBe(AgentConversationStatus.IDLE)+ expect(await db.findOneBy('waitpoint', { id: legacy.waitpointId })).not.toBeNull()+ })++ it('keeps a flow step whose release failed eligible for the next sweep, so one failed attempt does not strand the flow', async () => {+ const ctx = await createTestContext(app, { plan: { chatEnabled: true } })+ const unreleasable = await pausedFlowStepRun({ ctx, lastHeartbeat: new Date(Date.now() - 5 * 60 * 1_000), conversationProjectId: null })++ await agentHelpers.recoverAllStaleStreamingConversations({ log: app.log })++ const row = await db.findOneByOrFail<{ status: string }>('agent_conversation', { id: unreleasable.conversationId })+ expect(row.status).toBe(AgentConversationStatus.STREAMING)+ expect(await db.findOneBy('waitpoint', { id: unreleasable.waitpointId })).not.toBeNull()+ })++ it('stops retrying a release that keeps failing once the step has timed out, since the waitpoint timeout already ended the wait', async () => {+ const ctx = await createTestContext(app, { plan: { chatEnabled: true } })+ const pastTimeout = await pausedFlowStepRun({ ctx, lastHeartbeat: new Date(Date.now() - 4 * 60 * 60 * 1_000), conversationProjectId: null })++ await agentHelpers.recoverAllStaleStreamingConversations({ log: app.log })++ const row = await db.findOneByOrFail<{ status: string }>('agent_conversation', { id: pastTimeout.conversationId })+ expect(row.status).toBe(AgentConversationStatus.IDLE)+ })})++async function pausedFlowStepRun({ ctx, lastHeartbeat, conversationProjectId = ctx.project.id }: { ctx: TestContext, lastHeartbeat: Date, conversationProjectId?: string | null }): Promise<{ conversationId: string, waitpointId: string }> {+ const flow = createMockFlow({ projectId: ctx.project.id })+ await db.save('flow', flow)+ const flowVersion = createMockFlowVersion({ flowId: flow.id, state: FlowVersionState.LOCKED })+ await db.save('flow_version', flowVersion)+ const flowRun = createMockFlowRun({ projectId: ctx.project.id, flowId: flow.id, flowVersionId: flowVersion.id, status: FlowRunStatus.PAUSED })+ await db.save('flow_run', flowRun)+ const waitpointId = apId()+ await db.save('waitpoint', { id: waitpointId, flowRunId: flowRun.id, projectId: ctx.project.id, stepName: 'agent', type: 'WEBHOOK', status: WaitpointStatus.PENDING, httpRequestId: null, workerHandlerId: null })+ const conversationId = apId()+ await db.save('agent_conversation', {+ id: conversationId,+ platformId: ctx.platform.id,+ projectId: conversationProjectId,+ userId: ctx.user.id,+ source: AgentRunSource.FLOW_STEP,+ flowRunId: flowRun.id,+ waitpointId,+ activeRunId: apId(),+ status: AgentConversationStatus.STREAMING,+ messages: [],+ })+ await db.update('agent_conversation', conversationId, { created: lastHeartbeat.toISOString(), updated: lastHeartbeat.toISOString() })+ return { conversationId, waitpointId }+}diff --git a/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts b/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.tsindex 6cc1d73c129..992d7e32d1a 100644--- a/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts+++ b/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts@@ -103,6 +103,7 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForge...spreadIfDefined('messageSource', data.messageSource),...spreadIfDefined('agentId', data.agentId),...spreadIfDefined('flowRunId', flowRunId),+ ...spreadIfDefined('waitpointId', waitpointId),...spreadIfDefined('provider', data.provider),...spreadIfDefined('providerConfigId', data.providerConfigId),...spreadIfDefined('modelTierId', data.modelTierId),diff --git a/packages/server/worker/test/lib/execute/jobs/ee/agent/execute-agent-run-config-failure.test.ts b/packages/server/worker/test/lib/execute/jobs/ee/agent/execute-agent-run-config-failure.test.tsindex 3614e58af7b..c231375c953 100644--- a/packages/server/worker/test/lib/execute/jobs/ee/agent/execute-agent-run-config-failure.test.ts+++ b/packages/server/worker/test/lib/execute/jobs/ee/agent/execute-agent-run-config-failure.test.ts@@ -18,11 +18,15 @@ const noopLogger = {function buildContext(configError?: Error) {const events: AgentEvent[] = []const resumed: { flowRunId: string, waitpointId: string, output: unknown }[] = []+ const configRequests: unknown[] = []const apiClient = {- getAgentConfig: () => Promise.reject(configError ?? new ActivepiecesError({- code: ErrorCode.ENTITY_NOT_FOUND,- params: { entityId: 'OPENAI', entityType: 'AIProvider' },- })),+ getAgentConfig: (input: unknown) => {+ configRequests.push(input)+ return Promise.reject(configError ?? new ActivepiecesError({+ code: ErrorCode.ENTITY_NOT_FOUND,+ params: { entityId: 'OPENAI', entityType: 'AIProvider' },+ }))+ },resumeFlowStep: (input: { flowRunId: string, waitpointId: string, output: unknown }) => {resumed.push(input)return Promise.resolve()@@ -34,7 +38,7 @@ function buildContext(configError?: Error) {},}const ctx = { apiClient, log: noopLogger } as unknown as JobContext- return { ctx, events, resumed }+ return { ctx, events, resumed, configRequests }}function buildJobData(overrides: Partial<ExecuteAgentRunJobData>): ExecuteAgentRunJobData {@@ -70,6 +74,24 @@ describe('executeAgentRunJob — a config failure must not swallow the turn', ()expect(resumed[0].output).toMatchObject({ failure: expect.stringContaining('ENTITY_NOT_FOUND') })})+ it('hands the step\'s waitpoint to the server, so the stale sweep can release the flow if this worker dies', async () => {+ const { ctx, configRequests } = buildContext()++ await executeAgentRunJob.execute(ctx, buildJobData({ source: AgentRunSource.FLOW_STEP, flowRunId: 'flow-run-1', waitpointId: 'waitpoint-1' }))++ expect(configRequests[0]).toMatchObject({ flowRunId: 'flow-run-1', waitpointId: 'waitpoint-1' })+ })++ it('exits quietly without releasing the flow when it is a redelivered copy of a run still in progress', async () => {+ const { ctx, resumed, events } = buildContext(new ActivepiecesError({ code: ErrorCode.AGENT_RUN_SUPERSEDED, params: { message: 'This run is already in progress' } }))++ const result = await executeAgentRunJob.execute(ctx, buildJobData({ source: AgentRunSource.FLOW_STEP, flowRunId: 'flow-run-1', waitpointId: 'waitpoint-1' }))++ expect(result.status).toBe(EngineResponseStatus.OK)+ expect(resumed).toHaveLength(0)+ expect(events).toHaveLength(0)+ })+it('fails a flow step on a model our credits do not serve with the reason alone, as a user failure', async () => {const reason = 'The model "openai/gpt-4o" is not available on Activepieces AI credits. Available models: anthropic/claude-sonnet-4.6'const { ctx, resumed } = buildContext(Object.assign(new Error(`RPC [getAgentConfig] handler threw: ENTITY_NOT_FOUND: ${reason}`), {