Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(agents): AI agent steps no longer fail after a worker reconnects (#16880)

Hazem Adeldbf46098ca6e
diff --git a/brain/knowledge/flows-execution/chat.md b/brain/knowledge/flows-execution/chat.md
index 2aa23d97ce1..4ed667ee819 100644
--- a/brain/knowledge/flows-execution/chat.md
+++ b/brain/knowledge/flows-execution/chat.md
@@ -40,7 +40,7 @@ A turn is kept alive / reclaimed by three separate mechanisms in `execute-agent-
- **Chat has no fixed step cap.** A chat turn is bounded by credits (checked after every step), the runaway-token ceiling (`RUNAWAY_TURN_CONTEXT_MULTIPLE` × the context window), the identical-failure guard and the 2-hour `MAX_AGENT_TURN_WALL_CLOCK_MS`, so it finishes long jobs. A saved agent's author-set `maxSteps` is still honored on its last step: an agent with structured output is forced to call `TASK_COMPLETION_TOOL_NAME` so the flow gets a result, and any other agent runs with `toolChoice: 'none'` plus a wrap-up note, so it replies with what's done and what's left. Never a silent stop or an error.
- **A turn limit only works as a stop condition.** One `streamText` call runs every step of a turn, so a check after it returns can't stop a runaway. The token ceiling used to be checked only there, and a turn on 24 Oct 2026 ran 181 steps and 50.8M input tokens (AI-86). Put every per-turn bound in `loopStopCondition`. Test mocks must report usage in the V3 shape (`inputTokens: { total, ... }`); flat numbers read as 0 tokens.
- **The streaming lock is per run, not per conversation.** A re-send can claim `activeRunId` before the first job reaches `getAgentConfig`. `agentHelpers.acquireStreamingLock` only lets the owning run set STREAMING; an older run gets `AGENT_RUN_SUPERSEDED` and the worker exits quietly, with no ERROR save and no events. A lock that ignored `runId` let the old run stream unowned while the new one was rejected as busy and marked ERROR, which is how users saw nothing after tapping a starter card twice.
-- **A redelivered run is a duplicate, not busy.** A worker socket blip makes the app requeue a chat job whose worker is still running it (`releaseConnectionJobs`). The second delivery finds its own `activeRunId` STREAMING, so `acquireStreamingLock` returns `duplicate` and it exits quietly like a superseded run. When this came back as `busy` it saved ERROR, and users saw "An agent is already running for this conversation" in the middle of a working turn (prod, 2026-10-13). If the first worker really died, the 90s `CHAT_STALE_SWEEP` frees the conversation.
+- **A redelivered run is a duplicate, not busy.** A worker socket blip makes the app requeue a chat job whose worker is still running it (`releaseConnectionJobs`). The second delivery finds its own `activeRunId` STREAMING, so `acquireStreamingLock` returns `duplicate` and it exits quietly like a superseded run. When this came back as `busy` it saved ERROR, and users saw "An agent is already running for this conversation" in the middle of a working turn (prod, 2026-10-13). If the first worker really died, the 90s `CHAT_STALE_SWEEP` frees the conversation. This only works when the conversation carries the run's `activeRunId`: chat claims it in the controller, and a flow-step run claims it when `getAgentConfig` creates the conversation. Without that claim the redelivery read as `busy` and failed the job (prod, 2026-10-21). If the first worker really died, the redelivery is still dropped, so `CHAT_STALE_SWEEP` fails the step itself: a flow-step conversation keeps its `waitpointId`, and the sweep resumes that waitpoint with a `failure` result, so the flow doesn't sit paused until the 3h step timeout. It never re-runs the agent, because its tools may already have acted.
- **A failed turn used to leave status ERROR with an empty reply.** The error text only travelled as a websocket `ERROR` event, so a reload, another tab, or a user who was away saw a dead conversation (and often no user message either when the job died inside `getAgentConfig`, before it persists the turn). The `execute-agent-run.ts` catch now sends `failure: { message, userMessage? }` on the empty `saveAgentMessages` call and `conversation-rpc.ts` appends it to `uiMessages` as an assistant text reply; `use-chat.ts` shows the Retry banner when a loaded conversation is ERROR. Stale STREAMING to IDLE recovery, a worker that never picks the job up, and `assertAgentMessageRateLimitNotExceeded` still leave no reply of their own.
- **A built flow ends with a "Turn it on?" card, never an auto-publish**: an audit found 40% of built-but-unpublished flows ended on "open it to review", some claiming LIVE. Chat never publishes on its own; only a yes to the card calls `ap_lock_and_publish` (`build_flow.md` "Turn it on?"). Pinned by the `build-then-ask-turn-it-on` fixture (`askedToTurnItOn`, `noLiveClaimWithoutPublish`). The live-claim check is a regex plus "was `ap_lock_and_publish` called", so it cannot see whether publish succeeded.
- **Server-managed connections**: the LLM never sees connection externalIds; `ap_execute_action` auto-fills them from a Redis store.
diff --git a/bun.lock b/bun.lock
index d600caedfd7..05e37b4e221 100644
--- a/bun.lock
+++ b/bun.lock
@@ -122,7 +122,7 @@
},
"packages/core/execution": {
"name": "@activepieces/core-execution",
- "version": "0.28.0",
+ "version": "0.28.1",
"dependencies": {
"@activepieces/core-piece-types": "workspace:*",
"@activepieces/core-utils": "workspace:*",
@@ -167,7 +167,7 @@
},
"packages/core/shared": {
"name": "@activepieces/shared",
- "version": "0.208.0",
+ "version": "0.208.1",
"dependencies": {
"@activepieces/core-execution": "workspace:*",
"@activepieces/core-formula": "workspace:*",
diff --git a/packages/core/execution/package.json b/packages/core/execution/package.json
index b80b463c0cf..1ecfb5bc063 100644
--- a/packages/core/execution/package.json
+++ b/packages/core/execution/package.json
@@ -1,6 +1,6 @@
{
"name": "@activepieces/core-execution",
- "version": "0.28.0",
+ "version": "0.28.1",
"type": "commonjs",
"main": "./dist/src/index.js",
"scripts": {
diff --git a/packages/core/execution/src/lib/workers/worker-contract.ts b/packages/core/execution/src/lib/workers/worker-contract.ts
index 702ec9630d6..ceba0d8d7ae 100644
--- a/packages/core/execution/src/lib/workers/worker-contract.ts
+++ b/packages/core/execution/src/lib/workers/worker-contract.ts
@@ -137,6 +137,7 @@ export type GetAgentConfigRequest = {
messageSource?: 'onboarding'
agentId?: string
flowRunId?: string
+ waitpointId?: string
projectId?: string | null
userMessage: string
modelName: string | null
diff --git a/packages/core/shared/package.json b/packages/core/shared/package.json
index a6ce6d9e9a3..a61e76d1e5d 100644
--- a/packages/core/shared/package.json
+++ b/packages/core/shared/package.json
@@ -1,6 +1,6 @@
{
"name": "@activepieces/shared",
- "version": "0.208.0",
+ "version": "0.208.1",
"type": "commonjs",
"sideEffects": false,
"main": "./dist/src/index.js",
diff --git a/packages/core/shared/src/lib/ee/agent/index.ts b/packages/core/shared/src/lib/ee/agent/index.ts
index 76d7ab95c85..4d21b34dab3 100644
--- a/packages/core/shared/src/lib/ee/agent/index.ts
+++ b/packages/core/shared/src/lib/ee/agent/index.ts
@@ -201,6 +201,7 @@ export const AgentConversation = z.object({
userId: z.string(),
agentId: Nullable(z.string()),
flowRunId: Nullable(z.string()),
+ waitpointId: Nullable(z.string()),
aiCredits: Nullable(z.number().int()),
source: z.enum(AgentRunSource),
title: Nullable(z.string()),
diff --git a/packages/server/api/src/app/database/migration/postgres/1876000000000-AddAgentConversationWaitpointId.ts b/packages/server/api/src/app/database/migration/postgres/1876000000000-AddAgentConversationWaitpointId.ts
new file mode 100644
index 00000000000..284b17c88d0
--- /dev/null
+++ b/packages/server/api/src/app/database/migration/postgres/1876000000000-AddAgentConversationWaitpointId.ts
@@ -0,0 +1,22 @@
+import { QueryRunner } from 'typeorm'
+import { Migration } from '../../migration'
+
+export class AddAgentConversationWaitpointId1876000000000 implements Migration {
+ name = 'AddAgentConversationWaitpointId1876000000000'
+ breaking = false
+ release = '0.93.0'
+
+ public async up(queryRunner: QueryRunner): Promise<void> {
+ await queryRunner.query(`
+ ALTER TABLE "agent_conversation"
+ ADD COLUMN IF NOT EXISTS "waitpointId" character varying(21)
+ `)
+ }
+
+ public async down(queryRunner: QueryRunner): Promise<void> {
+ await queryRunner.query(`
+ ALTER TABLE "agent_conversation"
+ DROP COLUMN IF EXISTS "waitpointId"
+ `)
+ }
+}
diff --git a/packages/server/api/src/app/database/postgres-connection.ts b/packages/server/api/src/app/database/postgres-connection.ts
index aa073a6b41d..59a6101b845 100644
--- a/packages/server/api/src/app/database/postgres-connection.ts
+++ b/packages/server/api/src/app/database/postgres-connection.ts
@@ -462,6 +462,7 @@ import { DropChatRolloutUserTable1870000000000 } from './migration/postgres/1870
import { AddBarrierChildAttribution1871000000000 } from './migration/postgres/1871000000000-AddBarrierChildAttribution'
import { AddPieceTranslationTable1873000000000 } from './migration/postgres/1873000000000-AddPieceTranslationTable'
import { AddDefaultChatModelToPlatformConfiguration1875000000000 } from './migration/postgres/1875000000000-AddDefaultChatModelToPlatformConfiguration'
+import { AddAgentConversationWaitpointId1876000000000 } from './migration/postgres/1876000000000-AddAgentConversationWaitpointId'
import { AddAgentConversationProviderConfigIdIndex1869000000000 } from './migration/postgres/background/1869000000000-AddAgentConversationProviderConfigIdIndex'
import { ReshapeFlowRunListIndexesForBarrierChildren1871000000000 } from './migration/postgres/background/1871000000000-ReshapeFlowRunListIndexesForBarrierChildren'
import { BackfillPieceTranslation1874000000000 } from './migration/postgres/background/1874000000000-BackfillPieceTranslation'
@@ -949,6 +950,7 @@ export const getMigrations = (): (new () => Migration)[] => {
AddBarrierChildAttribution1871000000000,
AddPieceTranslationTable1873000000000,
AddDefaultChatModelToPlatformConfiguration1875000000000,
+ AddAgentConversationWaitpointId1876000000000,
]
return migrations
}
diff --git a/packages/server/api/src/app/ee/agent/agent-conversation-entity.ts b/packages/server/api/src/app/ee/agent/agent-conversation-entity.ts
index ee5f5eac237..976b863a031 100644
--- a/packages/server/api/src/app/ee/agent/agent-conversation-entity.ts
+++ b/packages/server/api/src/app/ee/agent/agent-conversation-entity.ts
@@ -29,6 +29,10 @@ export const AgentConversationEntity = new EntitySchema<AgentConversationWithRel
...ApIdSchema,
nullable: true,
},
+ waitpointId: {
+ ...ApIdSchema,
+ nullable: true,
+ },
aiCredits: {
type: Number,
nullable: true,
diff --git a/packages/server/api/src/app/ee/agent/agent-helpers.ts b/packages/server/api/src/app/ee/agent/agent-helpers.ts
index 4a639a4a7dd..437d5640b6c 100644
--- a/packages/server/api/src/app/ee/agent/agent-helpers.ts
+++ b/packages/server/api/src/app/ee/agent/agent-helpers.ts
@@ -1,8 +1,9 @@
import { setTimeout as delay } from 'node:timers/promises'
import { ExecuteAgentRunJobData } from '@activepieces/core-execution'
+import { AGENT_STEP_TIMEOUT_MS, AgentResult, AgentTaskStatus } from '@activepieces/core-piece-types'
import { ActivepiecesError, AIProviderName, apId, assertNotNullOrUndefined, ErrorCode, isNil, spreadIfDefined, tryCatch, tryCatchSync, unique } from '@activepieces/core-utils'
import { aiUtils, modelTierCatalog, ModelTierSurface } from '@activepieces/server-utils'
-import { AgentConfig, AgentConversation, AgentConversationStatus, AgentFlowTool, AgentTool, AgentToolType, AI_PROVIDER_CAPABILITIES, AI_PROVIDER_ENTITY_TYPES, AIProviderModelType, FlowVersionState, GetAgentMemoryResponse, GetProviderConfigResponse, Project, ProjectType, ResolvedAgentFlowTool, UserMemory } from '@activepieces/shared'
+import { AgentConfig, AgentConversation, AgentConversationStatus, AgentFlowTool, AgentRunSource, AgentTool, AgentToolType, AI_PROVIDER_CAPABILITIES, AI_PROVIDER_ENTITY_TYPES, AIProviderModelType, FlowVersionState, GetAgentMemoryResponse, GetProviderConfigResponse, Project, ProjectType, ResolvedAgentFlowTool, UserMemory } from '@activepieces/shared'
import { LanguageModel } from 'ai'
import { FastifyBaseLogger } from 'fastify'
import { Repository } from 'typeorm'
@@ -21,6 +22,7 @@ import { platformPlanService } from '../platform/platform-plan/platform-plan.ser
import { AgentConversationEntity, AgentConversationWithRelations } from './agent-conversation-entity'
import { agentModelResolution } from './agent-model-resolution'
import { agentModelTier } from './agent-model-tier'
+import { flowStepRpc } from './rpc/flow-step-rpc'
import { UserMemoryEntity } from './user-memory-entity'
const STREAMING_STALENESS_TIMEOUT_MS = 90 * 1_000
@@ -288,12 +290,33 @@ async function recoverAllStaleStreamingConversations({ log }: { log: FastifyBase
.where('status = :streaming', { streaming: AgentConversationStatus.STREAMING })
.andWhere('updated < NOW() - make_interval(secs => :staleSecs)', { staleSecs: STREAMING_STALENESS_TIMEOUT_MS / 1_000 })
.andWhere('id NOT LIKE :evalPrefix', { evalPrefix: `${EVAL_CONVERSATION_ID_PREFIX}%` })
+ .returning(['id', 'source', 'flowRunId', 'waitpointId'])
.execute()
- const recovered = result.affected ?? 0
- if (recovered > 0) {
- log.warn({ recovered }, '[agentHelpers] Swept stale STREAMING conversations to IDLE')
+ const swept = SweptConversation.array().parse(result.raw)
+ if (swept.length > 0) {
+ log.warn({ recovered: swept.length }, '[agentHelpers] Swept stale STREAMING conversations to IDLE')
+ }
+ await Promise.all(swept.map((conversation) => failAbandonedFlowStep({ conversation, log })))
+ return { recovered: swept.length }
+}
+
+async function failAbandonedFlowStep({ conversation, log }: { conversation: SweptConversation, log: FastifyBaseLogger }): Promise<void> {
+ const { id, source, flowRunId, waitpointId } = conversation
+ const holdsAPausedFlow = source === AgentRunSource.FLOW_STEP && !isNil(flowRunId) && !isNil(waitpointId)
+ if (!holdsAPausedFlow) {
+ return
+ }
+ const { error } = await tryCatch(() => flowStepRpc(log).resumeFlowStep({ conversationId: id, flowRunId, waitpointId, output: ABANDONED_FLOW_STEP_RESULT }))
+ if (!isNil(error)) {
+ log.error({ error, conversation: { id }, flowRun: { id: flowRunId }, waitpoint: { id: waitpointId } }, '[agentHelpers] Could not release the flow step of an abandoned agent run, the next sweep retries it until the step times out')
+ await conversationRepo()
+ .createQueryBuilder()
+ .update()
+ .set({ status: AgentConversationStatus.STREAMING })
+ .where('id = :id', { id })
+ .andWhere('created > NOW() - make_interval(secs => :stepTimeoutSecs)', { stepTimeoutSecs: AGENT_STEP_TIMEOUT_MS / 1_000 })
+ .execute()
}
- return { recovered }
}
async function incrementAndCheckLimit({ key, limit, ttlSeconds }: { key: string, limit: number, ttlSeconds: number }): Promise<{ allowed: boolean, count: number }> {
@@ -507,6 +530,11 @@ export const agentHelpers = {
saveUserMemory,
}
+const ABANDONED_FLOW_STEP_RESULT: AgentResult = { prompt: '', steps: [], status: AgentTaskStatus.FAILED, failure: 'The agent stopped responding before it finished. Run the flow again to retry this step.' }
+
+const SweptConversation = AgentConversation.pick({ id: true, source: true, flowRunId: true, waitpointId: true })
+type SweptConversation = z.infer<typeof SweptConversation>
+
type AgentJobConfigFields = Pick<ExecuteAgentRunJobData, 'tools' | 'structuredOutput' | 'maxSteps' | 'modelName' | 'provider' | 'providerConfigId' | 'modelTierId' | 'promptOverride'>
export type StreamingLockResult = 'acquired' | 'busy' | 'superseded' | 'duplicate'
diff --git a/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts b/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts
index bbb06d8d2ea..17216651a5b 100644
--- a/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts
+++ b/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts
@@ -29,7 +29,7 @@ import { CONNECTION_INVENTORY_LIMIT, loadOrStartConversation } from './rpc-share
export const agentConfigRpc = (log: FastifyBaseLogger) => ({
async getAgentConfig(input: GetAgentConfigRequest): Promise<AgentConfigResponse> {
- const { conversationId, platformId, userId, userMessage, modelName, files, promptOverride, dryRun, discoveryOnly, source: requestedSource, projectId: requestedProjectId, agentId: linkedAgentId, flowRunId } = input
+ const { conversationId, platformId, userId, userMessage, modelName, files, promptOverride, dryRun, discoveryOnly, source: requestedSource, projectId: requestedProjectId, agentId: linkedAgentId, flowRunId, waitpointId } = input
// A flow-step run gets none of the owner's chat context, so it is not fetched. Reading it
// anyway meant an owner without an MCP token or a user record failed the run outright.
@@ -40,7 +40,7 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({
const carriesChatContext = requestedSource !== AgentRunSource.FLOW_STEP && requestedSource !== AgentRunSource.AGENT && !isBuilder
const [conversation, userProjects, enabledAiTools, providerChoices] = await Promise.all([
- loadOrStartConversation({ conversationId, platformId, userId, source: requestedSource, projectId: requestedProjectId, modelName, agentId: linkedAgentId, flowRunId }),
+ loadOrStartConversation({ conversationId, runId: input.runId, platformId, userId, source: requestedSource, projectId: requestedProjectId, modelName, agentId: linkedAgentId, flowRunId, waitpointId }),
agentHelpers.getUserProjects({ platformId, userId, log }),
aiToolConfigService(log).getEnabledTools({ platformId }),
aiToolConfigService(log).getProviderChoices({ platformId }),
diff --git a/packages/server/api/src/app/ee/agent/rpc/rpc-shared.ts b/packages/server/api/src/app/ee/agent/rpc/rpc-shared.ts
index 904da56b1ee..be9eb2922d4 100644
--- a/packages/server/api/src/app/ee/agent/rpc/rpc-shared.ts
+++ b/packages/server/api/src/app/ee/agent/rpc/rpc-shared.ts
@@ -74,8 +74,9 @@ export async function configuredToolConversationOrThrow({ conversationId }: { co
}
}
-export async function loadOrStartConversation({ conversationId, platformId, userId, source, projectId, modelName, agentId, flowRunId }: {
+export async function loadOrStartConversation({ conversationId, runId, platformId, userId, source, projectId, modelName, agentId, flowRunId, waitpointId }: {
conversationId: string
+ runId?: string
platformId: string
userId: string
source?: AgentRunSource
@@ -83,12 +84,13 @@ export async function loadOrStartConversation({ conversationId, platformId, user
modelName?: string | null
agentId?: string
flowRunId?: string
+ waitpointId?: string
}): Promise<AgentConversation> {
if (source !== AgentRunSource.FLOW_STEP) {
return agentHelpers.getConversationOrThrow({ id: conversationId, platformId, userId })
}
const conversation = await agentHelpers.conversationRepo().findOneBy({ id: conversationId })
- ?? await insertFlowStepConversation({ conversationId, platformId, userId, projectId, modelName, agentId, flowRunId })
+ ?? await insertFlowStepConversation({ conversationId, runId, platformId, userId, projectId, modelName, agentId, flowRunId, waitpointId })
const ownedByThisRun = conversation.platformId === platformId && conversation.userId === userId && conversation.source === AgentRunSource.FLOW_STEP
if (!ownedByThisRun) {
throw new ActivepiecesError({ code: ErrorCode.AUTHORIZATION, params: { message: 'That conversation belongs to someone else' } })
@@ -283,14 +285,16 @@ export function recordAgentFlowToolUse({ run, conversationId, flow, tool, outcom
})
}
-async function insertFlowStepConversation({ conversationId, platformId, userId, projectId, modelName, agentId, flowRunId }: {
+async function insertFlowStepConversation({ conversationId, runId, platformId, userId, projectId, modelName, agentId, flowRunId, waitpointId }: {
conversationId: string
+ runId?: string
platformId: string
userId: string
projectId?: string | null
modelName?: string | null
agentId?: string
flowRunId?: string
+ waitpointId?: string
}): Promise<AgentConversation> {
await agentHelpers.conversationRepo().createQueryBuilder().insert().values({
id: conversationId,
@@ -300,10 +304,12 @@ async function insertFlowStepConversation({ conversationId, platformId, userId,
source: AgentRunSource.FLOW_STEP,
agentId: agentId ?? null,
flowRunId: flowRunId ?? null,
+ waitpointId: waitpointId ?? null,
title: null,
modelName: modelName ?? null,
messages: [],
status: AgentConversationStatus.IDLE,
+ activeRunId: runId ?? null,
}).orIgnore().execute()
return agentHelpers.conversationRepo().findOneByOrFail({ id: conversationId })
}
diff --git a/packages/server/api/test/integration/ce/agent/agent-run-worker-handoff.test.ts b/packages/server/api/test/integration/ce/agent/agent-run-worker-handoff.test.ts
index 8521c22db82..ec4bd1be7d3 100644
--- a/packages/server/api/test/integration/ce/agent/agent-run-worker-handoff.test.ts
+++ b/packages/server/api/test/integration/ce/agent/agent-run-worker-handoff.test.ts
@@ -1,10 +1,11 @@
-import { AIProviderName, apId } from '@activepieces/core-utils'
+import { AIProviderName, apId, ErrorCode } from '@activepieces/core-utils'
import { AgentRunSource } from '@activepieces/shared'
import { FastifyInstance } from 'fastify'
import { StatusCodes } from 'http-status-codes'
import { afterAll, beforeAll, describe, expect, it } from 'vitest'
-import { agentRpcHandlers } from '../../../../src/app/ee/agent/agent-rpc-handlers'
import { accessTokenManager } from '../../../../src/app/authentication/lib/access-token-manager'
+import { agentRpcHandlers } from '../../../../src/app/ee/agent/agent-rpc-handlers'
+import { db } from '../../../helpers/db'
import { mockAndSaveAIProvider } from '../../../helpers/mocks'
import { createTestContext, TestContext } from '../../../helpers/test-context'
import { setupTestEnvironment, teardownTestEnvironment } from '../../../helpers/test-setup'
@@ -83,4 +84,30 @@ describe('community edition: the worker picking up an agent step', () => {
expect(config.agentsAvailable).toBe(false)
})
+
+ it('drops a redelivered copy of a running step as superseded, so it is not failed as a concurrent run', async () => {
+ const ctx = await createTestContext(app)
+ await mockAndSaveAIProvider({ platformId: ctx.platform.id, provider: AIProviderName.OPENAI })
+ const conversationId = await startFlowStepRun(ctx, 'ce-handoff-redelivery')
+ const delivery = {
+ conversationId,
+ runId: apId(),
+ platformId: ctx.platform.id,
+ userId: ctx.user.id,
+ projectId: ctx.project.id,
+ userMessage: 'summarise the thread',
+ modelName: 'gpt-4o',
+ provider: AIProviderName.OPENAI,
+ source: AgentRunSource.FLOW_STEP,
+ waitpointId: apId(),
+ }
+
+ await agentRpcHandlers(app.log).getAgentConfig(delivery)
+ const conversation = await db.findOneByOrFail<{ activeRunId: string, waitpointId: string }>('agent_conversation', { id: conversationId })
+ expect(conversation).toMatchObject({ activeRunId: delivery.runId, waitpointId: delivery.waitpointId })
+
+ await expect(agentRpcHandlers(app.log).getAgentConfig(delivery)).rejects.toMatchObject({
+ error: { code: ErrorCode.AGENT_RUN_SUPERSEDED },
+ })
+ })
})
diff --git a/packages/server/api/test/integration/cloud/agent/agent-stuck-streaming.test.ts b/packages/server/api/test/integration/cloud/agent/agent-stuck-streaming.test.ts
index b01fc3a91c0..dee1c9f6627 100644
--- a/packages/server/api/test/integration/cloud/agent/agent-stuck-streaming.test.ts
+++ b/packages/server/api/test/integration/cloud/agent/agent-stuck-streaming.test.ts
@@ -1,10 +1,13 @@
-import { apId, AgentConversationStatus } from '@activepieces/shared'
+import { AgentConversationStatus, AgentRunSource, apId, FlowRunStatus, FlowVersionState } from '@activepieces/shared'
import { FastifyInstance } from 'fastify'
import { StatusCodes } from 'http-status-codes'
-import { afterAll, beforeAll, describe, expect, it } from 'vitest'
+import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest'
import { agentHelpers } from '../../../../src/app/ee/agent/agent-helpers'
+import { payloadOffloader } from '../../../../src/app/workers/payload-offloader'
+import { WaitpointStatus } from '../../../../src/app/waitpoints/waitpoint-types'
import { db } from '../../../helpers/db'
-import { createTestContext } from '../../../helpers/test-context'
+import { createMockFlow, createMockFlowRun, createMockFlowVersion } from '../../../helpers/mocks'
+import { createTestContext, TestContext } from '../../../helpers/test-context'
import { setupTestEnvironment, teardownTestEnvironment } from '../../../helpers/test-setup'
let app: FastifyInstance
@@ -189,4 +192,98 @@ describe('CHAT_STALE_SWEEP proactive recovery', () => {
const row = await db.findOneByOrFail<{ status: string }>('agent_conversation', { id: evalId })
expect(row.status).toBe(AgentConversationStatus.STREAMING)
})
+
+ it('releases the paused flow of a flow-step run whose worker died, so the flow does not wait out the step timeout', async () => {
+ const ctx = await createTestContext(app, { plan: { chatEnabled: true } })
+ const abandoned = await pausedFlowStepRun({ ctx, lastHeartbeat: new Date(Date.now() - 5 * 60 * 1_000) })
+ const running = await pausedFlowStepRun({ ctx, lastHeartbeat: new Date(Date.now() - 30 * 1_000) })
+
+ await agentHelpers.recoverAllStaleStreamingConversations({ log: app.log })
+ await agentHelpers.recoverAllStaleStreamingConversations({ log: app.log })
+
+ expect(await db.findOneBy('waitpoint', { id: abandoned.waitpointId })).toBeNull()
+ expect(await db.findOneBy('waitpoint', { id: running.waitpointId })).not.toBeNull()
+ })
+
+ it('resumes the flow with a failure that says the agent stopped, so the step fails with a reason', async () => {
+ const ctx = await createTestContext(app, { plan: { chatEnabled: true } })
+ await pausedFlowStepRun({ ctx, lastHeartbeat: new Date(Date.now() - 5 * 60 * 1_000) })
+ const offload = vi.spyOn(payloadOffloader, 'offloadPayload')
+
+ await agentHelpers.recoverAllStaleStreamingConversations({ log: app.log })
+
+ const resumedWith = offload.mock.calls.map(([, payload]) => payload)
+ offload.mockRestore()
+ expect(resumedWith).toContainEqual(expect.objectContaining({ body: expect.objectContaining({ failure: expect.stringContaining('The agent stopped responding before it finished') }) }))
+ })
+
+ it('never resumes a flow outside the conversation\'s own project, even if the row points at one', async () => {
+ const victim = await createTestContext(app, { plan: { chatEnabled: true } })
+ const attacker = await createTestContext(app, { plan: { chatEnabled: true } })
+ const foreign = await pausedFlowStepRun({ ctx: victim, lastHeartbeat: new Date(Date.now() - 5 * 60 * 1_000), conversationProjectId: attacker.project.id })
+
+ await agentHelpers.recoverAllStaleStreamingConversations({ log: app.log })
+
+ expect(await db.findOneBy('waitpoint', { id: foreign.waitpointId })).not.toBeNull()
+ })
+
+ it('frees a flow-step row written by an older worker without a waitpoint, and leaves the step to its own timeout', async () => {
+ const ctx = await createTestContext(app, { plan: { chatEnabled: true } })
+ const fiveMinutesAgo = new Date(Date.now() - 5 * 60 * 1_000)
+ const legacy = await pausedFlowStepRun({ ctx, lastHeartbeat: fiveMinutesAgo })
+ await db.update('agent_conversation', legacy.conversationId, { waitpointId: null, updated: fiveMinutesAgo.toISOString() })
+
+ await agentHelpers.recoverAllStaleStreamingConversations({ log: app.log })
+
+ const row = await db.findOneByOrFail<{ status: string }>('agent_conversation', { id: legacy.conversationId })
+ expect(row.status).toBe(AgentConversationStatus.IDLE)
+ expect(await db.findOneBy('waitpoint', { id: legacy.waitpointId })).not.toBeNull()
+ })
+
+ it('keeps a flow step whose release failed eligible for the next sweep, so one failed attempt does not strand the flow', async () => {
+ const ctx = await createTestContext(app, { plan: { chatEnabled: true } })
+ const unreleasable = await pausedFlowStepRun({ ctx, lastHeartbeat: new Date(Date.now() - 5 * 60 * 1_000), conversationProjectId: null })
+
+ await agentHelpers.recoverAllStaleStreamingConversations({ log: app.log })
+
+ const row = await db.findOneByOrFail<{ status: string }>('agent_conversation', { id: unreleasable.conversationId })
+ expect(row.status).toBe(AgentConversationStatus.STREAMING)
+ expect(await db.findOneBy('waitpoint', { id: unreleasable.waitpointId })).not.toBeNull()
+ })
+
+ it('stops retrying a release that keeps failing once the step has timed out, since the waitpoint timeout already ended the wait', async () => {
+ const ctx = await createTestContext(app, { plan: { chatEnabled: true } })
+ const pastTimeout = await pausedFlowStepRun({ ctx, lastHeartbeat: new Date(Date.now() - 4 * 60 * 60 * 1_000), conversationProjectId: null })
+
+ await agentHelpers.recoverAllStaleStreamingConversations({ log: app.log })
+
+ const row = await db.findOneByOrFail<{ status: string }>('agent_conversation', { id: pastTimeout.conversationId })
+ expect(row.status).toBe(AgentConversationStatus.IDLE)
+ })
})
+
+async function pausedFlowStepRun({ ctx, lastHeartbeat, conversationProjectId = ctx.project.id }: { ctx: TestContext, lastHeartbeat: Date, conversationProjectId?: string | null }): Promise<{ conversationId: string, waitpointId: string }> {
+ const flow = createMockFlow({ projectId: ctx.project.id })
+ await db.save('flow', flow)
+ const flowVersion = createMockFlowVersion({ flowId: flow.id, state: FlowVersionState.LOCKED })
+ await db.save('flow_version', flowVersion)
+ const flowRun = createMockFlowRun({ projectId: ctx.project.id, flowId: flow.id, flowVersionId: flowVersion.id, status: FlowRunStatus.PAUSED })
+ await db.save('flow_run', flowRun)
+ const waitpointId = apId()
+ await db.save('waitpoint', { id: waitpointId, flowRunId: flowRun.id, projectId: ctx.project.id, stepName: 'agent', type: 'WEBHOOK', status: WaitpointStatus.PENDING, httpRequestId: null, workerHandlerId: null })
+ const conversationId = apId()
+ await db.save('agent_conversation', {
+ id: conversationId,
+ platformId: ctx.platform.id,
+ projectId: conversationProjectId,
+ userId: ctx.user.id,
+ source: AgentRunSource.FLOW_STEP,
+ flowRunId: flowRun.id,
+ waitpointId,
+ activeRunId: apId(),
+ status: AgentConversationStatus.STREAMING,
+ messages: [],
+ })
+ await db.update('agent_conversation', conversationId, { created: lastHeartbeat.toISOString(), updated: lastHeartbeat.toISOString() })
+ return { conversationId, waitpointId }
+}
diff --git a/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts b/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts
index 6cc1d73c129..992d7e32d1a 100644
--- a/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts
+++ b/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts
@@ -103,6 +103,7 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForge
...spreadIfDefined('messageSource', data.messageSource),
...spreadIfDefined('agentId', data.agentId),
...spreadIfDefined('flowRunId', flowRunId),
+ ...spreadIfDefined('waitpointId', waitpointId),
...spreadIfDefined('provider', data.provider),
...spreadIfDefined('providerConfigId', data.providerConfigId),
...spreadIfDefined('modelTierId', data.modelTierId),
diff --git a/packages/server/worker/test/lib/execute/jobs/ee/agent/execute-agent-run-config-failure.test.ts b/packages/server/worker/test/lib/execute/jobs/ee/agent/execute-agent-run-config-failure.test.ts
index 3614e58af7b..c231375c953 100644
--- a/packages/server/worker/test/lib/execute/jobs/ee/agent/execute-agent-run-config-failure.test.ts
+++ b/packages/server/worker/test/lib/execute/jobs/ee/agent/execute-agent-run-config-failure.test.ts
@@ -18,11 +18,15 @@ const noopLogger = {
function buildContext(configError?: Error) {
const events: AgentEvent[] = []
const resumed: { flowRunId: string, waitpointId: string, output: unknown }[] = []
+ const configRequests: unknown[] = []
const apiClient = {
- getAgentConfig: () => Promise.reject(configError ?? new ActivepiecesError({
- code: ErrorCode.ENTITY_NOT_FOUND,
- params: { entityId: 'OPENAI', entityType: 'AIProvider' },
- })),
+ getAgentConfig: (input: unknown) => {
+ configRequests.push(input)
+ return Promise.reject(configError ?? new ActivepiecesError({
+ code: ErrorCode.ENTITY_NOT_FOUND,
+ params: { entityId: 'OPENAI', entityType: 'AIProvider' },
+ }))
+ },
resumeFlowStep: (input: { flowRunId: string, waitpointId: string, output: unknown }) => {
resumed.push(input)
return Promise.resolve()
@@ -34,7 +38,7 @@ function buildContext(configError?: Error) {
},
}
const ctx = { apiClient, log: noopLogger } as unknown as JobContext
- return { ctx, events, resumed }
+ return { ctx, events, resumed, configRequests }
}
function buildJobData(overrides: Partial<ExecuteAgentRunJobData>): ExecuteAgentRunJobData {
@@ -70,6 +74,24 @@ describe('executeAgentRunJob — a config failure must not swallow the turn', ()
expect(resumed[0].output).toMatchObject({ failure: expect.stringContaining('ENTITY_NOT_FOUND') })
})
+ it('hands the step\'s waitpoint to the server, so the stale sweep can release the flow if this worker dies', async () => {
+ const { ctx, configRequests } = buildContext()
+
+ await executeAgentRunJob.execute(ctx, buildJobData({ source: AgentRunSource.FLOW_STEP, flowRunId: 'flow-run-1', waitpointId: 'waitpoint-1' }))
+
+ expect(configRequests[0]).toMatchObject({ flowRunId: 'flow-run-1', waitpointId: 'waitpoint-1' })
+ })
+
+ it('exits quietly without releasing the flow when it is a redelivered copy of a run still in progress', async () => {
+ const { ctx, resumed, events } = buildContext(new ActivepiecesError({ code: ErrorCode.AGENT_RUN_SUPERSEDED, params: { message: 'This run is already in progress' } }))
+
+ const result = await executeAgentRunJob.execute(ctx, buildJobData({ source: AgentRunSource.FLOW_STEP, flowRunId: 'flow-run-1', waitpointId: 'waitpoint-1' }))
+
+ expect(result.status).toBe(EngineResponseStatus.OK)
+ expect(resumed).toHaveLength(0)
+ expect(events).toHaveLength(0)
+ })
+
it('fails a flow step on a model our credits do not serve with the reason alone, as a user failure', async () => {
const reason = 'The model "openai/gpt-4o" is not available on Activepieces AI credits. Available models: anthropic/claude-sonnet-4.6'
const { ctx, resumed } = buildContext(Object.assign(new Error(`RPC [getAgentConfig] handler threw: ENTITY_NOT_FOUND: ${reason}`), {