Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(chat): a short connection drop no longer fails the reply with 'An agent is already running' (#16492)

Hazem Adele2012c1f76f5
diff --git a/brain/knowledge/execution-runtime/workers.md b/brain/knowledge/execution-runtime/workers.md
index 873e91c8b39..82bfc5ca36d 100644
--- a/brain/knowledge/execution-runtime/workers.md
+++ b/brain/knowledge/execution-runtime/workers.md
@@ -12,6 +12,7 @@ The deep `Resolver`/`Runtime` concurrency and bundle-caching model lives on the
- Workers connect over Socket.IO; on connect they emit `FETCH_WORKER_SETTINGS`, get a `WorkerSettingsResponse` (incl. `APP_VERSION`), and the app registers an RPC server (`WorkerToApiContract`) per socket.
- Jobs are **pulled** via `poll()`, not pushed. Handing out a job moves it to BullMQ `active`, so the app records it under the polling `socket.id` (`jobAssignmentTracker`, keyed queue+jobId). Worker executes → periodic `extendLock` → `completeJob` clears the assignment.
- On disconnect, `connectionGeneration++` stops the loops; the app returns that socket's still-held jobs to the queue (`releaseConnectionJobs(socket.id)`) so they don't sit orphaned in `active` (the "Job stalled" storm); the worker aborts its in-flight runtime. Reconnect recreates the runtime fresh (kills any lingering job rather than colliding on the reused box).
+- **A requeue on disconnect can run a job twice.** `releaseConnectionJobs` fires as soon as the socket closes, with no grace period. On 2026-10-13 a fleet-wide `transport close` blip requeued live jobs while the original worker kept running them for another ~90s and then lost its RPC channel (`Missing lock` on completion). Any job type can execute twice this way, not just chat.
- Graceful `stop()` (SIGTERM) drains in-flight jobs first (≤25s); the stalled-scan is the backstop for abrupt death.
### Config & routing
diff --git a/brain/knowledge/flows-execution/chat.md b/brain/knowledge/flows-execution/chat.md
index 37ea51a459e..07cc0225d13 100644
--- a/brain/knowledge/flows-execution/chat.md
+++ b/brain/knowledge/flows-execution/chat.md
@@ -40,6 +40,7 @@ A turn is kept alive / reclaimed by three separate mechanisms in `execute-agent-
- **A step's `response.messages` holds only that step's messages (AI SDK v7).** The older SDK made it cumulative, so `collectStepMessages` once took the last step; under v7 that saved only a turn's closing text and dropped every tool call and result. The next turn then saw replies like "New agent is updated" with no tool call behind them, and repeated the pattern by claiming changes it never made. It now joins every step, and `run-agent-turn-history.test.ts` runs a real `streamText` turn so a future SDK change to these semantics fails loudly.
- **Chat has no fixed step cap.** A chat turn is bounded by credits (checked after every step), the runaway-token ceiling (`RUNAWAY_TURN_CONTEXT_MULTIPLE` × the context window), the identical-failure guard and the 2-hour `MAX_AGENT_TURN_WALL_CLOCK_MS`, so it finishes long jobs. A saved agent's author-set `maxSteps` is still honored on its last step: an agent with structured output is forced to call `TASK_COMPLETION_TOOL_NAME` so the flow gets a result, and any other agent runs with `toolChoice: 'none'` plus a wrap-up note, so it replies with what's done and what's left. Never a silent stop or an error.
- **The streaming lock is per run, not per conversation.** A re-send can claim `activeRunId` before the first job reaches `getAgentConfig`. `agentHelpers.acquireStreamingLock` only lets the owning run set STREAMING; an older run gets `AGENT_RUN_SUPERSEDED` and the worker exits quietly, with no ERROR save and no events. A lock that ignored `runId` let the old run stream unowned while the new one was rejected as busy and marked ERROR, which is how users saw nothing after tapping a starter card twice.
+- **A redelivered run is a duplicate, not busy.** A worker socket blip makes the app requeue a chat job whose worker is still running it (`releaseConnectionJobs`). The second delivery finds its own `activeRunId` STREAMING, so `acquireStreamingLock` returns `duplicate` and it exits quietly like a superseded run. When this came back as `busy` it saved ERROR, and users saw "An agent is already running for this conversation" in the middle of a working turn (prod, 2026-10-13). If the first worker really died, the 90s `CHAT_STALE_SWEEP` frees the conversation.
- **A failed turn used to leave status ERROR with an empty reply.** The error text only travelled as a websocket `ERROR` event, so a reload, another tab, or a user who was away saw a dead conversation (and often no user message either when the job died inside `getAgentConfig`, before it persists the turn). The `execute-agent-run.ts` catch now sends `failure: { message, userMessage? }` on the empty `saveAgentMessages` call and `conversation-rpc.ts` appends it to `uiMessages` as an assistant text reply; `use-chat.ts` shows the Retry banner when a loaded conversation is ERROR. Stale STREAMING to IDLE recovery, a worker that never picks the job up, and `assertAgentMessageRateLimitNotExceeded` still leave no reply of their own.
- **A built flow ends with a "Turn it on?" card, never an auto-publish**: an audit found 40% of built-but-unpublished flows ended on "open it to review", some claiming LIVE. Chat never publishes on its own; only a yes to the card calls `ap_lock_and_publish` (`build_flow.md` "Turn it on?"). Pinned by the `build-then-ask-turn-it-on` fixture (`askedToTurnItOn`, `noLiveClaimWithoutPublish`). The live-claim check is a regex plus "was `ap_lock_and_publish` called", so it cannot see whether publish succeeded.
- **Server-managed connections**: the LLM never sees connection externalIds; `ap_execute_action` auto-fills them from a Redis store.
diff --git a/packages/server/api/src/app/ee/agent/agent-helpers.ts b/packages/server/api/src/app/ee/agent/agent-helpers.ts
index 496455c20d5..5bf17f4e458 100644
--- a/packages/server/api/src/app/ee/agent/agent-helpers.ts
+++ b/packages/server/api/src/app/ee/agent/agent-helpers.ts
@@ -371,8 +371,11 @@ async function acquireStreamingLock({ conversationId, runId }: { conversationId:
return 'acquired'
}
const current = await conversationRepo().findOneBy({ id: conversationId })
- const supersededByNewerRun = !isNil(runId) && !isNil(current?.activeRunId) && current.activeRunId !== runId
- return supersededByNewerRun ? 'superseded' : 'busy'
+ const ownedByARun = !isNil(runId) && !isNil(current?.activeRunId)
+ if (!ownedByARun) {
+ return 'busy'
+ }
+ return current.activeRunId === runId ? 'duplicate' : 'superseded'
}
async function resolveFlowTools({ projectId, tools, log }: { projectId: string, tools: AgentTool[], log: FastifyBaseLogger }): Promise<ResolvedAgentFlowTool[]> {
@@ -466,4 +469,4 @@ export const agentHelpers = {
type AgentJobConfigFields = Pick<ExecuteAgentRunJobData, 'tools' | 'structuredOutput' | 'maxSteps' | 'modelName' | 'provider' | 'providerConfigId' | 'modelTierId' | 'promptOverride'>
-export type StreamingLockResult = 'acquired' | 'busy' | 'superseded'
+export type StreamingLockResult = 'acquired' | 'busy' | 'superseded' | 'duplicate'
diff --git a/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts b/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts
index 7732a72d007..1e89fc3e2c5 100644
--- a/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts
+++ b/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts
@@ -123,6 +123,13 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({
params: { message: 'This message was replaced by a newer one' },
})
}
+ if (lock === 'duplicate') {
+ log.warn({ conversation: { id: conversationId }, run: { id: input.runId } }, '[agentRpc#getAgentConfig] Run redelivered while its first delivery is still streaming, dropping the duplicate')
+ throw new ActivepiecesError({
+ code: ErrorCode.AGENT_RUN_SUPERSEDED,
+ params: { message: 'This run is already in progress' },
+ })
+ }
if (lock === 'busy') {
log.warn({ conversation: { id: conversationId } }, '[agentRpc#getAgentConfig] Concurrent run rejected (conversation already STREAMING)')
throw new ActivepiecesError({
diff --git a/packages/server/api/test/integration/cloud/agent/agent-resend-race.test.ts b/packages/server/api/test/integration/cloud/agent/agent-resend-race.test.ts
index 4154973d78b..d2bd8566bed 100644
--- a/packages/server/api/test/integration/cloud/agent/agent-resend-race.test.ts
+++ b/packages/server/api/test/integration/cloud/agent/agent-resend-race.test.ts
@@ -35,13 +35,25 @@ describe('re-sending a message while the first run has not started yet', () => {
expect(stored.activeRunId).toBe('run-2')
})
- it('still reports busy when the same run finds the conversation already streaming', async () => {
+ it('reports a redelivered run as a duplicate and leaves the first delivery streaming', async () => {
const ctx = await createTestContext(app, { plan: { chatEnabled: true } })
- const created = await ctx.post('/v1/agents/conversations', { title: 'Busy' })
+ const created = await ctx.post('/v1/agents/conversations', { title: 'Duplicate' })
const conversationId = created.json().id
await agentHelpers.claimConversationForRun({ conversationId, runId: 'run-1' })
expect(await agentHelpers.acquireStreamingLock({ conversationId, runId: 'run-1' })).toBe('acquired')
- expect(await agentHelpers.acquireStreamingLock({ conversationId, runId: 'run-1' })).toBe('busy')
+ expect(await agentHelpers.acquireStreamingLock({ conversationId, runId: 'run-1' })).toBe('duplicate')
+ const stored = await db.findOneByOrFail<{ status: string, activeRunId: string }>('agent_conversation', { id: conversationId })
+ expect(stored.status).toBe(AgentConversationStatus.STREAMING)
+ expect(stored.activeRunId).toBe('run-1')
+ })
+
+ it('still reports busy when a run without an id finds the conversation already streaming', async () => {
+ const ctx = await createTestContext(app, { plan: { chatEnabled: true } })
+ const created = await ctx.post('/v1/agents/conversations', { title: 'Busy' })
+ const conversationId = created.json().id
+
+ expect(await agentHelpers.acquireStreamingLock({ conversationId })).toBe('acquired')
+ expect(await agentHelpers.acquireStreamingLock({ conversationId })).toBe('busy')
})
})