Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(agents): an AI agent no longer runs twice when its job is delivered late (#16898)

Hazem Adeledf428d15acf
diff --git a/brain/knowledge/flows-execution/chat.md b/brain/knowledge/flows-execution/chat.md
index 4ed667ee819..146acfb0982 100644
--- a/brain/knowledge/flows-execution/chat.md
+++ b/brain/knowledge/flows-execution/chat.md
@@ -40,7 +40,7 @@ A turn is kept alive / reclaimed by three separate mechanisms in `execute-agent-
- **Chat has no fixed step cap.** A chat turn is bounded by credits (checked after every step), the runaway-token ceiling (`RUNAWAY_TURN_CONTEXT_MULTIPLE` × the context window), the identical-failure guard and the 2-hour `MAX_AGENT_TURN_WALL_CLOCK_MS`, so it finishes long jobs. A saved agent's author-set `maxSteps` is still honored on its last step: an agent with structured output is forced to call `TASK_COMPLETION_TOOL_NAME` so the flow gets a result, and any other agent runs with `toolChoice: 'none'` plus a wrap-up note, so it replies with what's done and what's left. Never a silent stop or an error.
- **A turn limit only works as a stop condition.** One `streamText` call runs every step of a turn, so a check after it returns can't stop a runaway. The token ceiling used to be checked only there, and a turn on 24 Oct 2026 ran 181 steps and 50.8M input tokens (AI-86). Put every per-turn bound in `loopStopCondition`. Test mocks must report usage in the V3 shape (`inputTokens: { total, ... }`); flat numbers read as 0 tokens.
- **The streaming lock is per run, not per conversation.** A re-send can claim `activeRunId` before the first job reaches `getAgentConfig`. `agentHelpers.acquireStreamingLock` only lets the owning run set STREAMING; an older run gets `AGENT_RUN_SUPERSEDED` and the worker exits quietly, with no ERROR save and no events. A lock that ignored `runId` let the old run stream unowned while the new one was rejected as busy and marked ERROR, which is how users saw nothing after tapping a starter card twice.
-- **A redelivered run is a duplicate, not busy.** A worker socket blip makes the app requeue a chat job whose worker is still running it (`releaseConnectionJobs`). The second delivery finds its own `activeRunId` STREAMING, so `acquireStreamingLock` returns `duplicate` and it exits quietly like a superseded run. When this came back as `busy` it saved ERROR, and users saw "An agent is already running for this conversation" in the middle of a working turn (prod, 2026-10-13). If the first worker really died, the 90s `CHAT_STALE_SWEEP` frees the conversation. This only works when the conversation carries the run's `activeRunId`: chat claims it in the controller, and a flow-step run claims it when `getAgentConfig` creates the conversation. Without that claim the redelivery read as `busy` and failed the job (prod, 2026-10-21). If the first worker really died, the redelivery is still dropped, so `CHAT_STALE_SWEEP` fails the step itself: a flow-step conversation keeps its `waitpointId`, and the sweep resumes that waitpoint with a `failure` result, so the flow doesn't sit paused until the 3h step timeout. It never re-runs the agent, because its tools may already have acted.
+- **A redelivered run is a duplicate, not busy.** A worker socket blip makes the app requeue a chat job whose worker is still running it (`releaseConnectionJobs`). The second delivery finds its own `activeRunId` STREAMING, so `acquireStreamingLock` returns `duplicate` and it exits quietly like a superseded run. When this came back as `busy` it saved ERROR, and users saw "An agent is already running for this conversation" in the middle of a working turn (prod, 2026-10-13). If the first worker really died, the 90s `CHAT_STALE_SWEEP` frees the conversation. This only works when the conversation carries the run's `activeRunId`: chat claims it in the controller, and a flow-step run claims it when `getAgentConfig` creates the conversation. Without that claim the redelivery read as `busy` and failed the job (prod, 2026-10-21). A copy that arrives after its run already finished finds the row IDLE, so the lock alone would let the agent run again and its tools act twice (AI-153). `acquireStreamingLock` therefore also sets a Redis marker per conversation and run (`agent-run-started:<conversationId>:<runId>`, 24h) with `putIfAbsent`, and any later delivery of that run is a `duplicate`. That's safe because every enqueue mints a fresh `runId` and agent jobs run with `attempts: 1`. If the first worker really died, the redelivery is still dropped, so `CHAT_STALE_SWEEP` fails the step itself: a flow-step conversation keeps its `waitpointId`, and the sweep resumes that waitpoint with a `failure` result, so the flow doesn't sit paused until the 3h step timeout. It never re-runs the agent, because its tools may already have acted.
- **A failed turn used to leave status ERROR with an empty reply.** The error text only travelled as a websocket `ERROR` event, so a reload, another tab, or a user who was away saw a dead conversation (and often no user message either when the job died inside `getAgentConfig`, before it persists the turn). The `execute-agent-run.ts` catch now sends `failure: { message, userMessage? }` on the empty `saveAgentMessages` call and `conversation-rpc.ts` appends it to `uiMessages` as an assistant text reply; `use-chat.ts` shows the Retry banner when a loaded conversation is ERROR. Stale STREAMING to IDLE recovery, a worker that never picks the job up, and `assertAgentMessageRateLimitNotExceeded` still leave no reply of their own.
- **A built flow ends with a "Turn it on?" card, never an auto-publish**: an audit found 40% of built-but-unpublished flows ended on "open it to review", some claiming LIVE. Chat never publishes on its own; only a yes to the card calls `ap_lock_and_publish` (`build_flow.md` "Turn it on?"). Pinned by the `build-then-ask-turn-it-on` fixture (`askedToTurnItOn`, `noLiveClaimWithoutPublish`). The live-claim check is a regex plus "was `ap_lock_and_publish` called", so it cannot see whether publish succeeded.
- **Server-managed connections**: the LLM never sees connection externalIds; `ap_execute_action` auto-fills them from a Redis store.
diff --git a/packages/server/api/src/app/ee/agent/agent-helpers.ts b/packages/server/api/src/app/ee/agent/agent-helpers.ts
index 437d5640b6c..e753e7418f7 100644
--- a/packages/server/api/src/app/ee/agent/agent-helpers.ts
+++ b/packages/server/api/src/app/ee/agent/agent-helpers.ts
@@ -12,7 +12,7 @@ import { aiProviderService, ProviderScope } from '../../ai/ai-provider-service'
import { platformModelTierService, TierForRun } from '../../ai/platform-model-tier-service'
import { repoFactory } from '../../core/db/repo-factory'
import { transaction } from '../../core/db/transaction'
-import { redisConnections } from '../../database/redis-connections'
+import { distributedStore, redisConnections } from '../../database/redis-connections'
import { flowService } from '../../flows/flow/flow.service'
import { extractMcpTriggerInput } from '../../mcp/mcp-server-builder'
import { mcpToolInput } from '../../mcp/mcp-tool-input'
@@ -26,6 +26,7 @@ import { flowStepRpc } from './rpc/flow-step-rpc'
import { UserMemoryEntity } from './user-memory-entity'
const STREAMING_STALENESS_TIMEOUT_MS = 90 * 1_000
+const STARTED_RUN_MARKER_TTL_SECONDS = 24 * 60 * 60
// Interactive-eval conversations carry this id prefix (within the 21-char id column) so both the
// eval endpoints and the regular chat path can tell them apart from real user conversations.
@@ -417,6 +418,12 @@ async function claimConversationForRun({ conversationId, runId }: { conversation
}
async function acquireStreamingLock({ conversationId, runId }: { conversationId: string, runId?: string }): Promise<StreamingLockResult> {
+ if (!isNil(runId)) {
+ const firstDelivery = await distributedStore.putIfAbsent(`agent-run-started:${conversationId}:${runId}`, true, STARTED_RUN_MARKER_TTL_SECONDS)
+ if (!firstDelivery) {
+ return 'duplicate'
+ }
+ }
const builder = conversationRepo()
.createQueryBuilder()
.update()
diff --git a/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts b/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts
index 17216651a5b..05d741690d3 100644
--- a/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts
+++ b/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts
@@ -135,10 +135,10 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({
})
}
if (lock === 'duplicate') {
- log.warn({ conversation: { id: conversationId }, run: { id: input.runId } }, '[agentRpc#getAgentConfig] Run redelivered while its first delivery is still streaming, dropping the duplicate')
+ log.warn({ conversation: { id: conversationId }, run: { id: input.runId } }, '[agentRpc#getAgentConfig] Run redelivered after its first delivery started, dropping the duplicate')
throw new ActivepiecesError({
code: ErrorCode.AGENT_RUN_SUPERSEDED,
- params: { message: 'This run is already in progress' },
+ params: { message: 'This run was already delivered' },
})
}
if (lock === 'busy') {
diff --git a/packages/server/api/test/integration/cloud/agent/agent-resend-race.test.ts b/packages/server/api/test/integration/cloud/agent/agent-resend-race.test.ts
index d2bd8566bed..b17c0b7f43b 100644
--- a/packages/server/api/test/integration/cloud/agent/agent-resend-race.test.ts
+++ b/packages/server/api/test/integration/cloud/agent/agent-resend-race.test.ts
@@ -48,6 +48,19 @@ describe('re-sending a message while the first run has not started yet', () => {
expect(stored.activeRunId).toBe('run-1')
})
+ it('drops a copy delivered after its run finished, so the agent and its tools do not run twice', async () => {
+ const ctx = await createTestContext(app, { plan: { chatEnabled: true } })
+ const created = await ctx.post('/v1/agents/conversations', { title: 'Late copy' })
+ const conversationId = created.json().id
+ await agentHelpers.claimConversationForRun({ conversationId, runId: 'run-1' })
+ expect(await agentHelpers.acquireStreamingLock({ conversationId, runId: 'run-1' })).toBe('acquired')
+ await db.update('agent_conversation', conversationId, { status: AgentConversationStatus.IDLE })
+
+ expect(await agentHelpers.acquireStreamingLock({ conversationId, runId: 'run-1' })).toBe('duplicate')
+ const stored = await db.findOneByOrFail<{ status: string }>('agent_conversation', { id: conversationId })
+ expect(stored.status).toBe(AgentConversationStatus.IDLE)
+ })
+
it('still reports busy when a run without an id finds the conversation already streaming', async () => {
const ctx = await createTestContext(app, { plan: { chatEnabled: true } })
const created = await ctx.post('/v1/agents/conversations', { title: 'Busy' })
diff --git a/packages/server/worker/test/lib/execute/jobs/ee/agent/execute-agent-run-config-failure.test.ts b/packages/server/worker/test/lib/execute/jobs/ee/agent/execute-agent-run-config-failure.test.ts
index c231375c953..4f1196a03c2 100644
--- a/packages/server/worker/test/lib/execute/jobs/ee/agent/execute-agent-run-config-failure.test.ts
+++ b/packages/server/worker/test/lib/execute/jobs/ee/agent/execute-agent-run-config-failure.test.ts
@@ -83,7 +83,7 @@ describe('executeAgentRunJob — a config failure must not swallow the turn', ()
})
it('exits quietly without releasing the flow when it is a redelivered copy of a run still in progress', async () => {
- const { ctx, resumed, events } = buildContext(new ActivepiecesError({ code: ErrorCode.AGENT_RUN_SUPERSEDED, params: { message: 'This run is already in progress' } }))
+ const { ctx, resumed, events } = buildContext(new ActivepiecesError({ code: ErrorCode.AGENT_RUN_SUPERSEDED, params: { message: 'This run was already delivered' } }))
const result = await executeAgentRunJob.execute(ctx, buildJobData({ source: AgentRunSource.FLOW_STEP, flowRunId: 'flow-run-1', waitpointId: 'waitpoint-1' }))