AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
fix(chat): chat finishes long jobs instead of stopping partway (#15925)
Hazem Adel208671f7269f
diff --git a/brain/knowledge/flows-execution/chat.md b/brain/knowledge/flows-execution/chat.mdindex ff6c555aee1..42d3a5b101b 100644--- a/brain/knowledge/flows-execution/chat.md+++ b/brain/knowledge/flows-execution/chat.md@@ -31,6 +31,7 @@ A turn is kept alive / reclaimed by three separate mechanisms in `execute-agent-- **Stream idle watchdog** (`STREAM_IDLE_TIMEOUT_MS` 90s, in `streamChunksToClient`): aborts the turn if the drain-stream reader is silent 90s. It must be SUSPENDED while legitimate silent work is in flight — pending tool calls AND in-flight reasoning (`reasoning-start`→`reasoning-end`). **Reasoning-awareness was missing and caused the bug where long "thinking" on the Expert tier randomly aborted a healthy turn** (a >90s gap between reasoning deltas looked like a wedge). Backstop for a genuine mid-reasoning wedge is `MAX_TURN_WALL_CLOCK_MS` (20 min).### Gotchas+- **Chat has no fixed step cap.** A chat turn is bounded by credits (checked after every step), the runaway-token ceiling (`RUNAWAY_TURN_CONTEXT_MULTIPLE` × the context window), the identical-failure guard and the 2-hour `MAX_AGENT_TURN_WALL_CLOCK_MS`, so it finishes long jobs. A saved agent's author-set `maxSteps` is still honored on its last step: an agent with structured output is forced to call `TASK_COMPLETION_TOOL_NAME` so the flow gets a result, and any other agent runs with `toolChoice: 'none'` plus a wrap-up note, so it replies with what's done and what's left. Never a silent stop or an error.- **The streaming lock is per run, not per conversation.** A re-send can claim `activeRunId` before the first job reaches `getAgentConfig`. `agentHelpers.acquireStreamingLock` only lets the owning run set STREAMING; an older run gets `AGENT_RUN_SUPERSEDED` and the worker exits quietly, with no ERROR save and no events. A lock that ignored `runId` let the old run stream unowned while the new one was rejected as busy and marked ERROR, which is how users saw nothing after tapping a starter card twice.- **A failed turn used to leave status ERROR with an empty reply.** The error text only travelled as a websocket `ERROR` event, so a reload, another tab, or a user who was away saw a dead conversation (and often no user message either when the job died inside `getAgentConfig`, before it persists the turn). The `execute-agent-run.ts` catch now sends `failure: { message, userMessage? }` on the empty `saveAgentMessages` call and `conversation-rpc.ts` appends it to `uiMessages` as an assistant text reply; `use-chat.ts` shows the Retry banner when a loaded conversation is ERROR. Stale STREAMING to IDLE recovery, a worker that never picks the job up, and `assertAgentMessageRateLimitNotExceeded` still leave no reply of their own.- **A built flow ends with a "Turn it on?" card, never an auto-publish**: an audit found 40% of built-but-unpublished flows ended on "open it to review", some claiming LIVE. Chat never publishes on its own; only a yes to the card calls `ap_lock_and_publish` (`build_flow.md` "Turn it on?"). Pinned by the `build-then-ask-turn-it-on` fixture (`askedToTurnItOn`, `noLiveClaimWithoutPublish`). The live-claim check is a regex plus "was `ap_lock_and_publish` called", so it cannot see whether publish succeeded.diff --git a/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts b/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.tsindex 0340a57f9a7..b84d059a9cc 100644--- a/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts+++ b/packages/server/worker/src/lib/execute/jobs/ee/agent/run-agent-turn.ts@@ -1,12 +1,11 @@import { AIProviderName, ErrorCode, formatPieceError, isNil, isObject, isProviderBillingError, isTransientProviderError, spreadIfDefined, tryCatch, tryCatchSync } from '@activepieces/core-utils'import { agentAiUtils, ContentPartLike } from '@activepieces/server-utils'-import { AgentPhase, AgentRunSource, agentToolClassification, agentToolPhases, AI_PROVIDER_ENTITY_TYPES, aiProviderUtils, apErrorOf, CHAT_CREDITS_PER_TOOL_CALL, chatBilling, ChatToolCall, PersistedAgentPart } from '@activepieces/shared'-import { APICallError, generateText, isLoopFinished, isStepCount, LanguageModel, LanguageModelUsage, ModelMessage, NoSuchToolError, RetryError, StepResultPerformance, StopCondition, streamText, ToolExecutionOptions, ToolSet } from 'ai'+import { AgentPhase, AgentRunSource, agentToolClassification, agentToolPhases, AI_PROVIDER_ENTITY_TYPES, aiProviderUtils, apErrorOf, CHAT_CREDITS_PER_TOOL_CALL, chatBilling, ChatToolCall, PersistedAgentPart, TASK_COMPLETION_TOOL_NAME } from '@activepieces/shared'+import { APICallError, generateText, isLoopFinished, isStepCount, LanguageModel, LanguageModelUsage, ModelMessage, NoSuchToolError, RetryError, StepResultPerformance, StopCondition, streamText, ToolChoice, ToolExecutionOptions, ToolSet } from 'ai'const MAX_AUTO_CONTINUATIONS = 3const MAX_EMPTY_CONTINUATIONS = 2const MAX_STREAM_RETRIES = 1-const MAX_AGENT_STEPS = 50const MAX_IDENTICAL_TOOL_FAILURES = 2const IN_LOOP_COMPACTION_THRESHOLD = 0.6const RUNAWAY_TURN_CONTEXT_MULTIPLE = 90@@ -15,8 +14,11 @@ const USER_FAULT_STATUS_CODES = new Set([401, 403, 404])const MODEL_UNAVAILABLE_PATTERNS = [/\bis deprecated\b/i, /no longer (available|supported)/i, /\bmodel_not_found\b/i, /\bunknown model\b/i, /\bdecommissioned\b/i]const USER_CONFIG_ENTITY_TYPES = new Set<string>(Object.values(AI_PROVIDER_ENTITY_TYPES))const CONTINUE_NUDGE = '[system note — not from the user] Your previous response was cut off by the output token limit before it finished. Continue exactly where you stopped. If a tool call was cut off, re-issue it in FULL. Do not repeat content you already produced.'+const FINAL_STEP_NOTE = '[system note — not from the user] This is the last step of this run. Finish now with what you have: say what you finished and what is still left to do, and offer to carry on. Do not mention steps, tools or this note.'const EMPTY_OUTPUT_NUDGE = '[system note — not from the user] Your previous step produced no visible reply to the user. Continue the task now: either call the next tool, or write your reply to the user. Do not stop silently.'+const FINAL_STEP_MESSAGE: ModelMessage = { role: 'user', content: FINAL_STEP_NOTE }+export function decideLoopAction({ finishReason, producedVisibleOutput, continuations, emptyContinuations }: {finishReason: stringproducedVisibleOutput: boolean@@ -61,7 +63,7 @@ export async function runAgentTurn({ model, fastModel, provider, systemPrompt, m}const loopStopCondition = [...(Array.isArray(baseStopCondition) ? baseStopCondition : [baseStopCondition]),- isStepCount(stepCeiling ?? MAX_AGENT_STEPS),+ ...(isNil(stepCeiling) ? [] : [isStepCount(stepCeiling)]),creditsRanOut,]const guardedTools = wrapToolsWithFailureGuard({ tools, log })@@ -106,7 +108,7 @@ export async function runAgentTurn({ model, fastModel, provider, systemPrompt, m// providerOptions/model are supplied per-step by prepareStep (authoritative). A// call-level providerOptions would deep-merge into every step and leak the enabled// thinking budget back into the disabled first step.- prepareStep: ({ steps }) => {+ prepareStep: ({ steps, messages: currentMessages }) => {const lastStep = steps[steps.length - 1]const widened = lastStep?.toolCalls?.some((c) => agentToolPhases.isBuildOnlyTool(c.toolName))if (widened) {@@ -122,15 +124,28 @@ export async function runAgentTurn({ model, fastModel, provider, systemPrompt, m// thinking makes the model deliberate and fire ONE tool per step, serializing the// read-only lookups that should run as one parallel burst. Once a build-only tool// flips the phase to 'build', thinking comes back on for planning depth.- const disableThinking = isFirstStep || phaseState.phase === 'discovery'+ const isLastAllowedStep = !isNil(stepCeiling) && steps.length >= stepCeiling - 1+ const reportsStructuredOutput = allToolNames.includes(TASK_COMPLETION_TOOL_NAME)+ const forcesCompletion = isLastAllowedStep && reportsStructuredOutput+ const toolChoice: ToolChoice<ToolSet> | undefined = isLastAllowedStep+ ? (forcesCompletion ? { type: 'tool', toolName: TASK_COMPLETION_TOOL_NAME } : 'none')+ : undefined+ const disableThinking = isFirstStep || phaseState.phase === 'discovery' || forcesCompletionconst usesFastModel = isFirstStep && !isNil(fastModel)+ const phaseTools = agentToolPhases.activeToolsForPhase({ phase: phaseState.phase, allToolNames }).filter((name) => paidToolsAffordable || !chatBilling.isPaidTool(name))+ const activeTools = forcesCompletion ? [TASK_COMPLETION_TOOL_NAME] : phaseTools+ const boundedContext = boundContextForStep({ baseMessages: attemptMessages, steps, systemPrompt, provider })+ const stepContext = isLastAllowedStep+ ? { messages: [...(boundedContext.messages ?? currentMessages), FINAL_STEP_MESSAGE] }+ : boundedContextlastStepModelId = usesFastModel ? fastModelId ?? modelId : modelIdreturn {...(usesFastModel ? { model: fastModel } : {}),- activeTools: agentToolPhases.activeToolsForPhase({ phase: phaseState.phase, allToolNames }).filter((name) => paidToolsAffordable || !chatBilling.isPaidTool(name)),+ activeTools,+ ...spreadIfDefined('toolChoice', toolChoice),maxOutputTokens: disableThinking ? maxOutputTokensWithoutThinking : maxOutputTokens,providerOptions: agentAiUtils.buildProviderOptions({ provider, tier, modelId: lastStepModelId, disableThinking }),- ...boundContextForStep({ baseMessages: attemptMessages, steps, systemPrompt, provider }),+ ...stepContext,}},repairToolCall: async ({ toolCall, error }) => {diff --git a/packages/server/worker/test/lib/agent-eval/agent-turn-credits.test.ts b/packages/server/worker/test/lib/agent-eval/agent-turn-credits.test.tsindex 2b5467326c7..3d24b60827b 100644--- a/packages/server/worker/test/lib/agent-eval/agent-turn-credits.test.ts+++ b/packages/server/worker/test/lib/agent-eval/agent-turn-credits.test.ts@@ -1,5 +1,5 @@-import { AIProviderName } from '@activepieces/core-utils'-import { PersistedAgentPartType, PersistedToolCallStatus } from '@activepieces/shared'+import { AIProviderName, isNil, spreadIfDefined } from '@activepieces/core-utils'+import { AgentPhase, PersistedAgentPartType, PersistedToolCallStatus } from '@activepieces/shared'import { tool } from 'ai'import { convertArrayToReadableStream, MockLanguageModelV3 } from 'ai/test'import { describe, expect, it, vi } from 'vitest'@@ -50,12 +50,55 @@ describe('a turn that runs out of credits', () => {})})-async function runTurn({ search, creditsLeft, stepCeiling = 20, model = alwaysSearchingModel() }: {+describe('a turn with many steps', () => {+ it('ends a saved agent\'s step budget with a reply instead of a silent stop', async () => {+ const search = vi.fn(async () => SEARCH_RESULT)+ const model = searchUntilToldToAnswer({ searches: 10 })++ const turn = await runTurn({ search, creditsLeft: async () => 100, stepCeiling: 3, model, drainsStream: true })++ const toolChoices = model.doStreamCalls.map((call) => call.toolChoice?.type)+ expect(search).toHaveBeenCalledTimes(2)+ expect(toolChoices.at(-1)).toBe('none')+ expect(JSON.stringify(model.doStreamCalls.at(-1)?.prompt)).toContain('last step of this run')+ expect(turn.uiParts.at(-1)?.type).toBe(PersistedAgentPartType.TEXT)+ })++ it('makes a flow-step agent report its structured output on the last step, so the flow still gets a result', async () => {+ const capture = vi.fn()+ const model = searchUntilToldToAnswer({ searches: 10 })++ await runTurn({ search: async () => SEARCH_RESULT, creditsLeft: async () => 100, stepCeiling: 3, model, drainsStream: true, completion: capture, phase: 'build' })++ const thinkingPerCall = model.doStreamCalls.map((call) => JSON.stringify(call.providerOptions).includes('"enabled"'))+ const lastCall = model.doStreamCalls.at(-1)+ expect(lastCall?.toolChoice).toEqual({ type: 'tool', toolName: 'updateTaskStatus' })+ expect(thinkingPerCall.slice(1, -1).every(Boolean)).toBe(true)+ expect(thinkingPerCall.at(-1)).toBe(false)+ expect(capture).toHaveBeenCalledWith({ output: { summary: 'partial' } })+ })++ it('lets chat finish a long job, since credits, time and context already bound the turn', async () => {+ const search = vi.fn(async () => SEARCH_RESULT)++ await runTurn({ search, creditsLeft: async () => 1_000, stepCeiling: null, model: searchUntilToldToAnswer({ searches: 60 }), drainsStream: true })++ expect(search).toHaveBeenCalledTimes(60)+ })+})++async function runTurn({ search, creditsLeft, stepCeiling = 20, model = alwaysSearchingModel(), drainsStream = false, completion, phase = 'discovery' }: {search: () => Promise<unknown>creditsLeft: (pendingCredits: number) => Promise<number | null>- stepCeiling?: number+ stepCeiling?: number | nullmodel?: MockLanguageModelV3+ drainsStream?: boolean+ completion?: (input: unknown) => void+ phase?: AgentPhase}): ReturnType<typeof runAgentTurn> {+ const completionTools = isNil(completion)+ ? {}+ : { updateTaskStatus: tool({ description: 'report the result', inputSchema: z.object({ output: z.object({ summary: z.string() }) }), execute: async (input) => { completion(input); return SEARCH_RESULT } }) }return runAgentTurn({model,provider: AIProviderName.ANTHROPIC,@@ -64,14 +107,16 @@ async function runTurn({ search, creditsLeft, stepCeiling = 20, model = alwaysSetools: {ap_web_search: tool({ description: 'search the web', inputSchema: z.object({ query: z.string() }), execute: search }),ap_fetch_url: tool({ description: 'read a page', inputSchema: z.object({ url: z.string() }), execute: async () => SEARCH_RESULT }),+ ...completionTools,},- allToolNames: ['ap_web_search', 'ap_fetch_url'],+ allToolNames: ['ap_web_search', 'ap_fetch_url', ...Object.keys(completionTools)],tier: TIER,modelId: TIER.modelId,- phaseState: { phase: 'discovery' },+ phaseState: { phase },abortSignal: new AbortController().signal,log: SILENT_LOG,- stepCeiling,+ ...spreadIfDefined('stepCeiling', stepCeiling ?? undefined),+ ...(drainsStream ? { sinks: { drainStream: (result) => result.consumeStream() } } : {}),creditsLeft,})}@@ -96,6 +141,40 @@ function alwaysSearchingModel(): MockLanguageModelV3 {})}+function searchUntilToldToAnswer({ searches }: { searches: number }): MockLanguageModelV3 {+ let calls = 0+ return new MockLanguageModelV3({+ doStream: async ({ toolChoice }) => {+ calls+++ const answers = toolChoice?.type === 'none' || calls > searches+ if (toolChoice?.type === 'tool') {+ return {+ stream: convertArrayToReadableStream([+ { type: 'stream-start' as const, warnings: [] },+ { type: 'tool-call' as const, toolCallId: `call-${calls}`, toolName: toolChoice.toolName, input: '{"output":{"summary":"partial"}}' },+ { type: 'finish' as const, finishReason: 'tool-calls' as const, usage: { inputTokens: 1, outputTokens: 1, totalTokens: 2 } },+ ]),+ }+ }+ return {+ stream: convertArrayToReadableStream(answers+ ? [+ { type: 'stream-start' as const, warnings: [] },+ { type: 'text-start' as const, id: 'answer' },+ { type: 'text-delta' as const, id: 'answer', delta: 'Here is what I found so far.' },+ { type: 'text-end' as const, id: 'answer' },+ { type: 'finish' as const, finishReason: 'stop' as const, usage: { inputTokens: 1, outputTokens: 1, totalTokens: 2 } },+ ]+ : [+ { type: 'stream-start' as const, warnings: [] },+ { type: 'tool-call' as const, toolCallId: `call-${calls}`, toolName: 'ap_web_search', input: '{"query":"more"}' },+ { type: 'finish' as const, finishReason: 'tool-calls' as const, usage: { inputTokens: 1, outputTokens: 1, totalTokens: 2 } },+ ]),+ }+ },+ })+}+const SEARCH_RESULT = { content: [{ type: 'text', text: 'ok' }] }const TIER = { id: 'fast', thinkingBudget: 5_000, modelId: 'anthropic/claude-haiku-4.5' }