AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
fix(chat): ask before turning on the automations it builds (#15926)
Hazem Adel1e69fe0d97a0
diff --git a/brain/knowledge/flows-execution/chat.md b/brain/knowledge/flows-execution/chat.mdindex f5310b32fa3..ab6278b87cd 100644--- a/brain/knowledge/flows-execution/chat.md+++ b/brain/knowledge/flows-execution/chat.md@@ -31,6 +31,7 @@ A turn is kept alive / reclaimed by three separate mechanisms in `execute-agent-- **Stream idle watchdog** (`STREAM_IDLE_TIMEOUT_MS` 90s, in `streamChunksToClient`): aborts the turn if the drain-stream reader is silent 90s. It must be SUSPENDED while legitimate silent work is in flight — pending tool calls AND in-flight reasoning (`reasoning-start`→`reasoning-end`). **Reasoning-awareness was missing and caused the bug where long "thinking" on the Expert tier randomly aborted a healthy turn** (a >90s gap between reasoning deltas looked like a wedge). Backstop for a genuine mid-reasoning wedge is `MAX_TURN_WALL_CLOCK_MS` (20 min).### Gotchas+- **A built flow ends with a "Turn it on?" card, never an auto-publish**: an audit found 40% of built-but-unpublished flows ended on "open it to review", some claiming LIVE. Chat never publishes on its own; only a yes to the card calls `ap_lock_and_publish` (`build_flow.md` "Turn it on?"). Pinned by the `build-then-ask-turn-it-on` fixture (`askedToTurnItOn`, `noLiveClaimWithoutPublish`). The live-claim check is a regex plus "was `ap_lock_and_publish` called", so it cannot see whether publish succeeded.- **Server-managed connections**: the LLM never sees connection externalIds; `ap_execute_action` auto-fills them from a Redis store.- **Prompt-injection taint**: a per-turn `taintState` flips to `tainted` once the turn reads outside or user data (MCP tools, web/search/scrape, `ap_explore_data`, `ap_execute_action`, `ap_run_code`, `ap_list_across_projects`, configured tools). It forces the action-preview gate on non-read-only actions, blocks saved-agent edits, and makes `ap_remember` ask before saving (memory reaches every future conversation). It carries one message: each saved reply stores `tainted` (this reply's own reads, via `createTaintState`), and the next message starts from it. Deletes are confirmed always, not by taint.- **Write-check gate**: before a live `ap_test_flow`, `__flow_write_check` RPC flags write/destructive PIECE steps; read-only flows run ungated; gate fails open on RPC error.diff --git a/packages/server/api/src/assets/prompts/chat-system-prompt.md b/packages/server/api/src/assets/prompts/chat-system-prompt.mdindex fad24e413d9..905a718a3d5 100644--- a/packages/server/api/src/assets/prompts/chat-system-prompt.md+++ b/packages/server/api/src/assets/prompts/chat-system-prompt.md@@ -326,7 +326,7 @@ Every request starts with `<discovery>` — understand WHAT and WHY first.**One-time vs recurring — read the signal, don't default to recurring.** A one-time task and a scheduled automation are different builds; misreading the cadence wastes the whole turn. Decide from the wording:- **One-time** (→ `one_time_task`): an imperative to act *now* on data that already exists — "categorize *these* tickets", "send *this* now", "check my inbox", "score the rows in my sheet". No recurrence cue → treat as one-time.-- **Recurring** (→ `<automation_build>`): an explicit ongoing/triggered cue — "when/whenever/each time/every new …", "every day/hour", "on a schedule", "from now on".+- **Recurring** (→ `<automation_build>`): an explicit ongoing/triggered cue — "when/whenever/each time/every new …", "every day/hour", "on a schedule", "from now on", or the words "workflow", "automation", "monitor", "keep track of" used for something ongoing (an imperative to act once, like "track down this issue now", is still one-time). Build a flow for these; never answer with a one-off `ap_run_code` / `ap_web_search` or the showcase instead.- Only when the cadence is genuinely ambiguous, ask the single binary business question ("Run this once, or every time?") via `ap_show_questions` — never guess recurring. (Converting a one-time task into a recurring one later: `build_flow.md` → "Converting a one-time task".)- **Once it's recurring, guarantee it won't reprocess.** A recurring flow that reads data which persists between runs (a sheet, a table, an inbox, a record set) sees the SAME data again on the next run — so during discovery you MUST settle *what makes run N+1 skip what run N already handled*: a "new item"/dedup trigger instead of read-all, a processed-flag filter, delete/archive-after, or a stored high-water mark. Treating a recurring job as if it runs once — read-all → act → done, with no such mechanism — is the #1 silent logic bug (e.g. a weekly "summarize hours and tell me who to pay" that re-pays everyone every week because nobody was marked paid). This shapes the trigger and steps, so decide it now, not after building; the mechanisms are in `build_flow.md` → "Recurring flows must not reprocess".@@ -350,7 +350,7 @@ This applies ONLY to deliverables the user asked you to produce. Normal conversa1. **DISCOVER** — follow `<discovery>`: understand the goal, then INFER the business logic from context (their company/domain, their data via `ap_explore_data`, market practice). Don't ask the user to supply categories, routing, or thresholds. If it's recurring and reads persistent data, also settle how run N+1 avoids reprocessing run N's data (see `<decision_framework>`) — a build-shaping decision, not an afterthought.2. **RESEARCH (the 2-call happy path)** — `ap_research_pieces({pieceNames:[app], forIntent:"<what you're doing>"})` for the apps involved (missing app → `http_fallback`); read the returned `recommendedActions` + each action's AI hint to pick the right action in one shot. Then call `ap_get_piece_props({pieceName, actionName, auth})` ONCE with the connection — this resolves the dropdowns/dynamic sub-fields AND returns `requiredInputs` + a ready-to-run `exampleInput`. Don't call `ap_resolve_property_options` separately when that one `ap_get_piece_props(auth)` call already resolved the field. Fill the example, then execute.3. **HANDOFF** — write a one-line recap of what you'll build and go. Make sure each app has a connection the user selected (`ap_show_connection_picker`/`ap_show_connection_required`). Pick sensible context-grounded defaults for every open choice; only show a choice-only `ap_show_questions` for a single make-or-break business choice you truly can't infer. No separate approval step, no upfront questionnaire.-4. **BUILD** — `ap_set_phase('build')`, publish the live build plan with `ap_set_build_plan` (`phase: 'detecting'`, all steps `pending`), load `build_flow`, and execute it — keep the plan updated as you go and finish it with `phase: 'done'` + `flowId`. A short real-text line between build phases is welcome to keep the user in the loop (e.g. "Connections are set — wiring up the steps now"), but keep it light: the build card carries the detailed play-by-play, so don't narrate every step. When all steps are done and the link is shared, deliver the closing brief: what you built, the assumptions you made (each editable), and the obvious next improvements.+4. **BUILD** — `ap_set_phase('build')`, publish the live build plan with `ap_set_build_plan` (`phase: 'detecting'`, all steps `pending`), load `build_flow`, and execute it — keep the plan updated as you go and finish it with `phase: 'done'` + `flowId`. A short real-text line between build phases is welcome to keep the user in the loop (e.g. "Connections are set — wiring up the steps now"), but keep it light: the build card carries the detailed play-by-play, so don't narrate every step. When all steps are done and the link is shared, deliver the closing brief: what you built, the assumptions you made (each editable), and the obvious next improvements. Chat never publishes on its own: end every validated flow with one "Turn it on?" quick-reply card, and only a yes calls `ap_lock_and_publish` (see `build_flow` → "Turn it on?"). Never end on "open it to review", and never call a flow you just built live, running or active unless publish succeeded for it (for other flows, report the status the tools show).</automation_build><links>diff --git a/packages/server/api/src/assets/prompts/guides/build_flow.md b/packages/server/api/src/assets/prompts/guides/build_flow.mdindex bc032855e2b..ac2cb0f61e9 100644--- a/packages/server/api/src/assets/prompts/guides/build_flow.md+++ b/packages/server/api/src/assets/prompts/guides/build_flow.md@@ -111,7 +111,11 @@ Wire the **specific fields** a step consumes, never an entire upstream output. A- **Simple flows** (linear, no branches/loops): `ap_build_flow` → validate every step (below) → test for real with cases (below) → reflect (below) → `ap_manage_notes`.- **Flows with loops**: `ap_build_flow` supports nesting. For steps inside a loop, set `parentStepName` to the loop step's name and `stepLocationRelativeToParent` to `INSIDE_LOOP`. Steps that omit `parentStepName` are placed after the last top-level step (not inside the loop).- **Complex flows** (branches, routers, many steps): `ap_create_flow` → configure trigger → validate → for each action: `ap_add_step` → validate → test for real with cases (below) → reflect → `ap_manage_notes`.-- Share the flow link. The flow is a draft — do NOT auto-publish.+- Share the flow link, then finish per "Turn it on?" below. Never auto-publish, and never leave a validated flow ending on "open it to review".++## Turn it on? — the one way every built flow ends+Chat NEVER publishes on its own. A flow only runs once published, so once it validates (and passes its test, when a test ran), end with exactly one `ap_show_quick_replies` (or `ap_show_questions`) card: "Turn it on?" with chips like "Turn it on" / "Not yet". Only a yes publishes: call `ap_lock_and_publish({flowId})` right away, then say it is live. "Not yet" leaves it a draft.+**Never say a flow you just built is live, running, active or turned on unless `ap_lock_and_publish` succeeded for it.** If publish returned an error, say so and fix it. Until then call it "a draft, not running yet". For a flow you did not just build, report the status the tools show (`ap_list_flows` shows it), never a guess.**After `ap_build_flow`** it creates the skeleton but does NOT validate configs or field mappings. You MUST: (1) `ap_validate_step_config` on the trigger and each step, (2) fix any errors with `ap_update_step`/`ap_update_trigger`, (3) `ap_validate_flow` to confirm all steps are valid.diff --git a/packages/server/worker/test/lib/agent-eval/core/fixture.ts b/packages/server/worker/test/lib/agent-eval/core/fixture.tsindex bafa8f9e793..4ddc67a3740 100644--- a/packages/server/worker/test/lib/agent-eval/core/fixture.ts+++ b/packages/server/worker/test/lib/agent-eval/core/fixture.ts@@ -8,6 +8,9 @@ export type ChatEvalAssertion =| { type: 'calledBefore', a: string, b: string }| { type: 'reachedToolWithin', toolName: string, n: number }| { type: 'maxQuestionCards', n: number, toolNames?: string[] }+ | { type: 'neverCalledTool', toolName: string }+ | { type: 'askedToTurnItOn' }+ | { type: 'noLiveClaimWithoutPublish' }export type ChatEvalJudgeDimension = {dimension: stringdiff --git a/packages/server/worker/test/lib/agent-eval/core/transcript-assertions.test.ts b/packages/server/worker/test/lib/agent-eval/core/transcript-assertions.test.tsindex 6ee15cfa0e0..e514f6c77d8 100644--- a/packages/server/worker/test/lib/agent-eval/core/transcript-assertions.test.ts+++ b/packages/server/worker/test/lib/agent-eval/core/transcript-assertions.test.ts@@ -131,3 +131,32 @@ describe('transcriptAssertions.maxQuestionCards', () => {expect(transcriptAssertions.maxQuestionCards(result, 1, ['ap_show_questions']).pass).toBe(true)})})++describe('transcriptAssertions publish behaviour', () => {+ const build = toolCall({ toolName: 'ap_build_flow', order: 0, phase: 'build' })+ const publish = toolCall({ toolName: 'ap_lock_and_publish', order: 1, phase: 'build' })++ it('fails when a flow is built and ends on open it to review, or is published without asking', () => {+ const review = makeResult({ toolCalls: [build], uiParts: [textPart('Open it to review.')] })+ expect(transcriptAssertions.askedToTurnItOn(review).pass).toBe(false)+ expect(transcriptAssertions.askedToTurnItOn(makeResult({ toolCalls: [build, publish] })).pass).toBe(false)+ })++ it('passes when a card is shown before any publish', () => {+ const card = { ...toolCall({ toolName: 'ap_show_quick_replies', order: 1 }), input: { replies: ['Turn it on', 'Not yet'] } }+ const unrelated = { ...toolCall({ toolName: 'ap_show_quick_replies', order: 1 }), input: { replies: ['Open it to review'] } }+ expect(transcriptAssertions.askedToTurnItOn(makeResult({ toolCalls: [build, card] })).pass).toBe(true)+ expect(transcriptAssertions.askedToTurnItOn(makeResult({ toolCalls: [build, unrelated] })).pass).toBe(false)+ })++ it('fails a live claim without a publish call and allows it after one', () => {+ const uiParts = [textPart('Your flow is live and will run every morning.')]+ expect(transcriptAssertions.noLiveClaimWithoutPublish(makeResult({ uiParts })).pass).toBe(false)+ expect(transcriptAssertions.noLiveClaimWithoutPublish(makeResult({ uiParts, toolCalls: [build, publish] })).pass).toBe(true)+ })++ it('flags a forbidden one-off tool', () => {+ const result = makeResult({ toolCalls: [toolCall({ toolName: 'ap_run_code', order: 0 })] })+ expect(transcriptAssertions.neverCalledTool(result, 'ap_run_code').pass).toBe(false)+ })+})diff --git a/packages/server/worker/test/lib/agent-eval/core/transcript-assertions.ts b/packages/server/worker/test/lib/agent-eval/core/transcript-assertions.tsindex 6370667c232..b101fdea4f3 100644--- a/packages/server/worker/test/lib/agent-eval/core/transcript-assertions.ts+++ b/packages/server/worker/test/lib/agent-eval/core/transcript-assertions.ts@@ -10,6 +10,16 @@ const ASKED_HOW_PATTERNS = [const DEFAULT_QUESTION_CARD_PATTERN = /question|quick_repl/i+const PUBLISH_TOOL_NAME = 'ap_lock_and_publish'++const TURN_IT_ON_PATTERN = /turn (it )?on|publish|go live|activate|enable/i++const LIVE_CLAIM_PATTERNS = [+ /\b(is|it's|it is|now|already|currently) (live|running|active|enabled|published)\b/i,+ /\b(is|are) now (on|turned on)\b/i,+ /\bgoing live\b/i,+]+function assistantText(result: AgentTurnResult): string {return result.uiParts.filter((part): part is Extract<PersistedAgentPart, { type: PersistedAgentPartType.TEXT }> => part.type === PersistedAgentPartType.TEXT)@@ -78,6 +88,31 @@ function maxQuestionCards(result: AgentTurnResult, n: number, toolNames?: string: { pass: false, reason: `${count} question card(s) shown (> ${n})` }}+function neverCalledTool(result: AgentTurnResult, toolName: string): AssertionOutcome {+ return result.toolCalls.some((call) => call.toolName === toolName)+ ? { pass: false, reason: `"${toolName}" was called` }+ : { pass: true, reason: `"${toolName}" was never called` }+}++function askedToTurnItOn(result: AgentTurnResult): AssertionOutcome {+ const cardOrder = result.toolCalls.find((call) => DEFAULT_QUESTION_CARD_PATTERN.test(call.toolName) && TURN_IT_ON_PATTERN.test(JSON.stringify(call.input)))?.order+ if (cardOrder === undefined) {+ return { pass: false, reason: 'no "Turn it on?" card was shown' }+ }+ const publishedBeforeAsking = result.toolCalls.some((call) => call.toolName === PUBLISH_TOOL_NAME && call.order < cardOrder)+ return publishedBeforeAsking+ ? { pass: false, reason: 'flow was published before the user said yes' }+ : { pass: true, reason: 'a "Turn it on?" card was shown before any publish' }+}++function noLiveClaimWithoutPublish(result: AgentTurnResult): AssertionOutcome {+ const published = result.toolCalls.some((call) => call.toolName === PUBLISH_TOOL_NAME)+ const matched = LIVE_CLAIM_PATTERNS.find((pattern) => pattern.test(assistantText(result)))+ return !published && matched+ ? { pass: false, reason: `claimed the flow is live (${matched}) without calling ${PUBLISH_TOOL_NAME}` }+ : { pass: true, reason: 'no live claim without a publish' }+}+function runAssertion(result: AgentTurnResult, assertion: ChatEvalAssertion): AssertionResult {switch (assertion.type) {case 'neverAskedHow':@@ -92,6 +127,12 @@ function runAssertion(result: AgentTurnResult, assertion: ChatEvalAssertion): Asreturn { type: assertion.type, ...reachedToolWithin(result, assertion.toolName, assertion.n) }case 'maxQuestionCards':return { type: assertion.type, ...maxQuestionCards(result, assertion.n, assertion.toolNames) }+ case 'neverCalledTool':+ return { type: assertion.type, ...neverCalledTool(result, assertion.toolName) }+ case 'askedToTurnItOn':+ return { type: assertion.type, ...askedToTurnItOn(result) }+ case 'noLiveClaimWithoutPublish':+ return { type: assertion.type, ...noLiveClaimWithoutPublish(result) }}}@@ -102,6 +143,9 @@ export const transcriptAssertions = {calledBefore,reachedToolWithin,maxQuestionCards,+ neverCalledTool,+ askedToTurnItOn,+ noLiveClaimWithoutPublish,runAssertion,}diff --git a/packages/server/worker/test/lib/agent-eval/fixtures/build-then-ask-turn-it-on.json b/packages/server/worker/test/lib/agent-eval/fixtures/build-then-ask-turn-it-on.jsonnew file mode 100644index 00000000000..e7503f24514--- /dev/null+++ b/packages/server/worker/test/lib/agent-eval/fixtures/build-then-ask-turn-it-on.json@@ -0,0 +1,30 @@+{+ "id": "build-then-ask-turn-it-on",+ "description": "After a validated, tested flow is built, chat ends with one 'Turn it on?' card and never auto-publishes or claims the flow is live. The build sits in initialMessages because the eval runner has no flow-building tools.",+ "kind": "capability",+ "initialMessages": [+ { "role": "user", "content": "Every weekday at 9am, post a reminder in #standup on Slack asking the team for updates." },+ { "role": "assistant", "content": "Built it: \"Standup reminder\" (flow flow_standup_1) runs on a weekday 9am schedule and posts to #standup. I validated every step and ran a test that posted the message successfully. The flow is a draft and has not been published." }+ ],+ "userTurns": [+ "Nice, thanks."+ ],+ "recordedToolCalls": [],+ "model": {+ "provider": "openrouter",+ "modelId": "anthropic/claude-sonnet-4.6",+ "tier": { "id": "balanced", "thinkingBudget": 2000, "modelId": "anthropic/claude-sonnet-4.6" }+ },+ "assertions": [+ { "type": "neverCutOff" },+ { "type": "askedToTurnItOn" },+ { "type": "noLiveClaimWithoutPublish" }+ ],+ "judge": [+ {+ "dimension": "ends_with_turn_it_on_card",+ "rubric": "A flow was built, validated and tested but is unpublished. PASS if the assistant asks once whether to turn it on (a single 'Turn it on?' choice) without publishing it first. FAIL if it publishes without the user saying yes, ends by telling the user to open or review the flow, or says the flow is live, running or active.",+ "expectedLabel": "pass"+ }+ ]+}diff --git a/packages/server/worker/test/lib/agent-eval/fixtures/intent-automation-builds-flow.json b/packages/server/worker/test/lib/agent-eval/fixtures/intent-automation-builds-flow.jsonnew file mode 100644index 00000000000..6e6791bcb25--- /dev/null+++ b/packages/server/worker/test/lib/agent-eval/fixtures/intent-automation-builds-flow.json@@ -0,0 +1,28 @@+{+ "id": "intent-automation-builds-flow",+ "description": "A stated recurring or automation intent (workflow, every day, track) must lead to a built flow, not a one-off code run or web search.",+ "kind": "capability",+ "initialMessages": [],+ "userTurns": [+ "I want a workflow that fetches the release notes of our dependencies every Monday and posts them to Slack."+ ],+ "recordedToolCalls": [],+ "model": {+ "provider": "openrouter",+ "modelId": "anthropic/claude-sonnet-4.6",+ "tier": { "id": "balanced", "thinkingBudget": 2000, "modelId": "anthropic/claude-sonnet-4.6" }+ },+ "assertions": [+ { "type": "neverCutOff" },+ { "type": "neverCalledTool", "toolName": "ap_run_code" },+ { "type": "neverCalledTool", "toolName": "ap_show_showcase" },+ { "type": "reachedToolWithin", "toolName": "ap_set_phase", "n": 10 }+ ],+ "judge": [+ {+ "dimension": "builds_flow_not_one_off",+ "rubric": "The user asked for a workflow with a recurring cadence. PASS if the assistant commits to building an automation: researching pieces, asking for a needed connection, or calling the build tools. FAIL if it answers with a one-off lookup, a web search summary or a code run, or only shows a showcase of what it can do without building.",+ "expectedLabel": "pass"+ }+ ]+}