Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
Make Temporal Stop and Resume durable across execution trees
trohitg516ecacad52d
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex 65e9ad5d..f23b758b 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -30,7 +30,14 @@ At a glance:[WorkflowService](server/services/workflow.py) is a thin facade that routes each run through Temporal when available, falling back to a plain sequential walk otherwise. Every run has an isolated `ExecutionContext` with no shared global state. Nodes are scheduled continuously — when any node completes, its newly-ready dependents start immediately (`FIRST_COMPLETED` pattern) instead of waiting for a whole layer to finish.-Deep dives: [DESIGN.md](docs-internal/DESIGN.md) - [TEMPORAL_ARCHITECTURE.md](docs-internal/TEMPORAL_ARCHITECTURE.md) - [event_framework.md](docs-internal/event_framework.md)+Newly started Temporal generations use revision-ordered cooperative control:+Stop closes admission across the execution tree, admitted work finishes with+its bookkeeping, and Resume releases the same pending continuation. Signals+queue events, completed Updates acknowledge control and root membership, and+Queries expose read-only state. Preserve the legacy/version-marker command+paths when editing Workflow code; validate them with native history replay.++Deep dives: [DESIGN.md](docs-internal/DESIGN.md) - [TEMPORAL_ARCHITECTURE.md](docs-internal/TEMPORAL_ARCHITECTURE.md) - [temporal-workflow-control.md](docs-internal/temporal-workflow-control.md) - [event_framework.md](docs-internal/event_framework.md)## AI Agent System@@ -144,6 +151,11 @@ The diagram above shows the full lifecycle of a workflow node: one self-containe- **Behavioral tests** live per category in `server/tests/nodes/test_<category>.py`, driven through `NodeTestHarness` ([server/tests/nodes/_harness.py](server/tests/nodes/_harness.py)) — it executes any node via `NodeExecutor` with mocked services and asserts the result envelope.- **Import sanity:** `uv run pytest --collect-only` (from `server/`) is the live plugin-count invariant — it fails if any plugin errors at import.- **Credential tests** follow the numbered-invariant style documented in [server/tests/credentials/README.md](server/tests/credentials/README.md).+- **Temporal control tests** include real SDK workflows with stub Activities,+ worker restart, Continue-As-New and captured-history replay. Run the+ [control verification commands](docs-internal/temporal-workflow-control.md#verification-and-operations)+ when changing admission gates, root enrollment, event queues or messaging.+ Replay hooks must remain deterministic and unchanged while replaying.- Run everything: `uv run pytest` from `server/`, `bun run --filter react-flow-client test` from the repo root, `uv run pytest cli/tests` from the repo root.- **Desktop shell:** from `desktop/`, `bun run typecheck && bun run test && bun run test:invariants` (the last needs `bun run stage` first), and `bun run build && bun run test:e2e` for the Playwright Electron smoke. In an editor-hosted terminal unset `ELECTRON_RUN_AS_NODE` before launching Electron by hand.@@ -179,6 +191,8 @@ Full setup and scripts reference: [SETUP.md](docs-internal/SETUP.md) - [SCRIPTS.|---|---|| [DESIGN.md](docs-internal/DESIGN.md) | Execution engine architecture, design patterns, execution modes || [TEMPORAL_ARCHITECTURE.md](docs-internal/TEMPORAL_ARCHITECTURE.md) | Distributed execution via Temporal activities |+| [temporal-workflow-control.md](docs-internal/temporal-workflow-control.md) | Versioned Stop/drain/Resume, execution roots, messaging, event accumulation, compatibility and verification |+| [chat_protocol.md](docs-internal/chat_protocol.md) | Tracked chat runs, controlled Stop/Resume, subscriptions, legacy Stop and watchdog behavior || [workflow-schema.md](docs-internal/workflow-schema.md) | Workflow JSON schema and node catalog (live count = `len(services.node_registry.NODE_METADATA)` after importing `nodes`) || [ROADMAP.md](docs-internal/ARCHIVE/ROADMAP.md) | *Archived* status snapshot; current state is in DESIGN.md and the Temporal docs || [SETUP.md](docs-internal/SETUP.md) | Development environment setup |diff --git a/README.md b/README.mdindex e66704b1..d646f7c8 100644--- a/README.md+++ b/README.md@@ -135,7 +135,7 @@ Nothing here is a black box. What an employee learns lives in its memory, its no- **Speech and translation**, so employees can listen, talk, and work in other languages.- **A local Android phone workspace** on Windows x64: set up a persistent emulator, watch it live, use it yourself, or let an AI agent use the **Android** tool. Optional setup requires SDK license acceptance; iOS is not yet supported. [Setup, controls and troubleshooting](docs/mobile-workspace.md).- **78 skills** that ship ready to use, and a place to drop your own.-- **Built to keep running.** Employees survive restarts, pause and resume from Home or the canvas, and catch up on missed schedules. Your API keys are stored encrypted on your machine. Add a login for shared or cloud use; one command deploys to Google Cloud.+- **Built to keep running.** Employees survive restarts and catch up on missed schedules. For newly started Temporal generations, Stop lets current work finish, retains its result, and Resume continues with the next pending action. Home, chat and canvas use the same control. [Stop and Resume behavior](docs-internal/temporal-workflow-control.md). Your API keys are stored encrypted on your machine. Add a login for shared or cloud use; one command deploys to Google Cloud.- **A canvas you will want to look at**, with 12 visual themes.## For Developersdiff --git a/client/src/components/ui/CommandPaletteHost.tsx b/client/src/components/ui/CommandPaletteHost.tsxindex fd7e75c5..f167ed41 100644--- a/client/src/components/ui/CommandPaletteHost.tsx+++ b/client/src/components/ui/CommandPaletteHost.tsx@@ -105,7 +105,7 @@ export const buildWorkflowLifecycleCommands = (if (handlers.workflowControl.can_pause || state === 'pausing') {commands.push({id: 'run.pause',- label: state === 'pausing' ? 'Retry Pause Workflow' : 'Pause Workflow',+ label: state === 'pausing' ? 'Stopping Workflow…' : 'Stop Workflow',group: 'Run',icon: Pause,onRun: handlers.pause,diff --git a/client/src/components/ui/TopToolbar.tsx b/client/src/components/ui/TopToolbar.tsxindex 646459af..bfe3a7cb 100644--- a/client/src/components/ui/TopToolbar.tsx+++ b/client/src/components/ui/TopToolbar.tsx@@ -140,9 +140,9 @@ const TopToolbar: React.FC<TopToolbarProps> = ({const canPauseWorkflow = workflowControl.can_pause || workflowControl.state === 'pausing';const canResumeWorkflow = workflowControl.can_resume || workflowControl.state === 'resuming';const isRetryingReset = workflowControl.state === 'resetting';- const controlTransitionLabel = (- workflowControlPending?.state ?? workflowControl.state- ).replace('_', ' ');+ const transitionState = workflowControlPending?.state ?? workflowControl.state;+ const controlTransitionLabel = transitionState === 'pausing' ? 'Stopping…'+ : transitionState === 'resuming' ? 'Resuming…' : transitionState.replace('_', ' ');// Global Model Selector stateconst { getValidatedAiProviders, saveGlobalModel, isConnected: apiKeysConnected } = useApiKeys();@@ -466,11 +466,11 @@ const TopToolbar: React.FC<TopToolbarProps> = ({intent="stop"onClick={onPauseWorkflow}title={workflowControl.state === 'pausing'- ? 'Retry the interrupted pause transition'- : 'Pause new workflow scheduling after in-flight work finishes'}+ ? 'Check progress of the Stop request'+ : 'Stop after current work finishes; Resume continues from the next pending action'}><Pause className="h-3 w-3 fill-current" />- {workflowControl.state === 'pausing' ? 'Retry Pause' : 'Pause'}+ {workflowControl.state === 'pausing' ? 'Stopping…' : 'Stop'}</ActionButton>) : isAuthoritativeTransition ? (<ActionButtondiff --git a/client/src/components/ui/__tests__/CommandPaletteHost.workflowControl.test.ts b/client/src/components/ui/__tests__/CommandPaletteHost.workflowControl.test.tsindex 7a9d7780..12347509 100644--- a/client/src/components/ui/__tests__/CommandPaletteHost.workflowControl.test.ts+++ b/client/src/components/ui/__tests__/CommandPaletteHost.workflowControl.test.ts@@ -62,7 +62,7 @@ describe('command palette workflow lifecycle capabilities', () => {});it.each([- ['pausing' as const, 'run.pause', 'Retry Pause Workflow'],+ ['pausing' as const, 'run.pause', 'Stopping Workflow…'],['resuming' as const, 'run.resume', 'Retry Resume Workflow'],['resetting' as const, 'run.reset', 'Retry Reset Workflow Execution'],])('makes an authoritative %s transition retryable', (state, id, label) => {diff --git a/client/src/contexts/WebSocketContext.tsx b/client/src/contexts/WebSocketContext.tsxindex 457d42a9..cbb86496 100644--- a/client/src/contexts/WebSocketContext.tsx+++ b/client/src/contexts/WebSocketContext.tsx@@ -162,6 +162,8 @@ export interface WorkflowControlStatus {data_scope_id?: string | null;controller_workflow_id?: string | null;controller_run_id?: string | null;+ execution_control_version?: number;+ last_resumed_at?: string | null;state: WorkflowControlState;revision: number;workspace_epoch?: number;@@ -530,6 +532,7 @@ interface WebSocketContextValue {getWorkflowStatus: (workflowId: string) => Promise<{ executing: boolean }>;startWorkflow: (workflowId: string, nodes: any[], edges: any[], sessionId?: string, expectedRevision?: number) => Promise<WorkflowStartResult>;pauseWorkflow: (workflowId: string, expectedRevision: number) => Promise<WorkflowControlStatus>;+ stopChatRun: (workflowId: string, expectedRevision: number, runId: string) => Promise<WorkflowControlStatus>;resumeWorkflow: (workflowId: string, expectedRevision: number) => Promise<WorkflowControlStatus>;resetWorkflow: (workflowId: string, expectedRevision: number) => Promise<WorkflowControlStatus>;getWorkflowControlStatus: (workflowId: string) => Promise<WorkflowControlStatus>;@@ -704,6 +707,7 @@ type WorkflowControlMutationRequest =| 'start_workflow'| 'start_employee'| 'pause_workflow'+ | 'stop_chat_run'| 'resume_workflow'| 'reset_workflow';@@ -716,6 +720,7 @@ const WORKFLOW_CONTROL_PENDING_BY_REQUEST: Record<// loads the graph; the editor's canvas is not involved).start_employee: { action: 'start', state: 'starting' },pause_workflow: { action: 'pause', state: 'pausing' },+ stop_chat_run: { action: 'pause', state: 'pausing' },resume_workflow: { action: 'resume', state: 'resuming' },reset_workflow: { action: 'reset', state: 'resetting' },};@@ -765,6 +770,7 @@ export interface WebSocketActions {* refuses while an app it needs is not connected. */startEmployee: (workflowId: string, expectedRevision: number) => Promise<WorkflowControlStatus>;pauseWorkflow: WebSocketContextValue['pauseWorkflow'];+ stopChatRun: WebSocketContextValue['stopChatRun'];resumeWorkflow: WebSocketContextValue['resumeWorkflow'];resetWorkflow: WebSocketContextValue['resetWorkflow'];getWorkflowControlStatus: WebSocketContextValue['getWorkflowControlStatus'];@@ -2982,6 +2988,9 @@ export const WebSocketProvider: React.FC<{ children: React.ReactNode }> = ({ chiconst pauseWorkflowAsync = useCallback((workflowId: string, expectedRevision: number) =>controlMutation('pause_workflow', workflowId, { expected_revision: expectedRevision }), [controlMutation]);+ const stopChatRunAsync = useCallback((workflowId: string, expectedRevision: number, runId: string) =>+ controlMutation('stop_chat_run', workflowId, { expected_revision: expectedRevision, run_id: runId }), [controlMutation]);+const resumeWorkflowAsync = useCallback((workflowId: string, expectedRevision: number) =>controlMutation('resume_workflow', workflowId, { expected_revision: expectedRevision }), [controlMutation]);@@ -3619,13 +3628,14 @@ export const WebSocketProvider: React.FC<{ children: React.ReactNode }> = ({ chistartWorkflow: startWorkflowAsync,startEmployee: startEmployeeAsync,pauseWorkflow: pauseWorkflowAsync,+ stopChatRun: stopChatRunAsync,resumeWorkflow: resumeWorkflowAsync,resetWorkflow: resetWorkflowAsync,getWorkflowControlStatus: getWorkflowControlStatusAsync,getWorkflowStatus: getWorkflowStatusAsync,}), [isConnected, isReady, sendRequest, addEventListener,- startWorkflowAsync, startEmployeeAsync, pauseWorkflowAsync,+ startWorkflowAsync, startEmployeeAsync, pauseWorkflowAsync, stopChatRunAsync,resumeWorkflowAsync, resetWorkflowAsync, getWorkflowControlStatusAsync,getWorkflowStatusAsync,]);@@ -3699,6 +3709,7 @@ export const WebSocketProvider: React.FC<{ children: React.ReactNode }> = ({ chigetWorkflowStatus: getWorkflowStatusAsync,startWorkflow: startWorkflowAsync,pauseWorkflow: pauseWorkflowAsync,+ stopChatRun: stopChatRunAsync,resumeWorkflow: resumeWorkflowAsync,resetWorkflow: resetWorkflowAsync,getWorkflowControlStatus: getWorkflowControlStatusAsync,@@ -3760,7 +3771,7 @@ export const WebSocketProvider: React.FC<{ children: React.ReactNode }> = ({ chicancelEventWaitAsync,deployWorkflowAsync, cancelDeploymentAsync, getDeploymentStatusAsync,cancelExecutionAsync, getWorkflowStatusAsync,- startWorkflowAsync, pauseWorkflowAsync, resumeWorkflowAsync, resetWorkflowAsync,+ startWorkflowAsync, pauseWorkflowAsync, stopChatRunAsync, resumeWorkflowAsync, resetWorkflowAsync,getWorkflowControlStatusAsync, getTeamTaskTraceAsync,executeAiNodeAsync, getAiModelsAsync,validateApiKeyAsync, getStoredApiKeyAsync, saveApiKeyAsync, deleteApiKeyAsync,diff --git a/client/src/contexts/__tests__/webSocketLifecycle.test.tsx b/client/src/contexts/__tests__/webSocketLifecycle.test.tsxindex 19f02c30..d5b02e54 100644--- a/client/src/contexts/__tests__/webSocketLifecycle.test.tsx+++ b/client/src/contexts/__tests__/webSocketLifecycle.test.tsx@@ -116,6 +116,31 @@ describe('WebSocket recovery and ownership', () => {expect(clear).not.toHaveBeenCalled();});+ it('uses the generation control protocol for chat Stop and reconciles a lost acknowledgement', async () => {+ await mount();+ await open();+ const running = { workflow_id: 'chat', state: 'running', revision: 3, generation: 1,+ root_execution_id: 'g1', execution_control_version: 1, in_flight_count: 2 };+ act(() => latest().message({ type: 'workflow_control_status', data: running }));+ let result!: Promise<unknown>;+ act(() => { result = actions.stopChatRun('chat', 3, 'r1'); });+ const request = latest().sent.find(item => item.type === 'stop_chat_run')!;+ expect(request).toMatchObject({ workflow_id: 'chat', run_id: 'r1', expected_revision: 3 });+ expect(request.idempotency_key).toBeTruthy();+ await act(async () => {+ latest().message({ request_id: request.request_id, success: false, error: 'workflow_control_transition_pending',+ status: { ...running, state: 'pausing', revision: 4 } });+ });+ const resync = latest().sent.filter(item => item.type === 'get_workflow_control_status').at(-1)!;+ expect(resync).toBeDefined();+ await act(async () => {+ latest().message({ request_id: resync.request_id, success: true,+ status: { ...running, state: 'paused', revision: 5, in_flight_count: 0 } });+ });+ await expect(result).resolves.toMatchObject({ state: 'paused', revision: 5 });+ expect(latest().sent.filter(item => item.type === 'stop_chat_run')).toHaveLength(1);+ });+it.each([false, true])('confirms an exact phone reset ID only after cleanup ends (resetting=%s)', async (workspaceResetting) => {await mount();await open();diff --git a/client/src/features/chat/ChatPane.tsx b/client/src/features/chat/ChatPane.tsxindex da58d2ea..d39bc943 100644--- a/client/src/features/chat/ChatPane.tsx+++ b/client/src/features/chat/ChatPane.tsx@@ -60,6 +60,7 @@ import {type DecideInput,} from './data/approvals';import { useConversation } from './data/conversation';+import { useChatWorkflowControl, useResumeChatGeneration } from './data/control';import {useEditChatMessage,useRegenerateChatReply,@@ -83,6 +84,12 @@ import { branchRefusalText, feedbackThanks } from './turns/runCopy';export function ChatPane({ host, ref }: { host: ChatHost; ref?: Ref<ChatPaneHandle> }) {const { sessionId, scope, persona, composer, notify, onSendRefused, compact = false } = host;const { thread, turns, lane } = useConversation(sessionId, scope);+ const workflowId = sessionId === 'default' ? null : sessionId;+ const control = useChatWorkflowControl(workflowId, lane);+ const controlled = control?.execution_control_version === 1;+ const suspended = controlled && control.state === 'paused';+ const resuming = controlled && control.state === 'resuming';+ const resume = useResumeChatGeneration(control, () => notify('Couldn’t resume. Try again.', 'error'));const boxRef = useRef<HTMLTextAreaElement>(null);useImperativeHandle(ref, () => ({ focusComposer: () => boxRef.current?.focus() }), []);@@ -94,7 +101,7 @@ export function ChatPane({ host, ref }: { host: ChatHost; ref?: Ref<ChatPaneHand[notify, onSendRefused],);const send = useSendChatMessage(sessionId, scope, onRefused);- const stop = useStopChatRun(sessionId, () => notify('Couldn’t stop the reply. Try again.', 'error'));+ const stop = useStopChatRun(sessionId, () => notify('Couldn’t stop the reply. Try again.', 'error'), control);// Changing the conversation: edit, try again, another version, a rating.const [editingId, setEditingId] = useState<string | null>(null);@@ -121,7 +128,7 @@ export function ChatPane({ host, ref }: { host: ChatHost; ref?: Ref<ChatPaneHandconst changing = edit.isPending || regenerate.isPending || switchBranch.isPending;const busy = send.isPending || lane !== null || changing;- const stopping = lane?.state === 'stopping' || stop.isPending;+ const stopping = lane?.state === 'stopping' || stop.isPending || (controlled && control.state === 'pausing');const editMutate = edit.mutate;const regenerateMutate = regenerate.mutate;@@ -174,7 +181,7 @@ export function ChatPane({ host, ref }: { host: ChatHost; ref?: Ref<ChatPaneHand};const stopAnswer = () => {- if (lane && !stopping) stop.mutate(lane);+ if (lane && !stopping && !suspended && !resuming) stop.mutate(lane);};// What an interface's buttons do. Read through a ref, so the handlers an@@ -230,7 +237,6 @@ export function ChatPane({ host, ref }: { host: ChatHost; ref?: Ref<ChatPaneHand// Drafts waiting for the owner: on the replies that made them, and the// rest (the employee's own work) after the conversation.- const workflowId = sessionId === 'default' ? null : sessionId;useApprovalEvents();const approvalsQuery = useChatApprovals(workflowId);const approvals = approvalsQuery.data ?? NO_APPROVALS;@@ -288,7 +294,7 @@ export function ChatPane({ host, ref }: { host: ChatHost; ref?: Ref<ChatPaneHandif (sendNewestDraft()) event.preventDefault();return;}- if (event.key !== 'Escape' || !lane || stopping) return;+ if (event.key !== 'Escape' || !lane || stopping || suspended || resuming) return;event.preventDefault();stopAnswer();};@@ -439,7 +445,9 @@ export function ChatPane({ host, ref }: { host: ChatHost; ref?: Ref<ChatPaneHandready={Boolean(thread.data)}busy={busy}onSend={submit}- onStop={lane ? stopAnswer : undefined}+ onStop={lane && !suspended && !resuming ? stopAnswer : undefined}+ onResume={suspended || resuming ? () => resume.mutate() : undefined}+ resuming={resume.isPending || resuming}stopping={stopping}compact={compact}boxRef={boxRef}diff --git a/client/src/features/chat/__tests__/chatPane.test.tsx b/client/src/features/chat/__tests__/chatPane.test.tsxindex 1bf24492..8ff65cab 100644--- a/client/src/features/chat/__tests__/chatPane.test.tsx+++ b/client/src/features/chat/__tests__/chatPane.test.tsx@@ -12,16 +12,21 @@ import { act, fireEvent, render, screen, waitFor } from '@testing-library/react'import { QueryClient, QueryClientProvider } from '@tanstack/react-query';const sendRequest = vi.fn();+const getWorkflowControlStatus = vi.fn();+const resumeWorkflow = vi.fn();+const stopChatRun = vi.fn();vi.mock('@/contexts/WebSocketContext', async (importOriginal) => ({...(await importOriginal<typeof import('@/contexts/WebSocketContext')>()),- useWebSocketActions: () => ({ sendRequest, isReady: true, addEventListener: () => () => {} }),+ useWebSocketActions: () => ({ sendRequest, getWorkflowControlStatus, resumeWorkflow, stopChatRun, isReady: true, addEventListener: () => () => {} }),}));import { resetChatRunStore, useChatRunStore } from '@/stores/chatRunStore';import { ChatPane } from '../ChatPane';import type { ChatHost } from '../host';import { useComposerStore } from '../state/composerStore';+import { normalizeWorkflowControlStatus } from '@/contexts/WebSocketContext';+import { useWorkflowControlStore } from '@/stores/workflowControlStore';// Loaded up front so the turns' lazy markdown resolves from the module cache.import '../markdown/ReplyMarkdown';@@ -97,6 +102,10 @@ async function write(text: string) {beforeEach(() => {resetChatRunStore();+ useWorkflowControlStore.setState({ statuses: {}, pending: {} });+ getWorkflowControlStatus.mockReset();+ resumeWorkflow.mockReset();+ stopChatRun.mockReset();useComposerStore.setState({ drafts: {} });server = { messages: [], send: { success: true, message_id: 'm1', run_id: 'r1', delivery: 'now' }, activeRuns: [] };sendRequest.mockReset().mockImplementation(async (kind: string, data: Wire) => {@@ -317,6 +326,55 @@ describe('ChatPane', () => {await waitFor(() => expect(chat.notify).toHaveBeenCalledWith('Couldn’t stop the reply. Try again.', 'error'));});+ it('suspends a controlled reply without ending its run and resumes its existing lane', async () => {+ const running = normalizeWorkflowControlStatus({ state: 'running', revision: 3, generation: 1,+ execution_control_version: 1, root_execution_id: 'g1' }, 'w1');+ const paused = { ...running, state: 'paused' as const, revision: 5, can_pause: false, can_resume: true };+ server.messages = [row('m1', 'user', 'Book Saturday', { run_id: 'r1' })];+ server.activeRuns = [{ run_id: 'r1', workflow_id: 'w1', session_id: 'w1', state: 'running', seq: 2, hub_epoch: 'e1',+ workflow_control: running, segments: [{ message_id: 'a_r1', text: 'Checking the calendar', final: null }] }];+ const answer = sendRequest.getMockImplementation()!;+ sendRequest.mockImplementation(async (kind: string, data: Wire) => kind === 'stop_chat_run'+ ? { success: true, resumable: true, ...paused } : answer(kind, data));+ getWorkflowControlStatus.mockImplementation(async () => {+ useWorkflowControlStore.getState().replaceStatuses({ w1: paused });+ return paused;+ });+ stopChatRun.mockImplementation(async () => {+ useWorkflowControlStore.getState().replaceStatuses({ w1: paused });+ return paused;+ });+ resumeWorkflow.mockImplementation(async () => {+ const resumed = { ...running, revision: 7 };+ useWorkflowControlStore.getState().replaceStatuses({ w1: resumed });+ return resumed;+ });+ renderPane();+ fireEvent.click(await screen.findByRole('button', { name: 'Stop reply' }));+ expect(await screen.findByRole('button', { name: 'Resume' })).toBeEnabled();+ expect(screen.getByText('Checking the calendar')).toBeInTheDocument();+ expect(screen.queryByText('You stopped this reply.')).not.toBeInTheDocument();+ expect(useChatRunStore.getState().sessions.w1.runs.r1.state).toBe('running');+ expect(stopChatRun).toHaveBeenCalledWith('w1', 3, 'r1');+ fireEvent.click(screen.getByRole('button', { name: 'Resume' }));+ await waitFor(() => expect(resumeWorkflow).toHaveBeenCalledWith('w1', 5));+ expect(await screen.findByRole('button', { name: 'Stop reply' })).toBeEnabled();+ expect(useChatRunStore.getState().sessions.w1.runs.r1.state).toBe('running');+ });++ it('restores a suspended generation from its run snapshot after reload', async () => {+ server.activeRuns = [{ run_id: 'r1', workflow_id: 'w1', session_id: 'w1', state: 'running', seq: 2,+ workflow_control: normalizeWorkflowControlStatus({ state: 'paused', revision: 5,+ execution_control_version: 1, root_execution_id: 'g1' }, 'w1'),+ segments: [{ message_id: 'a_r1', text: 'Saved partial output', final: false }] }];+ renderPane();+ expect(await screen.findByRole('button', { name: 'Resume' })).toBeEnabled();+ expect(screen.getByText('Saved partial output')).toBeInTheDocument();+ expect(screen.queryByText('Thinking')).not.toBeInTheDocument();+ fireEvent.keyDown(box(), { key: 'Escape' });+ expect(sendRequest.mock.calls.filter(([kind]) => kind === 'stop_chat_run')).toHaveLength(0);+ });+it('offers the next questions the employee suggested, under its latest answer only', async () => {const followups = ['What about Sunday?', 'Move it to 3pm', 'What about Sunday?', ' '];server.messages = [diff --git a/client/src/features/chat/composer/Composer.tsx b/client/src/features/chat/composer/Composer.tsxindex 572f376e..f46729fe 100644--- a/client/src/features/chat/composer/Composer.tsx+++ b/client/src/features/chat/composer/Composer.tsx@@ -18,7 +18,7 @@* still answering the last message the box takes text but Send waits.*/-import { ArrowUp, Mic, Plus, Square } from 'lucide-react';+import { ArrowUp, Mic, Play, Plus, Square } from 'lucide-react';import { useId, useRef, useState, type ClipboardEvent, type KeyboardEvent, type ReactNode, type RefObject } from 'react';import { Button } from '@/components/ui/button';import { Textarea } from '@/components/ui/textarea';@@ -47,6 +47,8 @@ export interface ComposerProps {onSend: () => void;/** Stops the answer under way; absent while there is none. */onStop?: () => void;+ onResume?: () => void;+ resuming?: boolean;/** The answer is already stopping. */stopping?: boolean;compact: boolean;@@ -72,6 +74,8 @@ export function Composer({busy,onSend,onStop,+ onResume,+ resuming = false,stopping = false,compact,boxRef,@@ -156,7 +160,12 @@ export function Composer({const iconSize = compact ? 'size-3.5' : 'size-4';const round = cn('shrink-0 rounded-full', compact ? 'size-7' : 'size-8.5');- const sendButton = onStop ? (+ const sendButton = onResume ? (+ <Button variant="invert" size="icon" disabled={resuming} onClick={onResume}+ aria-label={resuming ? 'Resuming' : 'Resume'} title="Resume from the next pending action" className={round}>+ <Play aria-hidden fill="currentColor" className={iconSize} />+ </Button>+ ) : onStop ? (<Buttonvariant="invert"size="icon"diff --git a/client/src/features/chat/data/control.ts b/client/src/features/chat/data/control.tsnew file mode 100644index 00000000..60fffd14--- /dev/null+++ b/client/src/features/chat/data/control.ts@@ -0,0 +1,25 @@+import { useMutation } from '@tanstack/react-query';+import { mergeWorkflowControlStatus, useWebSocketActions, type WorkflowControlStatus } from '@/contexts/WebSocketContext';+import type { RunSnapshot } from '@/lib/agui/reduceRun';+import { useWorkflowControl } from '@/stores/workflowControlStore';++/** Prefer live generation broadcasts while retaining the reload snapshot. */+export function useChatWorkflowControl(workflowId: string | null, run: RunSnapshot | null): WorkflowControlStatus | undefined {+ const latest = useWorkflowControl(workflowId);+ const owned = run?.workflowControl;+ if (owned && latest?.root_execution_id !== owned.root_execution_id) return owned;+ return owned && latest ? mergeWorkflowControlStatus(owned, latest) : owned ?? latest;+}++export function useResumeChatGeneration(control: WorkflowControlStatus | undefined, onFailed: () => void) {+ const { getWorkflowControlStatus, resumeWorkflow } = useWebSocketActions();+ return useMutation({+ mutationFn: async () => {+ if (!control?.workflow_id) throw new Error('missing_control');+ const current = await getWorkflowControlStatus(control.workflow_id);+ if (current.root_execution_id !== control.root_execution_id) throw new Error('control_generation_conflict');+ return resumeWorkflow(control.workflow_id, current.revision);+ },+ onError: onFailed,+ });+}diff --git a/client/src/features/chat/data/stop.ts b/client/src/features/chat/data/stop.tsindex b489ed63..48349512 100644--- a/client/src/features/chat/data/stop.ts+++ b/client/src/features/chat/data/stop.ts@@ -1,22 +1,26 @@/**- * Stopping the run that is answering (`stop_chat_run`). One nothing picked- * up yet ends at once; a working one stops at its next step and keeps what- * it wrote. The run's own events say what happens next; the server's answer- * is applied at once too, so the Stop button settles without waiting for- * them. A run that ended meanwhile (`not_stoppable`) is not a failure.+ * Controlled Stop suspends the owning generation through the existing+ * workflow-control mutation flow and keeps the chat run alive. Legacy Stop+ * ends the answer using its existing chat lifecycle. A run that ended+ * meanwhile (`not_stoppable`) is not a failure.*/import { useMutation } from '@tanstack/react-query';-import { useWebSocketActions } from '@/contexts/WebSocketContext';+import { useWebSocketActions, type WorkflowControlStatus } from '@/contexts/WebSocketContext';import { isLiveRun, type RunSnapshot } from '@/lib/agui/reduceRun';import { useChatRunStore } from '@/stores/chatRunStore';-type StopReply = { success?: boolean; error?: string; state?: string };+type StopReply = { success?: boolean; error?: string; state?: string; resumable?: boolean; control?: WorkflowControlStatus };-export function useStopChatRun(sessionId: string, onFailed?: () => void) {- const { sendRequest } = useWebSocketActions();- return useMutation<string, Error, RunSnapshot>({+export function useStopChatRun(sessionId: string, onFailed?: () => void, control?: WorkflowControlStatus) {+ const { sendRequest, getWorkflowControlStatus, stopChatRun } = useWebSocketActions();+ return useMutation<StopReply, Error, RunSnapshot>({mutationFn: async (run) => {+ const currentControl = control ?? (run.workflowId ? await getWorkflowControlStatus(run.workflowId) : undefined);+ if (currentControl?.execution_control_version === 1 && run.workflowId) {+ await stopChatRun(run.workflowId, currentControl.revision, run.runId);+ return { resumable: true };+ }let reply: StopReply | undefined;try {reply = await sendRequest<StopReply>('stop_chat_run', { run_id: run.runId });@@ -24,9 +28,14 @@ export function useStopChatRun(sessionId: string, onFailed?: () => void) {throw new Error('transport');}if (reply?.success === false) throw new Error(reply.error || 'stop_failed');- return typeof reply?.state === 'string' ? reply.state : 'stopping';+ if (reply?.resumable && run.workflowId) {+ await getWorkflowControlStatus(run.workflowId);+ }+ return reply ?? { state: 'stopping' };},- onSuccess: (state, run) => {+ onSuccess: (reply, run) => {+ if (reply.resumable) return;+ const state = reply.state;const store = useChatRunStore.getState();const current = store.sessions[sessionId]?.runs[run.runId] ?? run;if (!isLiveRun(current)) return;diff --git a/client/src/features/chat/turns/AssistantTurn.tsx b/client/src/features/chat/turns/AssistantTurn.tsxindex 41ea84c0..17cf795e 100644--- a/client/src/features/chat/turns/AssistantTurn.tsx+++ b/client/src/features/chat/turns/AssistantTurn.tsx@@ -56,6 +56,7 @@ import { StatusLine } from './StatusLine';import { StepsDisclosure } from './StepsDisclosure';import { TurnMeta } from './TurnMeta';import { useWritingRate } from './useWritingRate';+import { useChatWorkflowControl } from '../data/control';const NO_UI: UiPart[] = [];@@ -130,7 +131,10 @@ export function AssistantTurn({onOpenArtifact?: (artifact: ArtifactRef) => void;}) {const queued = run?.state === 'queued';- const live = Boolean(run && isLiveRun(run) && !queued);+ const control = useChatWorkflowControl(run?.workflowId ?? null, run);+ const controlled = control?.execution_control_version === 1 && Boolean(run && isLiveRun(run));+ const suspended = controlled && control.state === 'paused';+ const live = Boolean(run && isLiveRun(run) && !queued && !suspended);// A saved answer is the answer; until then, what the run streamed.const streamed = liveText(message ? null : run);const text = message ? message.text : streamed.text;@@ -193,7 +197,7 @@ export function AssistantTurn({><ChatAvatar persona={persona} live={live} compact={compact} /><div className="flex min-w-0 flex-1 flex-col gap-2.5">- {work && <StepsDisclosure work={work} compact={compact} />}+ {work && <StepsDisclosure work={suspended ? { ...work, live: false } : work} compact={compact} />}{live && !text && <Thinking />}{text && (<div@@ -229,14 +233,15 @@ export function AssistantTurn({))}{live && run && (<StatusLine- label={liveLabel(run, streaming)}+ label={controlled && control.state === 'pausing' ? 'Stopping…'+ : controlled && control.state === 'resuming' ? 'Resuming…' : liveLabel(run, streaming)}rate={rate}note={liveNote}- stopHint={canStop && run.state !== 'stopping'}+ stopHint={canStop && run.state !== 'stopping' && !(controlled && control.state !== 'running')}compact={compact}/>)}- {queued && (+ {(queued || suspended) && (<Note icon="pause"><p className="m-0">Waiting for you to resume {persona.name}.</p></Note>diff --git a/client/src/features/home/__tests__/employeeView.test.tsx b/client/src/features/home/__tests__/employeeView.test.tsxindex 63828088..6bb85b26 100644--- a/client/src/features/home/__tests__/employeeView.test.tsx+++ b/client/src/features/home/__tests__/employeeView.test.tsx@@ -96,7 +96,7 @@ describe('EmployeeView', () => {);expect(screen.queryByRole('heading', { name: 'Maya' })).not.toBeInTheDocument();expect(screen.getByRole('textbox', { name: 'Message Maya' })).toBeInTheDocument();- for (const name of ['Pause', 'Resume', 'Start', 'Watch live']) {+ for (const name of ['Stop', 'Resume', 'Start', 'Watch live']) {expect(screen.queryByRole('button', { name })).not.toBeInTheDocument();}for (const text of ['Receptionist', 'Working', 'Waiting for new WhatsApp messages', 'WhatsApp']) {diff --git a/client/src/features/home/__tests__/presentation.test.ts b/client/src/features/home/__tests__/presentation.test.tsindex 371f35b7..c505c129 100644--- a/client/src/features/home/__tests__/presentation.test.ts+++ b/client/src/features/home/__tests__/presentation.test.ts@@ -31,7 +31,7 @@ const whatsapp = { app_id: 'whatsapp', provider_id: 'whatsapp', name: 'WhatsApp'describe('presentEmployee', () => {it.each([['working', { state: 'running' }, 'Working', 'working', 'pause'],- ['paused', { state: 'paused' }, 'Paused', 'paused', 'resume'],+ ['paused', { state: 'paused' }, 'Stopped', 'paused', 'resume'],['attention', { state: 'paused' }, 'Needs attention', 'attention', 'resume'],['attention', { state: 'failed', can_resume: false }, 'Needs attention', 'attention', 'start'],['attention', { state: 'pausing', can_resume: false }, 'Needs attention', 'attention', 'open_workflow'],@@ -85,7 +85,7 @@ describe('presentEmployee', () => {it('labels the button while a change is in flight', () => {const view = presentEmployee(employee({ status: 'working' }, { state: 'running' }), { action: 'pause', state: 'pausing' });- expect(view.busyLabel).toBe('Pausing…');+ expect(view.busyLabel).toBe('Stopping…');expect(busyLabelFor('start')).toBe('Starting…');});diff --git a/client/src/features/home/__tests__/workspace.test.tsx b/client/src/features/home/__tests__/workspace.test.tsxindex 74d7a764..2ac31592 100644--- a/client/src/features/home/__tests__/workspace.test.tsx+++ b/client/src/features/home/__tests__/workspace.test.tsx@@ -166,7 +166,7 @@ describe('WorkspaceDock', () => {it('shows their status and Resume when the employee is not working', () => {useHomeStore.setState({ workspaceOpen: true, workspaceTab: 'browser' });renderWith([employee({ status: 'paused' })]);- expect(screen.getByText('Paused')).toBeInTheDocument();+ expect(screen.getByText('Stopped')).toBeInTheDocument();expect(screen.queryByText('Live')).toBeNull();expect(screen.getByRole('button', { name: 'Resume' })).toBeEnabled();});@@ -175,9 +175,9 @@ describe('WorkspaceDock', () => {actions.pauseWorkflow.mockReturnValue(new Promise(() => {}));useHomeStore.setState({ workspaceOpen: true, workspaceTab: 'browser' });renderWith([employee({ control: normalizeWorkflowControlStatus({ generation: 1, state: 'running', revision: 4 }, 'w1') })]);- fireEvent.click(screen.getByRole('button', { name: 'Pause' }));+ fireEvent.click(screen.getByRole('button', { name: 'Stop' }));expect(actions.pauseWorkflow).toHaveBeenCalledWith('w1', 4);- expect(screen.getByRole('button', { name: 'Pausing…' })).toBeDisabled();+ expect(screen.getByRole('button', { name: 'Stopping…' })).toBeDisabled();});it('follows the employee last opened, else the first', () => {diff --git a/client/src/features/home/data/presentation.ts b/client/src/features/home/data/presentation.tsindex 8a34f1af..995246e2 100644--- a/client/src/features/home/data/presentation.ts+++ b/client/src/features/home/data/presentation.ts@@ -52,13 +52,13 @@ export interface EmployeePresentation {const PILL: Record<EmployeeSummary['status'], { label: string; tone: StatusTone }> = {working: { label: 'Working', tone: 'working' },ready: { label: 'Ready', tone: 'ready' },- paused: { label: 'Paused', tone: 'paused' },+ paused: { label: 'Stopped', tone: 'paused' },attention: { label: 'Needs attention', tone: 'attention' },};const BUSY: Record<WorkflowControlPendingMutation['action'], string> = {start: 'Starting…',- pause: 'Pausing…',+ pause: 'Stopping…',resume: 'Resuming…',reset: 'Resetting…',};@@ -100,7 +100,9 @@ export function presentEmployee(// Only a green working dot pulses (its ring is green).pulse: pill.tone === 'working',primary: primaryAction(employee),- busyLabel: pending ? BUSY[pending.action] : null,+ busyLabel: pending ? BUSY[pending.action]+ : employee.control.state === 'pausing' ? 'Stopping…'+ : employee.control.state === 'resuming' ? 'Resuming…' : null,};}@@ -115,7 +117,7 @@ export const OPEN_IN_DEV_LABEL = 'Open in Dev mode';export function primaryActionLabel(action: PrimaryAction): string {switch (action.kind) {case 'pause':- return 'Pause';+ return 'Stop';case 'resume':return 'Resume';case 'start':diff --git a/client/src/features/home/employee/useEmployeeControl.ts b/client/src/features/home/employee/useEmployeeControl.tsindex f4b8fad8..90c72cdf 100644--- a/client/src/features/home/employee/useEmployeeControl.ts+++ b/client/src/features/home/employee/useEmployeeControl.ts@@ -106,7 +106,7 @@ export function useEmployeeControl(employee: EmployeeSummary, onConnect: (providreturn {view,label: view.busyLabel ?? (inFlight ? busyLabelFor(inFlight) : primaryActionLabel(view.primary)),- busy: Boolean(pending) || Boolean(inFlight),+ busy: Boolean(pending) || Boolean(inFlight) || employee.control.state === 'pausing' || employee.control.state === 'resuming',act,};}diff --git a/client/src/features/home/genui/views.tsx b/client/src/features/home/genui/views.tsxindex c8b2f7a1..c73d43e8 100644--- a/client/src/features/home/genui/views.tsx+++ b/client/src/features/home/genui/views.tsx@@ -119,7 +119,7 @@ const AGENT_STATUS = {dot: 'bg-status-ready-dot shadow-[0_0_8px_var(--status-ready-dot)]',ink: 'text-status-ready-ink',},- paused: { label: 'Paused', dot: 'bg-status-paused-dot', ink: 'text-status-paused-ink' },+ paused: { label: 'Stopped', dot: 'bg-status-paused-dot', ink: 'text-status-paused-ink' },} as const;function Pip({ className }: { className?: string }) {diff --git a/client/src/lib/agui/reduceRun.ts b/client/src/lib/agui/reduceRun.tsindex cc97b89d..8f048e0b 100644--- a/client/src/lib/agui/reduceRun.ts+++ b/client/src/lib/agui/reduceRun.ts@@ -22,6 +22,7 @@ import {type StepState,} from './events';import { applyPatch } from './patch';+import type { WorkflowControlStatus } from '@/contexts/WebSocketContext';export interface RunStep {stepId: string;@@ -55,6 +56,8 @@ export interface RunSnapshot {runId: string;sessionId: string;workflowId: string | null;+ /** Derived from the run's owning generation, never a chat lifecycle state. */+ workflowControl?: WorkflowControlStatus;kind: RunKind;state: RunState;/** The newest event folded in; 0 before any. */@@ -191,6 +194,10 @@ export function snapshotFromWire(raw: unknown): RunSnapshot | null {runId,sessionId,workflowId: textOrNull(raw.workflow_id),+ ...(isRecord(raw.workflow_control) && typeof raw.workflow_control.revision === 'number'+ && typeof raw.workflow_control.state === 'string'+ ? { workflowControl: raw.workflow_control as unknown as WorkflowControlStatus }+ : {}),kind: oneOf(raw.kind, RUN_KINDS, 'message'),state: oneOf(raw.state, RUN_STATES, 'pending'),seq: typeof raw.seq === 'number' && Number.isInteger(raw.seq) && raw.seq >= 0 ? raw.seq : 0,diff --git a/docs-internal/DESIGN.md b/docs-internal/DESIGN.mdindex adbfa42c..a707fe8c 100644--- a/docs-internal/DESIGN.md+++ b/docs-internal/DESIGN.md@@ -5,7 +5,9 @@The OpenCompany execution engine implements a robust workflow orchestration system combining industry-standard patterns from **Netflix Conductor**, **Prefect 3.0**, **Temporal**, and **Redis Streams**. This document details the architectural decisions, design patterns, and standards used.Related docs:-- [temporal-execution-engine-rfc.md](ARCHIVE/temporal-execution-engine-rfc.md) - Temporal control, trigger, graph, agent-team, and trace architecture++- [temporal-workflow-control.md](temporal-workflow-control.md) - current generation lifecycle, acknowledged Stop/drain/Resume, messaging, event queues and verification+- [temporal-execution-engine-rfc.md](ARCHIVE/temporal-execution-engine-rfc.md) - archived execution design- [TEMPORAL_ARCHITECTURE.md](TEMPORAL_ARCHITECTURE.md) - distributed execution via Temporal activities- [event_waiter_system.md](event_waiter_system.md) - push-based trigger waiters- [native_llm_sdk.md](native_llm_sdk.md) - LLM provider layer@@ -44,6 +46,23 @@ When `TEMPORAL_ENABLED=true` and the Temporal server is reachable, every workfloFull dispatch matrix + activity inventory (legacy + per-type + the F4.B `agent.*` activities — read the live set from `collect_agent_activities()` in `services/temporal/agent_activities.py`) + worker configuration + heartbeat semantics live in [TEMPORAL_ARCHITECTURE.md](TEMPORAL_ARCHITECTURE.md). Tool-call dispatch under F4.A: [tool_building_pipeline.md §9](./tool_building_pipeline.md).+Newly admitted deployment generations record execution control version 1.+Stop closes scheduling gates before model requests, tools, polling, compaction+and child starts. Admitted parallel work finishes under its existing retry+policy, and its results/bookkeeping settle before the generation acknowledges+Stopped. Resume releases the same pending continuation. The generation+controller records independently living roots; native execution descriptions+discover attached descendants after child-start acknowledgement.++Signals retain accepted trigger events, completed Updates acknowledge control+and root membership, and Queries expose status. Event admission stays immediate+when running; no inactivity batching is introduced. Controller and agent+continuation carry explicit state across Continue-As-New. Existing generations+and pre-marker histories keep their recorded command paths. The+[control contract](temporal-workflow-control.md) defines timeout reconciliation,+fail-closed recovery, replay validation and operating limits. These Temporal+guarantees do not extend to the sequential fallback.+### 2. Sequential FallbackWhen Temporal is not available, workflow execution falls back to a simple topological walk with no parallelism, no caching, and no retry. Intended only for minimal environments and CI smoke tests.diff --git a/docs-internal/SETUP.md b/docs-internal/SETUP.mdindex 80bb691c..bf17186f 100644--- a/docs-internal/SETUP.md+++ b/docs-internal/SETUP.md@@ -160,6 +160,11 @@ the arrangement is [desktop_host_contract.md](./desktop_host_contract.md).- Ports: gRPC `TEMPORAL_FRONTEND_GRPC_PORT`, Web UI `TEMPORAL_UI_PORT` (values in `.env.template`)- Embedded worker runs inside the Python backend (`TemporalWorkerManager` + `TemporalWorkerPool`, built by `services/temporal/lifecycle.py`; `main.py` only schedules `run_temporal_lifecycle`)- Running and paused deployments survive restarts: `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP=false` is the default, and the boot-time reconcile pass re-arms them. Setting it `true` is a debug-only sweep (history preserved; active control rows still veto it)+- New generations record execution control version 1. Stop drains admitted work+ before reporting Stopped; Resume releases existing continuations. Existing+ generations retain legacy behavior until Reset and Start. A lost version 1+ controller fails closed rather than rebuilding an empty root registry. See+ [control lifecycle, recovery and verification](temporal-workflow-control.md).- See [Temporal Architecture](./TEMPORAL_ARCHITECTURE.md) and [CLI Services Guide](./cli_services_integration.md)### Database (SQLite)@@ -267,4 +272,6 @@ Delete `workflow.db` there to reset all data.- **WebSocket-First**: WS message handlers replace most REST APIs (live set = `MESSAGE_HANDLERS` in `server/routers/websocket.py` + plugin-registered handlers)- **n8n-inspired**: Node definitions follow n8n INodeProperties pattern- **Cache Fallback**: Redis (production) → SQLite (dev) → Memory-- **Event-Driven**: Trigger nodes use asyncio.Future for event waiting+- **Event-Driven**: Controlled Temporal deployments queue Signals and polling+ results in their generation controller. Direct/legacy paths retain in-memory+ event waiters; see [event routing](event_framework.md).diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.mdindex 987b6b21..fc9ba5cb 100644--- a/docs-internal/TEMPORAL_ARCHITECTURE.md+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md@@ -1,20 +1,23 @@# Temporal Distributed Node Execution Architecture-> **Canonical system design:** See-> [Temporal Execution Engine RFC](temporal-execution-engine-rfc.md) for the-> current control-generation, trigger-hub, agent-team, pause/reset, trace, and-> security architecture. This document remains the detailed node dispatch and-> activity/worker inventory.+> **Current control contract:** [Temporal workflow control](temporal-workflow-control.md)+> defines generation ownership, acknowledged Stop/Resume, Reset, and recovery.+> This document describes execution, Activity dispatch, worker routing, and+> continuation. The [original execution-engine RFC](ARCHIVE/temporal-execution-engine-rfc.md)+> is historical design context; use the current docs and implementation for behavior.## Overview-Each workflow node executes as a **Temporal activity** with its own isolated context, enabling horizontal scaling across distributed workers. The orchestrator dispatches in one of three ways depending on settings flags:+Executable workflow nodes run as **Temporal Activities** or, for supported agents,+child Workflows. Each receives its own execution context, enabling distributed+workers to run independent steps. The orchestrator dispatches in one of three+ways depending on the run's frozen settings flags:| Dispatch | Trigger | Use case ||---|---|---|| **Legacy single activity** (`execute_node_activity`) | `TEMPORAL_PER_TYPE_DISPATCH=false` | Every node routed through one dispatcher activity. WebSocket round-trip back to the FastAPI server. Stable since Wave 11; kept as the fallback path. |-| **Per-type activity** (`node.{type}.v{version}`) | `TEMPORAL_PER_TYPE_DISPATCH=true` (production default) | Each plugin gets its own `@activity.defn`. Per-plugin retry / timeout / heartbeat configs apply. With `TEMPORAL_WORKER_POOL_ENABLED=true` (default since Wave 16.4) the activity also carries `task_queue=cls.task_queue`, landing it on its specialised `TemporalWorkerPool` worker (browser / code-exec / ai-heavy / ...). Shipped in F4.A (commit `8261b05`); queue routing activated in Wave 16. |-| **Agent-as-child-workflow** (`AgentWorkflow`) | `TEMPORAL_AGENT_WORKFLOW_ENABLED=true` | AI Agents (every type in `AGENT_WORKFLOW_TYPES` in `services/temporal/workflow.py`: aiAgent, chatAgent, the specialized agents and the team leads) run as Temporal child workflows. Each LLM turn = activity; each tool call = per-type activity. Mirrors Temporal's AI Cookbook canonical pattern. F4.B infrastructure shipped (commit `a4d009e`); per-agent migrations follow. |+| **Per-type activity** (`node.{type}.v{version}`) | `TEMPORAL_PER_TYPE_DISPATCH=true` (production default) | Each plugin gets its own Activity definition. Graph dispatch applies plugin retry and, with the frozen worker-pool flag on, `cls.task_queue`. Ordinary graph timeout/heartbeat remain 24 h / 2 min; Workspace-task patches use plugin values. Version 1 agent-tool dispatch records and uses all declared Activity policies. |+| **Agent-as-child-workflow** (`AgentWorkflow`) | `TEMPORAL_AGENT_WORKFLOW_ENABLED=true` | Agent types in `AGENT_WORKFLOW_TYPES` in `services/temporal/workflow.py` run as child Workflows. Each LLM turn and ordinary tool call is a regular Activity; direct delegations can start attached agent children and Task Manager assignments start detached runners. |`rlm_agent`, `claude_code_agent` and `vertex_managed_agent` are intentionally excluded from AgentWorkflow — their externalised loops (RLM REPL / Claude CLI `--resume` / Vertex Interactions API `previous_interaction_id` chaining) require single-process state continuity. The authoritative list is `AGENT_WORKFLOW_TYPES` in `services/temporal/workflow.py`; any agent type outside it runs as an ordinary per-type activity.@@ -96,12 +99,12 @@ This invokes `run_standalone_worker()` from `services/temporal/worker.py`. It al## Key Architecture Principles-### 1. Node = Independent Activity+### 1. External Work = Independent Activity-Each node runs as a separate Temporal activity with:+Ordinary nodes and the external steps of agent child Workflows run as Activities with:- **Own context** - No shared mutable state between nodes-- **Own retry policy** - Failed nodes retry independently (up to 3 attempts)-- **Own timeout** - Long AI nodes don't block short nodes (`_NODE_ACTIVITY_START_TO_CLOSE` = 24 h ceiling in `services/temporal/workflow.py`; liveness comes from the 2-minute heartbeat, not the start-to-close cap)+- **Own retry policy** - Plugin declarations take precedence over ordinary defaults. Agent LLM steps use unlimited transient retries; a declared one-attempt tool keeps that policy.+- **Own timeout** - Ordinary graph dispatch uses a 24 h ceiling and 2-minute heartbeat; Workspace-task nodes and version 1 agent-tool dispatch consume plugin values. Different Activities can progress independently.- **Own worker** - Can execute on any available worker in the cluster### 2. Workflow = Pure Orchestrator@@ -113,7 +116,9 @@ The workflow ONLY orchestrates:- Schedules activities using FIRST_COMPLETED pattern- Collects results and routes outputs to dependent nodes-**NO business logic in workflow** - all execution happens in activities.+Provider requests, tool execution, database access, and external effects happen+in Activities. Workflow code owns deterministic scheduling, admission gates,+result bookkeeping, and durable continuation.### 3. Context Passing (Immutable)@@ -141,6 +146,96 @@ context = {}```+## Cooperative generation Stop and Resume++New generations record `resource_manifest.execution_control_version=1` and+propagate that protocol in their Workflow input. The version is a generation+contract, independent of routing flags. Existing generations keep their prior+command paths; deployment of new code does not silently give old histories the+stronger acknowledgement guarantee.++Stop means **finish admitted work, retain its result, and wait before the next+action**. For a model response with tools A and B, Stop during A lets A finish+and records its result; B stays pending. Resume admits B from the existing+continuation. The model response and A are not explicitly scheduled again.+Already-admitted parallel work drains concurrently under its original retry+policies. Controller events remain queued while stopped.++`services/temporal/execution_control.py::ExecutionControl` holds the revision,+pause posture, producer hold, active-action count, and pending child-start+count in Workflow state. Owners expose completed `set_control_state` and+`wait_for_checkpoint` Updates and a read-only `execution_control_status` Query.+Initialization binds the input before early handlers can run; pure validators+reject malformed or wrong-generation/chain requests before acceptance. The+helper does not intercept SDK commands: callers use admission gates at their+existing scheduling boundaries.++The boundary accounting is deliberate:++| Work | Admission and acknowledgement |+|---|---|+| Model request, ordinary node/tool, polling fetch, compaction, tool refresh | Gate before scheduling; count admitted work through result consumption and its normal bookkeeping. |+| Child Workflow start | Gate and count until Temporal acknowledges the start, allowing complete native child enumeration afterward. |+| Child result or subagent-permit wait | Excluded from local drain count; the child is controlled separately and a waiting parent must not block Stop. |+| Control enrollment, checkpoint waits, maintenance | Excluded from business-work drain; completed results can still be recorded while stopped. |++The generation controller records only **independent roots**: graph runs,+cron/job roots, and detached `DelegatedTaskWorkflow` runners that can outlive+their parent. Roots enroll through an acknowledged controller Update before+business work; a regular maintenance Activity bridges Workflow code to the+Temporal client because Workflows cannot directly send Updates. Attached+agent children inherit the scope and are discovered from native execution+descriptions. There is no SQL participant registry, recurring participant+heartbeat, or separate checkpoint store.++`services/deployment/execution_control.py::transition_generation` first holds+controller producers and fences every reachable Workflow, then waits for+their checkpoint acknowledgements. Setters acknowledge already-admitted child+starts before the next Describe call. It reconciles new root registrations+until the controller membership epoch is stable. `pausing` means Stop has been+requested; `paused` is published only after admitted business work and+bookkeeping settle throughout that topology. A request timeout leaves+`pausing` for reconciliation and does not cancel the draining Activity.++Resume publishes a newer intent with producers held, releases existing+descendants and independent roots, reconciles membership, and releases the+controller producers last. Local collectors and schedules reopen through the+generation control handler. Revisions prevent delayed enrollment replies or+stale control messages from overriding newer intent. Control uses current+Workflow-ID handles with first-execution-chain checks, so continue-as-new+Run-ID changes remain addressable and reused IDs cannot control another chain.+An unexpectedly lost version 1 controller fails closed; an empty replacement+root map cannot prove a generation is safely resumable.++`rlm_agent`, `claude_code_agent`, and `vertex_managed_agent` remain whole-node+Activities. Stop waits for the admitted Activity, including its internal tool+work, to settle. It does not suspend the CLI, REPL, or cloud agent between+internal calls. Direct Workspace tasks and separately approved sends have+their own execution ownership; generation Stop does not control those+independent executions.++Heartbeats provide liveness and cancellation/progress information when an+Activity implements that contract. A heartbeat status string cannot checkpoint+an opaque request or process session. We use regular Activities for provider+and tool work, preserving retries, heartbeats, and worker routing. The+[Long-running Activity pattern](https://docs.temporal.io/design-patterns/long-running-activity)+explains those contracts; [Resumable Activity](https://docs.temporal.io/design-patterns/resumable-activity)+addresses retrying a failed Activity after corrective input, which is a+different operation from releasing this suspended continuation. The+[Python messaging guide](https://docs.temporal.io/develop/python/workflows/message-passing)+defines the Signal/Update/Query acknowledgement and handler rules used here.++Stop can remain `pausing` during a persistent provider outage because+`LLM_STEP_RETRY` intentionally has no attempt limit. A worker loss or ambiguous+external effect before Temporal records completion can still trigger an+Activity retry according to the tool's policy; external-effect deduplication+belongs to the tool's idempotency contract. Resume does not promise exactly-once+delivery to external systems.++For API shapes, chat lifecycle, recovery, and verification, see+[Temporal workflow control](temporal-workflow-control.md) and+[Chat protocol](chat_protocol.md).+## Execution Flow### 1. Workflow Receives Request@@ -226,7 +321,7 @@ while True:**Legacy path** (`execute_node_activity`): The activity round-trips through the local WebSocket back to FastAPI, which dispatches to the plugin handler. This was the only path before F4.A.-**Per-type path** (`node.{type}.v{version}`, F4.A): The activity body lives on the plugin class via `BaseNode.as_activity()` and calls `workflow_service.execute_node(...)` **directly** — no WebSocket round-trip. Same DI container (the worker shares the FastAPI process), same broadcasting + parameter-fetch pipeline. Each plugin class declares its own `start_to_close_timeout` / `retry_policy` / `heartbeat_timeout` so they're applied at activity definition time. See `server/services/plugin/base.py:as_activity`.+**Per-type path** (`node.{type}.v{version}`, F4.A): The Activity body lives on the plugin class via `BaseNode.as_activity()` and calls `workflow_service.execute_node(...)` **directly** — no WebSocket round-trip. Same DI container (the embedded worker shares the FastAPI process), same broadcasting + parameter-fetch pipeline. Activity options are selected by the caller, not the registration decorator. Graph dispatch uses plugin retry/queue with ordinary 24 h / 2 min timeout/heartbeat defaults, except the Workspace-task patch. Version 1 AgentWorkflow tool bindings record and use the plugin's timeout, heartbeat, retry, and queue declarations during payload preparation or explicit refresh. See `server/services/plugin/base.py:as_activity`, `server/services/temporal/workflow.py`, and `server/services/temporal/agent_workflow.py::_execute_plugin_tool_activity`.```python# F4.A per-type activity body (server/services/plugin/base.py)@@ -378,7 +473,8 @@ When `TEMPORAL_AGENT_WORKFLOW_ENABLED=true` and the node type is in `AGENT_WORKF```AgentWorkflow.run(context):- 0. execute_activity("agent.prepare_payload")+ 0. restore the carried prepared_payload on a version 1 continuation;+ otherwise execute_activity("agent.prepare_payload")resolves the DB-backed payload from the canvas context — runsworkflow_service._param_resolver.resolve so {{node.field}}templates in prompt / system_message become real values BEFORE@@ -476,13 +572,48 @@ request starts.`emit_phase(phase, status?)` is a thin helper that schedules `agent.broadcast_progress`. The activity emits `WorkflowEvent.agent_progress` (CloudEvents v1.0, `type="com.opencompany.agent.progress"`) for FE consumers; when `status` is supplied it also drives a raw-dict `update_node_status` for the canvas-glow color (executing / success / error). Same dual-channel pattern F4.A's `_node_activity` uses. When this workflow is itself a delegated child (`context["parent_node_id"]` set), every `emit_phase` call ALSO schedules a second broadcast against the parent's `node_id` with `phase="delegating"` — the parent's canvas badge then advances in real time while the child loops, instead of freezing at "executing" glow until the child completes.-Each LLM step is one activity and each ordinary tool call is one per-type activity. Team-lead Task Manager assignments are different: persistence happens first, then the lead starts a deterministic detached `DelegatedTaskWorkflow` with `ParentClosePolicy.ABANDON` and receives `queued` immediately. The runner owns the root-wide permit, claim, child `AgentWorkflow`, terminal result/usage persistence, `taskTrigger`, and permit release, so the assigning lead can return without polling. Direct non-team delegation retains the child-workflow path. Non-agent tools and excluded types (any agent type outside `AGENT_WORKFLOW_TYPES`: `rlm_agent`, `claude_code_agent`, `vertex_managed_agent`) still go through `execute_activity`. Failures surface as durable task failures and trigger review rather than being lost when the lead invocation closes.+Each LLM step is one Activity and each ordinary tool call is one per-type+Activity. In version 1, gates surround model requests, tool admission, tool+refresh, compaction, and child starts; admitted result and transcript+bookkeeping finish while stopped. Unstarted calls from a recorded response+stay in the current Workflow continuation. Stop preserves the existing+parallel scheduling and never converts child-result waits into drain work.++Team-lead Task Manager assignments persist first, then the lead starts a+deterministic detached `DelegatedTaskWorkflow` with+`ParentClosePolicy.ABANDON` and receives `queued` immediately. The runner owns+the root-wide permit, claim, child `AgentWorkflow`, terminal result/usage+persistence, `taskTrigger`, and permit release, so the assigning lead can+return without polling. A version 1 runner independently enrolls with the+generation controller; its attached agent child needs no separate enrollment.+Direct non-team delegation retains the attached child-workflow path.+Non-agent tools and excluded types (`rlm_agent`, `claude_code_agent`,+`vertex_managed_agent`) still use `execute_activity`. Failures surface as+durable task failures and trigger review rather than being lost when the lead+invocation closes.Which graph a firing runs depends on the deployment kind. Controlled generations (the listener payload carries a `data_scope_id`) execute the graph snapshot carried in their trigger registration on every firing, with no per-firing graph lookup. Legacy uncontrolled deployments (no `data_scope_id`) keep the hot lookup: each push or poll firing in `TriggerListenerWorkflow._spawn_child_run` / `PollingTriggerWorkflow._spawn_child_run` resolves the latest persisted workflow graph through `load_persisted_workflow_graph_activity` before filtering downstream nodes, falling back to the deployment snapshot if the lookup fails; that is what makes tools added after deployment available to `taskTrigger` and other triggered agent runs on those deployments. Edge traversal accepts both canonical `targetHandle` and legacy `target_handle`; tool choice remains entirely with the agent and no trigger-specific tool-use prompt is injected.**Delegation input contract (input-vs-config separation).** The LLM's `{task, context}` args are per-invocation *input*, not node configuration, and travel as the child workflow input's `invocation` field. `prepare_agent_payload` applies it AFTER its config resolution (`{**node_data, **db_params}` — DB wins for config liveness): `task` → system_message, `context`-or-`task` → prompt — the same semantics as the legacy `handlers.tools._execute_delegated_agent`. Stored node parameters (including the empty default `prompt` the frontend persists on drop) therefore never override the delegated task. A call with both fields empty is rejected at the parent's call boundary (tool-error message to the LLM, no child spawn). Bypass agents dispatched as plain activities (any type outside `AGENT_WORKFLOW_TYPES`: `rlm_agent` / `claude_code_agent` / `vertex_managed_agent`) instead receive the remap directly in `node_data` — their per-type activity consumes `node_data` verbatim with no DB re-merge.-**Canvas-aware tools** opt into receiving the parent workflow's `nodes`/`edges` by declaring `needs_canvas: ClassVar[bool] = True` on their `BaseNode` subclass. The F4.B tool dispatch reads this via `services.node_registry.get_node_class(node_type).needs_canvas` and forwards `context.get("nodes")` / `context.get("edges")` into `tool_payload`; default plugins keep the empty-canvas optimisation. Today only `agentBuilder` opts in: `add_tool` compares the run's canvas with the saved graph, so a tool saved earlier but missing from the generation's snapshot comes back as an `add_node` op to bind, with nothing saved. Its calling agent is the `invoking_agent_node_id` this dispatch supplies, else `parent_node_id`, never an edge walk. Operations inside agentBuilder reload via `database.get_workflow(workflow_id)` so in-run duplicate detection sees mutations from earlier calls in the same workflow run — see the [agentBuilder card](./node-logic-flows/ai_tools/agentBuilder.md).+**Canvas-aware tools** opt into receiving the parent Workflow's `nodes`/`edges`+with `needs_canvas: ClassVar[bool] = True`. Version 1 records that declaration+with the resolved tool binding; legacy F4.B reads the plugin class at dispatch.+Opted-in tools receive the parent's canvas; others keep the empty-canvas+optimization. `agentBuilder.add_tool` compares the run's canvas with the saved+graph, so a saved tool missing from the generation snapshot returns a bind-only+`add_node` operation. Its calling agent is `invoking_agent_node_id`, else+`parent_node_id`. Operations reload the saved graph for in-run duplicate+detection; see the [agentBuilder card](./node-logic-flows/ai_tools/agentBuilder.md).++Version 1 also records each ordinary tool's timeout, heartbeat, retry, and+queue policy in `tool_info.activity_policy`. Both initial preparation and+explicit refresh produce these bindings; continue-as-new carries the resolved+set. Scheduling uses those recorded declarations, with specialized queue+routing only when the frozen worker-pool flag is on. A plugin update during+Stop or worker restart does not implicitly replace an already-bound policy.+Legacy histories keep their prior scheduling options, including the existing+Workspace-task policy patch.`collect_agent_activities()` registers the agent activities — read the live setfrom that function rather than trusting a count here, which drifts on every@@ -520,8 +651,10 @@ pipeline: `agent.prepare_payload` loads the stored conversation for`conversation_key` (load failures are LOUD — `ConversationLoadFailed`retryable / `ConversationTooLarge` non-retryable at 1 MB), and`agent.execute_llm_step` saves `[...sent, assistant]` best-effort after each-provider call. A continue-as-new rollover carries the live transcript in its-resume marker, so nothing reconstructs from the store mid-run. The+provider call. A version 1 continue-as-new rollover carries the live transcript+and prepared state, so it does not reload changed configuration or reconstruct+the continuation from the conversation store. Legacy rollover behavior is+documented in [agent context flow](agent_context_flow.md#message-seeding-what-a-runs-initial-messages-list-is). Thejournal-era activities (`agent.prepare_context`,`agent.reconstruct_context_messages`, `agent.append_context`) are retired;`agent.compact_context` (the shared client-side summarizer that bounds the@@ -544,6 +677,43 @@ the tool-result cap apply whether or not compaction is on (seeReferences: [Temporal AI Cookbook](https://docs.temporal.io/ai-cookbook), [`temporal-community/temporal-ai-agent`](https://github.com/temporal-community/temporal-ai-agent), [`temporalio.contrib.openai_agents`](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/openai_agents).+### Agent continuation under history pressure++`AgentWorkflow` rolls over at a clean completed-turn boundary, with no live+delegation handles or Task Manager tasks. Version 1 parks at that boundary+while stopped and waits for all control handlers to finish before+continue-as-new. The next input carries pause/revision/hold state, application+execution identity, transcript, accumulated thinking, iteration, usage,+context usage, prepared payload, and the resolved tool bindings. A rollover+changes Temporal's Run ID while preserving the application's execution scope.++The converter measures the **complete encoded continuation input**, including+graph, preparation, and bindings, against `_CAN_INPUT_MAX_BYTES` (1,900,000+bytes). Existing post-turn result relief and compaction run before this clean+boundary. If the argument still cannot fit, version 1 fails explicitly with+`AgentContinuationTooLarge`; it never returns to the opening prompt. A large+graph or binding set can overflow even when the transcript alone fits.+Legacy histories retain their previous preparation and 1 MB transcript-only+fallback for replay compatibility. The new capacity guarantee is limited to+version 1 generations.++The [Continue-As-New guidance](https://docs.temporal.io/design-patterns/continue-as-new)+motivates bounded histories and explicit carried state. Controller rollover+can still happen while stopped because it carries pending events and root+membership; the agent rule above specifically avoids rolling over a live or+suspended tool/delegation turn.++The focused tests use stubbed business Activities with real Temporal test+Workflows, including tool drain/continuation, worker restart, actual rollover,+and replay: [agent execution control](../server/tests/temporal/test_agent_execution_control_integration.py),+[generation topology](../server/tests/temporal/test_generation_execution_control_integration.py),+and [execution-control replay](../server/tests/temporal/test_execution_control_replay.py).+Plugin options and capacity errors are also covered by+[AgentWorkflow tests](../server/tests/temporal/test_agent_workflow.py).+These checks do not establish external-tool exactly-once effects or eliminate+the ingress and duplicate-trigger naming limits recorded in the+[control guide](temporal-workflow-control.md).+### Direct Workspace tasksA node class that declares `workspace_task = True` (the phone nodes in `server/nodes/mobile/`) can also run a single task outside any graph run. The Workspace's **Ask AI to use the phone** box posts to `/api/mobile/{workflow_id}/{node_id}/tasks` (`server/nodes/mobile/_router.py`), and `services/node_invocations.py::submit` admits the task.@@ -627,10 +797,10 @@ Trigger nodes that aren't the firing trigger are:| Scenario | Behavior ||----------|----------|-| Ordinary node or agent-support activity fails transiently | Temporal retries up to 3 attempts with backoff unless the error type is non-retryable. |+| Ordinary node or agent-support Activity fails transiently | The shared ordinary default is 3 attempts with backoff; declared plugin policies and specialized support policies take precedence. || `AgentWorkflow` LLM-step activity fails | `LLM_STEP_RETRY` retries without limit (`maximum_attempts=0`, 5 s → 5 min backoff; `services/temporal/_retry_policies.py`) unless the provider error category is non-retryable (invalid_request, authentication, ...). Provider SDK retries stay disabled so Temporal owns the retry schedule and `retry_after` hints. |-| Worker crashes mid-execution | Temporal reschedules on another worker |-| Ordinary node times out | Temporal applies that activity's retry policy; plugin timeouts vary by node type. |+| Worker crashes mid-execution | Workflow state replays from history; an unfinished Activity retries according to its timeout/retry policy on an eligible worker. Completed recorded work is not explicitly rescheduled by Resume. |+| A node/tool Activity times out | Temporal applies the scheduled retry policy. Graph timeout/heartbeat defaults and version 1 agent-tool/plugin options differ as described above. || All retries exhausted | Workflow receives failure, stops execution |## File Structure@@ -651,6 +821,8 @@ server/services/temporal/│ ├── _find_ready_nodes() # Dependency resolution│ └── _wait_any_complete() # FIRST_COMPLETED wait├── workflow_control_workflow.py # WorkflowControlWorkflow (per-generation controller: triggers, signals, polling, pause)+├── execution_control.py # Revision-ordered business admission and checkpoint accounting+├── execution_control_activities.py # Client-side Update bridge for independent-root enrollment├── trigger_listener_workflow.py # TriggerListenerWorkflow (legacy push-trigger listener)├── polling_trigger_workflow.py # PollingTriggerWorkflow (legacy polling listener)├── workspace_tasks_workflow.py # WorkspaceTaskControllerWorkflow (per-workflow admission + Reset for direct Workspace tasks)@@ -729,13 +901,25 @@ The Temporal binary + persistence are managed in-process by the plugin-folder pa**WS surface**: `_handlers.py` registers `temporal_status` / `temporal_start` / `temporal_stop` via `services.ws_handler_registry.register_ws_handlers`. `_refresh.py` registers a status-refresh callback via `services.status_broadcaster.register_service_refresh`, which seeds the FE health indicator once at startup (refresh callbacks run once, in a background task, not on each WebSocket connect).-**Months-long durability contract**: running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. None of the mechanisms below carries a `workflow.patched` marker: the lifetime-cap removal and the poll-interval floor apply unconditionally, including when an older history is replayed.+**Months-long durability contract**: running and paused deployments retain their+Temporal histories across backend restarts. New child starts have no Workflow+lifetime cap; individual Activities retain their declared attempt timeouts and+retry limits. History pressure triggers continuation rather than a periodic+restart. Existing executions can still have timers recorded by an older start;+removing a timeout from new start options does not erase those timers. The+poll-interval floor is unconditional, while controller queue and execution+control changes use the version gates below. The strongest Stop/Resume+guarantee applies to version 1 generations with their controller chain intact.- **No lifetime caps on new child runs.** Trigger/cron-spawned `MachinaWorkflow` runs, agent children, and delegated-task runners previously carried 1-2h `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so any pause longer than the cap silently terminated the run (and a timed-out delegated runner skipped its compensation: leaked permit + stuck task row). New executions start children unbounded (no patch marker was retained for this change). The live `workflow.patched` markers are `machina-conditional-edges-v1`, `machina-run-record-v1`, `machina-chat-run-v1`, `workspace-node-cancellation-v1` and `workflow-node-user-action-error-v1` in `services/temporal/workflow.py`, `machina-trigger-listener-node-filter` in `services/temporal/trigger_listener_workflow.py`, and `workspace-task-activity-policy-v1`, `agent-user-action-error-v1` and `agent-blocked-tool-turn-save-v1` in `services/temporal/agent_workflow.py`; the list grows, so check it with `grep -rn "workflow.patched(" server/services/temporal`. `machina-run-record-v1` gates the `workflow_runs.record_completion` activity that trigger-spawned runs schedule when they finish, for Normal mode's "done today" (see [normal_mode.md](./normal_mode.md#done-today)). `machina-chat-run-v1` gates the `chat_run.start` / `chat_run.finish` activities a run spawned by the owner's chat message schedules to claim and finish its chat run; the run id is read only from an event whose source is `opencompany://services/chat` (see [chat_protocol.md](./chat_protocol.md#runs)). `workflow-node-user-action-error-v1` gates `MachinaWorkflow._node_failure`: when a node's activity or child workflow raises, it follows the error's causes to an `LLMError.*` or `MissingAgentProviderCredential` failure whose details carry `requires_user_action` (a missing or rejected key, billing, quota, permission, an unknown model or a wrong base URL) and returns a `NodeUserError` result with that message, its `hint` and `retryable: false` instead of the bare error text, so `chat_run.finish` records the hint and `workflow_control.pause_on_failure` pauses the deployment on that first failure rather than counting it toward the threshold. `agent-user-action-error-v1` gates the matching stop in `AgentWorkflow`: a tool result (a delegated agent's included) with an `error` and `requires_user_action` ends the run after that tool turn, before another model request or compaction, and the agent clears its active skills, emits `failed` and returns the error as a `NodeUserError`. `agent-blocked-tool-turn-save-v1` gates the `agent.persist_turn` activity that this stop runs first when the agent has a conversation key (a Context node in a started generation): it appends the turn's tool results to the stored conversation (`append_tool_results`), so the conversation does not end on unanswered tool calls. The two `workspace-*` markers are described under [Direct Workspace tasks](#direct-workspace-tasks). Liveness is the activity layer's job: node activities heartbeat every 30s against a 2-minute `heartbeat_timeout`; their `start_to_close` is a generous 24h ceiling, not 10 minutes. The subagent-permit wait uses `PERMIT_WAIT_RETRY` (unlimited attempts) so a queued delegation waits as long as admission takes instead of failing after ~3h.- **History-pressure continue-as-new everywhere.** Temporal terminates any workflow around ~51,200 history events. `WorkflowControlWorkflow` (which multiplexes all of a deployment's triggers into one history) now rolls over on `is_continue_as_new_suggested()` / a 10K-event soft cap, carrying trigger specs, per-trigger provider `seen_ids` (written back into the spec after every poll cycle), queued push events, the bounded dedup baseline, and the control state — a rollover works mid-pause too, since a paused controller still accretes signal history. `TriggerListenerWorkflow`/`PollingTriggerWorkflow` gained the same pressure check (the old `_processed_count >= 16_000` gate was unreachable: real spawns cost ~15-25 events each, and polling counted only emitted events while a quiet mailbox burned ~11 events/cycle — dead in ~3 days at the 60s default). Poll intervals are clamped to a 30s floor (`_MIN_POLL_INTERVAL_S` in `polling_trigger_workflow.py` and `workflow_control_workflow.py`), with no patch gate. Because run ids change on rollover, **controller handles are addressed by workflow id only, never run_id-pinned** (`_controller_handle`, manager `register_trigger`).- **dispatch.emit controller narrowing.** Controllers advertise their push event types via the `ControlEventTypes` keyword-list Search Attribute (upserted as triggers register); `dispatch.emit` skips controllers with no matching trigger instead of signalling every running controller with every platform event (each unmatched signal was ~4 immutable history events — one busy deployment burned every other controller's rollover budget). Controllers without the attribute (pre-upgrade histories) keep match-all behaviour.- **Boot-time reconcile** (`reconcile_active_controls_on_boot`, called from the lifecycle module after workers start): runs the lazy `_reconcile_control` over every active control row, converges `starting` rows a crash left behind (controller alive with triggers registered → `running`; alive-but-empty for a graph that declares triggers → `failed` + controller closed; vanished → `failed`), and re-arms the process-local half of running/paused generations from the persisted graph snapshot — DeploymentManager state, in-process collectors for non-canary trigger types, cron pause posture. Idempotent by construction (controller `register_trigger` keyed by listener id, legacy starts use `USE_EXISTING`, cron creation preserves server-owned pause state).+**Controller event accumulation.** `controller-durable-queue-v2` preserves FIFO across carried events and the existing durable spillway. New execution-control version 1 histories also record `controller-event-accumulator-v1`: pending IDs remain protected beyond the recent-ID window, rollover includes Signals received during spill Activities, and a duplicate deterministic child start cannot block subsequent events. Legacy and pre-marker histories retain their recorded paths. Trigger processing stays immediate; no inactivity batching is added. See [event accumulation during Stop and rollover](./temporal-workflow-control.md#event-accumulation-during-stop-and-rollover) for scope and delivery limits.++**Controller Update traffic.** Fresh version 1 histories record `controller-messaging-v1` so successful Updates can request history-pressure rollover even without event Signals or pollers. The same gate prevents producer-only Safe Apply compatibility messages from reopening a stopped generation. Pure Update validators reject malformed control requests before acceptance; the main loop waits for handlers before rollover. See [workflow messaging review](./temporal-workflow-control.md#workflow-messaging-review).+**Startup sweep (debug-only escape hatch)**: [`TemporalClientWrapper.terminate_running_workflows`](../server/services/temporal/client.py) — gated on `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` (default **`false`**; keep it false — setting `true` converts every boot into a namespace-wide terminate sweep). Even when enabled, any control row in an active state (the shared `WORKFLOW_CONTROL_ACTIVE_STATES`, which includes `resetting`) vetoes the sweep. History is preserved (UI shows workflows as `Terminated`, not deleted); only active execution stops.**Port management**: Temporal owns `TEMPORAL_FRONTEND_GRPC_PORT` (gRPC) + `TEMPORAL_UI_PORT` (Web UI). Both bound by the same `temporal.exe` process; both listed in `cli.config.Config.all_ports` so `company stop`'s port-freeing pre-flight covers them.@@ -764,7 +948,7 @@ The Temporal binary + persistence are managed in-process by the plugin-folder pa| `temporal_sweep_attempts` | `TEMPORAL_SWEEP_ATTEMPTS` | `4` | Boot-time terminate-running sweep: retries for the Visibility query that races shard acquisition ("shard status unknown") before giving up for that boot. Only relevant when `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP=true`. || `temporal_sweep_backoff_seconds` | `TEMPORAL_SWEEP_BACKOFF_SECONDS` | `0.5` | Linear backoff base for the sweep retries (`attempt x base`). || `workflow_control_crash_recovery` | `WORKFLOW_CONTROL_CRASH_RECOVERY` | `pause` | After an UNCLEAN shutdown (kill/crash, dirty-bit marker), boot pauses generations still `running` so the user consciously resumes; `resume` restores them running. Clean restarts always restore as-is. |-| `workflow_control_missing_controller` | `WORKFLOW_CONTROL_MISSING_CONTROLLER` | `pause` | A live generation whose controller vanished converges to `paused` (Resume rebuilds the controller); `fail` preserves the legacy Reset-only behaviour. |+| `workflow_control_missing_controller` | `WORKFLOW_CONTROL_MISSING_CONTROLLER` | `pause` | Legacy generations can converge to `paused` and rebuild on Resume; `fail` keeps Reset-only recovery. Version 1 fails closed when its controller is lost because an empty replacement cannot recover authoritative root membership. || `workflow_control_pause_on_failure` | `WORKFLOW_CONTROL_PAUSE_ON_FAILURE` | `true` | Circuit breaker: repeatedly-failing trigger-spawned runs pause their deployment (fix + Resume) instead of firing into the same error indefinitely. Evaluated activity-side; never touches recorded commands. || `workflow_control_pause_on_failure_threshold` | `WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD` | `3` | Failed runs inside the rolling window required to trip the breaker — one node hiccup never pauses a deployment. `1` = pause on the first failure. Resume resets the streak. || `workflow_control_pause_on_failure_window_seconds` | `WORKFLOW_CONTROL_PAUSE_ON_FAILURE_WINDOW_SECONDS` | `600` | Rolling window for the failure streak; older failures age out (streak state lives in the cache table with a matching TTL). |diff --git a/docs-internal/agent_architecture.md b/docs-internal/agent_architecture.mdindex e230f3c5..8a3c665a 100644--- a/docs-internal/agent_architecture.md+++ b/docs-internal/agent_architecture.md@@ -225,7 +225,13 @@ Closure responsibilities:`tool_executor` can dispatch the new call.- The closure is gated on the user toggle: `UserSettings.auto_rebind_tools_after_canvas_change` (default `True`). When off, the LLM is told "Available on your next turn" in the operation summary and the closure isn't wired.-For the F4.B Temporal path, the in-process closure is replaced by the `agent.refresh_tools` activity, which applies the same filter and also binds agent delegates; see [TEMPORAL_ARCHITECTURE.md](TEMPORAL_ARCHITECTURE.md). It cannot see what the run already holds, so re-adding a tool added earlier in the same run makes `AgentWorkflow` report a duplicate tool name (the tool stays callable).+For the F4.B Temporal path, the closure is replaced by `agent.refresh_tools`.+The `agent-node-binding-refresh-v2` path passes bound node IDs, the run's graph,+and parameter snapshot, then filters repeated node bindings. A distinct node+that would introduce a conflicting provider-visible tool name is rejected+with conflict details. Older histories preserve their recorded refresh path.+Version 1 preparation/refresh also records plugin Activity policies for each+bound tool. See [TEMPORAL_ARCHITECTURE.md](TEMPORAL_ARCHITECTURE.md).### Where it's called@@ -554,10 +560,11 @@ Two settings flags route agent execution through different Temporal paths (see [| Flag | Off | On (default) ||---|---|---|-| `TEMPORAL_PER_TYPE_DISPATCH` | Every node routes through the legacy `execute_node_activity` single dispatcher (WS round-trip to the FastAPI handler). | Each node routes through its per-type activity `node.{type}.v{version}` registered via `BaseNode.as_activity()`. Per-plugin retry / timeout / heartbeat configs apply. |-| `TEMPORAL_AGENT_WORKFLOW_ENABLED` | All specialized + team leads + base agents (`aiAgent` / `chatAgent`) run inside `execute_node_activity` using the in-process native loop. | The migrating agent types become Temporal **child workflows** (`AgentWorkflow`). LLM steps + tool calls become activities; `agent.prepare_payload` resolves the DB-backed payload as the workflow's first step; `agent.refresh_tools` refreshes the native tool surface after canvas mutations. `rlm_agent` / `claude_code_agent` stay on the F4.A per-type activity path (externalised session state). |+| `TEMPORAL_PER_TYPE_DISPATCH` | Ordinary nodes use the legacy `execute_node_activity` dispatcher (WS round-trip); enabled agent child-Workflow routing still takes precedence for supported types. | Ordinary nodes use per-type `node.{type}.v{version}` Activities registered by `BaseNode.as_activity()`. Graph dispatch honors plugin retry/queue; Workspace-task patches and version 1 agent-tool dispatch also use plugin timeout/heartbeat. |+| `TEMPORAL_AGENT_WORKFLOW_ENABLED` | Supported agents run the in-process loop inside a node Activity: per-type when per-type dispatch is on, otherwise the legacy single dispatcher. | Types in `AGENT_WORKFLOW_TYPES` become **child Workflows** (`AgentWorkflow`). LLM steps and ordinary tools are Activities; preparation resolves configuration and explicit refresh binds canvas changes. `rlm_agent`, `claude_code_agent`, and `vertex_managed_agent` remain whole-node Activities. |-Both flags default to `true` in `.env.template`.+Both flags default to `true` in `.env.template`; the starter freezes their+values and the worker-pool routing flag into each execution's input.`AgentWorkflow` executions carry messages in the single `MessageWire` shape;there is one engine and one wire standard, with no `llm_engine` or@@ -566,10 +573,44 @@ there is one engine and one wire standard, with no `llm_engine` orafter a provider request starts, and changing the environment does not alter anexecution whose `agent.prepare_payload` result is already in history.-**Team leads** (`orchestrator_agent`, `ai_employee`) run the same-`execute_chat_agent` path as the other specialized agents but add an-`input-teammates` handle. Connected agents become `delegate_to_<type>` tools-automatically via `collect_teammate_connections()` (in+### Version 1 cooperative Stop and Resume++New generation manifests enable `execution_control_version=1`, separately+from the dispatch flags. Workflow gates close before model requests, tools,+compaction, explicit binding refresh, polling fetches, and child starts. Stop+drains already-admitted parallel work and its normal result bookkeeping;+unstarted actions wait. If a recorded response requested A and B, stopping+during A retains A's result and leaves B pending. Resume continues with B,+preserving the current model response and transcript.++The controller tracks graph/cron/job roots and detached delegation runners+that can outlive their parent. Attached agent descendants are discovered by+native execution descriptions after their child-start acknowledgement fence.+Completed Updates acknowledge admission closure and drain; Queries report+state. Resume holds producers while descendants/roots are released, then+releases controller, local, and schedule producers. Revision and chain checks+prevent stale intent or reused Workflow IDs from controlling another execution.++At clean continue-as-new boundaries, version 1 carries prepared configuration,+resolved tools and plugin timeout/retry/heartbeat/queue declarations, transcript,+thinking, iteration, usage, execution identity, and control state. A stopped+agent parks before rollover. The complete encoded next input is checked against+1,900,000 bytes; an oversized continuation raises `AgentContinuationTooLarge`+instead of returning to the opening prompt. Old generations retain their+legacy command paths and capacity fallback for replay compatibility.++Unlimited transient LLM retries can keep a Stop in `pausing`. A tool's retry+and idempotency contract still governs effects lost before recorded completion.+Opaque CLI/REPL/managed-agent nodes stop after the whole Activity settles;+heartbeats do not checkpoint their internal sessions. Direct Workspace tasks+and separately approved sends retain independent ownership. See+[Temporal workflow control](temporal-workflow-control.md) for lifecycle/API+details and [Temporal architecture](TEMPORAL_ARCHITECTURE.md#cooperative-generation-stop-and-resume)+for accounting, topology, and continuation.++**Team leads** (`orchestrator_agent`, `ai_employee`) share the supported agent+runtime and add an `input-teammates` handle. Connected agents become authorized+Task Manager assignees via `collect_teammate_connections()` (in`server/services/plugin/edge_walker.py`, called from`server/nodes/agent/_inline.py:prepare_agent_call`). See [agent_teams.md](agent_teams.md).@@ -708,7 +749,7 @@ Handle layouts are declared in `server/nodes/agent/_handles.py` (local to `nodesAI Agents delegate to other agents wired to their `input-tools` handle, enabling hierarchical agent trees. There are two execution paths depending on whether the parent runs under Temporal F4.B:-- **F4.B (default, `TEMPORAL_AGENT_WORKFLOW_ENABLED=true`)**: the parent `AgentWorkflow` spawns the child as a **child `AgentWorkflow`** via `workflow.execute_child_workflow` when the child type is in `AGENT_WORKFLOW_TYPES`. The parent's `node_id` rides in `child_context["parent_node_id"]` so the child's `_emit_phase` mirrors progress onto the parent's canvas badge. The LLM's `{task, context}` args travel as `child_context["invocation"]`, applied AFTER config resolution in `prepare_agent_payload` (so stored node parameters never override the delegated task; empty-task calls are rejected at the boundary without spawning). Deterministic child id `f"{parent_workflow_id}-delegate-{child_node_id}-{iteration}"`. The parent waits for the child result synchronously through the child-workflow handle.+- **F4.B (default, `TEMPORAL_AGENT_WORKFLOW_ENABLED=true`)**: direct non-team delegation starts attached `AgentWorkflow` children when the target type is supported. Parent identity mirrors progress and per-invocation `{task, context}` wins over saved configuration. The deterministic child ID includes parent Workflow ID, child node ID, iteration, and call index. Team leads use Task Manager instead: assignment persists, a detached `DelegatedTaskWorkflow` owns the permit and attached child, and the lead receives `queued` without waiting. Version 1 detached runners independently enroll with generation control; attached children are controlled through the native parent tree.- **Legacy / F4.A-only**: fire-and-forget via `asyncio.create_task`. Still the path for `rlm_agent` / `claude_code_agent` (excluded from `AGENT_WORKFLOW_TYPES`) and any deployment with `TEMPORAL_AGENT_WORKFLOW_ENABLED=false`. The `delegate_to_*` tool spawns the child as a background task and returns immediately with `{"status": "delegated", "task_id": "..."}`; the parent keeps working while the child executes independently and broadcasts its own status (executing / success / error).Design decisions (legacy path): **memory isolation** (child uses its own connected memory, not the parent's), **error isolation** (child errors are logged + broadcast, never propagated to the parent), **task tracking** (background tasks live in `_delegated_tasks`, cleaned up on completion).diff --git a/docs-internal/agent_context_flow.md b/docs-internal/agent_context_flow.mdindex 05880e46..5e6490de 100644--- a/docs-internal/agent_context_flow.md+++ b/docs-internal/agent_context_flow.md@@ -17,7 +17,7 @@ Companion documents: [RFC-0002](../RFC-0002-AGENT-CONTEXT-AND-MEMORY.md)Context is **plain JSON messages**. One table, `agent_conversations`, keyedby **`(workflow_id, generation, agent_node_id)`** → `messages` (a JSON list-of `MessageWire` dicts) + `updated_at`. A run **loads** the row at start and+of `MessageWire` dicts) + `updated_at`. A fresh firing **loads** the row at start and**saves** the whole live message list back after every turn. Every firing ofan agent — chat messages AND taskTrigger completion reviews — continues theone conversation; a workflow Reset admits a new generation, which is a new@@ -26,8 +26,10 @@ opt-in switch and the viewing panel, nothing more. This is the industrypattern (LangGraph `thread_id`, OpenAI `conversation_id`, Claude Codesession files): a conversation id maps to a message list, and nothing else.-There are no threads, sessions, epochs, hash chains, checkpoints, blobs, or-operation ids. Memory (`simpleMemory`) is an ordinary tool — explicit+The conversation store adds no threads, sessions, epochs, checkpoint tables,+blobs, or operation IDs. Temporal control revisions and suspended continuations+live in Workflow history and serve a separate execution contract.+Memory (`simpleMemory`) is an ordinary tool — explicitremember/recall — and plays no automatic part in continuity.## The store (`services/agent_context/`)@@ -85,7 +87,7 @@ thread resolution is involved — there is nothing to route.```mermaidflowchart TDR[AgentWorkflow run start] --> P{resume marker has<br/>carried transcript?}- P -- "yes (continue-as-new)" --> C1["messages = carried transcript verbatim<br/>(exact live conversation, size-guarded ≤ 1 MB)"]+ P -- "yes (continue-as-new)" --> C1["messages = carried transcript verbatim<br/>(version 1 guards the complete continuation input)"]P -- no --> Q{payload carries a stored<br/>conversation? (Context node,<br/>generation > 0, non-empty row)}Q -- yes --> C2["system (this firing's)<br/>+ stored wires minus system messages<br/>+ THIS firing's user prompt last"]Q -- no --> C3["bare build:<br/>system + memory markdown (legacy) + prompt"]@@ -94,44 +96,63 @@ flowchart TDPrecedence is exactly **carried transcript > stored conversation > barebuild** and is intentional:-- A rollover resumes the *same* run mid-flight — the live transcript is the- truth and must win over any store read.+- A rollover resumes the *same application execution* mid-flight with a new+ Temporal Run ID — the live transcript is the truth and wins over store reads.- A fresh firing with a Context node continues the *conversation* — thestored row is the truth.- The bare build is the cold-start floor.-The carried transcript caps at ~1 MB serialized-(`_CAN_TRANSCRIPT_MAX_BYTES`) because Temporal's payload error limit is-2 MiB for the whole continue-as-new argument. Over the cap, the transcript is-dropped with a warning and the resumed run seeds like a fresh firing: from-the stored conversation when a Context node is connected (with the firing's-prompt appended again), else from the opening prompt.--A continue-as-new starts `run()` from the top, so `agent.prepare_payload`-runs again. The resumed run picks up the node's current configuration (tools,-system message, model) and reloads the stored conversation with the load-checks below, even when the carried transcript is what seeds it. If that-stored row is over the 1 MB seed cap, the resumed run fails with-`ConversationTooLarge`.+Version 1 generations carry the prepared payload and resolved tools along with+the transcript. `AgentWorkflow` restores these values and skips+`agent.prepare_payload` on continuation, preserving provider/model, tool+definitions, plugin Activity policies, and explicit in-run binding changes.+Configuration edited during a pause or restart cannot reconstruct a different+continuation. Iteration, usage, context usage, thinking, application execution+identity, and control revision/hold state are carried too.++Rollover happens at a completed-turn boundary without live child handles or+Task Manager tasks. A stopped agent waits there for Resume and for control+handlers to finish. Existing result relief and compaction bound the transcript;+the payload converter then measures the **whole next input** against+`_CAN_INPUT_MAX_BYTES` (1,900,000 bytes). If it still cannot fit, version 1+raises non-retryable `AgentContinuationTooLarge`. It never drops the transcript+or repeats the opening prompt. The graph and prepared binding set count toward+this limit, so transcript size alone cannot establish that rollover will fit.++Legacy generations retain their recorded path: `agent.prepare_payload` runs+again at rollover and can resolve current configuration; a carried transcript+over `_CAN_TRANSCRIPT_MAX_BYTES` (1,000,000 bytes) is dropped with a warning,+then seeding falls back to the stored row or opening prompt. A stored row over+the seed cap can still raise `ConversationTooLarge`. These behaviors remain+for replay compatibility and are not the version 1 continuation contract.+See [Temporal architecture → Agent continuation](TEMPORAL_ARCHITECTURE.md#agent-continuation-under-history-pressure)+and [Temporal Continue-As-New](https://docs.temporal.io/design-patterns/continue-as-new).### Stopped turns and unanswered calls-The owner can stop a chat run mid-answer ([Chat Protocol → Streaming, steps-and Stop](./chat_protocol.md#streaming-steps-and-stop)). The LLM step then-saves the turn as far as it went, like any other step: the exact sent list,-plus the partial answer when the model had written something (its text block-carries `metadata: {"stopped": true}`); with nothing written, the sent list-alone, so the owner's message stays in the conversation unanswered. A tool call-of a stopped run gets a result saying it was not run, so the transcript stays-whole.--A run cancelled while a tool ran (the watchdog cancels a run still stopping-after the grace; a crash) can leave the last assistant turn with calls that-have no result, which no provider will continue from. The load therefore+For a chat run owned by a version 1 generation, Stop suspends the generation+after admitted Activities and their result bookkeeping finish. An admitted+LLM step completes under its existing retry policy, and its exact assistant+message and pending tool calls stay in the existing Workflow continuation.+An admitted tool's result is consumed normally; unstarted tools wait for+Resume and are not replaced by synthetic "not run" results. Stop does not+terminalize the reply, release its lane, or start a fresh firing. The watchdog+does not expire an intentionally stopped controlled run by age; Resume gives+it a fresh timeout window. See [Chat Protocol → Streaming, steps and Stop](./chat_protocol.md#streaming-steps-and-stop)+and [Temporal workflow control](./temporal-workflow-control.md).++Legacy/uncontrolled chat Stop keeps the terminal cooperative-stop behavior:+the LLM step saves the sent list and any partial answer (text metadata+`{"stopped": true}`), and an unstarted tool receives a "not run" result.+The legacy grace-period watchdog can cancel an unfinished run. A cancellation+or other terminal interruption while a tool runs can leave calls without+results, which no provider will continue from. A later fresh load thereforepasses the stored row through `close_unanswered_tool_calls`(`services/agent_context/conversation.py`), which answers each open call of thelatest assistant turn with `UNANSWERED_TOOL_RESULT`, after the results the turn-already has. A closed conversation is returned unchanged.+already has. A closed conversation is returned unchanged. This repair applies+to a stored row loaded for a new firing; it is not the checkpoint or Resume+mechanism for a suspended version 1 agent.### Chat branches rewind and restore@@ -333,7 +354,7 @@ workflow, not only the removed Context's agent.| 8 | Specialized bridges record the ORIGINAL prompt, never the augmented one. | Recording the rendered transcript nests the conversation inside itself and grows without bound. || 9 | The store never imports `nodes/`; the plugin registers its broadcaster via `register_conversation_listener`, and a listener failure can never fail a save. | Same layering rule as every plugin registry; a UI notification must not break execution. || 10 | External tool results are capped before they enter the transcript; the latest turn is never cleared or summarized; the pressure rules are chosen by the recorded `context_pressure_version`. | Uncapped results bricked every later firing (`errors.md` #28); an unread turn summarized away loses what the model asked for; a replay must schedule the commands it recorded. |-| 11 | A stopped turn is saved as far as it went, and a load answers any tool call the stored row left open. | A provider refuses a conversation with an unanswered call, so one cancelled run would otherwise break every later firing. |+| 11 | Version 1 Stop retains the live continuation and drains admitted results; a new stored-row load repairs calls left open by terminal interruptions. Legacy Stop saves the partial turn. | Suspension must preserve pending calls, while a later fresh firing must not send an unanswered-call conversation to the provider. || 12 | Only `save_conversation` and a chat branch move (under `conversation_lock`, checked against the run's cursor) rewrite a stored conversation. | A move that ignored the lock or the cursor would cut a turn being saved, or cut a summarized conversation at the wrong place. |## How this broke (August 2026 regression), and why the journal went awaydiff --git a/docs-internal/agent_delegation.md b/docs-internal/agent_delegation.mdindex e102f9f6..f82c3ba8 100644--- a/docs-internal/agent_delegation.md+++ b/docs-internal/agent_delegation.md@@ -26,6 +26,49 @@ AI Agents can delegate tasks to other agents connected to their `input-tools` haIn both paths the parent passes only a `task` string and optional `context` string — per-invocation input that always wins over stored configuration. Everything else -- the child's provider, model, memory, skills, and tools -- comes from the child's own configuration and workflow connections.+## Generation Stop/Resume ownership++The F4.B path has two lifetimes. Direct non-team delegation starts an attached+`AgentWorkflow` child and collects its result through the existing child+handle. Task Manager assignment starts a detached `DelegatedTaskWorkflow`+runner with `ParentClosePolicy.ABANDON`; the runner can outlive the assigning+lead and owns its attached agent child, permit, and terminal bookkeeping.+Native parent discovery alone cannot find the runner after the lead closes.++For new `execution_control_version=1` generations, every detached runner+enrolls with the existing generation controller through an acknowledged Update+before admitting business work. The maintenance Activity/client bridge returns+current pause/revision/hold posture so a late runner cannot bypass Stop.+Attached agent children inherit the scope and require no separate registration.+The controller's independent-root map and membership epoch survive rollover;+control checks the execution chain while addressing the current Workflow ID.++Stop closes admission throughout the discovered topology before waiting for+admitted Activities and result bookkeeping to drain. A child-start gate counts+only until start acknowledgement, enabling reliable native enumeration. A+parent waiting for child completion or a runner waiting for a subagent permit+does not hold the local drain count; its child is acknowledged separately.+Stopped children therefore do not deadlock a parent's checkpoint. Completion+persistence and permit cleanup for work already completed still run normally.+No task is cancelled or re-queued merely because its generation is stopped.++Resume publishes a newer revision with producers held, releases descendants and+registered roots, reconciles membership, and releases controller/local/schedule+producers. It uses existing handles and pending continuations rather than+creating replacement runners or repeating completed tools. Pending task events+remain queued for later processing. Agent continuation carries prepared+configuration and bindings; rollover is deferred while child handles or Task+Manager tasks remain live.++The stronger guarantee is limited to version 1 Temporal generations. It does+not turn process-local `asyncio.Task` delegation inside an opaque node into+independent Temporal roots. Claude Code, RLM, and Vertex managed nodes remain+whole-Activity boundaries; their internal calls may continue until that+Activity settles. Unlimited LLM retries can delay complete Stop acknowledgement,+and ambiguous external effects remain subject to tool retries/idempotency.+See [Temporal workflow control](temporal-workflow-control.md) and+[Agent continuation](TEMPORAL_ARCHITECTURE.md#agent-continuation-under-history-pressure).+## Architecture```diff --git a/docs-internal/agent_teams.md b/docs-internal/agent_teams.mdindex 09f2d59e..bf2496e7 100644--- a/docs-internal/agent_teams.md+++ b/docs-internal/agent_teams.md@@ -106,6 +106,38 @@ bounded mission, relevant context, and its own connected tools, skills, andmemory. The parent receives a compact result; full child output remainsinspectable through task attempts and execution traces.+## Generation Stop and Resume++New Temporal generations carry `execution_control_version=1`. Detached+`DelegatedTaskWorkflow` runners enroll independently with the generation+controller before business work, so Stop still reaches their attached agents+after the assigning lead completes or the controller rolls over. Attached+children use their native parent relationships rather than a per-tool registry.++Stop preserves assigned tasks, lane/permit ownership, and existing continuations.+Already-admitted model/tool Activities and their result bookkeeping drain;+queued work, later calls, and new child starts wait behind the admission gate.+Permit waits and child-result waits do not count as local business drain, so+paused children cannot deadlock the checkpoint. Completion persistence and+permit release for completed work continue normally. `pausing` describes the+request/drain phase; `paused` requires topology-wide checkpoint acknowledgement.+Stop adds no second persisted task pause lifecycle and does not cancel tasks.++Resume applies the next revision with producers held, releases descendants and+independent roots, reconciles membership, then releases producers. It continues+the existing task attempt without another model response or completed tool+being explicitly scheduled. Completion events accumulated while stopped remain+pending for review; handlers do not batch inactivity or discard them.++Existing generations preserve their legacy protocol. Unlimited LLM retries may+leave Stop in `pausing` during an outage; a tool's policy/idempotency still governs+unrecorded external effects. Opaque managed agents finish the whole node+Activity before acknowledging Stop. Direct Workspace tasks and separately+approved sends remain independent. See+[Temporal workflow control](temporal-workflow-control.md),+[Delegation ownership](agent_delegation.md#generation-stopresume-ownership), and+[Agent context flow](agent_context_flow.md#stopped-turns-and-unanswered-calls).+## Completion and eventsAfter durable persistence, lifecycle transitions emit deterministic events suchdiff --git a/docs-internal/chat_protocol.md b/docs-internal/chat_protocol.mdindex 58fe6357..ab5dd63f 100644--- a/docs-internal/chat_protocol.md+++ b/docs-internal/chat_protocol.md@@ -1,4 +1,4 @@-# Chat protocol (v1)+# Chat protocol (v2)The wire contract between the server's chat runtime (`server/services/chat/`) and the shared chat UI(`client/src/features/chat/`). Home's employee page and Dev's console Chat pane both speak it. It follows AG-UI's@@ -9,10 +9,13 @@ AG-UI JSON.Status: built. Change this document in the same commit as any change to the shapes below, and bump`protocol_version` (returned by `get_chat_messages`) when an existing field changes meaning.+Version 2 distinguishes resumable generation Stop from terminal legacy chat Stop. The generation's+`execution_control_version`, rather than the wire version alone, selects the behavior.Only the AG-UI events the chat uses are sent. A tool call shows as a step, not as `TOOL_CALL_*` events; reasoning isnever streamed; and a draft waiting for the owner is an approval card that follows `approval_lifecycle`-([Approvals](#approvals)), so no run ends with AG-UI's `interrupt` outcome and nothing resumes one.+([Approvals](#approvals)), so no run ends with AG-UI's `interrupt` outcome. A controlled generation can suspend+and resume its existing run through workflow control without ending the chat run.## Concepts@@ -20,7 +23,7 @@ never streamed; and a draft waiting for the owner is an approval card that follo|---|---|| Session | One conversation. Its id is the workflow id (`"default"` with no workflow open). || Run | One answer the employee works on: an owner message, an edit, a regenerate, or a button press in generated UI (kinds `message`, `edit`, `regenerate`, `action`). An approved send is not a run: `ApprovedToolCallWorkflow` sends it. |-| Lane | At most one non-terminal run per session. A second send is refused with `run_in_progress`; the composer shows Stop instead of Send while a run is live. |+| Lane | At most one non-terminal run per session. A second send is refused with `run_in_progress`; the composer shows Stop while a run is live and Resume while its controlled generation is stopped. Suspension keeps the lane occupied. || Message tree | Messages link to their parent. Owner messages that share a parent are branches; replies under the same owner message are versions. The session's active leaf picks the path shown. || Part | Structured content attached to a reply: steps, generated UI, artifacts, approvals, sources, follow-ups. || Generation | The deployment generation a message was written in (`run_key`). A Reset clears the thread. |@@ -83,7 +86,8 @@ Every event's `data` carries `{workflow_id, session_id, run_id, seq, hub_epoch}`(a Canvas document the run wrote). Sources and follow-ups arrive with the saved reply, in its `parts`.`custom` names: `opencompany.segment_discarded` (`{message_id}`: a retried LLM attempt replaces this segment),-`opencompany.stopping` and `opencompany.resync`.+`opencompany.stopping` (legacy terminal Stop) and `opencompany.resync`. Controlled Stop derives its display from+generation control; it does not emit a second chat pause lifecycle.**Text segments.** Only the agent that answers the owner streams text. A segment that ends with `final: false` waswritten beside tool calls (narration, "Let me check the calendar."); the segment with `final: true` is the reply.@@ -95,12 +99,13 @@ streamed.| `outcome.type` | Meaning ||---|---|| `success` | The run ended. `result.reply_message_id` names the saved reply, or `result.no_reply` is true. |-| `stopped` | The owner pressed Stop (an OpenCompany addition to AG-UI's outcomes). |+| `stopped` | A legacy or uncontrolled chat run ended after Stop (an OpenCompany addition to AG-UI's outcomes). Controlled Stop does not emit this terminal outcome. |Runs nothing will finish end with `run.failed`: code `not_delivered` (never picked up, or the employee stopped-first), `timed_out` (running longer than `runs.max_running_s`), `interrupted` (its workflow closed without finishing+first), `timed_out` (running longer than its current `runs.max_running_s` window), `interrupted` (its workflow closed without finishingit and no reply was saved; with a saved reply it finishes instead). A Reset ends live runs with code `reset`, the-owner's Clear with `cleared`, and both delete them.+owner's Clear with `cleared`, and both delete them. Intentional suspension of a controlled generation excludes+its live runs from pickup and running expiry; Resume starts a fresh timeout window, as described below.## Handlers@@ -116,7 +121,7 @@ All are WebSocket request/response handlers with snake_case payloads. Failures a| `chat_subscribe` | `{session_id}` | `{success, session_id, hub_epoch, active_runs: [RunSnapshot]}` || `chat_unsubscribe` | `{session_id}` | `{success, session_id, hub_epoch}` || `get_chat_run` | `{run_id}` | `{success, run: RunSnapshot}`, or `not_found` |-| `stop_chat_run` | `{run_id}` | `{success, run_id, state: "stopping" \| "stopped"}`; `not_stoppable` (with `state`) for a run that ended otherwise |+| `stop_chat_run` | `{run_id, expected_revision?, idempotency_key?}` (revision/key required for controlled generations) | Controlled: generation control snapshot plus `{run_id, resumable: true}`, `state` is `pausing` or `paused`. Legacy: `{success, run_id, state: "stopping" \| "stopped"}`. `not_stoppable` for a terminal run. || `chat_ui_state` | `{session_id, part_id, changes: [{path, value}]}` (at most 32; the last value per path wins) | `{success, part_id, state_revision}`; `not_found`, `invalid_request` || `edit_chat_message` | `{session_id, message_id, message, expected_revision?, client_message_id?}` | `{success, message_id, run_id, delivery}` (the edit's id; a resent `client_message_id` answers the first edit) || `regenerate_chat_reply` | `{session_id, message_id, expected_revision?}` (the latest answer, or the owner's last message when its run gave none) | `{success, message_id, run_id, delivery}` (`message_id`: the owner's message answered again) |@@ -136,6 +141,13 @@ activity_type, content, patches}], outcome, result, error, error_code}`. `state``running`, `stopping`, `finished`, `error` or `stopped`. A snapshot read after a server restart has its steps but notext segments or activities (those are stored with the reply).+For a run whose owning generation declares `execution_control_version: 1`, snapshots from `get_chat_messages`,+`chat_subscribe` and `get_chat_run` also include `workflow_control`, serialized from that generation's existing+control row. It includes `workflow_id`, `root_execution_id`, `generation`, `execution_control_version`, `state`,+`revision`, the control capabilities, and `last_resumed_at`. This is derived control state, not an additional+`ChatRun.state`: a suspended run may still have state `running`, `pending` or `queued`. Terminal states in the+chat reducer remain terminal even when a control snapshot is present.+### Approvals (`server/services/approvals/handlers.py`)| Handler | Request | Response |@@ -227,8 +239,8 @@ Memory updates) still reach only that process.- `status`: `complete`, `stopped`, `error`.A reply shows, in this order whatever order they were produced in: its run's steps (`run.steps` above), its text,-then its parts: generated UI, artifacts, approvals, sources, follow-ups. A reply the owner stopped has `status:-"stopped"`.+then its parts: generated UI, artifacts, approvals, sources, follow-ups. A reply ended by legacy Stop has+`status: "stopped"`. Controlled suspension does not change a saved reply's status or allocate a replacement reply.```parts: {@@ -309,10 +321,16 @@ newer version wins in place.filter (`event_waiter.build_filter`). Otherwise it is saved and dispatched without a run, like the editor's`"default"` session, which keeps its unscoped delivery.- **Dispatch.** The `chat_message_received` event (`services/chat/events.py`, source `opencompany://services/chat`)- has the run id as its CloudEvent id and carries `message_id` and `run_id` in `data`, so the listener's child run- id is `<slug>-<trigger label>-<run id>`. It is never broadcast. Open threads hear of the message (`chat.updated`)- only after it is dispatched, so the sender's answer, which admits the run into their store, reaches them before+ has the run id as its CloudEvent id when tracked, otherwise the saved message id, and carries `message_id` and+ `run_id` in `data`. The generation controller receives it through `on_event`, deduplicates each registered+ trigger's event key, and queues or starts its child `MachinaWorkflow`; legacy standalone listeners retain their+ existing route. The child id is `<slug>-<trigger label>-<event id>`. It is never broadcast. Open threads hear of the message (`chat.updated`)+ only after the dispatch attempt, so the sender's answer, which admits the run into their store, reaches them beforetheir thread can read the run: told first, the thread showed the run live and read it with `get_chat_run`.+ The ingress path still uses eventually consistent Visibility and logs delivery errors: saving a message or+ returning `delivery` does not prove Temporal accepted its Signal. The accumulator's durable pending-event+ guarantee begins at Signal acceptance; no ingress outbox was added. A duplicate `client_message_id` returns+ the original message and run without redispatching it.- **Start and finish.** MachinaWorkflow, behind the `machina-chat-run-v1` patch, reads the run id only from an eventwith that source and type, claims the run (`chat_run.start`: `pending` or `queued` to `running`, recording theTemporal workflow and run ids) once the firing trigger's output is stored, passes `run_scope {run_id, session_id}`@@ -325,8 +343,13 @@ newer version wins in place.- **One chain.** Every message is appended after the session's active leaf (`chat_threads`) inside the same reservedtransaction, so concurrent writes never fork the thread.- **The watchdog** (`services/chat/watchdog.py`, started by `main.py`) sweeps every `runs.watchdog_interval_s` and- ends the runs nothing will finish (codes above). A pending run's wait (`runs.pickup_timeout_s`) counts from the- later of its creation and the server's start.+ ends the runs nothing will finish (codes above). It resolves each controlled run's owning generation, not the+ workflow's latest successor. While that generation is `pausing`, `paused` or `resuming`, its pending, queued and+ running runs are excluded from age expiry. After successful Resume, the existing generation manifest stores+ `last_resumed_at`: pending pickup counts from the latest of creation, server start and Resume; queued pickup+ follows the resumed delivery window; running expiry counts from the later of run start and Resume. Stop does+ not move controlled runs into the legacy `stopping` state, so they bypass grace-period cancellation. Temporal+ closure checks follow the current Workflow ID and compare execution chains across continue-as-new.## Streaming, steps and Stop@@ -349,7 +372,17 @@ Settings are in `server/config/chat_defaults.json` (`stream`, `steps`, `runs`).none, and skill loads (`agent.skill.invoke`) never pass through it. A tool may put a short line in its result as`_step_detail` ("3 events on Saturday"): it is shown under the step and taken out before the model reads theresult.-- **Stop** (`stop_chat_run`, `ledger.request_stop`). A run nothing has picked up (pending, or queued until Resume)+- **Controlled Stop/Resume** (`execution_control_version: 1` on the owning generation). `stop_chat_run` resolves+ the run's exact generation using `workflow_id` and `run_key` matched to `root_execution_id`, then calls generation Stop with the revision,+ idempotency key and expected generation root. Already admitted work finishes and its results are recorded;+ queued work waits. The run remains nonterminal with the same reply identity, subscriptions, partial output and+ occupied lane. Live snapshots include the derived `workflow_control`; no additional persisted chat pause state+ exists. The watchdog excludes `pausing`, `paused` and `resuming` generations from pickup and running expiry;+ successful Resume supplies a fresh timeout window using lifecycle metadata `last_resumed_at`. Resume uses+ `resume_workflow` and releases the existing continuation. Independent Workspace tasks and separately approved+ sends remain independent. Full execution-tree admission, checkpoint, rollover and compatibility rules are in+ [Temporal Workflow Control](./temporal-workflow-control.md).+- **Legacy Stop** (`stop_chat_run`, `ledger.request_stop`). A run nothing has picked up (pending, or queued until Resume)ends `stopped` at once and frees the lane; a workflow that picks it up later claims it still `stopped`(`chat_run.start`) and its agent answers nothing. A running run moves to `stopping` (`custom``opencompany.stopping`) and stops itself:@@ -367,8 +400,35 @@ Settings are in `server/config/chat_defaults.json` (`stream`, `steps`, `runs`).- **Agents that are not AgentWorkflows.** Only the agents in `AGENT_WORKFLOW_TYPES` (`services/temporal/workflow.py`)run as an AgentWorkflow and are prepared by `agent.prepare_payload`. The others (Claude Code, Codex, RLM, Vertex)run as one activity and get no `chat_stream`: nothing they write streams, their tool calls show no steps, they- cannot show UI, and their answer appears when Reply in Chat saves it. Stop moves their run to `stopping`, nothing- in them checks it, and the watchdog ends it `runs.stop_grace_s` later.+ cannot show UI, and their answer appears when Reply in Chat saves it. Controlled Stop drains their whole activity+ before suspension. Legacy Stop moves their run to `stopping`, and the watchdog ends it `runs.stop_grace_s` later.++### Controlled Stop acknowledgement++The client sends `stop_chat_run {run_id, expected_revision, idempotency_key}`. The server authorizes the run's+session, resolves its immutable owning generation and supplies `expected_root_execution_id: run.run_key` to+`pause_workflow`. A stale chat run cannot stop a newly started generation. The response is the existing workflow+control payload, with `run_id` and `resumable: true` added; it is not a terminal chat-run snapshot.++`pausing` means Stop was requested and admission is closing or work is draining. Only `paused` acknowledges+that admitted business work and result bookkeeping have settled across the controlled execution tree. For+example, Stop during tool A lets A finish once and records its result; tool B from the same model response stays+pending. Resume starts B in the same continuation without requesting that model response again. Admitted parallel+tools drain concurrently under their declared retry policies. An LLM call keeps its existing unlimited retries,+so a persistent provider outage can leave the generation Stopping until that call settles.++The request's acknowledgement deadline may expire while that durable transition continues. Keep the returned+or refreshed `pausing` state, reconcile with `get_workflow_control_status`, and do not synthesize a chat `stopped`+outcome or cancel the workflow. Resume sends `resume_workflow {workflow_id, expected_revision, idempotency_key}`+after checking that the latest control still names the run's owning root; it releases existing descendants before+producer admission reopens. The protocol uses completed Temporal Updates for control acknowledgement and+read-only Queries for status, following [Workflow Messaging Patterns](https://docs.temporal.io/design-patterns/workflow-messaging-patterns).++Run and reply ids, saved parts, completed steps, subscriptions and lane occupancy survive suspension. Live text+already received stays in the mounted conversation and in the process's hub snapshot. A reload restores the+owning generation's control and persisted chat state; a server process restart still cannot reconstruct text+deltas that were never saved in a reply. This existing streaming limitation is separate from durable Temporal+continuation: Resume does not explicitly rerun completed tools to reconstruct the display.## Client@@ -402,7 +462,8 @@ editor's console pane (`ConsoleChat`, compact, scope `live`) are the two hosts.the key its run's turn had, so neither remounts.- **Sending** (`data/send.ts`): the message shows at once; the server's answer admits its run into the store(`queued` or `pending`), so the employee shows working before the run's first event. While the lane is held, Send- is Stop. A refused or failed send takes the message out of the thread and puts its text back in the box+ is Stop in a running generation, disabled Stopping during `pausing`, and Resume during controlled suspension.+ A refused or failed send takes the message out of the thread and puts its text back in the box(`state/composerStore.ts`, a draft per session that survives switching conversations); after a failure in transitthe draft keeps its `client_message_id`, so sending it again is the same message: the server answers with thefirst send's message and run, and when the thread already holds that message the local copy gives way to it.@@ -412,8 +473,15 @@ editor's console pane (`ConsoleChat`, compact, scope `live`) are the two hosts.steps disclosure (`turns/StepsDisclosure.tsx`) sits on the run's first turn: "Working…" and open while the runworks, "Worked for 12s · 3 steps" after; a run read back later starts it closed. A finished run keeps its turnuntil its saved reply lands in the thread, so the streamed answer and the reply stay one element.-- **Stop** (`data/stop.ts`): the Stop button, or Esc anywhere in the pane, sends `stop_chat_run` for the lane's run- and applies the answer to the store at once; the run's events take it from there.+- **Stop/Resume** (`data/stop.ts`, `data/control.ts`): Stop or Esc sends `stop_chat_run` for the lane's run.+ Controlled requests use `WebSocketContext.controlMutation`, with a fresh idempotency key per bounded request+ attempt and the current control revision. Success and failure responses merge authoritative control snapshots+ before the request is judged; a transport or acknowledgement failure triggers a status resync. Controlled runs+ retain their chat state: the status line shows Stopping, the stopped turn says it waits for Resume, and the+ composer offers Resume, then disabled Resuming. `useChatWorkflowControl` merges a reload snapshot with newer+ broadcasts for the same root and retains the run's owning snapshot when a successor root differs. Resume+ re-reads status and checks the owning root before sending the current revision. Esc has no effect while stopped,+ stopping or resuming. Legacy responses still update the run's terminal lifecycle.- **The box's extras** (`composer/`): Attach, a paste and a drop on the chat (`DropOverlay`) all call`addAttachments` (`composer/attachments.ts`), which uploads each file at once into the box(`state/attachmentStore.ts`, chips with progress and Remove); Send waits while one uploads and sends the finished@@ -631,6 +699,10 @@ and server tests alike:| `not_found` | get_chat_run, stop_chat_run | No such run. || `attachment_rejected` | send | A file outside `uploads/`, gone, or more than six; `detail` says which. || `not_stoppable` | stop_chat_run | The run ended before Stop reached it; `state` says how. |+| `idempotency_key_required` / `expected_revision_required` | controlled stop_chat_run | The generation mutation lacks its request identity or revision. |+| `control_generation_conflict` | controlled stop_chat_run, Resume | The run's owning root differs from the generation currently being controlled. Refresh control; do not target the successor using the old run. |+| `control_revision_conflict` | controlled stop_chat_run, Resume | The generation revision changed; the response/status supplies the authoritative control. This revision is distinct from the thread's revision. |+| `workflow_control_transition_pending` | controlled stop_chat_run, Resume | A durable lifecycle transition is still in progress. Reconcile control status; the chat run remains live. || `revision_conflict` | edit, regenerate, switch | `expected_revision` is not the thread's revision. || `rule_conflict` | set_ask_first | `expected_revision` is stale; the answer carries the current value. || `not_editable` | edit, regenerate, decide | Not the owner's text message a run answered, not the latest answer, the editor's `default` chat, or an edited argument that is not editable. |diff --git a/docs-internal/cli_agent_framework.md b/docs-internal/cli_agent_framework.mdindex 1a74a9b7..5bfecd25 100644--- a/docs-internal/cli_agent_framework.md+++ b/docs-internal/cli_agent_framework.md@@ -63,6 +63,25 @@ Reuses (do not duplicate):- `nodes/agent/claude_code_agent/_oauth.py` — Claude `auth login` / `auth status` / `auth logout` wrappers, `bun add` (via `core/js_runtime.add_package`, `trust=True`) into the shared OpenCompany packages tree at `<DATA_DIR>/packages/` (binary resolves to `<DATA_DIR>/packages/node_modules/.bin/claude[.exe]` and runs on bun), `CLAUDE_CONFIG_DIR=<DATA_DIR>/claude/`. The `login` spawn passes `stdin=PIPE` (un-written) so the native CLI's stdin reader blocks instead of EOFing — keeps its localhost OAuth callback server alive until the browser flow completes- `nodes/stripe/_handlers.py` — pattern reference for marker-token + catalogue broadcast+## Generation Stop and CLI session resume++CLI agent nodes remain whole-node Activities in Temporal, outside+`AGENT_WORKFLOW_TYPES`. New generation control (`execution_control_version=1`)+closes admission before later node/tool Activities and waits for an admitted CLI+Activity, including its subprocess and MCP work, to settle under its scheduled+policy. Result bookkeeping completes normally before the graph/parent agent+waits. Generation Resume releases the existing continuation; it does not create+a replacement CLI task or repeat a recorded completed Activity.++Claude's native `--resume <session_id>`, warm session reuse, and pool crash+recovery are separate provider/session mechanisms described below. Generation+Stop does not pause between CLI-internal tools, and heartbeat strings cannot+serialize the subprocess or JSONL session. External effects before recorded+completion remain governed by retry/idempotency. Legacy generations retain+their earlier control behavior. See [Temporal workflow control](temporal-workflow-control.md),+[Temporal architecture](TEMPORAL_ARCHITECTURE.md#cooperative-generation-stop-and-resume),+and the [Claude Code node card](node-logic-flows/specialized_agents/claude_code_agent.md#temporal-stopresume-boundary).+## Provider abstraction (mirrors `services/llm/`)```pythondiff --git a/docs-internal/employee_teams.md b/docs-internal/employee_teams.mdindex 4e168d2e..fc87ccf9 100644--- a/docs-internal/employee_teams.md+++ b/docs-internal/employee_teams.md@@ -34,6 +34,16 @@ five-second active snapshot polling and coalesced status refresh; idle pollingruns every fifteen seconds. Pauses, approvals, disconnects and slow work showplain-text status and last activity. Raw internal reasoning is never displayed.+For newly started Temporal generations, Stop closes admission across the+controller, enrolled job/graph/detached roots and their attached agent children.+Admitted work and bookkeeping drain before Stopped is acknowledged. Resume+releases the same pending model/tool continuation without repeating recorded+completed tools. Queued events and the chat run's lane remain occupied while+stopped. Independent Workspace tasks and separately approved sends retain their+own lifecycle. See the [control contract](temporal-workflow-control.md) and+[chat protocol](chat_protocol.md) for revision guards, timeout reconciliation,+restart behavior and legacy compatibility.+Creation uses a stable owner-scoped key and payload hash, renewable lease,atomic graph/parameter/metadata/grant save, and a recoverable activationintent. Failed or abandoned builds keep the same workflow identity. A saved@@ -46,8 +56,10 @@ remain authoritative. Capability grants are scoped and revocable in Settingsand checked again inside the graph mutation. Public app messages cannotapprove expanded access. Execution approval for sending remains separate.-Safe Apply stops new admissions while allowing existing work and its reviews-to finish. It replaces future snapshots without Reset, preserving Context,+Safe Apply pauses producer admission while allowing existing work and its reviews+to finish when generation execution admission is open. It cannot reopen a+stopped generation or drain reviews while Stop/held Resume closes admission.+It replaces future snapshots without Reset, preserving Context,memory, chats, files, approvals and queued events. Paused employees staypaused. “Stop work and apply” is the explicit interruption path. The editor'sReset continues to clear execution state as before.diff --git a/docs-internal/event_framework.md b/docs-internal/event_framework.mdindex ce5d261a..11e21fb7 100644--- a/docs-internal/event_framework.md+++ b/docs-internal/event_framework.md@@ -1,17 +1,19 @@-# Event Framework (Wave 12)+# Event Framework and Durable Trigger Queues> **Current architecture:** This document retains the Wave-12 rollout history.> Controlled deployments now consolidate push registration, event Signals, and> polling activities inside `WorkflowControlWorkflow`; see-> [Temporal Execution Engine RFC](temporal-execution-engine-rfc.md). Listener+> [workflow control contract](temporal-workflow-control.md). Listener> workflows described in the phase log are legacy compatibility contracts.Temporal-native event-routing layer for OpenCompany. Implements RFC sections6.3 (Temporal worker contract) + 6.4 (CloudEvents broadcast contract) from[plugin_authoring_rfc.md](./ARCHIVE/plugin_authoring_rfc.md).-This doc is the operator + plugin-author reference. The design rationale +-phase plan lives in `~/.claude/plans/properly-fix-the-tech-dreamy-tarjan.md`.+This is the operator and plugin-author reference. The phase log below is+historical; the current control and queue contract is documented here and in+[Temporal architecture](TEMPORAL_ARCHITECTURE.md). The original design is in the+[archived execution RFC](ARCHIVE/temporal-execution-engine-rfc.md).## Status (2026-05-15)@@ -57,18 +59,25 @@ Locked by `tests/test_event_framework_phase_a.py::TestEventFrameworkEnabledDefau## What this framework does-Every inbound event (HTTP webhook, Telegram message, Gmail poll result,-task completion, …) becomes a Temporal Signal delivered to whichever-running workflows are waiting on that event type. Routing happens via-the Temporal Visibility API — workflows tag themselves with custom-Search Attributes at start, and the dispatch helper queries-`ListWorkflows(query="EventType='X' AND ExecutionStatus='Running'")`-to find consumers.--Why Temporal: durability, replay safety, server-side dedup-(`WorkflowIDReusePolicy`), and zero custom infrastructure. The framework-adds ~300 LOC on top of Temporal primitives rather than reinventing an-EventBus + event_log + DLQ table.+Push producers build a `WorkflowEvent` and call `dispatch.emit`. Routing uses+Temporal Visibility to find running legacy `EventType` consumers and generation+controllers. A controller advertises `ControlEventTypes` and matches accepted+Signals against its registered trigger definitions. Polling controllers instead+call provider Activities and enqueue their returned events; cron uses Temporal+Schedules and the plugin's `CronTriggerWorkflow`.++Visibility is eventually consistent. The dispatch helper logs and suppresses+Temporal connection, lookup and individual Signal failures, and returns the+envelope rather than a delivery receipt. Its return therefore does not prove+that a consumer accepted the event. Once a Signal RPC succeeds, Temporal stores+the Signal durably; successful acceptance still does not acknowledge handler+processing or graph execution. There is no producer outbox in this path.++Controller queues preserve accepted events across Stop, worker restart and+continue-as-new. Deduplication is an application key contract, supplemented by+deterministic child Workflow IDs; Workflow ID policies do not deduplicate+arbitrary Signal payloads. Existing SQL overflow pages and atomic read receipts+are reused for queue spill, without adding an event bus or separate DLQ.## Architecture@@ -76,8 +85,9 @@ EventBus + event_log + DLQ table.Inbound source (FastAPI process)↓services/events/dispatch.py:emit(event: WorkflowEvent)- ├─→ Temporal Visibility query: workflows where EventType=event.type- ├─→ Signal each matching workflow+ ├─→ Temporal Visibility: legacy listeners + generation controllers+ ├─→ on_event Signal to matching consumers+ │ └─→ controller queue → admitted graph child└─→ status_broadcaster.broadcast() — direct in-process WS fan-out(skipped with emit(..., broadcast=False))```@@ -161,6 +171,50 @@ default attributes (`WorkflowType`, `WorkflowId`, `ExecutionStatus`, …).## Temporal contract for plugin authors+### Event keys and accumulator behavior++The controller borrows durable buffering, stable keys and continuation-state+carry from Temporal's [Event Accumulator pattern](https://docs.temporal.io/design-patterns/event-accumulator).+The complete inactivity-window batching pattern is unsuitable here: triggers+process individual events immediately when admission is open. Stop holds the+queue; Resume admits it in FIFO order without replaying completed graph work.++New version 1 histories use `controller-event-accumulator-v1`. Pending keys are+protected separately from the bounded recent-completion cache, including an+event currently being dispatched. Restored overflow pages retain their earlier+FIFO position over a same-key live redelivery. Continue-as-new rechecks arrivals+after every spill and after message handlers finish, then carries or spills the+accepted tail. Reset's close flag prevents rollover from reopening the controller.++Use stable source identities for redelivery of the same occurrence:++| Producer | Event ID |+|---|---|+| Telegram | `telegram:{chat_id}:{message_id}` |+| Discord gateway message | `discord:message:{message_id}` |+| Discord interaction | `discord:interaction:{interaction_id}` |+| WhatsApp | `whatsapp:{direction}:{chat_id_or_sender}:{message_id}` |+| Chat | Tracked run ID, otherwise saved message ID |++A missing component retains a fresh generated ID rather than collapsing+unrelated messages. Provider event factories own these mappings; do not generate+random IDs inside a Workflow. Dedup is bounded, not a permanent occurrence ledger.+An already-started deterministic child is a successful duplicate admission;+transient child-start failures keep the original event pending for retry.++Signal-With-Start is appropriate when creating a collector is part of the+producer contract. Here Start owns generation creation and execution-root+membership. Recreating a lost version 1 controller via Signal-With-Start would+produce an empty registry, so producers signal the existing controller and its+loss fails closed. Legacy deployment recovery remains separate.++See [workflow messaging](temporal-workflow-control.md#workflow-messaging-review)+for completed control Updates and read-only status Queries. The Signal queue+and controller rollover tests are listed in the+[verification guide](temporal-workflow-control.md#verification-and-operations).++### Activity policy and Stop boundaries+| Class attribute | Purpose | Default ||---|---|---|| `start_to_close_timeout` | Per-attempt budget (one activity execution) | Kind-base default: ActionNode=10m, TriggerNode=24h, ToolNode=10m |@@ -172,6 +226,20 @@ Override only when the kind-base default doesn't fit; an inline commentexplaining why is required (enforced by`tests/test_plugin_contract.py::TestStartToCloseTimeoutOverridesAreCommented`).+Ordinary agent tool dispatch carries the resolved plugin's timeout, heartbeat,+retry and task-queue policy in prepared tool bindings. Routing also respects the+run's frozen worker-pool setting. Node and tool operations remain regular+Activities. Stop closes admission for the next operation and lets admitted+operations finish under their existing retry policies, including bookkeeping.+Resume consumes recorded results and continues from the next pending action.++Heartbeats track liveness, enable cancellation delivery and can record real+recoverable progress after failure. A status string cannot checkpoint an opaque+HTTP request, subprocess or provider session. Do not add a generic heartbeat+cursor or cancel the current tool to implement Stop. See the+[node authoring contract](node_creation.md) and+[long-running Activity guidance](https://docs.temporal.io/design-patterns/long-running-activity).+### Worker graceful shutdownWorkers honor SIGTERM with a configurable grace window@@ -424,7 +492,8 @@ Locked by `TestCancelSweepsStuckNodeStatuses` in [`test_deployment_canary_listen## References-- Plan: `~/.claude/plans/properly-fix-the-tech-dreamy-tarjan.md`+- Current contract: [Workflow control](temporal-workflow-control.md), including+ admission/drain acknowledgements, compatibility gates and verification.- RFC: [`plugin_authoring_rfc.md`](./ARCHIVE/plugin_authoring_rfc.md)-- Temporal: [Search Attributes](https://docs.temporal.io/search-attribute) · [Signals](https://docs.temporal.io/develop/python/message-passing) · [Schedules](https://docs.temporal.io/develop/python/schedules) · [Retry Policies](https://docs.temporal.io/encyclopedia/retry-policies)+- Temporal: [Messaging patterns](https://docs.temporal.io/design-patterns/workflow-messaging-patterns) · [Python messages](https://docs.temporal.io/develop/python/workflows/message-passing) · [Event Accumulator](https://docs.temporal.io/design-patterns/event-accumulator) · [Search Attributes](https://docs.temporal.io/search-attribute) · [Schedules](https://docs.temporal.io/develop/python/schedules) · [Retry Policies](https://docs.temporal.io/encyclopedia/retry-policies)- CloudEvents: [v1.0.2 spec](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/spec.md)diff --git a/docs-internal/event_waiter_system.md b/docs-internal/event_waiter_system.mdindex 2c92f0a1..145d8061 100644--- a/docs-internal/event_waiter_system.md+++ b/docs-internal/event_waiter_system.md@@ -110,7 +110,7 @@ Durability note: waiter state does NOT survive a process restart — that is by## Deployed Triggers vs Canvas Run-When a workflow is deployed, a trigger type registered with `register_canary_trigger_type` (live list: `grep -rn "register_canary_trigger_type(" server/nodes`) does not use this module. It runs under the Temporal controller, and its producer delivers events with `services.events.dispatch.emit`. The event waiter is involved only for canvas Run and for trigger types that are not canary-registered:+When a workflow is deployed, a trigger type registered with `register_canary_trigger_type` (live list: `rg 'register_canary_trigger_type\(' server/nodes`) does not use this module. It runs under the Temporal controller, and its producer delivers events with `services.events.dispatch.emit`. The event waiter is involved only for canvas Run and for trigger types that are not canary-registered:| Trigger Type | Deployed | Canvas Run ||---|---|---|@@ -120,7 +120,34 @@ When a workflow is deployed, a trigger type registered with `register_canary_tri| `twitterReceive` | Not canary-registered. It is in `POLLING_TRIGGER_TYPES` but registers no polling factory, so deploy falls back to an in-process `event_waiter` collector | `event_waiter` waiter; see the known gap below || `cronScheduler` | Temporal Schedule (`services/temporal/schedules.py`) | Not a waiter |-Controlled polling triggers are registered with `WorkflowControlWorkflow`, which starts a graph only for deduplicated new events. Legacy uncontrolled polling still uses the deployment compatibility layer. See [temporal-execution-engine-rfc.md](ARCHIVE/temporal-execution-engine-rfc.md) for the deployment architecture.+Controlled polling triggers are registered with `WorkflowControlWorkflow`, which starts a graph only for deduplicated new events. Legacy uncontrolled polling still uses the deployment compatibility layer. See [Workflow control](./temporal-workflow-control.md) and [Temporal Architecture](./TEMPORAL_ARCHITECTURE.md) for the live deployment architecture; the [execution-engine RFC](ARCHIVE/temporal-execution-engine-rfc.md) is historical.++### Cooperative Stop and durable controller queues++For new generations (`execution_control_version=1`), Stop closes admission+before another polling fetch or graph child start and drains already-admitted+business work through result bookkeeping. It does not cancel an Activity or+an event waiter. Accepted push Signals remain queued while stopped; Resume+releases the next pending action in the same execution. Polling results that+finish during Stop update their seen-ID baseline and preserve new events+before the fetch leaves the drain count. Sleeps between fetches and control+waits do not count as admitted business work.++The versioned controller uses Event Accumulator queue/deduplication/carry+practices without inactivity batching. Pending deduplication keys remain+protected independently of the bounded recent-key window, overflow pages+restore queued keys, and Continue-As-New fences include events delivered+during spill awaits. This applies after Temporal accepts the Signal;+`dispatch.emit` still relies on eventually consistent Visibility and logs or+suppresses delivery failures, so it is not a durable producer outbox.+Same-label/type trigger Workflow-ID collisions remain a separate known limit.+See [Node creation: event contract](./node_creation.md#event-identity-and-delivery-contract)+and [Temporal Event Accumulator](https://docs.temporal.io/design-patterns/event-accumulator).++Events use Signals; acknowledged control and root membership use completed+Updates; status uses read-only Queries. A Signal response means receipt into+Temporal history, not handler completion. Those messages never operate on the+process-local `_waiters` map. See [Temporal workflow messaging](https://docs.temporal.io/design-patterns/workflow-messaging-patterns).### Known gap: canvas Run on canary push triggers@@ -139,6 +166,11 @@ Users can cancel a waiting trigger from the UI (Cancel button on the trigger nod3. `cancel()` sets `w.cancelled = True` and calls `future.cancel()`.4. The suspended `wait_for_event()` raises `asyncio.CancelledError`, which bubbles up through the node executor.+This explicit waiter cancellation is distinct from a controlled generation's+Stop. It ends the interactive wait; it cannot provide durable Resume after a+process restart. Use generation `pause_workflow` / `resume_workflow` for the+cooperative Temporal contract.+## Debugging`event_waiter.get_active_waiters()` returns a list of all currently active waiters with age and done/cancelled status. The `get_active_waiters` WebSocket handler exposes this to the frontend for the active-triggers debug view. (`get_backend_mode()` still exists for compatibility but always returns `"memory"` — the Redis-Streams backend was retired in Wave 15.3.)diff --git a/docs-internal/memory_compaction.md b/docs-internal/memory_compaction.mdindex e8803467..ab4b1e20 100644--- a/docs-internal/memory_compaction.md+++ b/docs-internal/memory_compaction.md@@ -75,6 +75,48 @@ SDK. On the Temporal path, `prepare_agent_payload` (the `agent.prepare_payload`activity) calls `anthropic_config()` only as a provider-agnostic way tocalculate the numeric threshold `AgentWorkflow` compacts at.+## Temporal Stop, Resume, and continuation capacity++New controlled generations record `execution_control_version=1`. Compaction+is an admitted business Activity in their `AgentWorkflow`: Stop closes the+gate before a new summarizer starts, while an already-admitted summarizer+finishes under its existing retry policy and its result is consumed normally.+Resume continues the suspended loop. It does not repeat a recorded completed+compaction, model response, or tool call. Controller maintenance and checkpoint+waits are excluded from business drain counts.++The agent only continues-as-new at a completed-turn boundary with no live+delegation handles or Task Manager tasks. A stopped agent parks before this+boundary until Resume; the Workflow waits for control handlers to finish before+rollover. Version 1 carries the transcript, prepared payload, resolved tool+bindings and plugin Activity policies, thinking, iteration, usage, execution+identity, and pause/revision/hold state. Restoring the continuation skips+`agent.prepare_payload`, so it does not reload a changed model, graph binding,+or stored conversation to reconstruct the current turn.++Transcript relief and compaction remain the existing post-turn mechanisms;+they are not a second checkpoint store. Before version 1 rollover, the payload+converter measures the complete next input against `_CAN_INPUT_MAX_BYTES`+(1,900,000 bytes). This includes the graph and bindings as well as messages.+If it remains oversized, the agent raises non-retryable+`AgentContinuationTooLarge` with capacity guidance. It never discards the+transcript and returns to the opening prompt. Compaction can reduce transcript+pressure, but cannot guarantee that a very large graph or prepared payload fits.++Legacy generations preserve their previous command path, including preparation+on continuation and the 1,000,000-byte transcript-only fallback that drops an+oversized carried transcript. That fallback is not used by version 1.+`ConversationTooLarge` remains a separate fresh-conversation-load guard.++Stop is not a compaction or heartbeat checkpoint. Provider/tool work remains+regular Activities with declared retries, heartbeat, timeout, and queue+policies. Current LLM retries are unlimited, so Stop may stay `pausing` during+a persistent transient provider failure. External effects lost before Temporal+records completion remain subject to the tool's retry and idempotency contract.+See [Temporal workflow control](temporal-workflow-control.md),+[Agent context flow](agent_context_flow.md), and the official+[Continue-As-New guidance](https://docs.temporal.io/design-patterns/continue-as-new).+## Database Schema### TokenUsageMetricdiff --git a/docs-internal/node-logic-flows/README.md b/docs-internal/node-logic-flows/README.mdindex 7a1091a8..05392331 100644--- a/docs-internal/node-logic-flows/README.md+++ b/docs-internal/node-logic-flows/README.md@@ -1,8 +1,8 @@# Node Logic Flow Documentation-Frozen behavioural contract for every workflow node. One file per node, grouped-by category. After the upcoming full-stack refactor each handler must still-match its doc here, and each doc must still describe what the code does.+Behavioural contract for every workflow node. One file per node, grouped+by category. Keep each card aligned with its plugin and execution path;+historical diagrams do not override the current shared execution contract.## How to use@@ -28,6 +28,34 @@ match its doc here, and each doc must still describe what the code does.- Keep "Side Effects" honest: every DB write, broadcast, subprocess, and HTTPcall must be listed.+## Shared execution contract++Cards describe a node's operation. The workflow layer owns cooperative+admission and continuation for new controlled generations+(`execution_control_version=1`): Stop allows admitted work and result+bookkeeping to finish, then waits; Resume begins the next pending action in+the same execution. Already-admitted parallel tools drain concurrently.+An operation's sleep or approval wait is part of its admitted Activity when+the node runs as an Activity; it is not automatically a control checkpoint.+Legacy cancellation branches in a card describe explicit cancellation or+older execution paths, not the new generation's Stop behavior.++Controlled push triggers are definitions and Signal queues in+`WorkflowControlWorkflow`; controller polling uses provider fetch Activities,+and cron uses Temporal Schedules plus `CronTriggerWorkflow`. Separate+listener/poller workflows remain legacy compatibility paths. Stable event+IDs, durable pending-key protection, and Continue-As-New fences apply to the+versioned controller; interactive canvas waiters remain process-local.+Signal acceptance acknowledges receipt into Temporal history, while completed+Updates acknowledge transitions and read-only Queries return status.++Read [Workflow control](../temporal-workflow-control.md),+[Node creation](../node_creation.md#temporal-execution-and-stopresume-contract),+and [Event Waiter System](../event_waiter_system.md) for topology, Activity+policies, compatibility, producer delivery gaps, and known naming limits.+This shared contract applies to all cards below; individual cards call out+provider identities and operation-specific stopping boundaries.+## IndexFor the current browser architecture, see [Browser runtime](../browser.md)diff --git a/docs-internal/node-logic-flows/ai_agents/aiAgent.md b/docs-internal/node-logic-flows/ai_agents/aiAgent.mdindex 95892b9b..43a50ad6 100644--- a/docs-internal/node-logic-flows/ai_agents/aiAgent.md+++ b/docs-internal/node-logic-flows/ai_agents/aiAgent.md@@ -3,7 +3,7 @@| Field | Value ||------|-------|| **Category** | ai_agents / agent |-| **Backend handler** | [`server/nodes/agent/ai_agent/__init__.py`](../../../server/nodes/agent/ai_agent/__init__.py) — dispatched via `BaseNode.execute()` + the `@Operation("execute")` method (`execute_op`). Pre-dispatch (edge walk + task inject + prompt fallback) in [`server/nodes/agent/_inline.py::prepare_agent_call`](../../../server/nodes/agent/_inline.py); LLM loop in `AIService.execute_agent`. |+| **Backend execution** | Temporal graph runs normally use [`AgentWorkflow`](../../../server/services/temporal/agent_workflow.py), with preparation in [`agent_activities.py`](../../../server/services/temporal/agent_activities.py). The node/Activity path uses [`AIAgentNode.execute_op`](../../../server/nodes/agent/ai_agent/__init__.py), [`prepare_agent_call`](../../../server/nodes/agent/_inline.py), and `AIService.execute_agent`. || **Tests** | [`server/tests/nodes/test_ai_agents.py`](../../../server/tests/nodes/test_ai_agents.py) || **Skill (if any)** | n/a (the agent consumes skills via `input-skill`) || **Dual-purpose tool** | no |@@ -14,12 +14,40 @@and system message, continues its stored conversation when a Context node isconnected on `input-context` (RFC-0002), loads instructions from connectedskill nodes, binds tool nodes as provider-neutral `AgentToolSpec` values, and-runs `run_native_agent_loop` until the LLM produces a final answer. All of the-heavy lifting (native SDK invocation, tool execution, conversation-persistence) lives behind `AIService.execute_agent`; the handler's job is-purely to gather the connected-node payloads via-`edge_walker.collect_agent_connections` (a 5-tuple: context, skill, tool,-input, task) and forward them.+runs until the LLM produces a final answer. The node/Activity path gathers+connected payloads through `edge_walker.collect_agent_connections` (context,+skill, tool, input, task) and calls `AIService.execute_agent` with+`run_native_agent_loop`. The default Temporal graph path uses the same native+provider boundary through durable `AgentWorkflow` scheduling: preparation,+each LLM step, and each ordinary tool call are regular Activities.++## Temporal execution and Stop/Resume++Routing is frozen in the graph run's input. `aiAgent` takes the child-Workflow+path when `TEMPORAL_AGENT_WORKFLOW_ENABLED` was captured as true; otherwise it+runs inside the ordinary node Activity. For new controlled generations with+`execution_control_version=1`, the child Workflow closes admission before each+model request, tool, refresh, compaction, and child start. Stop lets admitted+work and its result bookkeeping finish, including parallel admitted work.+Pending calls from a recorded model response remain in the live continuation.+Resume starts the next pending action with the same execution scope rather+than repeating the response or completed tools.++Attached agents are discovered through their parent's native child tree;+detached Task Manager runners enroll independently with the generation+controller. Version 1 continuation carries prepared configuration, resolved+tools and plugin Activity policies, transcript, thinking, usage, iteration,+execution identity, and control state. It restores this state at a clean turn+boundary and reports `AgentContinuationTooLarge` if the complete encoded input+cannot fit. It never falls back to the opening prompt. Existing generations+retain legacy paths and do not acquire this stronger acknowledgement contract.++Unlimited LLM retries can keep Stop in `pausing` during an outage. Tool effects+before recorded completion remain subject to retry/idempotency rules; a+heartbeat status string cannot checkpoint an opaque request. If child-Workflow+routing is disabled, the entire node Activity is the cooperative boundary.+See [Temporal workflow control](../../temporal-workflow-control.md) and+[Agent continuation](../../TEMPORAL_ARCHITECTURE.md#agent-continuation-under-history-pressure).## Inputs (handles)@@ -69,6 +97,9 @@ Wrapped in the standard envelope: `{ success: true, result: <payload>, execution## Logic Flow+This diagram describes the node/Activity entry point. The default Temporal+child-Workflow loop is documented in [Temporal architecture](../../TEMPORAL_ARCHITECTURE.md#agent-as-child-workflow-f4b).+```mermaidflowchart TDA[execute_op -> prepare_agent_call] --> B[edge_walker.collect_agent_connections]@@ -125,8 +156,9 @@ flowchart TD## Side Effects-- **Database writes**: none directly in `execute_op`. `AIService.execute_agent`- writes `token_usage_metrics` rows (via `CompactionService.track`). With a+- **Database writes**: none directly in `execute_op`. A memory-connected+ in-process `AIService.execute_agent` writes `token_usage_metrics` via+ `CompactionService.track`; Temporal aggregates usage in its result. With aContext node connected, the conversation is saved to `agent_conversations`after every turn of a started workflow; a manual node Run saves nothing.- **Broadcasts**: `prepare_agent_call` resolves `StatusBroadcaster` and passesdiff --git a/docs-internal/node-logic-flows/ai_agents/chatAgent.md b/docs-internal/node-logic-flows/ai_agents/chatAgent.mdindex c6e4ff0a..ea5d482f 100644--- a/docs-internal/node-logic-flows/ai_agents/chatAgent.md+++ b/docs-internal/node-logic-flows/ai_agents/chatAgent.md@@ -3,7 +3,7 @@| Field | Value ||------|-------|| **Category** | ai_agents / agent |-| **Backend handler** | [`server/nodes/agent/chat_agent/__init__.py`](../../../server/nodes/agent/chat_agent/__init__.py) — dispatched via `BaseNode.execute()` + `@Operation("execute")` (`execute_op`). Pre-dispatch in [`_inline.py::prepare_agent_call`](../../../server/nodes/agent/_inline.py); LLM loop in `AIService.execute_chat_agent`. |+| **Backend execution** | Default Temporal graph runs use [`AgentWorkflow`](../../../server/services/temporal/agent_workflow.py) and [`agent_activities.py`](../../../server/services/temporal/agent_activities.py). The node/Activity path uses [`ChatAgentNode.execute_op`](../../../server/nodes/agent/chat_agent/__init__.py), [`prepare_agent_call`](../../../server/nodes/agent/_inline.py), and `AIService.execute_chat_agent`. || **Tests** | [`server/tests/nodes/test_ai_agents.py`](../../../server/tests/nodes/test_ai_agents.py) || **Skill (if any)** | n/a (consumes skills via `input-skill`) || **Dual-purpose tool** | no |@@ -22,7 +22,29 @@ the service method it calls (`AIService.execute_chat_agent` instead oftakes the **same execution path** — they subclass[`SpecializedAgentBase`](../../../server/nodes/agent/_specialized.py) whose`execute_op` also runs `prepare_agent_call` + `execute_chat_agent`. Each is its-own plugin folder; there is no `functools.partial` wiring anymore.+own plugin folder; there is no `functools.partial` wiring anymore. In a default+Temporal graph run these supported types route to the same `AgentWorkflow`+child path instead of executing the whole service loop as one node Activity.++## Temporal execution and Stop/Resume++`chatAgent` and supported specialized/team-lead agents use the same version 1+control and continuation contract as [`aiAgent`](aiAgent.md#temporal-execution-and-stopresume).+Stop lets each admitted model/tool Activity finish and retains its response or+tool result; pending actions wait in the existing continuation. Resume releases+those pending actions and preserves chat run/reply identity, partial output,+subscriptions, and lane occupancy. The controlled chat Stop request targets+the exact owning generation using `run_id`, `expected_revision`, and+`idempotency_key`. It does not terminalize that reply or trigger the legacy+grace-period cancellation behavior.++The version 1 agent carries prepared configuration, resolved plugin policies,+and transcript through clean continue-as-new boundaries. Legacy/uncontrolled+chat retains terminal Stop behavior, and a run routed as a whole node Activity+stops at that Activity's boundary. Unlimited LLM retries can delay complete+Stop acknowledgement. See [Chat protocol](../../chat_protocol.md),+[Temporal workflow control](../../temporal-workflow-control.md), and+[Agent context flow](../../agent_context_flow.md#stopped-turns-and-unanswered-calls).## Inputs (handles)@@ -56,6 +78,9 @@ Differences:## Logic Flow+This diagram describes the node/Activity entry point; the Temporal child path+uses the [durable agent loop](../../TEMPORAL_ARCHITECTURE.md#agent-as-child-workflow-f4b).+```mermaidflowchart TDA[execute_op -> prepare_agent_call] --> B[edge_walker.collect_agent_connections]@@ -81,10 +106,11 @@ flowchart TD`edge_walker.collect_agent_connections`; same rules as `aiAgent` (see that docfor the Context descriptor, `masterSkill` expansion, Android service tools,child-agent tool discovery).-- **Native agent loop**: current executions call the shared- `run_native_agent_loop` / `run_native_llm_step` service through- `ChatUnifier`. It is the only loop; pre-cutover Temporal histories are- refused rather than replayed.+- **Native provider boundary**: current executions use `ChatUnifier` with+ `run_native_llm_step`; the in-process path owns `run_native_agent_loop`,+ while Temporal owns the durable scheduling loop. There is one current wire+ standard. Pre-cutover deployment recovery uses Reset; control version gates+ preserve compatible legacy command paths rather than upgrading them in place.- **Task context injection** mirrors `aiAgent`: `format_task_context` wrapsthe task result as a plain-English instruction that the LLM must "reportnaturally", then all tools are stripped if the task has already completeddiff --git a/docs-internal/node-logic-flows/discord/discordInteraction.md b/docs-internal/node-logic-flows/discord/discordInteraction.mdindex 79f9edc1..7bbdbbf7 100644--- a/docs-internal/node-logic-flows/discord/discordInteraction.md+++ b/docs-internal/node-logic-flows/discord/discordInteraction.md@@ -17,8 +17,9 @@ verified, acknowledged inside Discord's three-second deadline, and emitted asA second trigger node rather than a mode on `discordReceive`:`canary_registry` maps one node type to exactly one CloudEvents type, because-that string becomes the Temporal `EventType` Search Attribute the listener is-found by. One node cannot subscribe to both.+that string determines deployed event routing. Controlled controllers advertise+their event types and hold the trigger definition; legacy listeners use their+`EventType` Search Attribute. One registered node type cannot subscribe to both.## Inputs (handles)@@ -72,7 +73,7 @@ flowchart TDE -- other --> F[shape_interaction: token -> ref]F --> G[respond type 5, or 6 for a component]G --> H[background task: dispatch.emit]- H --> I[Temporal listener matched by EventType SA]+ H --> I[Controlled controller on_event Signal<br/>legacy listener compatibility]```## Decision Logic@@ -117,5 +118,15 @@ flowchart TD## Related+- **Event identity and Stop/Resume**: `_events.discord_interaction_created`+ uses `discord:interaction:{interaction_id}` when the provider ID is present;+ otherwise it keeps the random envelope ID. New versioned generations queue+ accepted interaction Signals while stopped and admit graphs on Resume.+ Discord's initial HTTP acknowledgement is independent of Temporal+ acceptance or graph completion; background producer delivery remains best+ effort. A pause can exceed the ref/token TTL below, so durable event receipt+ does not guarantee a later interaction response is still possible. See+ [Workflow control](../../temporal-workflow-control.md).+- **Downstream**: `discordAction[interaction_respond]` completes the reply.- **Architecture docs**: [discord_service.md](../../discord_service.md)diff --git a/docs-internal/node-logic-flows/discord/discordReceive.md b/docs-internal/node-logic-flows/discord/discordReceive.mdindex ccf18cd7..6f817c6b 100644--- a/docs-internal/node-logic-flows/discord/discordReceive.md+++ b/docs-internal/node-logic-flows/discord/discordReceive.md@@ -14,6 +14,10 @@ Fire when a message arrives in a server channel or a DM. Messages come over apersistent gateway connection held per bot account, are flattened, and emittedas CloudEvents of type `com.opencompany.discord.message.received`.+Controlled deployments keep this trigger's definition and accepted Signal+queue in `WorkflowControlWorkflow`, which applies the filter before graph+admission. Separate listener workflows remain the legacy compatibility path.+## Inputs (handles)None — this is a trigger; it is the head of a run.@@ -66,10 +70,10 @@ flowchart TDB -- yes --> Z[drop: would loop]B -- no --> C[shape_message: stringify snowflakes]C --> D[dispatch.emit CloudEvent]- D --> E[Temporal listener matched by EventType SA]+ D --> E[Controlled controller on_event Signal<br/>legacy listener compatibility]E --> F[node filter gates the spawn]F --> G[child MachinaWorkflow run]- D --> H[in-process WS broadcast for canvas Run]+ D --> H[in-process WS broadcast<br/>does not resolve canvas waiter]```## Decision Logic@@ -115,6 +119,16 @@ flowchart TD## Related+- **Event identity and Stop/Resume**: `_events.discord_message_received`+ uses `discord:message:{message_id}` when the provider snowflake is present;+ otherwise it keeps the random envelope ID. It cannot collide with the+ interaction namespace. Versioned stopped generations retain accepted+ Signals until graph admission resumes. Signal acceptance acknowledges+ receipt, not graph completion; Visibility discovery and delivery remain+ best effort before acceptance. See+ [Workflow control](../../temporal-workflow-control.md) and+ [canvas waiter delivery gap](../../event_waiter_system.md#known-gap-canvas-run-on-canary-push-triggers).+- **Downstream**: `discordAction` (download attachments), `discordSend` (reply).- **Architecture docs**: [discord_service.md](../../discord_service.md),[event_framework.md](../../event_framework.md)diff --git a/docs-internal/node-logic-flows/specialized_agents/claude_code_agent.md b/docs-internal/node-logic-flows/specialized_agents/claude_code_agent.mdindex 19861cb5..2219dc05 100644--- a/docs-internal/node-logic-flows/specialized_agents/claude_code_agent.md+++ b/docs-internal/node-logic-flows/specialized_agents/claude_code_agent.md@@ -22,6 +22,25 @@ which the CLI resolves only against interactive sessions) over a stable`cwd`, not by re-injecting markdown. This is the onlyspecialized-agent node that shells out; the others stay in-process.+## Temporal Stop/Resume boundary++Claude Code is excluded from `AGENT_WORKFLOW_TYPES`; its subprocess, CLI+session, and MCP tool calls execute within a whole node Activity. New controlled+generations (`execution_control_version=1`) close admission before the next+Activity. An already-admitted Claude Code Activity continues through its+internal tools until it settles under its scheduled timeout/retry policy, then+normal result bookkeeping finishes and the graph/parent agent waits.++Generation Resume releases that existing continuation. It does not launch a+replacement CLI task or repeat a recorded completed Activity. CLI+`--resume <session_id>` is the plugin's own conversation mechanism and is+distinct from generation Resume. No generic heartbeat cursor serializes a CLI+session or subprocess; ambiguous effects before recorded completion remain+subject to the plugin/tool retry and idempotency contract. Legacy generations+keep their previous control path. See+[Temporal workflow control](../../temporal-workflow-control.md) and+[Temporal architecture](../../TEMPORAL_ARCHITECTURE.md#cooperative-generation-stop-and-resume).+## Inputs (handles)Standard `std_agent_handles()` topology (same as the generic agents).diff --git a/docs-internal/node-logic-flows/specialized_agents/rlm_agent.md b/docs-internal/node-logic-flows/specialized_agents/rlm_agent.mdindex e91dc560..04e5128d 100644--- a/docs-internal/node-logic-flows/specialized_agents/rlm_agent.md+++ b/docs-internal/node-logic-flows/specialized_agents/rlm_agent.md@@ -15,6 +15,23 @@ LLM is prompted to emit Python code that is executed in a REPL and mayrecursively call `llm_query()` or `rlm_query()` and end with `FINAL(...)`to return a result. See [RLM Service](../../rlm_service.md).+## Temporal Stop/Resume boundary++`rlm_agent` is excluded from `AGENT_WORKFLOW_TYPES`: the REPL and recursive+provider calls run inside one node Activity. For a new controlled generation+with `execution_control_version=1`, Stop closes admission before new node+Activities but lets an admitted RLM Activity, including its internal calls,+settle under its scheduled timeout/retry policy. Its result is recorded normally+and the graph/parent agent then waits. Resume releases the next pending action;+it does not relaunch a recorded completed RLM Activity.++This does not pause between REPL iterations or individual internal tools.+Heartbeat status is liveness information, not a serialized REPL checkpoint;+failure recovery remains the Activity's own retry/idempotency contract. Legacy+generations retain their prior control path. See+[Temporal workflow control](../../temporal-workflow-control.md) and+[Temporal architecture](../../TEMPORAL_ARCHITECTURE.md#cooperative-generation-stop-and-resume).+## Inputs (handles)Same 5 shared handles as the generic specialized agents. **Nodiff --git a/docs-internal/node-logic-flows/specialized_agents/vertex_managed_agent.md b/docs-internal/node-logic-flows/specialized_agents/vertex_managed_agent.mdindex aef19a53..c5da20b9 100644--- a/docs-internal/node-logic-flows/specialized_agents/vertex_managed_agent.md+++ b/docs-internal/node-logic-flows/specialized_agents/vertex_managed_agent.md@@ -27,6 +27,26 @@ Context store when a Context node is wired (transcript rendered into theprompt); legacy `input-memory` graphs keep the recorded Simple Memory chain-id bridge(`vertex_interaction_id` / `vertex_environment_id`).+## Temporal Stop/Resume boundary++`vertex_managed_agent` stays outside `AGENT_WORKFLOW_TYPES`. A complete managed+interaction chain, including local `requires_action` tools and remote tool+work, runs inside one node Activity. In new controlled generations with+`execution_control_version=1`, Stop fences later Activity admission and lets an+admitted chain settle under the declared policy. Its result is consumed+normally before the graph/parent agent waits. Resume releases the next pending+action without starting the recorded completed interaction again.++Generation Stop does not interrupt a cloud interaction between internal tool+calls. `previous_interaction_id` chaining remains the plugin's provider-session+mechanism, not the generation checkpoint. Heartbeats do not serialize a remote+session. This plugin's one-attempt retry declaration is preserved when version 1+ordinary agent-tool dispatch uses its frozen policy. External-effect ambiguity+before completion remains subject to provider/tool idempotency. Legacy+generations retain their prior command path. See+[Temporal workflow control](../../temporal-workflow-control.md) and+[Temporal architecture](../../TEMPORAL_ARCHITECTURE.md#cooperative-generation-stop-and-resume).+## Inputs (handles)`std_agent_handles()` + `STD_AGENT_HINTS` (`width 300`, `height 200`,diff --git a/docs-internal/node-logic-flows/telegram_social/telegramReceive.md b/docs-internal/node-logic-flows/telegram_social/telegramReceive.mdindex 0e13491a..67f94f15 100644--- a/docs-internal/node-logic-flows/telegram_social/telegramReceive.md+++ b/docs-internal/node-logic-flows/telegram_social/telegramReceive.md@@ -4,7 +4,7 @@|------|-------|| **Category** | social / trigger || **Backend handler** | [`server/nodes/telegram/telegram_receive.py`](../../../server/nodes/telegram/telegram_receive.py) (`TelegramReceiveNode`, a `TriggerNode` with `event_type = "telegram_message_received"`); filter via `build_filter` -> [`build_telegram_filter`](../../../server/nodes/telegram/_filters.py) |-| **Tests** | [`server/tests/nodes/test_telegram_social.py`](../../../server/tests/nodes/test_telegram_social.py) (node level), [`test_telegram_service.py`](../../../server/tests/nodes/test_telegram_service.py) (service level: `_format_message` per content type) |+| **Tests** | [`server/tests/nodes/test_telegram_social.py`](../../../server/tests/nodes/test_telegram_social.py) (node level), [`test_telegram_service.py`](../../../server/tests/nodes/test_telegram_service.py) (service level: `_format_message` per content type), [`test_telegram_trigger_canary_producer.py`](../../../server/tests/test_telegram_trigger_canary_producer.py) (stable envelope identity and producer routing) || **Skill (if any)** | none || **Dual-purpose tool** | no (trigger only) |@@ -141,9 +141,10 @@ flowchart TDsubgraph ProducerP1[TelegramService polling loop] --> P2[_on_message_received]P2 --> P3[_format_message -> event_data]- P3 --> P4[event_waiter.dispatch<br/>'telegram_message_received']- P4 --> P5[matches filter<br/>per waiter]- P5 -- match --> P6[Waiter.future.set_result]+ P3 --> P4[_events.broadcast_telegram_message<br/>dispatch.emit CloudEvent]+ P4 --> P5[Controlled controller on_event Signal<br/>legacy listener compatibility]+ P5 --> P6[Queue and filter -> admit graph run]+ P4 --> P7[WebSocket message broadcast<br/>does not resolve canvas waiter]end```@@ -212,7 +213,7 @@ flowchart TDso bot-owners who are themselves bots still match.- **Cancellation path**: If the trigger is cancelled mid-wait, the handlerreturns `success=false, error="Cancelled by user"` rather than swallowing.-- **No timeout**: The node waits indefinitely. The only exit routes are an+- **No direct-wait timeout**: The node waits indefinitely. The only exit routes are anevent match, an explicit `cancel_event_wait` WebSocket call, or the serverrestarting.- **GIFs changed category**: they now arrive as `content_type: "animation"`@@ -233,6 +234,17 @@ flowchart TD## Related+- **Event identity and Stop/Resume**: `_events.telegram_message_received`+ uses `telegram:{chat_id}:{message_id}` when both IDs are present, since+ Telegram message IDs are unique within a chat. Missing identity keeps the+ random envelope ID; attachment `file_unique_id` is not the trigger event+ key. Accepted Signals stay queued while a versioned generation is stopped;+ Resume admits the pending graph without repeating recorded node/tool work.+ Durability starts after Signal acceptance; producer Visibility discovery+ and delivery remain best effort. See+ [Workflow control](../../temporal-workflow-control.md) and the+ [canvas waiter delivery gap](../../event_waiter_system.md#known-gap-canvas-run-on-canary-push-triggers).+- **Sibling nodes**: [`telegramSend`](./telegramSend.md), [`socialReceive`](./socialReceive.md)- **Event waiter architecture**: [Event Waiter System](../../event_waiter_system.md)- **Service**: [`server/nodes/telegram/_service.py`](../../../server/nodes/telegram/_service.py)diff --git a/docs-internal/node-logic-flows/whatsapp/whatsappReceive.md b/docs-internal/node-logic-flows/whatsapp/whatsappReceive.mdindex 6664b9e7..fe756e64 100644--- a/docs-internal/node-logic-flows/whatsapp/whatsappReceive.md+++ b/docs-internal/node-logic-flows/whatsapp/whatsappReceive.md@@ -4,7 +4,7 @@|------|-------|| **Category** | whatsapp / trigger || **Backend handler** | [`server/nodes/whatsapp/whatsapp_receive.py`](../../../server/nodes/whatsapp/whatsapp_receive.py) (`WhatsAppReceiveNode`, a `TriggerNode`); filtering is built by [`server/nodes/whatsapp/_filters.py`](../../../server/nodes/whatsapp/_filters.py). CloudEvents type: `com.opencompany.whatsapp.message.received`; controlled deployment routes through `WorkflowControlWorkflow`. |-| **Tests** | [`server/tests/nodes/test_whatsapp.py`](../../../server/tests/nodes/test_whatsapp.py) |+| **Tests** | [`server/tests/nodes/test_whatsapp.py`](../../../server/tests/nodes/test_whatsapp.py), [`test_whatsapp_trigger_canary_producer.py`](../../../server/tests/test_whatsapp_trigger_canary_producer.py) (stable envelope identity and producer routing) || **Skill (if any)** | n/a || **Dual-purpose tool** | no - pure trigger node |@@ -93,6 +93,11 @@ flowchart TDE -- exception --> Y[success=false<br/>error=str e]E -- event dict --> F[Filter closure applied inside wait_for_event<br/>only matching messages resolve the future]F --> OK[success=true<br/>result=event_data]++ P[Go RPC message] --> Q[_events.broadcast_whatsapp_message<br/>dispatch.emit CloudEvent]+ Q --> R[Controlled controller on_event Signal]+ R --> S[Queue and filter -> admit graph run]+ Q --> W[WebSocket frame<br/>does not resolve canvas waiter]```### Filter closure (`_filters.build_filter`)@@ -165,6 +170,17 @@ flowchart LR## Related+- **Event identity and Stop/Resume**: `_events.whatsapp_message_event` uses+ `whatsapp:{direction}:{conversation}:{message_id}` when the conversation+ (`chat_id`, else `sender`, else `from`) and message ID are available. Sent+ and received events have different namespaces. Missing identity keeps the+ random envelope ID. New versioned generations retain accepted Signals while+ stopped and admit their pending graphs on Resume; producer discovery and+ delivery remain best effort before acceptance. The direct canvas waiter has+ a separate delivery gap. See+ [Workflow control](../../temporal-workflow-control.md) and+ [Event Waiter System](../../event_waiter_system.md#known-gap-canvas-run-on-canary-push-triggers).+- **Companion nodes**: [`whatsappSend`](./whatsappSend.md), [`whatsappDb`](./whatsappDb.md)- **Architecture docs**: `CLAUDE.md` -> "Event-Driven Trigger Node System", "WhatsApp Integration"- **Trigger infrastructure**: [`event_waiter.py`](../../../server/services/event_waiter.py), [`_filters.py`](../../../server/nodes/whatsapp/_filters.py), [`_events.py`](../../../server/nodes/whatsapp/_events.py), canary path in [`canary_registry.py`](../../../server/services/deployment/canary_registry.py)diff --git a/docs-internal/node-logic-flows/workflow_triggers/approvalGate.md b/docs-internal/node-logic-flows/workflow_triggers/approvalGate.mdindex a182bc76..6dd567f4 100644--- a/docs-internal/node-logic-flows/workflow_triggers/approvalGate.md+++ b/docs-internal/node-logic-flows/workflow_triggers/approvalGate.md@@ -151,12 +151,20 @@ flowchart TD`reconcile_once`, which `list_approvals` runs: local rows created beforethis process started, and Temporal rows whose generation is no longer live.- Deciding while the employee is paused is allowed; the response carries- `will_send_on_resume: true` and the card says so.+ `will_send_on_resume: true` for an approved gate draft and the card says so.+ Gate drafts resume through their graph's downstream admission gate.+- In a new controlled generation, an approval wait already admitted as a node+ Activity counts as business work. Stop lets it settle on a decision or expiry+ under its existing retries, retains the outcome, and can remain Stopping+ while it waits. Resume admits the downstream action rather than recreating+ the draft. This differs from separately approved tool calls, whose+ `ApprovedToolCallWorkflow` send lifecycle is independent of generation Stop.+ See [Workflow control](../../temporal-workflow-control.md).## Related- **WebSocket handlers**- ([`_handlers.py`](../../../server/nodes/workflow/approval_gate/_handlers.py)):+ ([`services/approvals/handlers.py`](../../../server/services/approvals/handlers.py)):`list_approvals {workflow_id?, status?, limit <= 100}` returns the owner'sdrafts, per-workflow pending counts and `server_time`;`decide_approval {approval_id, decision: "send" | "discard", text?, subject?, decision_key}`diff --git a/docs-internal/node-logic-flows/workflow_triggers/chatTrigger.md b/docs-internal/node-logic-flows/workflow_triggers/chatTrigger.mdindex 661900a0..4a3cff97 100644--- a/docs-internal/node-logic-flows/workflow_triggers/chatTrigger.md+++ b/docs-internal/node-logic-flows/workflow_triggers/chatTrigger.md@@ -3,8 +3,8 @@| Field | Value ||------|-------|| **Category** | workflow / trigger / utility |-| **Backend handler** | Plugin [`server/nodes/trigger/chat_trigger/__init__.py`](../../../server/nodes/trigger/chat_trigger/__init__.py) (`ChatTriggerNode`); dispatch via `BaseNode.execute()`. The base `TriggerNode.execute` handles the event-waiter wait — the `@Operation("wait")` body is a stub. The generic legacy path is [`server/services/handlers/triggers.py::handle_trigger_node`](../../../server/services/handlers/triggers.py) (reachable only for the lone deferred-canary trigger, not chatTrigger). |-| **Tests** | [`server/tests/nodes/test_workflow_triggers.py`](../../../server/tests/nodes/test_workflow_triggers.py) |+| **Backend handler** | Plugin [`server/nodes/trigger/chat_trigger/__init__.py`](../../../server/nodes/trigger/chat_trigger/__init__.py) (`ChatTriggerNode`). Deployed generations register its filter with [`WorkflowControlWorkflow`](../../../server/services/temporal/workflow_control_workflow.py), which receives Signals and starts graph runs with the trigger pre-executed. The base `TriggerNode.execute` event-waiter wait is the direct execution path; its `@Operation("wait")` body is a stub. Standalone legacy deployments retain `TriggerListenerWorkflow`. |+| **Tests** | [`server/tests/nodes/test_workflow_triggers.py`](../../../server/tests/nodes/test_workflow_triggers.py), [`server/tests/services/chat/test_handlers.py`](../../../server/tests/services/chat/test_handlers.py), [`server/tests/services/chat/test_stop.py`](../../../server/tests/services/chat/test_stop.py), [`server/tests/temporal/test_controller_accumulator.py`](../../../server/tests/temporal/test_controller_accumulator.py) || **Skill (if any)** | none || **Dual-purpose tool** | no |@@ -15,13 +15,17 @@ Fires when the user sends a chat message: from the editor's chat paneproducer [`server/services/chat/events.py`](../../../server/services/chat/events.py)emits a CloudEvents `WorkflowEvent` (`type: com.opencompany.chat.message.received`,source `opencompany://services/chat`) via `dispatch.emit`, never broadcast to-sockets. `chatTrigger` is canary-registered-(`register_canary_trigger_type`), so `DeploymentManager` starts a-`TriggerListenerWorkflow` for it; the listener receives the event via Temporal-Signal and spawns a child `MachinaWorkflow` per matching event. Any `chatTrigger`-node whose `session_id` matches (or is `'default'`) receives the event and emits-it as output. This is the primary way a user feeds an interactive prompt into an-`aiAgent` or `chatAgent`.+sockets. `chatTrigger` is canary-registered (`register_canary_trigger_type`).+For a generation with a durable controller, `DeploymentManager` sends+`register_trigger`; the controller retains its filter and graph context rather+than starting a separate listener. `dispatch.emit` discovers the controller+through Temporal Visibility and sends `on_event`. The controller filters each+registered trigger, queues the accepted event and starts a child+`MachinaWorkflow` when producer admission is open. Deployments without a+durable controller keep the standalone `TriggerListenerWorkflow` route.+A matching `session_id` (or `'default'`) makes the payload the trigger's+pre-executed output. This is the primary way a user feeds an interactive prompt+into an `aiAgent` or `chatAgent`.A workflow's chat session id is the workflow id. Home's Talk and the editor'schat pane send with `session_id` = that id (`'default'` only when the editor@@ -66,7 +70,9 @@ trigger that Hire and Turn on Talk add beside another worker) has}```-Wrapped in the standard envelope, whose id is the run id when there is one.+Wrapped in the standard envelope, whose stable id is the run id when there is+one, otherwise the saved message id. Redelivery of that same envelope keeps its+identity; the controller's deduplication key is `<listener_id>:<event_id>`.A run spawned from it claims that chat run (MachinaWorkflow,`machina-chat-run-v1`) and passes `run_scope` to every node, so[`chatReply`](../chat_utility/chatReply.md) saves its answer as the run's reply.@@ -80,10 +86,14 @@ flowchart TDV -- yes --> X[not_running: nothing saved or sent]V -- no --> W[chat ledger: save the message,<br/>and its run when a chat trigger answers the session]W --> Q[services/chat/events.py dispatch.emit<br/>WorkflowEvent com.opencompany.chat.message.received]- Q --> R[TriggerListenerWorkflow receives via Temporal Signal]- R --> S[ChatTriggerNode.build_filter:<br/>if session_id != 'default' require exact match]+ Q --> R[WorkflowControlWorkflow receives on_event Signal<br/>legacy standalone listener retains its route]+ R --> K[match event type, deduplicate<br/>and queue accepted trigger event]+ K --> A{producer admission open?}+ A -- no --> B[retain event until Resume]+ B --> A+ A -- yes --> S[ChatTriggerNode.build_filter:<br/>if session_id != 'default' require exact match]S -- match --> T[spawn child MachinaWorkflow<br/>trigger pre-executed with event payload]- S -- no match --> R+ S -- no match --> D[consume this trigger's rejected event]```## Decision Logic@@ -97,7 +107,14 @@ flowchart TDSo `session_id='default'` (the frontend default) is a wildcard - thetrigger fires for every chat message regardless of which session the useris in.-- **Cancellation**: yields `success=False, error="Cancelled by user"`.+- **Controlled Stop**: suspends admission on the owning generation. It does+ not cancel the trigger, withdraw its accepted event or terminalize the chat+ run. Already admitted work drains before `paused` is acknowledged; Resume+ releases the same continuation or pending event. See+ [Chat protocol → Controlled Stop acknowledgement](../../chat_protocol.md#controlled-stop-acknowledgement).+- **Direct-path cancellation**: the event-waiter path can yield+ `success=False, error="Cancelled by user"`. Reset/Clear and legacy terminal+ chat Stop retain their separate lifecycle behavior.## Side Effects@@ -113,8 +130,10 @@ flowchart TD- **Broadcasts**: the producer emits a CloudEvents `WorkflowEvent` via`dispatch.emit` (Temporal Signal fan-out only: `broadcast=False`, since theenvelope carries the owner's text). The- `TriggerListenerWorkflow` emits firing-pulse status via- `broadcast_trigger_status_activity` before/after each child spawn.+ controller (or legacy listener) emits firing-pulse status via+ `broadcast_trigger_status_activity` around child admission. Chat-run events+ are delivered only to authorized session subscribers; generation control+ status is broadcast separately.- **External API calls**: none.- **File I/O**: none.- **Subprocess**: none.@@ -122,31 +141,55 @@ flowchart TD## External Dependencies- **Credentials**: none.-- **Services**: `services.event_waiter`, `services.status_broadcaster`.-- **Python packages**: stdlib only.+- **Services**: chat ledger/thread, `services.events.dispatch`, Temporal+ controller and graph execution, `services.status_broadcaster`; the direct+ event-waiter path uses `services.event_waiter`.+- **Python packages**: Temporal Python SDK for deployed execution; the plugin's+ filter/output logic uses the base node framework.- **Environment variables**: none.## Edge cases & known limits- `session_id='default'` behaves as a wildcard; setting a unique session IDper trigger is the only way to scope messages to a specific node.-- When multiple `chatTrigger` nodes exist with the same session ID, all of- them fire for a matching message, so one message starts one run per- trigger; the first run to start claims the chat run and the others run- untracked. Hire and Turn on Talk never add a second trigger on a+- Distinct registered `chatTrigger` nodes with the same session ID can all+ match a message: the first graph run to claim tracks the chat run and the+ others run untracked. Existing listener ids use the workflow slug and+ display-label slug, so duplicate trigger labels/types can collide before+ registration; this remains a naming limitation. Hire and Turn on Talk never add a second trigger on aworkflow's own session: a chat hire's "Chat" trigger is already its talkline.- The handler has no timeout; it waits forever until an event arrives orthe run is cancelled.- For a workflow's session, `send_chat_message` first reads the latest- control. While it runs, starts or resumes, the message goes now (response- `delivery: "now"`); while it is paused or pausing, the controller keeps it- and starts a run on Resume (`delivery: "queued"`). In any other state the+ control. While it runs, starts or resumes, the response says+ `delivery: "now"`; a versioned Resume still holds producer admission until+ descendants are released. While it is paused or pausing, an event accepted+ by the controller stays queued for Resume (`delivery: "queued"`). In any other state thehandler answers `not_running` and neither saves nor dispatches the message.Session `'default'` is saved and dispatched as before, so a message there isstored even when no `chatTrigger` is waiting. A workflow's message starts achat run only when a deployed `chatTrigger` accepts its session; while a run- is live the next message is refused with `run_in_progress`.+ is live the next message is refused with `run_in_progress`. Controlled Stop+ keeps that lane occupied and its run/reply ids unchanged; legacy Stop can+ withdraw a queued message and release the lane.+- The controller applies the [Event Accumulator](https://docs.temporal.io/design-patterns/event-accumulator)+ mechanics to accepted events: pending-key deduplication, bounded recent keys,+ FIFO overflow pages in the existing queue store, and queued-event carry+ through continue-as-new. Processing remains immediate when admission is open;+ there is no inactivity batching. Deduplication of completed historical events+ is bounded, not an indefinite exactly-once guarantee.+- `dispatch.emit` uses eventually consistent Visibility discovery and logs+ unavailable clients, query failures and Signal failures. Saving the message+ and returning `delivery` does not guarantee the Signal was accepted; no+ durable ingress outbox was added. Once Temporal accepts the Signal, replay+ and the controller's rollover fences preserve its accepted pending event.+ A repeated `client_message_id` returns the original message/run without+ redispatching, so client resend is not an ingress recovery mechanism.+- A control request timeout leaves `pausing` intact and lets status+ reconciliation continue; it does not cancel the current tool. Intentionally+ stopped controlled chat runs are protected from watchdog age expiry and get+ a fresh timeout window after successful Resume.## Related@@ -156,4 +199,6 @@ flowchart TD- **Answers into the thread**: [`chatReply`](../chat_utility/chatReply.md).- **Architecture docs**: [Event Waiter System](../../event_waiter_system.md),[Status Broadcaster](../../status_broadcaster.md),- [Normal mode → Talk](../../normal_mode.md#talk)+ [Normal mode → Talk](../../normal_mode.md#talk),+ [Temporal Workflow Control](../../temporal-workflow-control.md),+ [Temporal Workflow Messaging Patterns](https://docs.temporal.io/design-patterns/workflow-messaging-patterns)diff --git a/docs-internal/node-logic-flows/workflow_triggers/cronScheduler.md b/docs-internal/node-logic-flows/workflow_triggers/cronScheduler.mdindex 5bdadc24..7fdc85ac 100644--- a/docs-internal/node-logic-flows/workflow_triggers/cronScheduler.md+++ b/docs-internal/node-logic-flows/workflow_triggers/cronScheduler.md@@ -82,8 +82,7 @@ flowchart TDC --> D[_calculate_wait_seconds<br/>maps frequency to seconds]D --> E[get_status_broadcaster.update_node_status 'waiting']E --> F[await asyncio.sleep wait_seconds]- F -- CancelledError --> G[Return success=false<br/>error: Scheduler cancelled]- F -- Exception --> H[Return success=false<br/>error: str e]+ F -- CancelledError --> G[Raise RuntimeError: Scheduler cancelled<br/>BaseNode returns failed envelope]F -- ok --> I[Build result_data: timestamp / iteration=1 / frequency / schedule / waited_seconds]I --> J{frequency == 'once'?}J -- yes --> K[message: 'Triggered after waiting ...']@@ -106,8 +105,28 @@ flowchart TD- **Once vs recurring**: determines whether `next_run` is added to thepayload and which `message` template is used. Both branches still return`success=True`.-- **Cancellation / exception**: both produce a failed envelope with- `success=False`.+- **Explicit cancellation**: the operation raises `RuntimeError("Scheduler+ cancelled")`; `BaseNode.execute` turns that failure into its standard+ envelope. Cooperative generation Stop does not enter this branch.++## Deployed Stop/Resume boundary++For new controlled generations (`execution_control_version=1`), Stop pauses+the Temporal Schedule and closes producer admission. A cron firing already+started enrolls with the controller before graph work and respects its current+revision; Stop waits for any admitted child-start command's start+acknowledgement. The spawned `MachinaWorkflow` is `ABANDON` and enrolls as an+independent root, so it remains controlled after the short cron wrapper exits.+Already-admitted graph work drains through result bookkeeping. Resume releases+those existing continuations before producer/schedule admission reopens.++The manual or AI-tool operation above remains one regular Activity when+dispatched through Temporal. Its interval sleep is part of that Activity;+Stop waits for it to settle under its existing timeout/retry policy before+acknowledging Stopped. This differs from the workflow's maintenance/control+sleeps, which are excluded from the drain count. See+[Workflow control](../../temporal-workflow-control.md) and+[Temporal Python Schedules](https://docs.temporal.io/develop/python/schedules).## Side Effectsdiff --git a/docs-internal/node-logic-flows/workflow_triggers/start.md b/docs-internal/node-logic-flows/workflow_triggers/start.mdindex d0d47d01..63241613 100644--- a/docs-internal/node-logic-flows/workflow_triggers/start.md+++ b/docs-internal/node-logic-flows/workflow_triggers/start.md@@ -100,6 +100,11 @@ flowchart TD## Related+- **Stop/Resume**: `start` is admitted graph work, not a restart command. New+ controlled generations preserve its completed output while stopped and+ resume at the next pending node. See+ [Workflow control](../../temporal-workflow-control.md).+- **Skills using this as a tool**: none.- **Other nodes that consume this output**: any downstream node - typicalpattern is `start -> aiAgent` or `start -> httpRequest` in a hand-rundiff --git a/docs-internal/node-logic-flows/workflow_triggers/taskTrigger.md b/docs-internal/node-logic-flows/workflow_triggers/taskTrigger.mdindex 2ce8d2ec..0ef06675 100644--- a/docs-internal/node-logic-flows/workflow_triggers/taskTrigger.md+++ b/docs-internal/node-logic-flows/workflow_triggers/taskTrigger.md@@ -11,15 +11,18 @@## PurposeFires when a child agent dispatched from a durable Task Manager assignment completes or-errors. The delegation code path in-`server/services/handlers/tools.py::_execute_delegated_agent` calls-`nodes.agent._events.broadcast_agent_task_completed` for both the success and-error paths, which emits a CloudEvents `WorkflowEvent`+errors. The Temporal `agent.finish_delegation` Activity in+`server/services/temporal/agent_activities.py` and the legacy coordinator in+`server/services/handlers/tools.py` persist the task outcome before calling+`nodes.agent._events.broadcast_agent_task_completed` or+`broadcast_agent_task_failed`. Both emit a CloudEvents `WorkflowEvent`(`type: com.opencompany.agent.task.completed`, with the success/errordiscriminator carried in `data.status`) via `dispatch.emit`. `taskTrigger` is-canary-registered, so `DeploymentManager` starts a `TriggerListenerWorkflow`-that receives the event via Temporal Signal and spawns a child workflow per-match. The assigning lead has already returned after receiving `queued`. When+canary-registered. In controlled deployments its definition and accepted+Signals live in `WorkflowControlWorkflow`, which filters/queues the event and+starts a child graph per match. A separate `TriggerListenerWorkflow` remains+the legacy compatibility path. The assigning lead has already returned after+receiving `queued`. Whenthe detached runner submits or fails the task, this trigger starts the connectedlead's separate review invocation. The event preserves the owning execution,so Task Manager reads the original durable task rather than the trigger run.@@ -70,8 +73,8 @@ envelope.```mermaidflowchart TD- P[child agent completes/errors] --> Q[broadcast_agent_task_completed<br/>dispatch.emit com.opencompany.agent.task.completed]- Q --> R[TriggerListenerWorkflow receives via Temporal Signal]+ P[Persist terminal task outcome] --> Q[broadcast_agent_task_completed or failed<br/>dispatch.emit com.opencompany.agent.task.completed]+ Q --> R[Controlled controller receives on_event Signal]R --> S[TaskTriggerNode.build_filter<br/>task_id / status / agent_name / parent_node_id]S -- match --> T[spawn child MachinaWorkflow<br/>trigger pre-executed with event payload]S -- no match --> R@@ -96,8 +99,8 @@ If any filter rejects, the event is skipped and the waiter stays blocked.- **Database writes**: none.- **Broadcasts**: the producer emits a CloudEvents `WorkflowEvent` via- `dispatch.emit`. The `TriggerListenerWorkflow` emits firing-pulse status via- `broadcast_trigger_status_activity` around each child spawn.+ `dispatch.emit`. The controller (or legacy listener) emits firing-pulse+ status via `broadcast_trigger_status_activity` around each child spawn.- **External API calls**: none.- **File I/O**: none.- **Subprocess**: none.@@ -107,9 +110,10 @@ If any filter rejects, the event is skipped and the waiter stays blocked.- **Credentials**: none.- **Services**: `services.deployment` (canary listener), `services.events.dispatch`,`services.status_broadcaster`.-- **Upstream dispatcher**: `services.handlers.tools._execute_delegated_agent`- calls `nodes.agent._events.broadcast_agent_task_completed` for both success and- error paths.+- **Upstream dispatchers**: `services.temporal.agent_activities.finish_agent_delegation`+ and the legacy `services.handlers.tools._execute_delegated_agent` use+ `nodes.agent._events` after persisting the outcome. A failed attempt that is+ requeued does not publish a terminal failure event.- **Python packages**: stdlib only.- **Environment variables**: none.@@ -125,9 +129,22 @@ If any filter rejects, the event is skipped and the waiter stays blocked.trigger blocks until cancelled.- Lead review requires a connected matching trigger path. External consumersmay use the event without receiving authority to mutate another team.+- Terminal event IDs are stable (`terminal_event_id` when supplied, otherwise+ task ID plus persisted terminal status), so Activity redelivery can use the+ controller's per-trigger deduplication. This does not fix same-label/type+ trigger Workflow-ID collisions or guarantee producer delivery before a+ Signal is accepted.## Related+- **Stop/Resume**: new versioned generations retain accepted completion+ events while stopped and start the review graph after Resume. Detached+ delegation runners enroll as independent execution roots, so Stop controls+ them even after their assigning parent finishes. Safe Apply's producer-only+ review drain cannot reopen a whole-generation Stop. Signal acceptance is+ receipt, not completion; producer delivery remains best effort before+ acceptance. See [Workflow control](../../temporal-workflow-control.md).+- **Skills using this as a tool**: none.- **Architecture docs**: [Event Waiter System](../../event_waiter_system.md),[Agent Delegation](../../agent_delegation.md)diff --git a/docs-internal/node-logic-flows/workflow_triggers/timer.md b/docs-internal/node-logic-flows/workflow_triggers/timer.mdindex 8d5b1827..d61b51b8 100644--- a/docs-internal/node-logic-flows/workflow_triggers/timer.md+++ b/docs-internal/node-logic-flows/workflow_triggers/timer.md@@ -66,8 +66,7 @@ flowchart TDS3 --> DS4 --> DD --> E[await asyncio.sleep wait_seconds]- E -- CancelledError --> F[Return success=false<br/>error: Timer cancelled]- E -- Exception --> G[Return success=false<br/>error: str e]+ E -- CancelledError --> F[Raise RuntimeError: Timer cancelled<br/>BaseNode returns failed envelope]E -- ok --> H[elapsed_ms = now - start]H --> I[Return success envelope<br/>with timestamp / elapsed_ms / duration / unit / message]```@@ -77,10 +76,22 @@ flowchart TD- **Unit mapping**: `seconds` / `minutes` / `hours` multiply duration by1 / 60 / 3600. Any other value falls through to the `_` match case and istreated as raw seconds.-- **Cancellation**: `asyncio.CancelledError` is caught and returns- `success=False` with `error="Timer cancelled"` rather than propagating.-- **Generic errors**: any other `Exception` is stringified into the error- envelope.+- **Explicit cancellation**: `asyncio.CancelledError` becomes+ `RuntimeError("Timer cancelled")`; the node execution wrapper surfaces the+ failed envelope. Cooperative generation Stop does not cancel this sleep.++## Stop/Resume boundary++When dispatched through Temporal, this operation is one regular Activity.+For a new controlled generation, Stop before admission holds it; Stop after+admission lets the delay finish under its existing timeout/retry policy and+retains its output. Stopped is acknowledged after that result's bookkeeping,+and Resume admits the next pending graph node or agent tool. The duration is+not checkpointed halfway through, and Resume does not repeat a completed timer.+Workflow checkpoint/maintenance sleeps are separately excluded from the drain+count; this user-requested delay is admitted node work. See+[Workflow control](../../temporal-workflow-control.md) and+[Node creation](../../node_creation.md#temporal-execution-and-stopresume-contract).## Side Effectsdiff --git a/docs-internal/node-logic-flows/workflow_triggers/webhookResponse.md b/docs-internal/node-logic-flows/workflow_triggers/webhookResponse.mdindex f63c1269..3e3bee9b 100644--- a/docs-internal/node-logic-flows/workflow_triggers/webhookResponse.md+++ b/docs-internal/node-logic-flows/workflow_triggers/webhookResponse.md@@ -119,6 +119,12 @@ flowchart TD## Related+- **Stop/Resume**: new controlled generations gate admission before this node+ and retain an admitted result through bookkeeping. The originating HTTP+ request still has its own timeout, and its process-local response future+ does not survive a backend restart or an arbitrarily long pause. See+ [Workflow control](../../temporal-workflow-control.md).+- **Companion node**: [`webhookTrigger`](./webhookTrigger.md) -`webhookResponse` is only useful when the trigger's `responseMode` is`responseNode`.diff --git a/docs-internal/node-logic-flows/workflow_triggers/webhookTrigger.md b/docs-internal/node-logic-flows/workflow_triggers/webhookTrigger.mdindex c584f886..0815cf9e 100644--- a/docs-internal/node-logic-flows/workflow_triggers/webhookTrigger.md+++ b/docs-internal/node-logic-flows/workflow_triggers/webhookTrigger.md@@ -15,9 +15,10 @@ Start a workflow when an HTTP request hits `/webhook/{path}`. Theproducer [`server/nodes/trigger/webhook_trigger/_events.py`](../../../server/nodes/trigger/webhook_trigger/_events.py)emits a CloudEvents `WorkflowEvent` (`type: com.opencompany.webhook.received`)via `dispatch.emit`. `webhookTrigger` is canary-registered, so-`DeploymentManager` starts a `TriggerListenerWorkflow` that receives the event-via Temporal Signal and spawns a child `MachinaWorkflow` per matching event-(filtered by `path`).+the controlled `WorkflowControlWorkflow` receives the event via `on_event`+Signal and starts a child `MachinaWorkflow` per matching event (filtered by+`path`). Separate `TriggerListenerWorkflow` executions remain the legacy+compatibility path; the plugin's waiter body is used for direct canvas Run.## Inputs (handles)@@ -64,7 +65,7 @@ Wrapped in the standard envelope.```mermaidflowchart TDP[HTTP request hits /webhook/path] --> Q[routers/webhook.py + _events.py<br/>dispatch.emit com.opencompany.webhook.received]- Q --> R[TriggerListenerWorkflow receives via Temporal Signal]+ Q --> R[Controlled controller receives on_event Signal]R --> S[WebhookTriggerNode.build_filter:<br/>event.path == params.path]S -- match --> T[spawn child MachinaWorkflow<br/>trigger pre-executed with event payload]S -- no match --> R@@ -78,15 +79,15 @@ flowchart TD- **Method / authentication** are NOT enforced inside the filter - they aresupposed to be enforced at the router layer when the HTTP request lands.The handler/filter only looks at `path`.-- **Cancellation**: user-initiated cancel via `cancel_event_wait` produces+- **Direct waiter cancellation**: user-initiated cancel via `cancel_event_wait` produces`success=False, error="Cancelled by user"`.## Side Effects- **Database writes**: none inside the plugin.- **Broadcasts**: the producer emits a CloudEvents `WorkflowEvent` via- `dispatch.emit`. The `TriggerListenerWorkflow` emits firing-pulse status via- `broadcast_trigger_status_activity` around each child spawn.+ `dispatch.emit`. The controller (or legacy listener) emits firing-pulse+ status via `broadcast_trigger_status_activity` around each child spawn.- **External API calls**: none.- **File I/O**: none.- **Subprocess**: none.@@ -102,7 +103,7 @@ flowchart TD## Edge cases & known limits-- The handler registers a waiter with `timeout=None`, so it waits forever+- On direct canvas Run, the handler registers a waiter with `timeout=None`, so it waits foreveruntil either an event arrives or the run is cancelled. There is nohandler-side timeout.- When multiple `webhookTrigger` nodes share the same `path` (different@@ -116,6 +117,15 @@ flowchart TD## Related+- **Stop/Resume and delivery**: a versioned stopped generation queues accepted+ webhook Signals and delays graph admission until Resume. An HTTP response+ from the router is not an acknowledged graph-completion result, and+ `dispatch.emit` discovery/delivery remains best effort before Signal+ acceptance. A deferred HTTP response may time out during a long pause; its+ process-local future is not a durable Temporal result. See+ [Workflow control](../../temporal-workflow-control.md) and+ [Event Waiter delivery gap](../../event_waiter_system.md#known-gap-canvas-run-on-canary-push-triggers).+- **Skills using this as a tool**: none.- **Companion node**: [`webhookResponse`](./webhookResponse.md) for`responseMode=responseNode`.diff --git a/docs-internal/node_creation.md b/docs-internal/node_creation.mdindex e8492e18..fce754f6 100644--- a/docs-internal/node_creation.md+++ b/docs-internal/node_creation.md@@ -34,6 +34,103 @@ Simple plugins inherit directly from `ActionNode` /`TriggerNode` / `ToolNode`. Rich plugin folders (telegram,stripe) layer additional bases from `services.events` underneath them.+## Temporal execution and Stop/Resume contract++Read [Workflow control](./temporal-workflow-control.md) before adding a node+that starts external work, a trigger, or an agent tool. New deployment+generations use `execution_control_version=1`; existing histories retain their+recorded command paths. A plugin does not implement a second pause lifecycle.++The workflow owns admission. It checks the control gate before admitting a+node, model request, tool, polling fetch, compaction, or child start. Stop+closes that admission, lets already-admitted work finish under its existing+retry policy, records its result and normal bookkeeping, then acknowledges+Stopped. Resume releases the same continuation at the next pending action.+For an agent response containing tools A and B, Stop during A preserves A's+result and the pending B call; Resume executes B without repeating the model+response or A. Already-admitted parallel tools finish concurrently.++Declare the operation's Temporal policy on the node class:++| Declaration | Meaning |+|---|---|+| `task_queue` | Plugin worker pool when the run's frozen worker-pool setting is enabled; otherwise the default worker handles it. |+| `start_to_close_timeout` | Maximum duration of one Activity attempt. |+| `heartbeat_timeout` | Liveness window for a long Activity; independent of cooperative Stop. |+| `retry_policy` | Attempts, intervals, and non-retryable error types for the operation. |++Ordinary per-type agent-tool dispatch applies all four declarations for new+controlled generations. Graph node dispatch currently applies plugin retry+and queue declarations but uses generic 24-hour attempt / 2-minute heartbeat+defaults; the existing Workspace-task override uses that plugin's timeout and+heartbeat values. Do not assume a graph node's timeout declaration is enforced+on every path. Resolved tool bindings and policies are recorded during agent+preparation and carried through+Continue-As-New, so Resume does not rebuild the pending call from changed+configuration. The in-process executor does not provide Temporal's policies.+Keep provider/tool I/O in regular Activities. A heartbeat can report liveness+or a real recovery checkpoint supported by the operation; a status string+cannot restore an opaque HTTP request, subprocess, or provider session.+Managed agents such as Claude Code, RLM, and Vertex stop at their whole+Activity boundary. Stop can remain Stopping while an admitted operation is+waiting or retrying; the model request policy currently permits unlimited+retries. See [Temporal Activity failure detection](https://docs.temporal.io/develop/python/failure-detection)+and [Long-running Activity](https://docs.temporal.io/design-patterns/long-running-activity).++## Event identity and delivery contract++For a deployed event trigger, register its CloudEvents type with+`register_canary_trigger_type` and emit a `WorkflowEvent` through+`services.events.dispatch.emit`. Controlled deployments keep definitions and+queued events in `WorkflowControlWorkflow`; separate+`TriggerListenerWorkflow` / `PollingTriggerWorkflow` executions are legacy+compatibility paths. An interactive canvas waiter is a separate in-memory+mechanism; check the [Event Waiter delivery gap](./event_waiter_system.md#known-gap-canvas-run-on-canary-push-triggers)+before assuming `emit` resolves it.++Use a stable event ID only when the provider supplies the complete identity.+Keep it unchanged on redelivery and distinguish different messages or event+families. The current producer conventions are:++| Producer | `WorkflowEvent.id` when identity is complete |+|---|---|+| Telegram message | `telegram:{chat_id}:{message_id}` |+| Discord message | `discord:message:{message_id}` |+| Discord interaction | `discord:interaction:{interaction_id}` |+| WhatsApp message | `whatsapp:{direction}:{chat_id-or-sender-or-from}:{message_id}` |+| Tracked chat message | Existing chat `run_id` (untracked messages use their message UID). |++Missing identity uses the envelope's random ID. Do not synthesize a constant+or partial key that would merge different messages. Random fallback preserves+distinct messages but cannot deduplicate provider redelivery.++The controller adopts the [Event Accumulator](https://docs.temporal.io/design-patterns/event-accumulator)+pattern's durable queue, deduplication, and continuation practices while+keeping immediate per-event processing and accepted queue order through+overflow. It does not add an inactivity batching window. Deduplication is per+trigger and event ID: pending keys remain protected+independently of the bounded recent-key window, overflow pages restore pending+keys, and queued events remain pending while stopped. Carry and overflow+spill fences include Signals arriving during awaits before Continue-As-New.+This protection is versioned with `controller-event-accumulator-v1`; it does+not retrofit old runs' recorded decisions.++Temporal Signals carry events, completed Updates acknowledge control and+membership changes, and read-only Queries expose status. Successful Signal+delivery means Temporal accepted the event into history, not that its handler+or downstream node finished. `dispatch.emit` still discovers consumers through+eventually consistent Visibility and logs/suppresses delivery errors; the+durability guarantee begins after a target accepts its Signal. There is no+durable producer outbox or promise of external exactly-once effects. Do not+use Signal-With-Start to recreate a missing versioned controller with an empty+root registry. See [Workflow messaging](https://docs.temporal.io/design-patterns/workflow-messaging-patterns)+and [Python message passing](https://docs.temporal.io/develop/python/message-passing).++Known naming limit: triggers with the same display-label/type slug can still+collide in listener and child Workflow IDs. A stable provider event ID does+not fix that separate trigger-identity issue. See the control document's+operating limits when adding multiple similar triggers.+## Five-minute recipe — one folder, one `__init__.py`For nodes with no state, no daemon, no signed webhooks (the common@@ -267,6 +364,7 @@ What you **do** still write:| Polling triggers + event_waiter mechanics | [event_waiter_system.md](./event_waiter_system.md) || Memory lifecycle (markdown parse/append/trim, vector store, session resume) — *archived; describes the retired pre-RFC-0002 `input-memory` markdown model* | [ARCHIVE/memory_lifecycle.md](./ARCHIVE/memory_lifecycle.md) || Tool building pipeline (`_build_tool_from_node`, schema, per-type Temporal dispatch) | [tool_building_pipeline.md](./tool_building_pipeline.md) |+| Cooperative Stop/Resume, root registration, messaging, and rollover | [temporal-workflow-control.md](./temporal-workflow-control.md) || Process supervision (used by `DaemonEventSource`) | [server/services/process_service.py](../server/services/process_service.py) — singleton API || Multi-vendor node behind one `provider` dropdown (two registries, JSON capabilities, per-vendor modules) | [speech_provider_rfc.md](./speech_provider_rfc.md) || Returning media from a node without blowing the payload limits (`AudioRef`, workspace routes, containment) | [media_transport.md](./media_transport.md) |diff --git a/docs-internal/normal_mode.md b/docs-internal/normal_mode.mdindex 1a24a748..5771e430 100644--- a/docs-internal/normal_mode.md+++ b/docs-internal/normal_mode.md@@ -68,7 +68,7 @@ is the reference.| `sidebar/`, `header/` | The team list, New employee, the profile row; the view title, the Workspace pill, the mode toggle (on an employee's page, Dev opens their workflow) and the theme button || `hire/` | The hero, the composer, the template chips, the hire notice (`HireNotice.tsx`), and the starter bundles (`starters.json`), which the chips and Settings > Plugins both read || `genui/` | The setup draft under the composer, and the hire itself, from a setup or a starter (below) |-| `employee/` | One employee's page, which is the conversation with them (`EmployeeChat` over the shared chat, Talk below); the header names them. What to act on shows only while there is something to do: the drafts waiting for the owner after the conversation, and above the message box their main action while they can't read messages (Resume, Start, or connect what is missing; `useEmployeeControl`, `PrimaryActionButton`) or Help in browser while they wait there. Pausing them, and watching them work, is the Workspace's: the header's Workspace pill opens it on the employee on screen, and its header carries the same main action |+| `employee/` | One employee's page, which is the conversation with them (`EmployeeChat` over the shared chat, Talk below); the header names them. What to act on shows only while there is something to do: the drafts waiting for the owner after the conversation, and above the message box their main action while they can't read messages (Resume, Start, or connect what is missing; `useEmployeeControl`, `PrimaryActionButton`) or Help in browser while they wait there. The conversation's Stop controls its run's owning generation; the Workspace header offers generation Stop and Resume too || `connectAI/` | The guided Connect an AI model dialog (below) || `workspace/` | The Workspace dock (below), its header pill, and its Canvas tab, which loads in its own chunk || `settings/` | Settings pages (Profile, Billing, Skills, Connectors, Plugins). Catalog primitives live in `components/catalog`; Connectors embeds the shared `components/credentials/CredentialsBrowser`. Provider dialogs belong to AppShell. |@@ -79,7 +79,9 @@ is the reference.| `state/homeStore.ts` | UI state only: the view, the sidebar, Settings, the Connect an AI model dialog, the last hire's notice, the Workspace dock, one-shot glow and pulse signals |Status comes from the server: an employee summary is `working`, `ready`,-`paused` or `attention` (an automatic pause, see below), and a pending draft+`paused` or `attention` (an automatic pause, see below). The `paused` summary+is displayed as **Stopped**; durable control transitions display+**Stopping…** or **Resuming…** even after the local request has ended. A pending draftshows as "Needs you". So does a browser waiting for the owner: when an agentcalls `request_user`, the summary's `browser_request` (`{node_id, reason,since}`, never the agent's message, since summaries reach every socket) is@@ -167,10 +169,11 @@ Help in browser), else the first on the team.- **Header**: the avatar, "{Name}’s workspace", the live task line whenthere is one (`useLiveTask`, else the summary's task), and a pill: Live- while the employee works, otherwise their status (Ready, Paused, Needs- attention, Needs you). Then their main action (Pause, Resume, Start, or- connect what is missing), the one place to pause them in Normal mode, and- Expand and Close.+ while the employee works, otherwise their status (Ready, Stopped, Needs+ attention, Needs you). Then their main action (Stop, Resume, Start, or+ connect what is missing), and Expand and Close. Stop shows Stopping while+ admitted work drains; the header follows authoritative generation control+ instead of treating a completed browser request as a completed Stop.- **Canvas**: the board named by the summary's `canvas_node_id`, drawn bythe editor's Canvas renderer (`CanvasContent`, see [Canvas Node](./canvas_node.md)).It loads in its own chunk, which keeps the board's markdown, code and@@ -362,7 +365,7 @@ owner dismisses it or opens another view; `needs_ai` opens Connect an AImodel; every error code has plain words (`busy`: they are still being setup).-### 3. Start, pause, resume+### 3. Start, Stop, Resume`start_employee {workflow_id, expected_revision, idempotency_key}`([start.py](../server/services/employees/start.py)) refuses with@@ -371,8 +374,27 @@ and the agent it talks to the owner through, `node_roles` `agent` and`talk_agent`) onto a usable model, and calls `start_saved_workflow`, the samestart path the editor uses. One that stopped after a problem is reset first(`reset_if_failed`); the reset moves its control revision on, so the revision-the page sent is checked here instead. Pause and Resume are the editor's-`pause_workflow` / `resume_workflow`.+the page sent is checked here instead. Stop and Resume retain the editor's+`pause_workflow` / `resume_workflow` request names and require+`expected_revision` and `idempotency_key`.++For new generations with `execution_control_version: 1`, Stop first closes+admission throughout the generation's execution tree. Already admitted model+requests, parallel tools and whole-activity agents finish under their existing+retry policies; their results and bookkeeping are retained. **Stopping** is the+persisted `pausing` transition, and **Stopped** is acknowledged `paused` after+that work settles. Resume releases the same continuations at the next pending+action and then reopens event and schedule producers. It never starts the+agent again at its opening prompt. A request timeout leaves the durable intent+in place and status reads reconcile it. Unlimited model retries can keep Stop+in Stopping during a provider outage. Independent Workspace tasks and approved+sends have their own execution lifecycle.++Older generations retain their recorded control paths. A missing versioned+controller fails closed; Resume does not rebuild an empty registry and claim+the suspended work was recovered. See+[Temporal Workflow Control](./temporal-workflow-control.md) for acknowledgements,+rollover, recovery and operating limits.Automatic pauses now record why: `WorkflowControlExecution.pause_reason`(`failures` from the circuit breaker, `recovery` after a crash,@@ -461,10 +483,11 @@ sending and drafts belong to the chat (wire and client rules in"Jump to latest" button counts them. It refetches on `chat.updated`, after aruntime reset, and when the socket reopens. The drafts waiting for theowner's OK follow the conversation (the host's `afterThread`).-- **Working** follows the run the message started, not timers: from the+- **Working** follows the run the message started and its owning generation's+ control, not timers: from themoment the server admits it, the avatar spins its ring and skeleton linesstand where the answer will be, with "Thinking" under them, until that run- ends, whatever else lands meanwhile (a routine report does not end it). The+ ends or is intentionally suspended, whatever else lands meanwhile (a routine report does not end it). Thetalk agent's answer streams in as it is written ("Writing · N tok/s", acaret after the text), and the apps it uses show above it as steps("Working…", then "Worked for 12s · 3 steps"). A run that failed says so@@ -474,9 +497,19 @@ sending and drafts belong to the chat (wire and client rules incame. While the talk agent waits to retry after a failed attempt, its retrymessage sits on the status line (`useRetryNote`, from the agent's nodestatus).-- **Stop.** While the run works, Send is Stop, and Esc stops it too: the- answer so far stays, marked "You stopped this reply." A message still- waiting for Resume is withdrawn the same way.+- **Stop/Resume.** While the run works, Send is Stop, and Esc requests Stop+ too. In a controlled generation the current tool or model request finishes,+ its output stays, and the turn shows Stopping while work drains, then+ "Waiting for you to resume {Name}." The composer offers Resume, followed by+ disabled Resuming; Esc has no effect while stopping, stopped or resuming.+ The chat run keeps its run/reply identities and its occupied lane. Resume+ continues from the next pending action, so editing, trying another answer+ or sending a second message still waits for this run to end. The control+ snapshot and subscriptions reconcile on reload; unsaved streamed text still+ depends on the live server hub (see the chat protocol's streaming limits).+ For legacy or uncontrolled runs, Stop retains its terminal behavior: the+ partial answer says "You stopped this reply," or a queued message is+ withdrawn, and the lane is released.- **The box's extras**: Attach (or paste, or drop files anywhere on thepage) adds up to six files, uploaded at once and sent with the next message(a message can be files alone); the employee reads where they are in its@@ -499,14 +532,17 @@ sending and drafts belong to the chat (wire and client rules init next time, which is when the rating reaches them.- **The box** follows the control state the way `send_chat_message` does(`talkMode` in `presentation.ts`):- - *send* (running, starting, resuming): a message shows at once- ("Sending…") and goes to the employee; Send is then Stop until the run- ends, since overlapping runs would each save over the other's+ - *send* (running, starting, resuming): a message accepted by the server shows at once+ ("Sending…") and goes to the employee; Send is then Stop while the run+ works, since overlapping runs would each save over the other'sconversation (the server refuses a second message with`run_in_progress`).- - *queue* (paused, pausing): one message waits for Resume ("Your message is+ - *queue* (paused, pausing): the server can admit one queued message when+ the lane is free, and an accepted event waits for Resume ("Your message iswaiting…" above the box, "Waiting for you to resume {Name}." in the- thread), and Send is Stop until it has been answered or withdrawn.+ thread). A suspended controlled run keeps the lane occupied; the composer+ offers Resume while paused, Resuming during release, and disabled Stop+ while its run drains. Legacy queued runs can still be withdrawn with Stop.- *start* (never started, ready, resetting, failed): no box; a line saysthey can't read messages, beside their main action (Start, Start again,or what they are missing), which the Workspace header offers too.@@ -799,12 +835,14 @@ WebSocket requests (snake_case; failures come back as `success: false` with an| `cancel_employee_setup` | `{draft_token}` | `{cancelled}` || `hire_employee` | `HireEmployeeRequest` | `{employee, started, missing_apps, needs_ai, unsupported_apps, warnings, idempotent}`; errors include `busy` while the same key's first attempt is still building || `start_employee` | `{workflow_id, expected_revision, idempotency_key}` | as `start_workflow` |+| `pause_workflow` / `resume_workflow` | `{workflow_id, expected_revision, idempotency_key}` | Generation control snapshot: `state`, `revision`, root and execution identities, capabilities and `execution_control_version`; acknowledged `paused` means admitted work drained for version 1. Transitional status is reconciled after a timeout || `enable_employee_talk` | `{workflow_id, idempotency_key}` | `{employee}`. Errors: `invalid_request`, `not_found`, `unsupported`, `conflict` (a start, pause, resume or reset is under way, or the graph changed meanwhile), `restart_failed`; the last three carry `employee` too || `apply_employee_changes` | `{workflow_id, idempotency_key}` | `{employee}`: running ends running, paused or failed ends ready, ready is left alone. Errors: `invalid_request`, `not_found`, `conflict`, `restart_failed` (the last two with `employee`) || `rename_employee` | `{workflow_id, name}` (spaces collapsed, at most 40 characters) | `{employee}`: renames the workflow (a new slug, the workspace folder moved, `workflow.renamed` sent), and each agent's instructions a hire wrote take the new name in their opening ("You are <name>, ..."); instructions the owner rewrote keep their words. Agents read them on every run, so nothing restarts. Errors: `invalid_request`, `not_found`, `save_failed` || `set_employee_photo` | `{workflow_id, path \| null}` | `{employee}`: `path` is a PNG, JPEG, WebP or GIF the owner uploaded under `uploads/` (`POST /api/workspace/{workflow_id}/uploads`), at most 5 MB (`EMPLOYEE_PHOTO_MAX_BYTES`); `null` takes the photo away. Errors: `invalid_request`, `not_found`, `invalid_photo` (with `detail`), `unsupported` (a workflow built in the editor has no employee row to keep it on) || `send_chat_message` | `{message, role: "user", session_id: <workflow_id>, timestamp, client_message_id?}` | `{timestamp, delivery, message_id, run_id}`: `"now"` while running, starting or resuming; `"queued"` while paused or pausing (it runs on Resume). In any other state `not_running`, and nothing is saved or sent; `run_in_progress` (with the live `run_id`) while a run is live. `run_id` is null when no deployed chat trigger answers the session. Session `"default"` works as before, with no `delivery` |-| `get_chat_messages` | `{session_id, limit?, all_generations?}` | `{messages, thread, active_runs}`, messages oldest first, each `{id, role, message, timestamp, run_key, ...}` (the full shape is in [chat_protocol.md](./chat_protocol.md#messages)). Timestamps carry their UTC offset; `run_key` is the generation the row was written in. Without `all_generations`, only the latest generation's rows (none after a Reset; every row when the workflow was never started). A failed read answers `read_failed`, never an empty thread |+| `get_chat_messages` | `{session_id, limit?, all_generations?}` | `{protocol_version: 2, messages, thread, active_runs}`, messages oldest first, each `{id, role, message, timestamp, run_key, ...}` (the full shape is in [chat_protocol.md](./chat_protocol.md#messages)). Controlled active run snapshots also include their owning `workflow_control`. Timestamps carry their UTC offset; `run_key` is the generation the row was written in. Without `all_generations`, only the latest generation's rows (none after a Reset; every row when the workflow was never started). A failed read answers `read_failed`, never an empty thread |+| `stop_chat_run` | `{run_id, expected_revision, idempotency_key}` for controlled runs; `{run_id}` for legacy | Controlled: generation Stop payload with `run_id` and `resumable: true`; the owning root is checked. Legacy: terminal chat `stopping` / `stopped`. Full acknowledgement and error contract: [chat_protocol.md](./chat_protocol.md#controlled-stop-acknowledgement) || `list_approvals` | `{workflow_id?, status?, limit <= 100}` | `{approvals, counts, server_time}` || `decide_approval` | `{approval_id, decision, text?, subject?, decision_key}` | `{approval, will_send_on_resume}` || `delete_workflow` | `{workflow_id}` | `{workflow_id, contexts_archived, context_archives_pending}`; `DELETE /api/database/workflows/{id}` is the same handler. Error `workflow_shutdown_failed` (with `detail`) when stopping the employee failed: nothing was deleted |@@ -896,8 +934,12 @@ chat's rollback), `lib/__tests__/workflowOps.test.ts` andThe next message is a new run from the live generation's snapshot, whichlacks it until the agent asks the Agent Builder for it again (that bindsthe saved node, bind-only). The worker gets it only after Apply.-- Messages queued while paused each start a run on Resume; the page allows- one queued message at a time.+- A queued message whose Signal reached the controller starts its pending+ execution on Resume; a suspended run continues its existing execution.+ The lane permits one nonterminal run, including a stopped controlled run.+ Event ingress uses eventually consistent Visibility discovery and logs+ delivery failures; saving a message does not establish an end-to-end durable+ outbox guarantee. See [Chat Trigger](./node-logic-flows/workflow_triggers/chatTrigger.md#edge-cases--known-limits).- Turn on Talk and Apply reset the employee: the conversation starts fresh(its thread is cleared with the agent's Context), messages queued whilepaused are dropped, and drafts waiting for the owner are cancelled (thediff --git a/docs-internal/plugin_system.md b/docs-internal/plugin_system.mdindex 8d85c9d3..d79e916a 100644--- a/docs-internal/plugin_system.md+++ b/docs-internal/plugin_system.md@@ -84,8 +84,8 @@ class SpecializedAgentBase(ActionNode, abstract=True):| `approval` | `ApprovalSpec` (`services/plugin/approval.py`) for a tool that reaches someone outside: which operations send (`operations`, `when`), who it goes to, the body and subject the owner may edit, the card's other lines, `refuse_while_asking` (Stripe) or `restrict_while_asking` (the browser, read-only). While the workflow asks first, an agent's call that sends is held as a draft for the owner instead of running. See [Chat Protocol, Approvals](./chat_protocol.md#approvals). || `chat_sources` | `ClassVar[bool]` — the tool returns web results (`results: [{title, snippet, url}]`, the searches). When the agent answering a chat run calls it, the results are numbered for the conversation in what the model reads, so the answer can cite them as `[n]`. See [Chat Protocol](./chat_protocol.md#sources). || `task_queue` | Temporal worker pool. See `TaskQueue` constants. |-| `retry_policy` | `RetryPolicy` dataclass (mirrors `temporalio.common.RetryPolicy`). |-| `start_to_close_timeout` / `heartbeat_timeout` | Per-node Temporal knobs. |+| `retry_policy` | `RetryPolicy` dataclass (mirrors `temporalio.common.RetryPolicy`); attempts, intervals, and non-retryable error types. |+| `start_to_close_timeout` / `heartbeat_timeout` | Per-attempt deadline / liveness window for regular Temporal Activities. Heartbeats do not extend the attempt deadline. |### Polymorphic result envelope (`interpret_result`)@@ -465,6 +465,30 @@ per-type activity to its plugin-declared `cls.task_queue`. Setting`TEMPORAL_WORKER_POOL_ENABLED=false` stops the pool and routes activities backto the manager's single default queue.+Execution routing is recorded in the run's input; changing environment settings+does not redirect an already-running workflow. New controlled generations+(`execution_control_version=1`) also freeze resolved agent-tool bindings and+the plugin's timeout, heartbeat, retry, and queue declarations during payload+preparation. Ordinary per-type tools use these policies and carry them through+Continue-As-New. Existing histories retain their recorded policy decisions.+The in-process executor does not enforce Temporal Activity policies.++Graph Activity starts currently honor plugin retry and queue declarations,+while ordinary graph nodes retain generic 24-hour attempt / 2-minute heartbeat+defaults. The existing Workspace-task override applies its declared timeout+and heartbeat. The frozen v1 ordinary-agent-tool policy described above is+stronger; do not assume the graph path applies every timeout declaration.++Cooperative Stop is owned by workflow admission gates before node, tool,+model, polling, compaction, refresh, and child-start work. Admitted work finishes+under its existing retries and retains its result/bookkeeping; Resume releases+the existing continuation's next pending action. Already-admitted parallel work+drains concurrently. Keep provider I/O in regular Activities, and add heartbeat+progress recovery only when the operation actually supports it. A status+heartbeat cannot resume an opaque subprocess or HTTP request. See+[Workflow control](./temporal-workflow-control.md) and+[Node creation: execution contract](./node_creation.md#temporal-execution-and-stopresume-contract).+```python# Worker collection patterns (both supported).@@ -509,10 +533,11 @@ agents (`android_agent`, `coding_agent`, `web_agent`, `task_agent`,`payments_agent`, `consumer_agent`, `autonomous_agent`), 2 teamleads (`orchestrator_agent`, `ai_employee`).-**Agents that stay as single activities** (2): `rlm_agent`,-`claude_code_agent`. Their internal session state (RLM REPL / Claude-CLI `--resume` with stable `cwd`) requires single-process continuity-that would break across activity boundaries.+**Agents that stay as single activities**: `rlm_agent`,+`claude_code_agent`, and `vertex_managed_agent`. Their internal session state+(RLM REPL, Claude CLI `--resume` with stable `cwd`, or Vertex interaction and+environment chaining) remains inside one Activity. Cooperative Stop drains+that whole Activity; it cannot stop between tools hidden inside that session.Queue distribution (live count via`distinct_task_queues()` and `len(_NODE_CLASS_REGISTRY)`):@@ -1412,7 +1437,7 @@ project that publishes pre-built binaries via GitHub releases.|---|---|---|| HTTP webhook ingress | `routers/webhook.py` consults `WEBHOOK_SOURCES` registry | `register_webhook_source(MySource())` || Event dispatch into workflows | `event_waiter.dispatch(event)` from `WebhookSource.handle` (canvas Run, and a deploy that runs without Temporal) | provider-specific `shape()` returning `WorkflowEvent` |-| Deployed delivery (Temporal) | `services.events.dispatch.emit` signals the listeners whose `EventType` matches the envelope's `type` | `register_canary_trigger_type(node_type, <the envelope type>)` + an `emit` of each envelope (Stripe: `_events.emit_stripe_event`, called from `handle`) |+| Deployed delivery (Temporal) | `services.events.dispatch.emit` discovers consumers through Visibility and sends `on_event` Signals; controlled deployments queue/filter in `WorkflowControlWorkflow`, separate listeners are compatibility paths | `register_canary_trigger_type(node_type, <the envelope type>)` + an `emit` of each envelope (Stripe: `_events.emit_stripe_event`, called from `handle`) || Trigger waiting + filtering | `WebhookTriggerNode.build_filter` (CloudEvents glob); both callers pass the envelope's `data` | override `build_filter` to read the payload (the base rebuilds an envelope from it and raises); `_extra_filter(params)` only on top of the base || Deploy-time setup | `DeploymentManager._prepare_trigger_deployment` calls the trigger class's `prepare_deployment` at Start and at the boot re-arm; a raise is logged and the trigger armed anyway | optional `prepare_deployment(node_id=, workflow_id=, parameters=)` classmethod (Stripe starts `stripe listen` in a background task) || Daemon lifecycle | `DaemonEventSource.start/stop/restart` via `ProcessService` | `build_command(secrets)` + `parse_line(stream, line)` (subscribed via `ProcessService.start(line_handler=...)` — no log-file tailing) |@@ -1426,6 +1451,20 @@ project that publishes pre-built binaries via GitHub releases.| Connection state surfaced to the modal (CLI-managed auth) | `auth_service.store_oauth_tokens(provider, "cli-managed", "cli-managed")` + `StatusBroadcaster.broadcast_credential_event` (CloudEvents v1.0 envelope wrapping `WorkflowEvent`; locked by `tests/credentials/test_credential_broadcasts.py`) | `_mark_logged_in` / `_mark_logged_out` helper pair; one `broadcaster.broadcast_credential_event("credential.oauth.connected", provider="<id>")` after login and `…disconnected` after logout. Same shape Twitter / Google logout use. || Auto-install of an external CLI binary | `_install.py` with `ensure_<cli>_cli()` + GitHub-releases asset map | pinned `_VERSION` constant, `(system, machine) -> (asset, kind, member)` table, and an override on `DaemonEventSource.start()` that `await`s the helper before `super().start()` (also set `binary_name = ""` to skip the framework's `shutil.which` pre-check) |+Use a stable provider event ID when the complete identity is available and+preserve it on redelivery; otherwise retain the random envelope fallback.+The controller borrows durable pending-key, deduplication, and rollover+practices from the Event Accumulator pattern while preserving immediate+per-event processing. Accepted Signals queue while stopped, including across+overflow spill and Continue-As-New. Consumer discovery and delivery errors+remain best effort before Signal acceptance; there is no producer outbox or+external exactly-once guarantee. Same-label/type trigger Workflow-ID collisions+remain a known naming limit. See+[Node creation: event identity and delivery](./node_creation.md#event-identity-and-delivery-contract),+[Workflow control](./temporal-workflow-control.md),+[Event Accumulator](https://docs.temporal.io/design-patterns/event-accumulator),+and [Workflow messaging](https://docs.temporal.io/design-patterns/workflow-messaging-patterns).+### Tool / skill / visuals naming contractThree coordinates have to agree for both the LLM tool dispatcher anddiff --git a/docs-internal/rlm_service.md b/docs-internal/rlm_service.mdindex bdbcc764..32320300 100644--- a/docs-internal/rlm_service.md+++ b/docs-internal/rlm_service.md@@ -10,15 +10,29 @@ The RLM (Recursive Language Models) service integrates the `rlms` library into O**Paper**: https://arxiv.org/abs/2512.24601**Docs**: https://alexzhang13.github.io/rlm/+### Temporal generation Stop/Resume boundary++The RLM REPL and recursive provider/tool calls execute inside one node Activity;+`rlm_agent` is outside `AGENT_WORKFLOW_TYPES`. In new controlled generations+(`execution_control_version=1`), Stop lets that admitted Activity settle under+its scheduled policy, consumes its result normally, and fences the next graph+or parent-agent action. Resume releases the existing continuation without+relaunching the recorded completed RLM work. It does not pause at individual+REPL iterations or recursive calls, and heartbeats do not serialize a REPL+checkpoint. Failure recovery and unrecorded effects remain subject to the+Activity/tool retry and idempotency contract. Legacy generation behavior is+preserved. See [Temporal workflow control](temporal-workflow-control.md) and+[RLM node control boundary](node-logic-flows/specialized_agents/rlm_agent.md#temporal-stopresume-boundary).+### How RLM Differs from Standard Agents| Aspect | Standard Agents (aiAgent, chatAgent, etc.) | RLM Agent (rlm_agent) ||--------|---------------------------------------------|----------------------|| Execution model | LLM -> tool call -> LLM -> tool call | LLM -> `\`\`\`repl` code block -> exec() -> stdout -> LLM || Tool interface | Provider-neutral `AgentToolSpec` / `ToolDef` values with Pydantic validation | Python functions injected into REPL namespace |-| State management | Native message accumulation in `run_native_agent_loop` | Python variables in REPL namespace + `context` variable |+| State management | In-process `run_native_agent_loop` or durable Temporal `AgentWorkflow` transcript | Python variables in REPL namespace + `context` variable || Completion signal | LLM stops making tool calls | LLM calls `FINAL(answer)` or `FINAL_VAR(variable_name)` |-| Recursion | Agent delegation (fire-and-forget) | `rlm_query()` spawns child RLM with own REPL |+| Recursion | Attached Temporal children, detached Task Manager runners, or process-local delegation according to routing | `rlm_query()` spawns child RLM with own REPL || Strengths | Structured tool calling, provider-agnostic | Complex reasoning, code-driven decomposition, recursive sub-problems |---diff --git a/docs-internal/temporal-workflow-control.md b/docs-internal/temporal-workflow-control.mdindex d907cc70..40b5fcd3 100644--- a/docs-internal/temporal-workflow-control.md+++ b/docs-internal/temporal-workflow-control.md@@ -5,15 +5,20 @@ Workflow deployments use a persisted control generation and a long-livedcurrent generation, revision, graph snapshot, and UI authorization; Temporalis authoritative for execution history.+This is the current control contract. Use it with+[Temporal architecture](TEMPORAL_ARCHITECTURE.md),+[node authoring](node_creation.md), [event routing](event_framework.md), and+[chat protocol](chat_protocol.md). Archived RFCs describe earlier designs.+## Control lifecycle- **Start** creates the first generation and deploys its snapshotted graph.-- **Pause** cooperatively gates new trigger admissions, workflow nodes, agent+- **Stop** (the `pause_workflow` API) cooperatively gates new trigger admissions, workflow nodes, agentturns, tool calls, polling iterations, and delegated work. In-flight work mayfinish and remains durable. Push events stay queued in the controller,provider polling cannot launch graph runs, Temporal cron schedules arepaused, and armed trigger nodes switch to an explicit paused visual state.-- **Resume** signals the same running Temporal executions and drains buffered+- **Resume** releases the same running Temporal executions and drains bufferedtrigger events in FIFO order, unpauses cron schedules, and rearms triggernodes.- **Reset** revision-guards the old generation, closes controller and local@@ -40,7 +45,7 @@ Once the `running` compare-and-swap commits, a status-broadcast/projectionfailure does not tear down that live generation; the failed request is recoveredby the client's authoritative status resync.-Pause and Resume use the controller's acknowledged `set_control_state` Temporal+Legacy Pause and Resume use the controller's acknowledged `set_control_state` TemporalUpdate. The database first publishes `pausing`/`resuming`, then publishes thestable state only after the Update result confirms `paused`/`running`. If theUpdate is known to have been rejected before admission, the database and local@@ -65,6 +70,342 @@ already reports its requested stable state. Transitional Pause, Resume, andReset states expose an explicit retry rather than trapping the toolbar in apermanent spinner.+## Acknowledged Stop for new generations++New generations atomically store `resource_manifest.execution_control_version=1`.+Existing generations keep their recorded command paths and legacy Signals;+unordered Signals do not change version 1 control intent. Version 1 uses+revision-ordered `set_control_state` and `wait_for_checkpoint` Updates, with+`execution_control_status` Queries on controlled workflows.++Stop during tool A means A finishes under its existing retry policy, its result+and normal bookkeeping are retained, and the next pending action waits for+Resume. Completed LLM responses retain their pending tool calls. Parallel work+already admitted drains concurrently. Stop does not cancel, terminate, or+replace the execution. `pausing` means admission is closing or admitted work is+draining; `paused` means the checkpoint has been acknowledged throughout the+execution tree. A request timeout leaves `pausing` for reconciliation and does+not cancel the tool. Unlimited LLM retries can keep it there during an outage.++The existing controller stores only independently living roots: graph runs,+cron firings, employee jobs, and detached delegation runners. Each enrolls+through an acknowledged controller Update before business work. A small+maintenance Activity bridges the Workflow to the client Update API. Registry+entries contain Workflow ID and first execution Run ID, and are carried with+the membership epoch through controller continue-as-new. Terminal roots remove+themselves; control operations remove stale registrations after authoritative+Describe calls. There is no SQL participant registry or periodic lease loop.++Each admission setter closes local scheduling and waits for admitted child+starts to acknowledge. Native `DescribeWorkflowExecution.pending_children`+then discovers attached children. Stop closes the complete discovered tree+before awaiting any checkpoint. Checkpoints count admitted business Activities+through result bookkeeping; maintenance Activities, permit waits, sleeps and+child-result waits are excluded. Root membership is reconciled until stable.+Visibility scans remain part of destructive Reset, not Stop acknowledgement.++### Stop and Resume acknowledgement sequence++The orchestration lives in+[`transition_generation`](../server/services/deployment/execution_control.py);+the participant gate and counters live in+[`ExecutionControl`](../server/services/temporal/execution_control.py).++1. Stop persists the `pausing` revision and closes process-local producers.+ Cron schedules and trigger projections are paused before tree traversal.+2. A completed controller admission Update closes controller producers. Each+ independent root and attached descendant receives the same revision. Its+ setter waits only for child-start acknowledgements, allowing the subsequent+ native Describe to enumerate children already being started.+3. After admission is closed throughout the discovered tree, checkpoint+ Updates run concurrently. Each waits for local admitted work and result+ bookkeeping, without waiting for child completion. This lets a paused child+ remain alive without deadlocking its parent's acknowledgement.+4. A changed membership epoch restarts discovery. Stale registrations are+ removed only after native Describe confirms a terminal, missing, or+ different execution chain. The database publishes `paused` after the+ membership and checkpoint barriers settle.+5. Resume persists `resuming` and applies a new running revision with+ `producers_held=true`. Descendants are released before roots. Controller+ release checks the membership epoch atomically; a late registration causes+ another reconciliation pass. Local producers and cron schedules reopen+ after that release, and the database publishes `running`.++The admission acknowledgement is distinct from complete drain. Status counts+continue to report admitted business work during `pausing`; a zero UI count+does not establish the checkpoint. Checkpoint sleeps, permit waits, enrollment+maintenance and child-result waits are not business work. Completing a child+result still performs its normal bookkeeping while admission is closed.++Resume sets the next revision while holding producer admission, resumes+descendants before their roots, and atomically checks controller membership+before reopening producers. A late root inherits the controller's held state+and is included in reconciliation. Handles address current Workflow IDs;+native chain identity and Update payload checks prevent Workflow ID reuse+from controlling a different execution. Application revisions survive rollover+because Temporal Update-ID deduplication is scoped to one execution.++Agents roll over only at clean boundaries with no live delegation handles,+while running and after pending control handlers finish. Continuation carries+prepared configuration and tool policies, transcript, thinking, iteration,+usage and control state. The complete encoded continuation input is measured.+Existing compaction is used; an oversized version 1 continuation reports+`AgentContinuationTooLarge` rather than restarting at the opening prompt.+Ordinary tool dispatch respects frozen plugin retry, queue, heartbeat and+timeout declarations. Heartbeats retain their liveness/failure-recovery role;+they are not checkpoints for opaque provider or tool sessions.++Controlled chat Stop resolves the run's exact owning generation, preserves its+run/reply identity, partial output, subscription and occupied lane, and derives+Stopping/Stopped/Resume from workflow control. Suspension retains live output;+a server process restart cannot reconstruct unsaved text deltas from the hub.+See the [chat persistence limits](chat_protocol.md#controlled-stop-acknowledgement).+The watchdog excludes stopped+generations from age expiry; successful Resume persists `last_resumed_at` in+existing control metadata to give a fresh timeout window. Direct Workspace+tasks and separately approved delivery continuations remain independent.+Claude Code, RLM and Vertex managed agents drain at their whole Activity boundary.++Real SDK integration tests use stub Activities and replay captured new and+legacy histories. They exercise parallel drain, attached children, late roots,+worker restart, pending model tool calls and two actual agent rollovers.+Admission acknowledgement and complete drain latency are logged separately.++The guarantee applies to recorded completion. Resume does not schedule a+successfully recorded tool again. If an external effect happened before+Temporal recorded the Activity result, the Activity's existing retry and+idempotency contract still determines whether that effect can repeat.++The protocol follows Temporal's [entity lifecycle](https://docs.temporal.io/design-patterns/entity-lifecycle-patterns),+[event accumulator](https://docs.temporal.io/design-patterns/event-accumulator),+[long-running Activity](https://docs.temporal.io/design-patterns/long-running-activity),+and [continue-as-new](https://docs.temporal.io/develop/python/workflows/continue-as-new)+guidance. The [resumable Activity pattern](https://docs.temporal.io/design-patterns/resumable-activity)+reruns a failed Activity after corrective input; it does not implement Resume+after a successfully completed tool. Provider and tool operations remain+regular Activities rather than [Local Activities](https://docs.temporal.io/design-patterns/local-activities).++## Workflow messaging review++The [messaging patterns](https://docs.temporal.io/design-patterns/workflow-messaging-patterns)+and [sending reference](https://docs.temporal.io/sending-messages) support this+division of responsibilities:++| Operation | Message | Acknowledgement |+|---|---|---|+| Incoming trigger event | Signal with stable event ID | Temporal accepted the Signal; handler processing is asynchronous |+| Stop/Resume admission | Completed `set_control_state` Update | Requested revision applied and admitted child starts acknowledged |+| Stop drain | Completed `wait_for_checkpoint` Update | Admitted work and result bookkeeping settled |+| Independent root registration/removal | Update through a maintenance Activity | Controller membership mutation completed |+| UI status | Synchronous Query | Read-only snapshot; not a transition acknowledgement |++The Activity bridge is required because a Workflow cannot directly issue an+Update to another Workflow. `execute_update` waits for completion; merely+reaching the Accepted stage does not establish that work drained.++Pure control validators check message shape, state, revision, generation,+execution chain and admission-hold type before acceptance. Handler-side typed+`ApplicationError` checks remain for direct calls and replay. An ordinary+`ValueError` or `TypeError` in an accepted Python Update handler retries the+Workflow Task rather than rejecting only the Update. These distinctions follow+the [handler exception rules](https://docs.temporal.io/handling-messages#exceptions-in-message-handlers).++State/revision changes happen before the first await. Checkpoint waits use+`workflow.wait_condition` and allow newer control handlers to run; a lock held+through draining would prevent Resume from superseding the wait. Input state+is initialized with `@workflow.init`; pending handlers finish before return or+rollover. See the [Python async-handler guidance](https://docs.temporal.io/develop/python/workflows/message-passing#use-async-handlers)+and [official safe-handler sample](https://github.com/temporalio/samples-python/blob/main/message_passing/safe_message_handlers/workflow.py).++New version 1 histories record `controller-messaging-v1`. Successful controller+Updates check the existing history-pressure signal and wake the main loop to+continue-as-new, even when no events or polls occur. The Update handler never+performs the rollover itself. This accounts for [per-execution Update limits](https://docs.temporal.io/evaluate/cloud/limits#per-workflow-execution-update-limits)+in addition to event-history growth. Existing revisions, membership and events+carry into the next run; replay tests cover pre-marker histories.++Employee Safe Apply keeps its producer-only admission pause. Its compatibility+string Resume can reopen producers only while the versioned participant is+running and unheld; it cannot override Stop or a held Resume. Task-review+draining is likewise disabled while generation control closes admission.++A client deadline or cancellation does not cancel an accepted Update. Retain+the desired revision and transitional state and reconcile the outcome, as+specified by the [Update client contract](https://python.temporal.io/temporalio.client.WorkflowHandle.html#execute_update).+Update-ID deduplication covers a single run; application revision and membership+identity preserve idempotency across rollover and maintenance Activity retries.++### Control interfaces++The WebSocket APIs retain their existing names. Example Stop request data:++```json+{+ "workflow_id": "saved-workflow-id",+ "expected_revision": 12,+ "idempotency_key": "stop-request-unique-id"+}+```++Send this data with `type: "pause_workflow"`; Resume uses+`type: "resume_workflow"` and the latest returned revision. Status uses+`get_workflow_control_status`. Read the authoritative response and its+capabilities before the next mutation: a stable-state CAS also advances the+database revision. Do not infer the next revision from a local counter.++Controlled `stop_chat_run` takes `run_id`, `expected_revision` and+`idempotency_key`. It authorizes the session, resolves the run's workflow and+root execution identity, and delegates to Stop with an owning-generation+guard. It returns the generation control payload plus `run_id` and+`resumable: true`. Resume uses the generation's `resume_workflow` API. See+[chat protocol](chat_protocol.md) for legacy terminal Stop and wire details.++Internal Update payloads are not the browser API:++| Surface | Fields and result |+|---|---|+| `set_control_state` | `state` (`paused` or `running`), transition `revision`, `generation`, `first_execution_run_id`, and optional boolean `producers_held`; returns applied intent and admission acknowledgement |+| `wait_for_checkpoint` | Same intent fields, or omitted payload to wait for existing intent; returns counters and `checkpoint` |+| `register_execution` | `workflow_id`, `first_execution_run_id`, `generation`; returns controller participant state/revision and hold for the enrolling root |+| `unregister_execution` | Same membership identity; removes only a matching execution chain |+| `execution_control_status` | Read-only participant state, revision, hold, active actions, pending child starts and checkpoint |+| Controller `status` | Also includes `live_roots`, `membership_epoch`, producer state and event queue status |++Final controller release additionally includes `expected_membership_epoch`.+Rejected shape, state, revision, generation or execution identity produces a+typed Update error; no accepted mutation should be followed by a raw input+conversion failure. A stale revision cannot overwrite newer intent. Same-revision+registration results cannot undo an explicit release. Child inheritance omits+the parent's root-chain and release markers; same-Workflow continuation carries+them, so a new child's enrollment still learns the controller's current hold.++Separate identity finding: existing listener IDs and child display prefixes use+trigger labels. Two same-label trigger nodes can collide during registration.+That pre-existing naming contract needs an immutable trigger-node identity for+multi-trigger deployments; the messaging hardening above does not change names+or recorded child IDs.++## Event accumulation during Stop and rollover++The controller already serves as the generation's event accumulator: its+stable Workflow ID groups Signals, and its queue retains each accepted event+until trigger processing can resume. The [Event Accumulator pattern](https://docs.temporal.io/design-patterns/event-accumulator)+provides useful buffering and deduplication guidance. Inactivity timers and+batch processing would change the existing immediate, per-event trigger+behavior, so neither is introduced. Signal-With-Start is not used to replace+a lost version 1 controller with an empty execution-root registry.++New version 1 histories record `controller-event-accumulator-v1`. This marker+preserves recorded command decisions for legacy and earlier version 1 histories.+Pending event keys are protected independently of the bounded recent-ID cache,+including the event currently being dispatched. Restored overflow pages keep+their original FIFO position when the same pending event was redelivered.+Existing durable overflow storage and read receipts are reused.++Continue-as-new drains the appended queue tail after each spill and checks again+after control handlers finish, so Signals accepted during a spill are included+in the continuation or overflow store. Reset closes the controller and prevents+rollover from resurrecting it. An already-started deterministic child is treated+as a duplicate admission; transient start failures retain the event for retry+instead of blocking all later events permanently on a successful duplicate.++Message producers use their existing stable provider identities: Telegram's+chat/message pair, Discord message or interaction ID, WhatsApp chat/message+and direction, and chat run ID or saved message ID. Payloads without a complete+provider identity retain a fresh ID rather than merging unrelated messages.+Recent completed-ID memory remains bounded; deterministic child IDs and the+existing overflow keys also protect redelivery. This is not an unlimited+generation-wide deduplication ledger.++These queue guarantees begin when Temporal accepts an event Signal. The existing+Visibility-based, best-effort producer dispatch remains unchanged; durable+end-to-end producer delivery would require a separate admission/outbox design.++The published accumulator Python example currently assigns a boolean from+`workflow.wait_condition`. Follow the installed+[Python SDK contract](https://python.temporal.io/temporalio.workflow.html#wait_condition):+it returns normally without a value and raises `asyncio.TimeoutError` on timeout.+Use `try`/`except` if a timeout outcome is needed. This controller uses admission+and rollover conditions rather than the example's inactivity timer.++## Compatibility and rollout++Start writes `execution_control_version=1` atomically with a new generation.+Deploying this code does not upgrade an already-admitted generation or rewrite+its history. Reset followed by Start admits the stronger protocol for that+workflow. There is no migration of an old run's continuation into a new root.++| Gate | Purpose |+|---|---|+| Persisted generation `execution_control_version=1` | Root enrollment, cooperative gates, acknowledged execution-tree drain, held Resume, frozen agent continuation and controlled chat Stop |+| `controller-event-accumulator-v1` | Pending-key protection, duplicate-start handling and final Signal fences during controller rollover |+| `controller-messaging-v1` | Successful Updates request main-loop rollover and Safe Apply producer controls cannot override generation suspension |+| Existing controller durable-queue markers | Preserve the recorded overflow/spill command paths |++New markers are recorded at the relevant runtime command paths, not in+constructors, Queries or validators. Pre-marker and legacy histories retain+their previous paths; native replay tests cover these cases. Update handlers+request rollover, while the main Workflow waits for handlers and performs it.+Controller rollover may occur while paused; Agent rollover parks until running+at a clean boundary without live child handles.++No participant SQL registry, recurring lease heartbeat, separate checkpoint+store or generic tool progress cursor is added. The existing controller history,+native execution descriptions, queue overflow storage and application revisions+provide the necessary control state.++## Verification and operations++The regression suite separates participant behavior, whole-generation control,+event durability, messaging and chat/UI contracts:++| Tests | Guarantees exercised |+|---|---|+| [Participant unit tests](../server/tests/temporal/test_execution_control.py) | A finishes once, B waits; parallel drain; bookkeeping; child-start acknowledgements; delayed registration/revision guards; held release and carry |+| [Agent SDK integration](../server/tests/temporal/test_agent_execution_control_integration.py) | Recorded model calls and pending tools survive Stop, worker restart and two real agent rollovers; replay and complete-input capacity error |+| [Control replay integration](../server/tests/temporal/test_execution_control_replay.py) | Suspended worker restart, actual continuation runs, new and legacy history replay |+| [Generation SDK integration](../server/tests/temporal/test_generation_execution_control_integration.py) | Attached graph/agent children, cron roots and jobs, late enrollment, detached lifetime and controller rollover |+| [Accumulator unit tests](../server/tests/temporal/test_controller_accumulator.py) and [SDK integration](../server/tests/temporal/test_controller_accumulator_integration.py) | Signals during spill and handler fences, pending-key eviction, restored FIFO, duplicate child start, transient retry, Reset, restart and pre-marker replay |+| [Messaging unit tests](../server/tests/temporal/test_controller_messaging.py) and [SDK integration](../server/tests/temporal/test_controller_messaging_integration.py) | Malformed Updates rejected before acceptance without Workflow Task failure, valid subsequent messages, Update-only rollover, Safe Apply guards and replay |+| [Versioned handler tests](../server/tests/services/test_versioned_workflow_control.py) | Timeout keeps intent closed, Resume ordering/deadline metadata, missing controller fail-closed and owning-generation guard |+| [Chat Stop](../server/tests/services/chat/test_stop.py) and [watchdog](../server/tests/services/chat/test_watchdog.py) | Controlled run identity/lane retention, legacy terminal Stop, long-pause protection and fresh resumed age window |++Run from `server/` after the normal dependency setup:++```bash+uv run pytest -q -p no:cacheprovider tests/temporal/test_execution_control.py tests/temporal/test_controller_accumulator.py tests/temporal/test_controller_messaging.py tests/services/test_versioned_workflow_control.py+uv run pytest -q -p no:cacheprovider tests/temporal/test_agent_execution_control_integration.py tests/temporal/test_execution_control_replay.py tests/temporal/test_generation_execution_control_integration.py tests/temporal/test_controller_accumulator_integration.py tests/temporal/test_controller_messaging_integration.py+uv run pytest -q tests/temporal/test_agent_workflow.py tests/temporal/test_controller_queue.py tests/services/chat+```++Native tests launch Temporal's time-skipping test server in isolated processes+and use stub provider/tool Activities. They exercise actual Updates,+Continue-As-New and `Replayer`, without calling live provider APIs. The local+test server does not expose production dynamic Update-limit settings: rollover+tests use deterministic history-pressure hooks held unchanged during replay.+They validate the rollover path, not production load capacity.++Client control/chat regressions and type checking run from the repository root:++```bash+bun run --filter react-flow-client test src/components/ui/__tests__/CommandPaletteHost.workflowControl.test.ts src/contexts/__tests__/webSocketLifecycle.test.tsx src/features/chat/__tests__/chatPane.test.tsx src/features/home/__tests__/presentation.test.ts src/features/home/__tests__/employeeView.test.tsx src/features/home/__tests__/workspace.test.tsx+bun run --filter react-flow-client typecheck+```++When Stop remains `pausing`, inspect the controller participant revision/hold,+live-root membership and each participant's `active_actions` and+`pending_child_starts`. The `Generation admission acknowledged` log measures+time until admission closes; `Generation checkpoint acknowledged` measures+the complete transition. A retrying LLM or whole managed-agent Activity can+legitimately keep Stop in progress. Read status and retry reconciliation with+the returned revision; do not terminate a tool to make the UI report Stopped.++If a transition RPC times out, the durable Update may still complete. Keep the+transitional state and reconcile. If a version 1 controller is lost, its root+registry cannot be reconstructed safely by starting an empty controller; Reset+and Start are required. Explicit continuation capacity errors likewise surface+to the operator rather than silently rebuilding an agent conversation.+## Generation-scoped workflow dataThe editable `workflows` row is the stable canvas definition; it is not runtime@@ -180,10 +521,12 @@ the control plane is hardened against Temporal's per-run event-history ceilingunder history pressure (`is_continue_as_new_suggested()` or a 10K-event softcap), carrying trigger specs, per-trigger provider `seen_ids` (written backinto the carried spec after every poll cycle), queued push events, the- bounded dedup baseline, and the control state + revision. Rollover works- mid-pause; the paused state carries. Because run ids change on rollover,- every control surface addresses the controller **by workflow id only** —- `controller_run_id` is stored for provenance but never pinned on a handle.+ bounded recent dedup baseline, pending keys, durable overflow position,+ control intent/revision/hold, live roots and membership epoch. Rollover works+ mid-pause; the paused state carries. Because Run IDs change on rollover,+ control handles address the current controller **by Workflow ID**, with the+ first execution Run ID checked as the chain identity. The recorded+ `controller_run_id` is not pinned as the current Run ID on a handle.- **Signal narrowing.** The controller upserts the `ControlEventTypes`keyword-list Search Attribute as push triggers register; `dispatch.emit`skips controllers whose deployment has no matching trigger so other@@ -216,11 +559,11 @@ the control plane is hardened against Temporal's per-run event-history ceilingThree env-driven policies (canonical defaults + semantics in`.env.template`; consumed by `services/deployment/handlers.py`) govern how ageneration behaves around kills, crashes, and failures. All three reuse the-cooperative control-plane pause — Temporal's native Pause/Unpause (server-1.28+) is deliberately not used: it is an operational control with no Python-SDK client methods, and it halts workflow-task dispatch entirely, so a-natively-paused controller could not process the `set_control_state` Update-that Resume relies on.+cooperative control-plane pause. The application requires Workflow Tasks to+keep processing events, completed results and control messages while stopped;+it does not use a server-level pause of Workflow Task dispatch for this+handshake. The registered control Updates remain available for reconciliation+and Resume.- **`WORKFLOW_CONTROL_CRASH_RECOVERY`** (`pause` | `resume`, default`pause`): after an UNCLEAN shutdown — kill or crash, detected via a@@ -229,7 +572,7 @@ that Resume relies on.every generation still `running` so the user consciously resumes it. Aclean `company stop` + start always restores deployments as they were.- **`WORKFLOW_CONTROL_MISSING_CONTROLLER`** (`pause` | `fail`, default- `pause`): a live generation whose controller execution vanished+ `pause`): a legacy live generation whose controller execution vanished(terminated in the Temporal UI, killed, retention-deleted) converges to`paused` instead of `failed`. Resume then **rebuilds the controller**:same generation-scoped workflow id started with the documented@@ -240,6 +583,9 @@ that Resume relies on.`starting` rows always fail instead (nothing durable runs yet; Reset +Start rebuilds cleanly). `fail` preserves the legacy Reset-onlybehaviour.+ Version 1 generations always fail closed when their controller is lost.+ Resume cannot rebuild an empty registry and claim verified resumability;+ Reset followed by Start is required.- **`WORKFLOW_CONTROL_PAUSE_ON_FAILURE`** (default `true`): circuitbreaker — when trigger-spawned runs keep failing, MachinaWorkflowschedules `workflow_control.pause_on_failure` and the deployment pauses so the userdiff --git a/docs-internal/tool_building_pipeline.md b/docs-internal/tool_building_pipeline.mdindex d704df8b..b58b98a3 100644--- a/docs-internal/tool_building_pipeline.md+++ b/docs-internal/tool_building_pipeline.md@@ -229,13 +229,15 @@ When `TEMPORAL_PER_TYPE_DISPATCH=true` and the run is happening inside a Tempora1. `agent.prepare_payload` builds the payload; there is one wire standardwith no `llm_engine` / `message_wire_version` discriminators (locked by- `tests/llm/test_single_wire_standard.py`).+ `tests/llm/test_single_wire_standard.py`). New controlled generations also+ record resolved tool bindings and plugin Activity policies here.2. `AgentWorkflow` (the F4.B child workflow) gets the LLM's tool-calls listback from `agent.execute_llm_step`, whose native branch sends `ToolDef`declarations through `ChatUnifier`.3. For each tool call, the workflow schedules`f"node.{node_type}.v{version}"` as a Temporal activity on the plugin's- declared `task_queue`.+ declared `task_queue` when the run's frozen worker-pool setting is enabled;+ otherwise the default worker handles the Activity.4. The per-type activity body in[`server/services/plugin/base.py:as_activity`](../server/services/plugin/base.py)runs the same pipeline `execute_tool` would have — broadcasts, plugin@@ -243,7 +245,39 @@ When `TEMPORAL_PER_TYPE_DISPATCH=true` and the run is happening inside a Tempora5. The result returns to the workflow, is appended to the native messagehistory, and the next LLM step sees it.-This means tool calls inside an agent loop get Temporal's retry / timeout / heartbeat semantics independently of the parent agent's. A `code-exec` task burns its own retries; a `browser` task survives past the parent's `start_to_close_timeout` via its own heartbeat. See [TEMPORAL_ARCHITECTURE.md](./TEMPORAL_ARCHITECTURE.md).+Tool calls inside an agent workflow have their own regular Activity boundary.+For new generations (`execution_control_version=1`), ordinary plugin tools use+the recorded `start_to_close_timeout`, `heartbeat_timeout`, `retry_policy`,+and queue declaration. Heartbeats enforce liveness; they do not extend an+attempt's `start_to_close_timeout` or checkpoint an opaque provider call.+The parent is a workflow, so it has no Activity `start_to_close_timeout`.+Existing histories retain their previous policy decisions. See+[Temporal Architecture](./TEMPORAL_ARCHITECTURE.md) and+[Node creation](./node_creation.md#temporal-execution-and-stopresume-contract).++### Stop at the next pending tool++The workflow closes admission before each model request, tool, compaction,+refresh, or child start. An admitted tool finishes under its recorded retry+policy, and its result/transcript bookkeeping completes even while stopped.+For a recorded response requesting A then B, Stop during A preserves A's+result and B's pending call; Resume executes B once without repeating that+response or A. Already-admitted parallel tools drain concurrently. Permit+waits and child-result waits are excluded from local drain acknowledgement;+the attached child is controlled through the execution tree instead.++Clean Continue-As-New boundaries carry the prepared bindings, transcript,+thinking, iteration, usage, execution identity, and control revision. Stopped+agents park before rollover. If existing compaction cannot bring the complete+continuation input under the limit, the versioned path raises+`AgentContinuationTooLarge` rather than returning to the opening prompt.+Do not implement a second tool pause flag, blanket heartbeat cursor, or+rescheduling loop for already-recorded results. External effects before an+Activity's completion is recorded still depend on the tool's idempotency and+retry contract. Opaque Claude Code, RLM, and Vertex managed agents have a+whole-Activity Stop boundary; independently approved sends and Workspace+tasks retain their own lifecycle. The full topology and messaging protocol+is in [Workflow control](./temporal-workflow-control.md).The v1/v2 wire duality was purged; there is no engine-marker refusal path.diff --git a/server/nodes/README.md b/server/nodes/README.mdindex ab7bc998..41212d4c 100644--- a/server/nodes/README.md+++ b/server/nodes/README.md@@ -7,6 +7,9 @@ at import time; no other code needs to change.Full reference: [docs-internal/plugin_system.md](../../docs-internal/plugin_system.md).+Execution contract: [Workflow control](../../docs-internal/temporal-workflow-control.md)+and [Node creation](../../docs-internal/node_creation.md#temporal-execution-and-stopresume-contract).+---## Five-minute recipe@@ -105,6 +108,77 @@ On server restart this node is:No other edits. Zero frontend changes.+## Activity policy and cooperative Stop++Use the existing node declarations instead of introducing plugin-specific+pause state or a checkpoint store. `task_queue`, `start_to_close_timeout`,+`heartbeat_timeout`, and `retry_policy` live on the node class+([`services/plugin/scaling.py`](../services/plugin/scaling.py)). The queue is+used when the execution's frozen worker-pool setting is enabled; with pools+disabled the default worker receives the operation. The in-process executor+does not enforce Temporal Activity policies.++New deployment generations (`execution_control_version=1`) record resolved+tool bindings and these policies during agent preparation. Ordinary per-type+agent tools use that recorded policy, including attempt limits and+non-retryable error types. Continue-As-New carries the prepared bindings and+policy rather than reloading a pending tool from edited configuration. Old+histories retain their previous dispatch decisions.++Graph node dispatch has a narrower policy contract: plugin retry and queue+declarations apply, but its ordinary Activity starts retain generic 24-hour+attempt and 2-minute heartbeat defaults. The existing Workspace-task override+uses that plugin's timeout/heartbeat declarations. Test the intended execution+path instead of assuming a timeout declaration is honored everywhere.++Stop closes workflow admission before new node/tool/model/poll/compaction work+or a child start. An admitted Activity finishes, including configured retries;+its result and bookkeeping are retained before Stopped is acknowledged.+Resume runs the next pending action in the existing execution. Parallel tools+already admitted drain concurrently. A stopped generation can therefore remain+Stopping while a long operation or unlimited model retry is outstanding.+An opaque CLI or managed-agent invocation has a whole-Activity boundary.++Keep external I/O in regular Activities. Heartbeats support liveness and real+operation-specific retry recovery; they do not implement Stop-after-completion.+Do not add a generic heartbeat cursor to an HTTP call, subprocess, or provider+session that cannot actually resume from that cursor. External side effects+still require the plugin's retry/idempotency contract when completion was not+yet recorded by Temporal. Independent Workspace tasks and separately approved+sends retain their own lifecycle. See [Long-running Activity](https://docs.temporal.io/design-patterns/long-running-activity)+and [Python Activity failure detection](https://docs.temporal.io/develop/python/failure-detection).++## Trigger events and stable IDs++Registered deployed triggers send CloudEvents with `dispatch.emit`.+`WorkflowControlWorkflow` owns controlled deployments' definitions and queues;+separate listener/poller workflows remain compatibility paths. Interactive+canvas waits use the in-memory event waiter and have a+[known producer delivery gap](../../docs-internal/event_waiter_system.md#known-gap-canvas-run-on-canary-push-triggers).++Keep provider event IDs stable across redelivery only when the full identity+is available. Telegram uses chat plus message ID, Discord uses separate message+and interaction namespaces, and WhatsApp uses direction plus conversation plus+message ID. Missing identity leaves the envelope's random fallback in place.+Do not substitute a shared fallback key. Exact formats and producer references+are in [Node creation](../../docs-internal/node_creation.md#event-identity-and-delivery-contract).++The controller borrows queue/deduplication/carry practices from Temporal's+[Event Accumulator](https://docs.temporal.io/design-patterns/event-accumulator)+while processing each admitted event immediately. Pending keys are separate+from the bounded recent-key window; overflow restoration and Continue-As-New+retain queued work, including Signals arriving during spill awaits. Stopped+generations queue accepted events without starting their graph runs. Events+use Signals, acknowledged control uses completed Updates, and status uses+read-only Queries ([Workflow messaging](https://docs.temporal.io/design-patterns/workflow-messaging-patterns)).++Signal acceptance is not a node-completion acknowledgement. The producer's+Visibility lookup and signal fan-out remain best effort, so the durable queue+guarantee starts after Temporal accepts the Signal. Same-label/type trigger+Workflow-ID collisions also remain a known limit; stable provider IDs do not+resolve them. See [Workflow control](../../docs-internal/temporal-workflow-control.md)+for version gates, membership, rollover fences, and operating limits.+---## Folder mapdiff --git a/server/nodes/discord/_events.py b/server/nodes/discord/_events.pyindex 5bd7abbc..51fe2ae4 100644--- a/server/nodes/discord/_events.py+++ b/server/nodes/discord/_events.py@@ -29,7 +29,10 @@ INTERACTION_WIRE_KEY = "discord_interaction_created"def discord_message_received(event_data: Mapping[str, Any]) -> WorkflowEvent:"""Envelope for one inbound Discord message."""+ message_id = event_data.get("message_id")+ fields = {"id": f"discord:message:{message_id}"} if message_id else {}return WorkflowEvent(+ **fields,source=SOURCE,type=MESSAGE_RECEIVED_TYPE,# Snowflakes are stringified upstream, in _dispatch. The envelope@@ -47,7 +50,10 @@ def discord_interaction_created(event_data: Mapping[str, Any]) -> WorkflowEvent:Search Attribute its listener is found by. One type per trigger node isthe only shape the routing supports."""+ interaction_id = event_data.get("interaction_id")+ fields = {"id": f"discord:interaction:{interaction_id}"} if interaction_id else {}return WorkflowEvent(+ **fields,source=SOURCE,type=INTERACTION_CREATED_TYPE,subject=str(event_data.get("interaction_id") or ""),diff --git a/server/nodes/scheduler/cron_scheduler/_workflow.py b/server/nodes/scheduler/cron_scheduler/_workflow.pyindex 6d462371..999984dc 100644--- a/server/nodes/scheduler/cron_scheduler/_workflow.py+++ b/server/nodes/scheduler/cron_scheduler/_workflow.py@@ -34,6 +34,7 @@ from typing import Any, Dictfrom temporalio import workflowfrom temporalio.common import WorkflowIDReusePolicyfrom temporalio.workflow import ParentClosePolicy+from services.temporal.execution_control import ExecutionControl@workflow.defn(name="CronTriggerWorkflow", sandboxed=False)@@ -45,23 +46,66 @@ class CronTriggerWorkflow:everything else is computed from the static action args."""- def __init__(self) -> None:+ @workflow.init+ def __init__(self, listener_data: Dict[str, Any] | None = None) -> None:self._control_paused = False+ self._execution_control = ExecutionControl()+ if listener_data is not None:+ self._execution_control.bind(listener_data)@workflow.signalasync def pause(self) -> None:+ if self._execution_control.enabled:+ returnself._control_paused = True@workflow.signalasync def resume(self) -> None:+ if self._execution_control.enabled:+ returnself._control_paused = Falseasync def _wait_until_resumed(self) -> None:+ if self._execution_control.enabled:+ await self._execution_control.wait_until_running()+ returnif self._control_paused:await workflow.wait_condition(lambda: not self._control_paused)@workflow.run- async def run(self, listener_data: Dict[str, Any]) -> Dict[str, Any]:+ async def run(self, listener_data: Dict[str, Any] | None = None) -> Dict[str, Any]:+ listener_data = listener_data or {}+ self._execution_control.bind(listener_data)+ if not self._execution_control.enabled:+ return await self._run_firing(listener_data)+ await self._execution_control.register_root()+ try:+ return await self._run_firing(listener_data)+ finally:+ await self._execution_control.unregister_root()+ await workflow.wait_condition(workflow.all_handlers_finished)++ @workflow.update+ async def set_control_state(self, payload: Dict[str, Any]) -> Dict[str, Any]:+ return await self._execution_control.set_control_state(payload)++ @set_control_state.validator+ def validate_set_control_state(self, payload: Dict[str, Any]) -> None:+ self._execution_control.validate_control(payload)++ @workflow.update+ async def wait_for_checkpoint(self, payload: Dict[str, Any]) -> Dict[str, Any]:+ return await self._execution_control.wait_for_checkpoint(payload)++ @wait_for_checkpoint.validator+ def validate_wait_for_checkpoint(self, payload: Dict[str, Any]) -> None:+ self._execution_control.validate_control(payload)++ @workflow.query+ def execution_control_status(self) -> Dict[str, Any]:+ return self._execution_control.status()++ async def _run_firing(self, listener_data: Dict[str, Any]) -> Dict[str, Any]:"""Spawn one MachinaWorkflow child per cron firing.``listener_data`` shape (deployment-supplied, frozen at@@ -125,8 +169,6 @@ class CronTriggerWorkflow:from services.temporal.trigger_listener_workflow import (event_workflow_search_attributes,)- from services.temporal.workflow import TEMPORAL_ROUTING_INPUT_KEY-# A spawned run may execute — or stay cooperatively paused — for# months, and Temporal's timeout timer keeps ticking through a# pause, so children start without lifetime caps. Liveness is@@ -143,6 +185,21 @@ class CronTriggerWorkflow:}await self._wait_until_resumed()+ if self._execution_control.enabled:+ await self._execution_control.begin_child_start()+ try:+ await self._start_graph(listener_data, child_options, filtered_nodes, filtered_edges,+ session_id, deployment_workflow_id, workflow_slug, tenant_id)+ finally:+ if self._execution_control.enabled:+ self._execution_control.end_child_start()++ workflow.logger.info(f"CronTriggerWorkflow spawned child run: child_id={child_id} " f"timestamp={firing_iso}")+ return {"spawned_child_id": child_id, "timestamp": firing_iso}++ async def _start_graph(self, listener_data, child_options, filtered_nodes, filtered_edges,+ session_id, deployment_workflow_id, workflow_slug, tenant_id):+ from services.temporal.workflow import TEMPORAL_ROUTING_INPUT_KEYawait workflow.start_child_workflow("MachinaWorkflow",args=[@@ -159,7 +216,8 @@ class CronTriggerWorkflow:# default, silently bypassing per-queue rate limits.**{key: listener_data[key]- for key in ("user_id", "graphVersion", "generation")+ for key in ("user_id", "graphVersion", "generation", "execution_id", "root_execution_id",+ "data_scope_id", "parameter_snapshot", "execution_control_version", "controller_workflow_id")if key in listener_data},**(@@ -176,14 +234,6 @@ class CronTriggerWorkflow:**child_options,)- workflow.logger.info(f"CronTriggerWorkflow spawned child run: child_id={child_id} " f"timestamp={firing_iso}")-- return {- "spawned_child_id": child_id,- "timestamp": firing_iso,- }--def _build_trigger_output(listener_data: Dict[str, Any]) -> Dict[str, Any]:"""Construct the cron trigger's output payload for one firing.diff --git a/server/nodes/telegram/_events.py b/server/nodes/telegram/_events.pyindex 54cea70f..dd989508 100644--- a/server/nodes/telegram/_events.py+++ b/server/nodes/telegram/_events.py@@ -83,7 +83,11 @@ def telegram_message_received(event_data: Mapping[str, Any]) -> WorkflowEvent:requires a string subject)."""payload = dict(event_data)chat_id = payload.get("chat_id")+ message_id = payload.get("message_id")+ # Telegram message IDs are unique within a chat, not across chats.+ fields = {"id": f"telegram:{chat_id}:{message_id}"} if chat_id is not None and message_id is not None else {}return WorkflowEvent(+ **fields,source="opencompany://nodes/telegram",type="com.opencompany.telegram.message.received",subject=str(chat_id) if chat_id is not None else None,diff --git a/server/nodes/whatsapp/_events.py b/server/nodes/whatsapp/_events.pyindex 0cd0305a..424cee55 100644--- a/server/nodes/whatsapp/_events.py+++ b/server/nodes/whatsapp/_events.py@@ -92,7 +92,10 @@ def whatsapp_message_event(from the params payload (chat_id / sender / from)."""payload = dict(params)subject = payload.get("chat_id") or payload.get("sender") or payload.get("from")+ message_id = payload.get("message_id")+ fields = {"id": f"whatsapp:{direction}:{subject}:{message_id}"} if subject and message_id else {}return WorkflowEvent(+ **fields,source="opencompany://nodes/whatsapp",type=f"com.opencompany.whatsapp.message.{direction}",subject=str(subject) if subject else None,diff --git a/server/services/chat/handlers.py b/server/services/chat/handlers.pyindex caca40a4..f06aa0c3 100644--- a/server/services/chat/handlers.py+++ b/server/services/chat/handlers.py@@ -47,7 +47,7 @@ from services.ws_handler_registry import ws_handlerlogger = get_logger(__name__)-PROTOCOL_VERSION = 1+PROTOCOL_VERSION = 2_DENIED = {"success": False, "error": "access_denied"}#: A workflow's chat messages travel through Temporal (``dispatch.emit``#: signals its listeners) and only a Temporal activity claims their runs, so@@ -133,7 +133,17 @@ async def run_snapshots(database: Any, session_id: str) -> List[Dict[str, Any]]:hub = get_chat_hub()live = hub.session_live(session_id)rows = await ledger.live_runs(database, session_id)- return [{**reducer.merge_snapshot(reducer.snapshot_from_row(run), live.get(run.run_id)), "hub_epoch": hub.epoch} for run in rows]+ return [await _run_snapshot(database, run, live.get(run.run_id), hub.epoch) for run in rows]+++async def _run_snapshot(database: Any, run: Any, live: Any, hub_epoch: str) -> Dict[str, Any]:+ snapshot = {**reducer.merge_snapshot(reducer.snapshot_from_row(run), live), "hub_epoch": hub_epoch}+ control = await ledger.owning_control(database, run)+ if ledger.controlled_generation(control):+ from services.deployment.control import serialize_control++ snapshot["workflow_control"] = serialize_control(control)+ return snapshotasync def _thread_state(database: Any, session_id: str) -> Dict[str, Any]:@@ -207,13 +217,14 @@ async def _dispatch() -> None:"""Send the owner's message to the workflow's chat triggers: the run itstarts is the event's id, so the trigger's child workflow is the run's.+ Without a tracked run, the saved message supplies a stable event identity.Its attachments ride along (the trigger's output carries them)."""event_data: Dict[str, Any] = {"message": prompt, "timestamp": timestamp, "session_id": session_id, "message_id": message_uid}if run_id is not None:event_data["run_id"] = run_idif attachments:event_data["attachments"] = list(attachments)- await dispatch_chat_message_received(event_data, workflow_id=workflow_id, event_id=run_id)+ await dispatch_chat_message_received(event_data, workflow_id=workflow_id, event_id=run_id or message_uid)logger.info("Chat message dispatched", session_id=session_id, run_id=run_id)@@ -436,16 +447,12 @@ async def handle_get_chat_run(data: Dict[str, Any], websocket: WebSocket) -> Dichub = get_chat_hub()live = hub.live_snapshot(run.run_id)run = await ledger.get_run(database, run.run_id) or run- snapshot = reducer.merge_snapshot(reducer.snapshot_from_row(run), live)- return {"success": True, "run": {**snapshot, "hub_epoch": hub.epoch}}+ return {"success": True, "run": await _run_snapshot(database, run, live, hub.epoch)}@ws_handler("run_id")async def handle_stop_chat_run(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:- """Stop a live run (``ledger.request_stop``). One nothing picked up yet- ends ``stopped`` at once; a running one stops itself at its next step,- keeping its reply so far. Answers the run's state: ``stopping`` or- ``stopped``, ``not_stoppable`` for a run that ended otherwise."""+ """Suspend a controlled run's generation; retain legacy terminal Stop."""database = container.database()run = await ledger.get_run(database, str(data["run_id"]))if run is None:@@ -454,6 +461,24 @@ async def handle_stop_chat_run(data: Dict[str, Any], websocket: WebSocket) -> Diawait authorize_session(database, websocket, run.session_id)except ChatAccessDenied:return dict(_DENIED)+ control = await ledger.owning_control(database, run)+ if ledger.controlled_generation(control):+ if run.state not in ledger.LIVE_STATES:+ return {"success": False, "error": "not_stoppable", "run_id": run.run_id, "state": run.state}+ if not isinstance(data.get("idempotency_key"), str) or not data["idempotency_key"].strip():+ return {"success": False, "error": "idempotency_key_required"}+ from services.deployment.handlers import handle_pause_workflow++ result = await handle_pause_workflow(+ {+ "workflow_id": control.workflow_id,+ "expected_root_execution_id": run.run_key,+ "expected_revision": data.get("expected_revision"),+ "idempotency_key": data.get("idempotency_key"),+ },+ websocket,+ )+ return {**result, "run_id": run.run_id, "resumable": True}run = await ledger.request_stop(database, run.run_id) or runif run.state not in ("stopping", "stopped"):return {"success": False, "error": "not_stoppable", "run_id": run.run_id, "state": run.state}diff --git a/server/services/chat/ledger.py b/server/services/chat/ledger.pyindex 0981b276..8c4c1b38 100644--- a/server/services/chat/ledger.py+++ b/server/services/chat/ledger.py@@ -26,7 +26,11 @@ running run past ``runs.max_running_s`` (``timed_out``), and a running runwhose Temporal workflow closed without finishing it: finished when its replywas saved, ``interrupted`` otherwise.-**Stop** (``request_stop``) ends a run nothing has picked up at once+**Controlled Stop** suspends the owning generation while keeping the run+and its lane alive. The watchdog excludes intentionally suspended generations+and uses their successful Resume time for a fresh timeout window.++**Legacy Stop** (``request_stop``) ends a run nothing has picked up at once(``stopped``, no reply) and moves a running run to ``stopping``. A runningrun stops itself: its agent's next model step returns what it has written sofar, and a tool not started yet is skipped (``services/chat/stream.py``,@@ -60,7 +64,7 @@ from sqlmodel import selectfrom core.logging import get_loggerfrom models.chat import LIVE_STATES, TERMINAL_STATES, ChatRun-from models.database import ChatMessage+from models.database import ChatMessage, WorkflowControlExecutionfrom services.chat.config import runs_setting, steps_settingfrom services.chat.hub import publish_run_eventfrom services.chat_thread import delivery_for@@ -139,6 +143,35 @@ async def get_run(database: Any, run_id: str) -> Optional[ChatRun]:return await session.get(ChatRun, run_id)+async def owning_control(database: Any, run: ChatRun) -> Optional[WorkflowControlExecution]:+ """Resolve the immutable generation that admitted this run, not its successor."""+ if not run.workflow_id or not run.run_key:+ return None+ async with database.get_session() as session:+ result = await session.execute(+ select(WorkflowControlExecution).where(+ WorkflowControlExecution.workflow_id == run.workflow_id,+ WorkflowControlExecution.root_execution_id == run.run_key,+ ).limit(1)+ )+ return result.scalar_one_or_none()+++def controlled_generation(control: Any) -> bool:+ return bool(control and (control.resource_manifest or {}).get("execution_control_version") == 1)+++def resumed_at(control: Any) -> Optional[datetime]:+ """A successful Resume grants a fresh watchdog timeout window."""+ raw = (getattr(control, "resource_manifest", None) or {}).get("last_resumed_at")+ if not isinstance(raw, str):+ return None+ try:+ return _aware(datetime.fromisoformat(raw.replace("Z", "+00:00")))+ except ValueError:+ return None++async def lane_run(database: Any, session_id: str) -> Optional[ChatRun]:"""The run holding the session's lane, if any."""async with database.get_session() as session:@@ -704,13 +737,20 @@ async def _sweep_one(longest: timedelta,temporal_status: Optional[TemporalStatus],) -> Optional[ChatRun]:- created = max(_aware(run.created_at) or now, process_started)+ control = await owning_control(database, run)+ controlled = controlled_generation(control)+ if controlled and control.status in {"pausing", "paused", "resuming"}:+ return None+ resumed = resumed_at(control) if controlled else None+ created = max(_aware(run.created_at) or now, process_started, resumed or process_started)if run.state == "pending":if now - created > pickup:return await fail_run(database, run, code="not_delivered", message="The employee did not pick up this message.")return Noneif run.state == "queued":- control = await database.get_latest_workflow_control(run.workflow_id) if run.workflow_id else None+ # Legacy runs still use the historical deployment-delivery policy.+ if not controlled:+ control = await database.get_latest_workflow_control(run.workflow_id) if run.workflow_id else Nonedelivery = delivery_for(control)if delivery == "queued":return None@@ -721,6 +761,8 @@ async def _sweep_one(return Nonereturn await fail_run(database, run, code="not_delivered", message="The employee stopped before reading this message.")started = _aware(run.started_at) or created+ if resumed is not None:+ started = max(started, resumed)if now - started > longest:return await fail_run(database, run, code="timed_out", message="The employee took too long to answer.")if temporal_status is None or not run.temporal_workflow_id or now - started < _CLOSED_GRACE:diff --git a/server/services/chat/watchdog.py b/server/services/chat/watchdog.pyindex 8cb8028d..fce9313f 100644--- a/server/services/chat/watchdog.py+++ b/server/services/chat/watchdog.py@@ -36,7 +36,23 @@ async def temporal_workflow_closed(workflow_id: str, run_id: Optional[str]) -> Ofrom temporalio.service import RPCError, RPCStatusCodetry:- description = await client.get_workflow_handle(workflow_id, run_id=run_id).describe()+ description = await client.get_workflow_handle(workflow_id).describe()+ # Continue-as-new closes a run, not the execution chain. Compare the+ # claimant's chain when it differs so a reused ID cannot keep a chat alive.+ current_run = getattr(description, "run_id", None)+ if run_id and current_run and current_run != run_id:+ current_first = getattr(getattr(description, "raw_info", None), "first_run_id", None)+ if current_first != run_id:+ try:+ original = await client.get_workflow_handle(workflow_id, run_id=run_id).describe()+ except RPCError:+ # The older run may have aged out while its chain lives.+ return None+ original_first = getattr(getattr(original, "raw_info", None), "first_run_id", None)+ if not current_first or not original_first:+ return None+ if current_first != original_first:+ return Trueexcept RPCError as exc:return True if exc.status == RPCStatusCode.NOT_FOUND else Noneexcept Exception: # noqa: BLE001 - unknown is not closeddiff --git a/server/services/deployment/control.py b/server/services/deployment/control.pyindex 4943c2c9..a9580233 100644--- a/server/services/deployment/control.py+++ b/server/services/deployment/control.py@@ -42,6 +42,8 @@ def serialize_control(control: Optional[WorkflowControlExecution]) -> Dict[str,"can_edit": True,}state = "ready" if control.status == "reset" else control.status+ manifest = getattr(control, "resource_manifest", None)+ manifest = manifest if isinstance(manifest, dict) else {}return {"workflow_id": control.workflow_id,"generation": control.generation,@@ -50,6 +52,8 @@ def serialize_control(control: Optional[WorkflowControlExecution]) -> Dict[str,"data_scope_id": control.data_scope_id or control.execution_id,"controller_workflow_id": control.controller_workflow_id,"controller_run_id": control.controller_run_id,+ "execution_control_version": int(manifest.get("execution_control_version", 0)),+ "last_resumed_at": manifest.get("last_resumed_at"),"state": state,"revision": control.revision,"can_start": state == "ready",@@ -150,6 +154,7 @@ class WorkflowControlService:"workflow_slug": slug if isinstance(slug, str) and slug else None,},idempotency_key=idempotency_key,+ resource_manifest={"execution_control_version": 1},# Revisions are monotonic across generations so a delayed request# from an archived generation cannot pass CAS against a newer one# that happens to be at the same lifecycle step.diff --git a/server/services/deployment/execution_control.py b/server/services/deployment/execution_control.pynew file mode 100644index 00000000..f2dca0d4--- /dev/null+++ b/server/services/deployment/execution_control.py@@ -0,0 +1,174 @@+"""Acknowledged generation control using native Temporal execution trees.++The controller records only roots which can outlive a parent. Descriptions+discover attached children after each workflow has fenced child starts.+No visibility scan is used as evidence that a generation is quiescent.+"""+from __future__ import annotations++import asyncio+import time+from typing import Any++from temporalio.client import WorkflowExecutionStatus+from temporalio.service import RPCError, RPCStatusCode++from core.logging import get_logger++logger = get_logger(__name__)+++async def count_generation_actions(client: Any, controller_id: str, status: dict) -> int:+ """A read-only UI count; transition verification uses acknowledgements."""+ queue = [controller_id, *(status.get("live_roots") or {})]+ seen: set[str] = set()+ count = 0+ while queue:+ workflow_id = queue.pop()+ if workflow_id in seen:+ continue+ seen.add(workflow_id)+ handle = client.get_workflow_handle(workflow_id)+ desc = await handle.describe()+ if desc.status != WorkflowExecutionStatus.RUNNING:+ continue+ participant = await handle.query("execution_control_status")+ count += int(participant.get("active_actions", 0))+ queue.extend(child.workflow_id for child in desc.raw_description.pending_children)+ return count+++async def transition_generation(client: Any, control: Any, *, paused: bool, on_admission: Any = None) -> dict:+ controller = client.get_workflow_handle(control.controller_workflow_id)+ revision = int(control.revision)+ base = {"generation": int(control.generation), "revision": revision,+ "state": "paused" if paused else "running"}+ started = time.monotonic()++ async def describe(workflow_id: str, chain: str) -> Any:+ handle = client.get_workflow_handle(workflow_id)+ try:+ desc = await handle.describe()+ except RPCError as exc:+ if exc.status == RPCStatusCode.NOT_FOUND:+ return None+ raise+ actual = desc.raw_description.workflow_execution_info.first_run_id+ if chain and actual != chain:+ # Workflow ID reuse must never let an old root control a new chain.+ return None+ if desc.status != WorkflowExecutionStatus.RUNNING:+ return None+ return desc++ controller_description = await describe(control.controller_workflow_id, control.controller_run_id or "")+ if controller_description is None:+ raise RuntimeError("controller_execution_missing")+ controller_chain = controller_description.raw_description.workflow_execution_info.first_run_id+ controller_payload = {**base, "first_execution_run_id": controller_chain, "producers_held": True}+ status = await controller.execute_update("set_control_state", controller_payload,+ id=f"control:{revision}:admission")+ if status.get("participant_revision", status.get("revision")) != revision:+ raise RuntimeError("control_revision_conflict")++ targets: dict[str, str] = {}+ order: list[str] = []+ removed: set[tuple[str, str]] = set()+ while True:+ before = await controller.query("status")+ epoch = before["membership_epoch"]+ roots = before.get("live_roots", {})+ queue = [(control.controller_workflow_id, controller_chain)]+ queue.extend((r["workflow_id"], r["first_execution_run_id"]) for r in roots.values())+ visited: set[str] = set()+ # First fence every reachable workflow. Never await its activity drain+ # here: an unfenced sibling could otherwise continue admitting work.+ while queue:+ workflow_id, chain = queue.pop()+ if workflow_id in visited:+ continue+ visited.add(workflow_id)+ desc = await describe(workflow_id, chain)+ if desc is None:+ targets.pop(workflow_id, None)+ if workflow_id in roots and (workflow_id, chain) not in removed:+ await controller.execute_update("unregister_execution", {+ "workflow_id": workflow_id, "first_execution_run_id": chain,+ "generation": control.generation,+ })+ removed.add((workflow_id, chain))+ continue+ current_chain = desc.raw_description.workflow_execution_info.first_run_id+ handle = client.get_workflow_handle(workflow_id)+ if workflow_id != control.controller_workflow_id:+ payload = {**base, "first_execution_run_id": current_chain, "producers_held": True}+ try:+ ack = await handle.execute_update("set_control_state", payload,+ id=f"control:{revision}:admission")+ except RPCError as exc:+ if exc.status == RPCStatusCode.NOT_FOUND:+ continue+ raise+ if ack.get("revision") != revision or ack.get("state") != base["state"]:+ raise RuntimeError("temporal_control_ack_mismatch")+ targets[workflow_id] = current_chain+ if workflow_id not in order:+ order.append(workflow_id)+ # Setter acknowledges admitted child starts before this second+ # describe, so pending_children is a complete direct-child fence.+ desc = await describe(workflow_id, current_chain)+ if desc is not None:+ queue.extend((child.workflow_id, child.run_id)+ for child in desc.raw_description.pending_children)++ logger.info("Generation admission acknowledged", workflow_id=control.workflow_id,+ revision=revision, latency_seconds=time.monotonic() - started,+ controlled_executions=len(targets))+ if on_admission is not None:+ await on_admission(await controller.query("status"))+ if paused:+ async def checkpoint(workflow_id: str, chain: str) -> None:+ if await describe(workflow_id, chain) is None:+ return+ try:+ result = await client.get_workflow_handle(workflow_id).execute_update(+ "wait_for_checkpoint", {**base, "first_execution_run_id": chain},+ id=f"control:{revision}:checkpoint")+ except RPCError as exc:+ if exc.status == RPCStatusCode.NOT_FOUND:+ return+ raise+ if (not result.get("checkpoint") or result.get("state") != base["state"]+ or result.get("revision", result.get("participant_revision")) != revision):+ raise RuntimeError("temporal_checkpoint_ack_mismatch")+ await asyncio.gather(*(checkpoint(wid, chain) for wid, chain in targets.items()))+ after = await controller.query("status")+ if after["membership_epoch"] != epoch:+ continue+ if not paused:+ for workflow_id in reversed(order):+ if workflow_id == control.controller_workflow_id:+ continue+ chain = targets.get(workflow_id)+ if chain and await describe(workflow_id, chain) is not None:+ ack = await client.get_workflow_handle(workflow_id).execute_update(+ "set_control_state", {**base, "first_execution_run_id": chain, "producers_held": False},+ id=f"control:{revision}:release")+ if ack.get("revision") != revision or ack.get("state") != "running" or ack.get("producers_held"):+ raise RuntimeError("temporal_control_ack_mismatch")+ status = await controller.execute_update("set_control_state", {+ **controller_payload, "producers_held": False,+ "expected_membership_epoch": epoch,+ })+ if status.get("participant_revision") != revision or status.get("participant_state") != "running":+ raise RuntimeError("control_revision_conflict")+ if status.get("producers_held"):+ continue+ else:+ status = after+ if status["membership_epoch"] == epoch:+ break+ logger.info("Generation checkpoint acknowledged", workflow_id=control.workflow_id,+ revision=revision, latency_seconds=time.monotonic() - started,+ controlled_executions=len(targets))+ return {"controller_status": status, "controlled_executions": len(targets)}diff --git a/server/services/deployment/handlers.py b/server/services/deployment/handlers.pyindex 71a900b1..b3e76268 100644--- a/server/services/deployment/handlers.py+++ b/server/services/deployment/handlers.py@@ -513,6 +513,17 @@ def _control_service():return WorkflowControlService(container.database())+def _versioned_control(control) -> bool:+ return int((getattr(control, "resource_manifest", None) or {}).get("execution_control_version", 0)) >= 1+++def _control_scope(control) -> Dict[str, Any]:+ return {+ "execution_control_version": 1 if _versioned_control(control) else 0,+ "controller_workflow_id": control.controller_workflow_id,+ }++async def _start_controller(control, *, use_existing: bool = False) -> Optional[str]:"""Start the durable controller, or use local mode when Temporal is disabled.@@ -548,6 +559,8 @@ async def _start_controller(control, *, use_existing: bool = False) -> Optional["root_execution_id": control.root_execution_id,"data_scope_id": control.data_scope_id or control.execution_id,"state": "running",+ "revision": control.revision,+ **_control_scope(control),}],id=control.controller_workflow_id,task_queue=container.settings().temporal_task_queue,@@ -687,7 +700,12 @@ async def _fail_missing_controller(service: WorkflowControlService, control):"""if control.status not in _MISSING_CONTROLLER_FAILS:return control+ if _versioned_control(control):+ _close_local_admission(control.workflow_id)+ await _set_cron_pause(control.workflow_id, paused=True, strict=True)if (+ not _versioned_control(control)+ andcontrol.status in _MISSING_CONTROLLER_PAUSESand _missing_controller_policy() == "pause"):@@ -996,11 +1014,12 @@ async def _with_runtime_counts(payload: Dict[str, Any], workflow_id: str) -> Dicstatus = container.workflow_service().get_deployment_status(workflow_id)workspace = await _workspace_runtime_status(workflow_id)+ active = payload.get("generation_in_flight_count", status.get("active_runs", 0))return {**payload,**{key: value for key, value in workspace.items() if key != "workspace_active_count"},- "active_count": status.get("active_runs", 0) + workspace.get("workspace_active_count", 0),- "in_flight_count": status.get("active_runs", 0) + workspace.get("workspace_active_count", 0),+ "active_count": active + workspace.get("workspace_active_count", 0),+ "in_flight_count": active + workspace.get("workspace_active_count", 0),"queued_count": (int(payload.get("queued_count", 0) or 0)+ int(status.get("queued_events", 0) or 0)@@ -1073,6 +1092,17 @@ async def _control_payload("queued_count": controller_status.get("queued_events", 0),"temporal_available": True,})+ if _versioned_control(control):+ from core.container import container+ from services.deployment.execution_control import count_generation_actions+ wrapper = container.temporal_client()+ if wrapper is not None and wrapper.client is not None:+ try:+ payload["generation_in_flight_count"] = await count_generation_actions(+ wrapper.client, control.controller_workflow_id, controller_status)+ except Exception as exc:+ logger.debug("Generation activity count unavailable", workflow_id=control.workflow_id,+ error=str(exc))payload.update(extra or {})return await _with_runtime_counts(payload, control.workflow_id)@@ -1124,6 +1154,8 @@ async def _reconcile_control(service: WorkflowControlService, control):}.get(control.status)if transition_target is None:return control, controller_status+ if _versioned_control(control):+ return await _finish_versioned_transition(service, control)requested_state, stable_state = transition_targetif controller_status is None:@@ -1206,6 +1238,65 @@ async def _reconcile_control(service: WorkflowControlService, control):return control, controller_status+async def _finish_versioned_transition(service: WorkflowControlService, control):+ """Finish durable intent without cancelling admitted work on RPC timeout."""+ from core.container import container+ from services.deployment.execution_control import transition_generation++ paused = control.status == "pausing"+ runtime = container.workflow_service()+ runtime.pause_deployment(control.workflow_id)+ if paused:+ schedules = await _set_cron_pause(control.workflow_id, paused=True, strict=True)+ triggers = await runtime.update_trigger_pause_status(control.workflow_id, paused=True)+ wrapper = container.temporal_client()+ if wrapper is None or wrapper.client is None:+ if container.settings().temporal_enabled:+ raise TemporalControlUnavailable("temporal_control_unavailable")+ result = {"controller_status": None, "controlled_executions": 0}+ else:+ async def admitted(status):+ try:+ await _broadcast_control(control, controller_status=status)+ except Exception as exc:+ logger.warning("Control admission status broadcast failed", workflow_id=control.workflow_id,+ error=str(exc))+ result = await transition_generation(wrapper.client, control, paused=paused, on_admission=admitted)+ details = {"controlled_executions": result["controlled_executions"]}+ values: Dict[str, Any] = {}+ if paused:+ details.update(paused_schedules=schedules, paused_triggers=triggers)+ else:+ # The existing execution continuations are released before producers.+ details["resumed_schedules"] = await _set_cron_pause(control.workflow_id, paused=False, strict=True)+ details["resumed_queued_events"] = await runtime.resume_deployment(control.workflow_id)+ details["resumed_triggers"] = await runtime.update_trigger_pause_status(control.workflow_id, paused=False)+ values.update(CLEAR_PAUSE_REASON)+ values["resource_manifest"] = {**(control.resource_manifest or {}),+ "last_resumed_at": datetime.now(timezone.utc).isoformat()}+ try:+ control = await service.transition(control, expected_revision=control.revision,+ from_statuses={control.status}, status="paused" if paused else "running",+ values=values or None)+ except ValueError as exc:+ if str(exc) != "control_revision_conflict":+ raise+ latest = await service.database.get_latest_workflow_control(control.workflow_id)+ if latest is None or latest.generation != control.generation:+ raise+ return latest, result["controller_status"]+ if not paused:+ await _clear_failure_streak(service.database, control)+ payload = await _broadcast_control(control, controller_status=result["controller_status"], extra=details)+ from services.status_broadcaster import get_status_broadcaster+ broadcaster = get_status_broadcaster()+ await broadcaster.update_workflow_status(executing=bool(not paused and payload.get("in_flight_count")),+ current_node=None, progress=0, workflow_id=control.workflow_id)+ await broadcaster.update_deployment_status(is_running=True, status=control.status,+ active_runs=payload.get("generation_in_flight_count", 0), workflow_id=control.workflow_id)+ return control, result["controller_status"]++# ---------------------------------------------------------------------------# Boot-time reconcile — called once from services.temporal.lifecycle after# the workers start. Request-time reconciliation stays lazy; this pass closes@@ -1473,6 +1564,7 @@ async def _rearm_generation(control) -> None:"edges": edges,"parameters_by_id": snapshot.get("parameters") or {},"generation": control.generation,+ **_control_scope(control),# Snapshots created before the Context topology deliberately retain version 0,# so a process restart cannot mutate their Temporal command sequence."graphVersion": int(@@ -1579,6 +1671,9 @@ async def _rebuild_missing_controller(service: WorkflowControlService, control):"""from core.container import container+ if _versioned_control(control):+ raise ControllerExecutionMissing("Versioned execution registry cannot be reconstructed; Reset is required")+logger.warning("Rebuilding missing workflow controller",workflow_id=control.workflow_id,@@ -1767,8 +1862,8 @@ async def handle_start_workflow(data: Dict[str, Any], websocket: WebSocket) -> Dcontrol,controller_status=controller_status,)- await _broadcast_control(control)try:+ await _broadcast_control(control)run_id = await _start_controller(control)if run_id:control = await service.transition(@@ -1789,6 +1884,7 @@ async def handle_start_workflow(data: Dict[str, Any], websocket: WebSocket) -> D"parameters_by_id": normalization.node_parameters,"graphVersion": normalization.graph_version,"generation": control.generation,+ **_control_scope(control),"session_id": control.data_scope_id or control.execution_id,"execution_id": control.execution_id,"root_execution_id": control.root_execution_id,@@ -1919,6 +2015,8 @@ async def handle_pause_workflow(data: Dict[str, Any], websocket: WebSocket) -> Dcontrol = await service.database.get_latest_workflow_control(workflow_id)if control is None:return {"success": False, "error": "workflow_never_started"}+ if data.get("expected_root_execution_id") and data["expected_root_execution_id"] != control.root_execution_id:+ raise ValueError("control_generation_conflict")control, controller_status = await _reconcile_control(service, control)if control.status == "paused":return {@@ -1945,6 +2043,9 @@ async def handle_pause_workflow(data: Dict[str, Any], websocket: WebSocket) -> D)await _broadcast_control(control)container.workflow_service().pause_deployment(workflow_id)+ if _versioned_control(control):+ control, controller_status = await _finish_versioned_transition(service, control)+ return {"success": True, **await _control_payload(control, controller_status=controller_status)}try:controller_status = await _update_controller_state(control,@@ -2010,6 +2111,8 @@ async def handle_resume_workflow(data: Dict[str, Any], websocket: WebSocket) ->control = await service.database.get_latest_workflow_control(workflow_id)if control is None:return {"success": False, "error": "workflow_never_started"}+ if data.get("expected_root_execution_id") and data["expected_root_execution_id"] != control.root_execution_id:+ raise ValueError("control_generation_conflict")control, controller_status = await _reconcile_control(service, control)if control.status == "running":return {@@ -2027,6 +2130,9 @@ async def handle_resume_workflow(data: Dict[str, Any], websocket: WebSocket) ->control, expected_revision=_expected_revision(data, control), from_statuses={"paused"}, status="resuming")await _broadcast_control(control)+ if _versioned_control(control):+ control, controller_status = await _finish_versioned_transition(service, control)+ return {"success": True, **await _control_payload(control, controller_status=controller_status)}resume_update_key = data.get("idempotency_key") or uuid.uuid4().hexasync def _apply_resume_update(target):diff --git a/server/services/deployment/manager.py b/server/services/deployment/manager.pyindex 4e92f659..09031641 100644--- a/server/services/deployment/manager.py+++ b/server/services/deployment/manager.py@@ -906,6 +906,8 @@ class DeploymentManager:"root_execution_id": control.root_execution_id,"data_scope_id": control.data_scope_id or control.execution_id,"generation": control.generation,+ "execution_control_version": int((control.resource_manifest or {}).get("execution_control_version", 0)),+ "controller_workflow_id": control.controller_workflow_id,})# The controller owns listeners as child workflows. This keeps# trigger activity in the deployment's Temporal execution tree@@ -1058,6 +1060,17 @@ class DeploymentManager:**capture_temporal_routing_input(),}+ control_lookup = self.database.get_latest_workflow_control(workflow_id)+ control = await control_lookup if inspect.isawaitable(control_lookup) else None+ if control is not None and control.generation == state.generation:+ listener_data.update({+ "graphVersion": state.graph_version, "generation": control.generation,+ "execution_id": control.execution_id, "root_execution_id": control.root_execution_id,+ "data_scope_id": control.data_scope_id or control.execution_id,+ "execution_control_version": int((control.resource_manifest or {}).get("execution_control_version", 0)),+ "controller_workflow_id": control.controller_workflow_id,+ })+schedule_id = await create_cron_schedule(wrapper.client,workflow_id=workflow_id,diff --git a/server/services/employees/jobs.py b/server/services/employees/jobs.pyindex 6ffa6e33..58d6b828 100644--- a/server/services/employees/jobs.py+++ b/server/services/employees/jobs.py@@ -62,6 +62,7 @@ async def create_job(database: Any, ctx: Any, *, mission: str, lead_node_id: str"parameters": parameters, "outputs": ctx.outputs, "user_id": employee.owner_id,"session_id": ctx.session_id, "generation": ctx.raw.get("generation", 0),"data_scope_id": ctx.raw.get("data_scope_id"), "graphVersion": (graph.data or {}).get("graphVersion", 2), "team_plan": plan}+ source.update({key: ctx.raw[key] for key in ("execution_control_version", "controller_workflow_id") if key in ctx.raw})async with database.reserved_session() as session:row = await session.get(EmployeeJob, identifier)if row is not None:@@ -119,6 +120,7 @@ async def dispatch_job(database: Any, job_id: str) -> bool:"generation": job.source.get("generation", 0), "graphVersion": job.source.get("graphVersion", 2),"data_scope_id": job.source.get("data_scope_id"), "employee_job_id": job.id,"parameter_snapshot": job.source["parameters"]}+ context.update({key: job.source[key] for key in ("execution_control_version", "controller_workflow_id") if key in job.source})try:await container.temporal_client().client.start_workflow("EmployeeJobWorkflow", context,id=job.id, task_queue=container.settings().temporal_task_queue,diff --git a/server/services/temporal/agent_activities.py b/server/services/temporal/agent_activities.pyindex 9ccc31df..ae53b69a 100644--- a/server/services/temporal/agent_activities.py+++ b/server/services/temporal/agent_activities.py@@ -63,6 +63,27 @@ logger = logging.getLogger(__name__)# the rollover path._SEED_TRANSCRIPT_MAX_BYTES = 1_000_000++def _tool_activity_policy(cls: Any) -> Dict[str, Any]:+ """Freeze declaration values in history rather than consult code on replay."""+ from services.plugin.base import BaseNode++ cls = cls or BaseNode+ retry = cls.retry_policy+ heartbeat = cls.heartbeat_timeout+ return {+ "start_to_close_seconds": cls.start_to_close_timeout.total_seconds(),+ "heartbeat_seconds": heartbeat.total_seconds() if heartbeat else None,+ "task_queue": cls.task_queue,+ "retry_policy": {+ "initial_interval_seconds": retry.initial_interval.total_seconds(),+ "backoff_coefficient": retry.backoff_coefficient,+ "maximum_interval_seconds": retry.maximum_interval.total_seconds(),+ "maximum_attempts": retry.maximum_attempts,+ "non_retryable_error_types": list(retry.non_retryable_error_types),+ },+ }+# Activity result shapes — keep these in sync with AgentWorkflow's# expectations. Pydantic was considered but plain dicts keep the# payload-serialisation cost flat (Temporal serialises via JSON anyway)@@ -1434,6 +1455,7 @@ async def prepare_agent_payload(context: Dict[str, Any]) -> Dict[str, Any]:# through the workflow verbatim so ``execute_llm_step`` can# rebuild the real StructuredTool inside the activity."tool_info": tool_info,+ **({"activity_policy": _tool_activity_policy(cls), "needs_canvas": bool(getattr(cls, "needs_canvas", False))} if context.get("execution_control_version") == 1 else {}),# Team leads create and dispatch durable work through Task# Manager. Delegate descriptors stay in workflow state for# trusted assignee resolution, but are not callable directly@@ -1736,6 +1758,7 @@ async def refresh_agent_tools(payload: Dict[str, Any]) -> Dict[str, Any]:"tool_node_id": tool_info["node_id"],"parameters": tool_info["parameters"],"tool_info": tool_info,+ **({"activity_policy": _tool_activity_policy(cls), "needs_canvas": bool(getattr(cls, "needs_canvas", False))} if payload.get("execution_control_version") == 1 else {}),"llm_hidden": bool(team_lead_refresh and is_agent_delegate),})diff --git a/server/services/temporal/agent_workflow.py b/server/services/temporal/agent_workflow.pyindex 876a45bd..b1cd746e 100644--- a/server/services/temporal/agent_workflow.py+++ b/server/services/temporal/agent_workflow.py@@ -75,6 +75,7 @@ from .agent_context_pressure import (turn_tool_chars,)from .workflow import AGENT_WORKFLOW_TYPES+from .execution_control import ExecutionControl# Activity timeouts. LLM step can stream for several minutes on@@ -98,8 +99,9 @@ async def _execute_plugin_tool_activity(activity_id: str,tool_payload: Dict[str, Any],context: Dict[str, Any],+ tool_info: Optional[Dict[str, Any]] = None,) -> Any:- """Preserve legacy commands; honor embedded task engines' execution policy.+ """Preserve legacy commands; use frozen plugin policy for new generations.These plugins can perform irreversible device actions. Temporal's implicitretry defaults must not override their explicitly declared attempt limit.@@ -109,20 +111,22 @@ async def _execute_plugin_tool_activity("start_to_close_timeout": TOOL_STEP_TIMEOUT,"heartbeat_timeout": TOOL_HEARTBEAT_TIMEOUT,}- cls = get_node_class(tool_payload["node_type"])- if getattr(cls, "workspace_task", False) and workflow.patched("workspace-task-activity-policy-v1"):- options.update(- start_to_close_timeout=cls.start_to_close_timeout,- heartbeat_timeout=cls.heartbeat_timeout,- retry_policy=cls.retry_policy.to_temporal(),- cancellation_type=ActivityCancellationType.WAIT_CANCELLATION_COMPLETED,- )- if context.get("temporal_worker_pool_enabled") is True:- options["task_queue"] = cls.task_queue- # The caller's authenticated principal is not a model argument. Keep- # it on the leaf context so owner-only plugins cannot inherit the- # legacy default principal accidentally.+ policy = (tool_info or {}).get("activity_policy")+ if context.get("execution_control_version") == 1 and policy:+ options = _frozen_tool_activity_options(policy, context)tool_payload = {**tool_payload, **_inherited_scope(context), "user_id": str(context.get("user_id") or "owner")}+ else:+ cls = get_node_class(tool_payload["node_type"])+ if getattr(cls, "workspace_task", False) and workflow.patched("workspace-task-activity-policy-v1"):+ options.update(+ start_to_close_timeout=cls.start_to_close_timeout,+ heartbeat_timeout=cls.heartbeat_timeout,+ retry_policy=cls.retry_policy.to_temporal(),+ cancellation_type=ActivityCancellationType.WAIT_CANCELLATION_COMPLETED,+ )+ if context.get("temporal_worker_pool_enabled") is True:+ options["task_queue"] = cls.task_queue+ tool_payload = {**tool_payload, **_inherited_scope(context), "user_id": str(context.get("user_id") or "owner")}return await workflow.execute_activity(activity_name,args=[tool_payload],@@ -130,6 +134,26 @@ async def _execute_plugin_tool_activity(**options,)++def _frozen_tool_activity_options(policy: Dict[str, Any], context: Dict[str, Any]) -> Dict[str, Any]:+ """Use the plugin declaration recorded by payload preparation."""+ retry = policy["retry_policy"]+ options: Dict[str, Any] = {+ "start_to_close_timeout": timedelta(seconds=policy["start_to_close_seconds"]),+ "heartbeat_timeout": (timedelta(seconds=policy["heartbeat_seconds"]) if policy.get("heartbeat_seconds") else None),+ "retry_policy": RetryPolicy(+ initial_interval=timedelta(seconds=retry["initial_interval_seconds"]),+ backoff_coefficient=retry["backoff_coefficient"],+ maximum_interval=timedelta(seconds=retry["maximum_interval_seconds"]),+ maximum_attempts=retry["maximum_attempts"],+ non_retryable_error_types=retry["non_retryable_error_types"],+ ),+ "cancellation_type": ActivityCancellationType.WAIT_CANCELLATION_COMPLETED,+ }+ if context.get("temporal_worker_pool_enabled") is True:+ options["task_queue"] = policy["task_queue"]+ return options+# Bounded loop count to defend against a runaway LLM. Plugin classes# override via ``payload["max_iterations"]`` (set by# ``prepare_agent_payload`` from Settings.agent_recursion_limit). This@@ -281,6 +305,10 @@ _INHERITED_SCOPE_KEYS = ("run_scope","employee_job_id","parameter_snapshot",+ "execution_control_version",+ "controller_workflow_id",+ "execution_control_state",+ "execution_control_revision",)@@ -297,11 +325,10 @@ _AGENT_HISTORY_SOFT_CAP = 10_000# payload error limit._RESUME_MARKER = "_agent_resume"-# Byte ceiling for the carried transcript (Temporal's payload error limit is-# 2 MiB for the WHOLE continue_as_new argument, which also carries the-# original context). Past this the rollover restarts from the opening-# prompt with a warning instead of failing the rollover itself.+# Legacy transcript-only limit remains for replay. Version 1 checks the+# complete encoded continuation and fails explicitly if it cannot fit._CAN_TRANSCRIPT_MAX_BYTES = 1_000_000+_CAN_INPUT_MAX_BYTES = 1_900_000def _inherited_scope(context: Dict[str, Any]) -> Dict[str, Any]:@@ -309,6 +336,14 @@ def _inherited_scope(context: Dict[str, Any]) -> Dict[str, Any]:return {key: context[key] for key in _INHERITED_SCOPE_KEYS if key in context}+def _child_control_scope(control: ExecutionControl) -> Dict[str, Any]:+ """Fresh intent travels to children; each root owns its chain identity."""+ return {+ key: value for key, value in control.carry().items()+ if key not in {"execution_control_root_run_id", "execution_control_released_revision"}+ }++def _trusted_tool_scope(context: Dict[str, Any]) -> Dict[str, Any]:return {**_inherited_scope(context), "outputs": context.get("outputs") or context.get("inputs") or {}}@@ -383,8 +418,32 @@ def _duplicate_visible_tool_name_error(tools: List[Dict[str, Any]]) -> Optional[class DelegatedTaskWorkflow:"""Own one queued delegation after the lead returns to its caller."""- def __init__(self) -> None:+ @workflow.init+ def __init__(self, request: Dict[str, Any] = None) -> None:self._control_paused = False+ self._execution_control = ExecutionControl()+ if request is not None:+ self._execution_control.bind(request.get("child_context") or request)++ @workflow.update+ async def set_control_state(self, payload: Dict[str, Any]) -> Dict[str, Any]:+ return await self._execution_control.set_control_state(payload)++ @set_control_state.validator+ def validate_set_control_state(self, payload: Dict[str, Any]) -> None:+ self._execution_control.validate_control(payload)++ @workflow.update+ async def wait_for_checkpoint(self, payload: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:+ return await self._execution_control.wait_for_checkpoint(payload)++ @wait_for_checkpoint.validator+ def validate_wait_for_checkpoint(self, payload: Optional[Dict[str, Any]] = None) -> None:+ self._execution_control.validate_control(payload)++ @workflow.query+ def execution_control_status(self) -> Dict[str, Any]:+ return self._execution_control.status()@workflow.signalasync def pause(self) -> None:@@ -395,11 +454,16 @@ class DelegatedTaskWorkflow:self._control_paused = Falseasync def _wait_until_resumed(self) -> None:+ if self._execution_control.enabled:+ await self._execution_control.wait_until_running()+ returnif self._control_paused:await workflow.wait_condition(lambda: not self._control_paused)@workflow.run- async def run(self, request: Dict[str, Any]) -> Dict[str, Any]:+ async def run(self, request: Dict[str, Any] = None) -> Dict[str, Any]:+ self._execution_control.bind(request.get("child_context") or request)+ await self._execution_control.register_root()acquire_cancellation_options = {"cancellation_type": (ActivityCancellationType.WAIT_CANCELLATION_COMPLETED@@ -410,6 +474,7 @@ class DelegatedTaskWorkflow:root_id = lifecycle["root_execution_id"]acquired_permit_id: Optional[str] = Nonebegan = False+ completion_action_active = Falsetry:await self._wait_until_resumed()info = workflow.info()@@ -444,19 +509,21 @@ class DelegatedTaskWorkflow:or task_id)await self._wait_until_resumed()- await workflow.execute_activity(- "agent.begin_delegation", args=[lifecycle],- start_to_close_timeout=PERSIST_TURN_TIMEOUT,- retry_policy=AGENT_ACTIVITY_RETRY,- **acquire_cancellation_options,- )- began = True+ async with self._execution_control.action():+ await workflow.execute_activity(+ "agent.begin_delegation", args=[lifecycle],+ start_to_close_timeout=PERSIST_TURN_TIMEOUT,+ retry_policy=AGENT_ACTIVITY_RETRY,+ **acquire_cancellation_options,+ )+ began = Trueawait self._wait_until_resumed()request["child_context"]["team_permit_id"] = acquired_permit_id- child_handle = await workflow.start_child_workflow(- "AgentWorkflow", args=[request["child_context"]],- id=request["child_workflow_id"],- )+ async with self._execution_control.child_start():+ child_handle = await workflow.start_child_workflow(+ "AgentWorkflow", args=[request["child_context"]],+ id=request["child_workflow_id"],+ )await workflow.execute_activity("agent.register_task_execution",args=[{**lifecycle, "runner_workflow_id": info.workflow_id,@@ -467,6 +534,8 @@ class DelegatedTaskWorkflow:retry_policy=AGENT_ACTIVITY_RETRY,)result = await child_handle+ self._execution_control.begin_bookkeeping()+ completion_action_active = self._execution_control.enabledsucceeded = bool(result.get("success", True)) if isinstance(result, dict) else True_response, summary = _normalise_delegated_result(result)return await workflow.execute_activity(@@ -574,6 +643,13 @@ class DelegatedTaskWorkflow:"DelegatedTaskWorkflow final permit release failed "f"for {task_id}: {release_exc}")+ try:+ await self._execution_control.unregister_root()+ finally:+ if completion_action_active:+ self._execution_control.end_action()+ if self._execution_control.enabled:+ await workflow.wait_condition(workflow.all_handlers_finished)@workflow.defn(sandboxed=False, name="AgentWorkflow")@@ -589,8 +665,32 @@ class AgentWorkflow:as F4.A per-type activities."""- def __init__(self) -> None:+ @workflow.init+ def __init__(self, context: Dict[str, Any] = None) -> None:self._control_paused = False+ self._execution_control = ExecutionControl()+ if context is not None:+ self._execution_control.bind(context)++ @workflow.update+ async def set_control_state(self, payload: Dict[str, Any]) -> Dict[str, Any]:+ return await self._execution_control.set_control_state(payload)++ @set_control_state.validator+ def validate_set_control_state(self, payload: Dict[str, Any]) -> None:+ self._execution_control.validate_control(payload)++ @workflow.update+ async def wait_for_checkpoint(self, payload: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:+ return await self._execution_control.wait_for_checkpoint(payload)++ @wait_for_checkpoint.validator+ def validate_wait_for_checkpoint(self, payload: Optional[Dict[str, Any]] = None) -> None:+ self._execution_control.validate_control(payload)++ @workflow.query+ def execution_control_status(self) -> Dict[str, Any]:+ return self._execution_control.status()@workflow.signalasync def pause(self) -> None:@@ -601,11 +701,15 @@ class AgentWorkflow:self._control_paused = Falseasync def _wait_until_resumed(self) -> None:+ if self._execution_control.enabled:+ await self._execution_control.wait_until_running()+ returnif self._control_paused:await workflow.wait_condition(lambda: not self._control_paused)@workflow.run- async def run(self, context: Dict[str, Any]) -> Dict[str, Any]:+ async def run(self, context: Dict[str, Any] = None) -> Dict[str, Any]:+ self._execution_control.bind(context)runtime_v2 = any((node.get("data") or {}).get("employee_recipe_version") == 2 and node.get("id") == context.get("node_id") for node in context.get("nodes", []))runtime_v2 = runtime_v2 and workflow.patched("employee-task-manager-runtime-v2")self._employee_runtime_v2 = runtime_v2@@ -625,6 +729,9 @@ class AgentWorkflow:from temporalio.exceptions import is_cancelled_exceptionawait fail_job(cancelled=is_cancelled_exception(exc))raise+ finally:+ if self._execution_control.enabled:+ await workflow.wait_condition(workflow.all_handlers_finished)async def _run_impl(self, context: Dict[str, Any]) -> Dict[str, Any]:"""Run the agent loop.@@ -706,20 +813,24 @@ class AgentWorkflow:# ---- Step 0: Resolve payload via the prep activity --------------# DB lookups + edge walking + tool schema build happen here, NOT# in the workflow body (workflows must be deterministic).- payload = await workflow.execute_activity(- "agent.prepare_payload",- args=[context],- activity_id="prepare-payload",- start_to_close_timeout=PERSIST_TURN_TIMEOUT * 2, # 60s default- retry_policy=AGENT_ACTIVITY_RETRY,- )+ resume = dict(context.get(_RESUME_MARKER) or {})+ if self._execution_control.enabled and resume.get("prepared_payload"):+ payload = dict(resume["prepared_payload"])+ else:+ async with self._execution_control.action():+ payload = await workflow.execute_activity(+ "agent.prepare_payload",+ args=[context],+ activity_id="prepare-payload",+ start_to_close_timeout=PERSIST_TURN_TIMEOUT * 2,+ retry_policy=AGENT_ACTIVITY_RETRY,+ )# Stable per-run execution id, forwarded into every tool-call# activity so session-keyed nodes (browser) reuse one instance# across iterations instead of minting a fresh uuid per call# (node_executor.py fallback). Delegation children inherit it via# the ``child_context`` spread below. ``workflow.info().run_id``# is deterministic — safe inside workflow code.- resume = dict(context.get(_RESUME_MARKER) or {})# run_id CHANGES on continue-as-new, so a resumed run would mint a# different execution id and break browser-session reuse, permit# scoping and root_execution_id fallback. Carry it explicitly.@@ -755,7 +866,7 @@ class AgentWorkflow:if payload.get("employee_job_id"):context["employee_job_id"] = payload["employee_job_id"]binding_refresh_v2 = workflow.patched("agent-node-binding-refresh-v2")- tools = payload.get("tools") or []+ tools = (resume.get("tools") if self._execution_control.enabled and "tools" in resume else payload.get("tools")) or []if binding_refresh_v2:tools = unique_node_bindings(tools, id_key="tool_node_id")duplicate_tool_error = _duplicate_visible_tool_name_error(tools)@@ -861,7 +972,7 @@ class AgentWorkflow:info = tool_index.get(name)return tool_output_is_capped(info.get("node_type") if info else None)- thinking_accumulated = ""+ thinking_accumulated = str(resume.get("thinking") or "") if self._execution_control.enabled else ""final_content: Optional[str] = None# Billing/observability is cumulative for the entire execution and# survives continue_as_new via the resume marker — the final result@@ -962,171 +1073,172 @@ class AgentWorkflow:),}- # Transient provider failures (429 rate limit, 5xx, network)- # retry inside the activity under LLM_STEP_RETRY (unlimited,- # exponential backoff, provider retry_after honored via- # next_retry_delay). Only non-retryable classifications- # (invalid_request, authentication, ...) reach this except- # block, and those are genuinely terminal for the run.- try:- step_result = await workflow.execute_activity(- "agent.execute_llm_step",- args=[llm_payload],- activity_id=f"llm-step-{iteration + 1}",- start_to_close_timeout=LLM_STEP_TIMEOUT,- heartbeat_timeout=LLM_STEP_HEARTBEAT_TIMEOUT,- retry_policy=LLM_STEP_RETRY,- )- except Exception as e:- from temporalio.exceptions import is_cancelled_exception- if getattr(self, "_employee_runtime_v2", False) and is_cancelled_exception(e):- raise asyncio.CancelledError() from e- cause = getattr(e, "cause", None)- raw_detail = str(cause) if cause is not None else str(e)- cause_type = str(getattr(cause, "type", "") or "")- cause_message = str(- getattr(cause, "message", "") or ""- ).strip()- recovery = {}- if cause_type.startswith("LLMError.") or cause_type == "MissingAgentProviderCredential":- diagnostics = getattr(cause, "details", ())- if diagnostics and isinstance(diagnostics[0], dict):- recovery = {- key: diagnostics[0][key]- for key in ("hint", "requires_user_action", "retryable")- if diagnostics[0].get(key) is not None- }- if recovery:- recovery["error_type"] = "NodeUserError"- safe_activity_types = {- "MissingAgentProviderCredential",- "EmptyAgentPrompt",- }- if cause_type.startswith("LLMError."):- detail = (- cause_message- or "The language model request failed."- )- elif cause_type in safe_activity_types:- detail = (- cause_message- or "The language model request failed."- )- else:- detail = (- "The language model step failed unexpectedly. "- "Retry the run or check server logs."- )- workflow.logger.error(- f"AgentWorkflow LLM step failed terminally "- f"(iteration {iteration + 1}): {raw_detail}"- )- # A failed model step is terminal for this run. Clear any- # turn-scoped skill badges and publish an error status before- # returning; otherwise the normal agent can remain visually- # stuck on its last capability while the workflow has already- # ended. Keep the public event free of provider error text.- await workflow.execute_activity(- "agent.skill.clear",- args=[{- "workflow_id": payload.get("workflow_id"),- "execution_id": task_scope_execution_id,- "agent_node_id": agent_node_id,- }],- activity_id="clear-active-skills-failed",- start_to_close_timeout=PERSIST_TURN_TIMEOUT,- retry_policy=AGENT_ACTIVITY_RETRY,- )- await self._emit_phase(- agent_node_id,- agent_workflow_id,- iteration,- max_iterations,- phase="failed",- status="error",- extra={"error": detail, "error_type": "NodeUserError", **recovery}- if recovery else None,- )- return {- "success": False,- "error": f"LLM step failed: {detail}",- "error_type": "LLMStepError",- **recovery,- "result": {- "iterations": iteration + 1,- "usage": usage_total,- },- }-- # Accumulate usage + thinking for the eventual return value.- for k, v in (step_result.get("usage") or {}).items():- if isinstance(v, int):- usage_total[k] = usage_total.get(k, 0) + v- context_usage_total[k] = (- context_usage_total.get(k, 0) + v+ async with self._execution_control.action():+ # Transient provider failures (429 rate limit, 5xx, network)+ # retry inside the activity under LLM_STEP_RETRY (unlimited,+ # exponential backoff, provider retry_after honored via+ # next_retry_delay). Only non-retryable classifications+ # (invalid_request, authentication, ...) reach this except+ # block, and those are genuinely terminal for the run.+ try:+ step_result = await workflow.execute_activity(+ "agent.execute_llm_step",+ args=[llm_payload],+ activity_id=f"llm-step-{iteration + 1}",+ start_to_close_timeout=LLM_STEP_TIMEOUT,+ heartbeat_timeout=LLM_STEP_HEARTBEAT_TIMEOUT,+ retry_policy=LLM_STEP_RETRY,)- step_thinking = step_result.get("thinking")- if not step_thinking:- step_thinking = _native_assistant_thinking(- step_result.get("assistant_message")- )- if step_thinking:- if thinking_accumulated:- thinking_accumulated += (- f"\n\n--- Iteration {iteration + 1} ---\n"- + step_thinking+ except Exception as e:+ from temporalio.exceptions import is_cancelled_exception+ if getattr(self, "_employee_runtime_v2", False) and is_cancelled_exception(e):+ raise asyncio.CancelledError() from e+ cause = getattr(e, "cause", None)+ raw_detail = str(cause) if cause is not None else str(e)+ cause_type = str(getattr(cause, "type", "") or "")+ cause_message = str(+ getattr(cause, "message", "") or ""+ ).strip()+ recovery = {}+ if cause_type.startswith("LLMError.") or cause_type == "MissingAgentProviderCredential":+ diagnostics = getattr(cause, "details", ())+ if diagnostics and isinstance(diagnostics[0], dict):+ recovery = {+ key: diagnostics[0][key]+ for key in ("hint", "requires_user_action", "retryable")+ if diagnostics[0].get(key) is not None+ }+ if recovery:+ recovery["error_type"] = "NodeUserError"+ safe_activity_types = {+ "MissingAgentProviderCredential",+ "EmptyAgentPrompt",+ }+ if cause_type.startswith("LLMError."):+ detail = (+ cause_message+ or "The language model request failed."+ )+ elif cause_type in safe_activity_types:+ detail = (+ cause_message+ or "The language model request failed."+ )+ else:+ detail = (+ "The language model step failed unexpectedly. "+ "Retry the run or check server logs."+ )+ workflow.logger.error(+ f"AgentWorkflow LLM step failed terminally "+ f"(iteration {iteration + 1}): {raw_detail}")- else:- thinking_accumulated = step_thinking-- kind = step_result.get("kind")-- # The activity returns the FULL serialized assistant message- # (the legacy canonical {type, data} shape).- # Appending verbatim preserves Gemini thought_signature, Anthropic- # cache markers, OpenAI reasoning content — everything the next- # turn's request needs.- assistant_message = step_result.get("assistant_message")- # Where this turn starts. Transcript-pressure relief never clears- # or summarizes the turn whose tool results the model has not- # read yet.- turn_start = len(messages)- if assistant_message:- messages.append(assistant_message)-- # A provider that stops to compact has not answered. It carries- # no tool calls, so it would otherwise be classified "final" and- # its truncated content returned to the user as the response.- # Treat it as a no-op turn and let the loop request again.- finish_reason = str(step_result.get("finish_reason") or "").strip().lower()- if kind != "tool_calls" and finish_reason == "compaction":- workflow.logger.info(- f"Provider paused to compact at iteration {iteration + 1}; "- "continuing without treating the stop as a final answer"- )- continue-- if kind == "final":- final_content = step_result.get("content", "")- team_id = str(payload.get("team_id") or "")- if team_id and payload.get("owns_execution_team"):- # Finalization is opportunistic. Queued/running work keeps- # the durable team active, but must not force this lead- # invocation to wait. Completion will emit taskTrigger and- # start the separately scoped review invocation.+ # A failed model step is terminal for this run. Clear any+ # turn-scoped skill badges and publish an error status before+ # returning; otherwise the normal agent can remain visually+ # stuck on its last capability while the workflow has already+ # ended. Keep the public event free of provider error text.await workflow.execute_activity(- "agent.finalize_team",- args=[{"team_id": team_id}],- activity_id="finalize-agent-team",+ "agent.skill.clear",+ args=[{+ "workflow_id": payload.get("workflow_id"),+ "execution_id": task_scope_execution_id,+ "agent_node_id": agent_node_id,+ }],+ activity_id="clear-active-skills-failed",start_to_close_timeout=PERSIST_TURN_TIMEOUT,retry_policy=AGENT_ACTIVITY_RETRY,)- await self._persist_turn(payload, human_text=user_prompt, assistant_text=final_content)- break+ await self._emit_phase(+ agent_node_id,+ agent_workflow_id,+ iteration,+ max_iterations,+ phase="failed",+ status="error",+ extra={"error": detail, "error_type": "NodeUserError", **recovery}+ if recovery else None,+ )+ return {+ "success": False,+ "error": f"LLM step failed: {detail}",+ "error_type": "LLMStepError",+ **recovery,+ "result": {+ "iterations": iteration + 1,+ "usage": usage_total,+ },+ }++ # Accumulate usage + thinking for the eventual return value.+ for k, v in (step_result.get("usage") or {}).items():+ if isinstance(v, int):+ usage_total[k] = usage_total.get(k, 0) + v+ context_usage_total[k] = (+ context_usage_total.get(k, 0) + v+ )+ step_thinking = step_result.get("thinking")+ if not step_thinking:+ step_thinking = _native_assistant_thinking(+ step_result.get("assistant_message")+ )+ if step_thinking:+ if thinking_accumulated:+ thinking_accumulated += (+ f"\n\n--- Iteration {iteration + 1} ---\n"+ + step_thinking+ )+ else:+ thinking_accumulated = step_thinking++ kind = step_result.get("kind")++ # The activity returns the FULL serialized assistant message+ # (the legacy canonical {type, data} shape).+ # Appending verbatim preserves Gemini thought_signature, Anthropic+ # cache markers, OpenAI reasoning content — everything the next+ # turn's request needs.+ assistant_message = step_result.get("assistant_message")+ # Where this turn starts. Transcript-pressure relief never clears+ # or summarizes the turn whose tool results the model has not+ # read yet.+ turn_start = len(messages)+ if assistant_message:+ messages.append(assistant_message)++ # A provider that stops to compact has not answered. It carries+ # no tool calls, so it would otherwise be classified "final" and+ # its truncated content returned to the user as the response.+ # Treat it as a no-op turn and let the loop request again.+ finish_reason = str(step_result.get("finish_reason") or "").strip().lower()+ if kind != "tool_calls" and finish_reason == "compaction":+ workflow.logger.info(+ f"Provider paused to compact at iteration {iteration + 1}; "+ "continuing without treating the stop as a final answer"+ )+ continue++ if kind == "final":+ final_content = step_result.get("content", "")+ team_id = str(payload.get("team_id") or "")+ if team_id and payload.get("owns_execution_team"):+ # Finalization is opportunistic. Queued/running work keeps+ # the durable team active, but must not force this lead+ # invocation to wait. Completion will emit taskTrigger and+ # start the separately scoped review invocation.+ await workflow.execute_activity(+ "agent.finalize_team",+ args=[{"team_id": team_id}],+ activity_id="finalize-agent-team",+ start_to_close_timeout=PERSIST_TURN_TIMEOUT,+ retry_policy=AGENT_ACTIVITY_RETRY,+ )+ await self._persist_turn(payload, human_text=user_prompt, assistant_text=final_content)+ break- if kind != "tool_calls":- workflow.logger.error(f"AgentWorkflow: unexpected LLM step kind={kind!r}")- break+ if kind != "tool_calls":+ workflow.logger.error(f"AgentWorkflow: unexpected LLM step kind={kind!r}")+ break# ---- Schedule tool activities -------------------------------# The LLM activity and phase broadcasts yield long enough for@@ -1300,6 +1412,7 @@ class AgentWorkflow:child_context = {# Inherited scope first: explicit keys below win.**_inherited_scope(context),+ **_child_control_scope(self._execution_control),"node_id": candidate_tool["tool_node_id"],"node_type": candidate_tool["node_type"],"node_data": {@@ -1355,21 +1468,22 @@ class AgentWorkflow:lifecycle_payload)await self._wait_until_resumed()- await workflow.execute_activity(- "agent.queue_delegation",- args=[{- **lifecycle_payload,- "queued_event_id": (- f"{child_context['team_task_id']}:queued"+ async with self._execution_control.action():+ await workflow.execute_activity(+ "agent.queue_delegation",+ args=[{+ **lifecycle_payload,+ "queued_event_id": (+ f"{child_context['team_task_id']}:queued"+ ),+ }],+ activity_id=(+ f"queue-delegation-{iteration + 1}-{call_index + 1}"),- }],- activity_id=(- f"queue-delegation-{iteration + 1}-{call_index + 1}"- ),- start_to_close_timeout=PERSIST_TURN_TIMEOUT,- retry_policy=AGENT_ACTIVITY_RETRY,- **acquire_cancellation_options,- )+ start_to_close_timeout=PERSIST_TURN_TIMEOUT,+ retry_policy=AGENT_ACTIVITY_RETRY,+ **acquire_cancellation_options,+ )await self._wait_until_resumed()acquire_payload = {"root_execution_id": root_execution_id,@@ -1402,16 +1516,17 @@ class AgentWorkflow:# while the coordinator queues it.try:await self._wait_until_resumed()- await workflow.execute_activity(- "agent.begin_delegation",- args=[lifecycle_payload],- activity_id=(- f"begin-delegation-{iteration + 1}-{call_index + 1}"- ),- start_to_close_timeout=PERSIST_TURN_TIMEOUT,- retry_policy=AGENT_ACTIVITY_RETRY,- **acquire_cancellation_options,- )+ async with self._execution_control.action():+ await workflow.execute_activity(+ "agent.begin_delegation",+ args=[lifecycle_payload],+ activity_id=(+ f"begin-delegation-{iteration + 1}-{call_index + 1}"+ ),+ start_to_close_timeout=PERSIST_TURN_TIMEOUT,+ retry_policy=AGENT_ACTIVITY_RETRY,+ **acquire_cancellation_options,+ )except asyncio.CancelledError:# Cancellation cleanup is owned by the caller's# _cleanup_cancelled_delegations() sweep; releasing@@ -1428,11 +1543,12 @@ class AgentWorkflow:)try:await self._wait_until_resumed()- delegation_handles[call_index] = await workflow.start_child_workflow(- "AgentWorkflow",- args=[child_context],- id=child_id,- )+ async with self._execution_control.child_start():+ delegation_handles[call_index] = await workflow.start_child_workflow(+ "AgentWorkflow",+ args=[child_context],+ id=child_id,+ )if team_id:child_handle = delegation_handles[call_index]await workflow.execute_activity(@@ -1534,19 +1650,21 @@ class AgentWorkflow:# idempotent for the pre-created task and records the same# lifecycle event as direct delegation without duplicating it.await self._wait_until_resumed()- await workflow.execute_activity(- "agent.queue_delegation",- args=[{**lifecycle, "queued_event_id": f"{task_id}:queued"}],- activity_id=f"queue-task-manager-{iteration + 1}-{call_index + 1}",- start_to_close_timeout=PERSIST_TURN_TIMEOUT,- retry_policy=AGENT_ACTIVITY_RETRY,- )+ async with self._execution_control.action():+ await workflow.execute_activity(+ "agent.queue_delegation",+ args=[{**lifecycle, "queued_event_id": f"{task_id}:queued"}],+ activity_id=f"queue-task-manager-{iteration + 1}-{call_index + 1}",+ start_to_close_timeout=PERSIST_TURN_TIMEOUT,+ retry_policy=AGENT_ACTIVITY_RETRY,+ )if context.get("employee_job_id"):request_context = {**(request_context or {}), "acceptance_criteria": request.get("acceptance_criteria"), "depends_on": request.get("depends_on") or []}context_text = request_context if isinstance(request_context, str) else _serialise_tool_result(request_context)child_context = {# Inherited scope first: explicit keys below win.**_inherited_scope(context),+ **_child_control_scope(self._execution_control),"node_id": assignee_id, "node_type": delegate["node_type"],"node_data": {**(delegate.get("parameters") or {}),"system_message": mission, "prompt": context_text or mission},@@ -1578,12 +1696,13 @@ class AgentWorkflow:delegated_task_options["search_attributes"] = (search_attributes)- await workflow.start_child_workflow(- "DelegatedTaskWorkflow",- args=[{"lifecycle": lifecycle, "child_context": child_context,- "child_workflow_id": child_id, "limit": max_concurrent_subagents}],- **delegated_task_options,- )+ async with self._execution_control.child_start():+ await workflow.start_child_workflow(+ "DelegatedTaskWorkflow",+ args=[{"lifecycle": lifecycle, "child_context": child_context,+ "child_workflow_id": child_id, "limit": max_concurrent_subagents}],+ **delegated_task_options,+ )return {"status": "queued", "result": None, "runner_workflow_id": runner_id}# Preflight every Task Manager assignment activity in this LLM@@ -1592,6 +1711,8 @@ class AgentWorkflow:# are allowed to reach the child-workflow bridge below.task_manager_preflight_indices: List[int] = []task_manager_preflight_handles: List[Any] = []+ preflight_action_active = False+ recorded_tool_results: set[int] = set()for preflight_index, preflight_call in enumerate(calls):preflight_tool = tool_index.get(preflight_call.get("name", ""))preflight_args = preflight_call.get("args") or {}@@ -1631,6 +1752,14 @@ class AgentWorkflow:# ``start_activity`` does not yield; one admission# check protects this whole concurrent batch.await self._wait_until_resumed()+ await self._execution_control.begin_action()+ preflight_action_active = self._execution_control.enabled+ preflight_options = {+ "start_to_close_timeout": TOOL_STEP_TIMEOUT,+ "heartbeat_timeout": TOOL_HEARTBEAT_TIMEOUT,+ }+ if self._execution_control.enabled and preflight_tool.get("activity_policy"):+ preflight_options = _frozen_tool_activity_options(preflight_tool["activity_policy"], context)task_manager_preflight_handles.append(workflow.start_activity(f"node.taskManager.v{preflight_tool['version']}",@@ -1639,21 +1768,30 @@ class AgentWorkflow:f"task-manager-preflight-{iteration + 1}-"f"{preflight_index + 1}"),- start_to_close_timeout=TOOL_STEP_TIMEOUT,- heartbeat_timeout=TOOL_HEARTBEAT_TIMEOUT,+ **preflight_options,))task_manager_preflight_results: Dict[int, Any] = {}task_manager_delegation_tasks: Dict[int, asyncio.Task[Any]] = {}if task_manager_preflight_handles:- preflight_results = await asyncio.gather(- *task_manager_preflight_handles, return_exceptions=True- )- for preflight_index, preflight_result in zip(- task_manager_preflight_indices, preflight_results- ):- task_manager_preflight_results[preflight_index] = preflight_result+ try:+ preflight_results = await asyncio.gather(+ *task_manager_preflight_handles, return_exceptions=True+ )+ for preflight_index, preflight_result in zip(+ task_manager_preflight_indices, preflight_results+ ):+ task_manager_preflight_results[preflight_index] = preflight_result+ if self._execution_control.enabled:+ content = (_serialise_tool_result(preflight_result) if not isinstance(preflight_result, BaseException)+ else _serialise_tool_result({"error": str(preflight_result)}))+ _append_tool_result_message(messages, content=content,+ tool_call_id=calls[preflight_index].get("id", ""), name=calls[preflight_index].get("name", ""))+ recorded_tool_results.add(preflight_index)+ finally:+ if preflight_action_active:+ self._execution_control.end_action()# Yield a child lead's slot exactly once before descendant# assignment coroutines contend for root-wide permits.@@ -1808,8 +1946,12 @@ class AgentWorkflow:# via the BaseNode.needs_canvas# ClassVar. Default tools execute against their own# params alone and don't see the parent canvas.- plugin_cls = get_node_class(tool_info["node_type"])- if plugin_cls is not None and plugin_cls.needs_canvas:+ if self._execution_control.enabled:+ needs_canvas = bool(tool_info.get("needs_canvas"))+ else:+ plugin_cls = get_node_class(tool_info["node_type"])+ needs_canvas = bool(plugin_cls is not None and plugin_cls.needs_canvas)+ if needs_canvas:child_nodes = context.get("nodes") or []child_edges = context.get("edges") or []else:@@ -1899,328 +2041,166 @@ class AgentWorkflow:await _cleanup_cancelled_delegations()raise+ tool_action_active = Falsetry:- if is_delegation and tool_info["node_type"] in AGENT_WORKFLOW_TYPES:- handle = delegation_handles[call_index]- try:- tool_result = await handle- except asyncio.CancelledError:- # Cancellation cleanup is owned by the outer- # _cleanup_cancelled_delegations() sweep.- raise- except BaseException:- await _release_delegation_permit(call_index)- raise- else:- await _release_delegation_permit(call_index)- # The child is done — drop its handle so a completed- # delegation no longer blocks the rollover guard.- # (An agent that delegates every turn previously- # could never continue_as_new and grew until- # Temporal's hard history terminate.)- delegation_handles.pop(call_index, None)- child_succeeded = (- bool(tool_result.get("success", True))- if isinstance(tool_result, dict) else True- )- child_error = (- tool_result.get("error")- if isinstance(tool_result, dict) else None- )- child_response, child_summary = _normalise_delegated_result(- tool_result- )- team_id = str(payload.get("team_id") or context.get("team_id") or "")- if team_id:- task_id = (- f"task-{root_execution_id}-{agent_node_id}-"- f"{iteration + 1}-{call_index + 1}"- )- await workflow.execute_activity(- "agent.finish_delegation",- args=[{- "team_id": team_id,- "team_task_id": task_id,- "parent_agent_node_id": agent_node_id,- "child_agent_node_id": tool_info["tool_node_id"],- "child_agent_name": str(- (tool_info.get("tool_info") or {}).get("label")- or tool_info.get("node_type")- or "agent"- ),- "workflow_id": payload.get("workflow_id"),- "parent_agent_workflow_id": workflow.info().workflow_id,- "root_execution_id": root_execution_id,- "trace_id": str(call.get("id", "") or ""),- "success": child_succeeded,- "result": child_summary,- "error": child_error,- "terminal_event_id": f"{task_id}:terminal",- }],- activity_id=(- f"finish-delegation-{iteration + 1}-{call_index + 1}"- ),- start_to_close_timeout=PERSIST_TURN_TIMEOUT,- retry_policy=AGENT_ACTIVITY_RETRY,+ try:+ if is_delegation and tool_info["node_type"] in AGENT_WORKFLOW_TYPES:+ handle = delegation_handles[call_index]+ try:+ tool_result = await handle+ self._execution_control.begin_bookkeeping()+ tool_action_active = self._execution_control.enabled+ except asyncio.CancelledError:+ # Cancellation cleanup is owned by the outer+ # _cleanup_cancelled_delegations() sweep.+ raise+ except BaseException:+ self._execution_control.begin_bookkeeping()+ tool_action_active = self._execution_control.enabled+ await _release_delegation_permit(call_index)+ raise+ else:+ await _release_delegation_permit(call_index)+ # The child is done — drop its handle so a completed+ # delegation no longer blocks the rollover guard.+ # (An agent that delegates every turn previously+ # could never continue_as_new and grew until+ # Temporal's hard history terminate.)+ delegation_handles.pop(call_index, None)+ child_succeeded = (+ bool(tool_result.get("success", True))+ if isinstance(tool_result, dict) else True)- delegation_lifecycles.pop(- call_index,- None,+ child_error = (+ tool_result.get("error")+ if isinstance(tool_result, dict) else None)- child_recovery = {- key: tool_result[key]- for key in ("hint", "requires_user_action", "retryable")- if isinstance(tool_result, dict) and key in tool_result- }- tool_result = {- "success": child_succeeded,- "status": "submitted" if child_succeeded else "failed",- "result": child_response,- "usage": child_summary.get("usage"),- **({"error": child_error} if child_error else {}),- **child_recovery,- }- if next_delegation_to_start < len(delegation_call_indices):- await _start_delegation(- delegation_call_indices[next_delegation_to_start]+ child_response, child_summary = _normalise_delegated_result(+ tool_result)- next_delegation_to_start += 1- else:- tool_activity_id = _tool_activity_id(- tool_info["tool_node_id"],- iteration,- call_index,- )- if call_index in task_manager_preflight_results:- tool_result = task_manager_preflight_results[call_index]- if isinstance(tool_result, BaseException):- raise tool_result- else:- await self._wait_until_resumed()- tool_result = await _execute_plugin_tool_activity(- tool_activity_name,- tool_activity_id,- tool_payload,- context,- )- if (- tool_info["node_type"] == "taskManager"- and isinstance(tool_result, dict)- and isinstance(tool_result.get("delegation_request"), dict)- ):- if call_index in task_manager_delegation_tasks:- # Await in original tool-call order; every- # child was already started above, so slow- # earlier siblings do not prevent later work.- # pop: a finished task must not block the- # rollover guard for the rest of the turn.- delegated = await task_manager_delegation_tasks.pop(call_index)- else:- delegated = await _run_task_manager_delegation(- tool_result["delegation_request"], call_index, call+ team_id = str(payload.get("team_id") or context.get("team_id") or "")+ if team_id:+ task_id = (+ f"task-{root_execution_id}-{agent_node_id}-"+ f"{iteration + 1}-{call_index + 1}"+ )+ await workflow.execute_activity(+ "agent.finish_delegation",+ args=[{+ "team_id": team_id,+ "team_task_id": task_id,+ "parent_agent_node_id": agent_node_id,+ "child_agent_node_id": tool_info["tool_node_id"],+ "child_agent_name": str(+ (tool_info.get("tool_info") or {}).get("label")+ or tool_info.get("node_type")+ or "agent"+ ),+ "workflow_id": payload.get("workflow_id"),+ "parent_agent_workflow_id": workflow.info().workflow_id,+ "root_execution_id": root_execution_id,+ "trace_id": str(call.get("id", "") or ""),+ "success": child_succeeded,+ "result": child_summary,+ "error": child_error,+ "terminal_event_id": f"{task_id}:terminal",+ }],+ activity_id=(+ f"finish-delegation-{iteration + 1}-{call_index + 1}"+ ),+ start_to_close_timeout=PERSIST_TURN_TIMEOUT,+ retry_policy=AGENT_ACTIVITY_RETRY,)+ delegation_lifecycles.pop(+ call_index,+ None,+ )+ child_recovery = {+ key: tool_result[key]+ for key in ("hint", "requires_user_action", "retryable")+ if isinstance(tool_result, dict) and key in tool_result+ }tool_result = {- **tool_result,- "delegation_status": delegated["status"],- "delegation_result": delegated["result"],- "delegation_usage": delegated.get("usage"),+ "success": child_succeeded,+ "status": "submitted" if child_succeeded else "failed",+ "result": child_response,+ "usage": child_summary.get("usage"),+ **({"error": child_error} if child_error else {}),+ **child_recovery,}- if (- isinstance(tool_result, dict)- and tool_result.get("error")- and tool_result.get("requires_user_action") is True- and workflow.patched("agent-user-action-error-v1")- ):- blocked_error = blocked_error or {- "error": tool_result["error"],- "error_type": "NodeUserError",- "hint": tool_result.get("hint"),- "requires_user_action": True,- "retryable": False,- }- tool_content = _serialise_tool_result(tool_result)- if (- tool_output_limit- and not is_delegation- and tool_output_is_capped(tool_info["node_type"])- ):- # Only the model's copy is cut; the tool node keeps- # its whole result.- tool_content = bound_tool_output(- tool_content, tool_output_limit- )- await self._emit_phase(- agent_node_id,- agent_workflow_id,- iteration,- max_iterations,- phase="tool_completed",- extra={- "tool_name": call.get("name", ""),- "tool_node_id": tool_info["tool_node_id"],- "tool_call_id": str(- call.get("id") or f"{iteration + 1}:{call_index + 1}"- ),- },- )-- # Hot-rebind: if the tool returned ``operations`` (canvas- # mutation), schedule ``agent.refresh_tools.v1`` to build- # new tool_payload entries from the ops and splice them- # into the workflow's live ``tools`` / ``tool_index``.- # The next ``execute_llm_step`` invocation rebuilds the- # bound LLM surface from this updated list, so the new- # tool is callable in the very next iteration without a- # Run-stop-Run cycle.- auto_rebind_enabled = bool(payload.get("auto_rebind_tools", True))- if auto_rebind_enabled and isinstance(tool_result, dict):- ops_from_tool = tool_result.get("operations") or []- if ops_from_tool:- # Deliberately multi-attempt (unlike the LLM- # step's one-shot LLM_STEP_RETRY): rebuilding- # the tool surface from canvas state is fully- # idempotent, so retries are free.- refresh_activity_id = _refresh_tools_activity_id(+ if next_delegation_to_start < len(delegation_call_indices):+ if not self._execution_control.enabled:+ await _start_delegation(+ delegation_call_indices[next_delegation_to_start]+ )+ next_delegation_to_start += 1+ else:+ tool_activity_id = _tool_activity_id(tool_info["tool_node_id"],iteration,call_index,)- refresh_payload = {- "operations": ops_from_tool,- "agent_node_type": payload.get("node_type") or context.get("node_type"),- **call_metadata,- }- if binding_refresh_v2:- refresh_payload["bound_node_ids"] = [tool.get("tool_node_id") for tool in tools if tool.get("tool_node_id")]- refresh_payload["graph_snapshot"] = {"nodes": context.get("nodes") or [], "edges": context.get("edges") or []}- refresh_payload["parameter_snapshot"] = context.get("parameter_snapshot") or {}- await self._wait_until_resumed()- refresh_result = await workflow.execute_activity(- "agent.refresh_tools",- args=[refresh_payload],- activity_id=refresh_activity_id,- start_to_close_timeout=timedelta(seconds=30),- retry_policy=AGENT_ACTIVITY_RETRY,- )- if binding_refresh_v2 and refresh_result.get("graph_snapshot"):- refreshed_graph = refresh_result["graph_snapshot"]- context["nodes"] = refreshed_graph.get("nodes") or context.get("nodes") or []- context["edges"] = refreshed_graph.get("edges") or context.get("edges") or []- context["parameter_snapshot"] = {**(context.get("parameter_snapshot") or {}), **(refresh_result.get("parameter_updates") or {})}- added_tools = refresh_result.get("tools") or []- if binding_refresh_v2:- added_tools = unique_node_bindings(- added_tools, id_key="tool_node_id",- bound=[tool.get("tool_node_id") for tool in tools if tool.get("tool_node_id")],- )- refresh_duplicate_error = _duplicate_visible_tool_name_error(- [*tools, *added_tools]- )- refresh_duplicate_conflicts = (- _duplicate_visible_tool_name_conflicts([*tools, *added_tools])- if refresh_duplicate_error- else {}- )- if refresh_duplicate_error:- workflow.logger.warning(- "AgentWorkflow rejected hot-rebound tools: %s",- refresh_duplicate_error,- )- tool_content = _serialise_tool_result(- {- "error_type": DUPLICATE_TOOL_NAME_ERROR_TYPE,- "error": refresh_duplicate_error,- "conflicts": refresh_duplicate_conflicts,- }- )- await self._emit_phase(- agent_node_id,- agent_workflow_id,- iteration,- max_iterations,- phase="tool_error",- extra={- "error_type": DUPLICATE_TOOL_NAME_ERROR_TYPE,- "error": refresh_duplicate_error,- "conflicts": refresh_duplicate_conflicts,- **call_metadata,- },+ if call_index in task_manager_preflight_results:+ tool_result = task_manager_preflight_results[call_index]+ if isinstance(tool_result, BaseException):+ raise tool_result+ else:+ await self._wait_until_resumed()+ await self._execution_control.begin_action()+ tool_action_active = self._execution_control.enabled+ tool_result = await _execute_plugin_tool_activity(+ tool_activity_name,+ tool_activity_id,+ tool_payload,+ context,+ tool_info,)- added_tools = []- for new_tool in added_tools:- tools.append(new_tool)- tool_index[new_tool["name"]] = new_tool- if added_tools:- delegates = [tool for tool in tools if tool.get("name", "").startswith("delegate_to_")]- if delegates and binding_refresh_v2:- roster = "\n".join(- f"{tool.get('tool_node_id')}: {(tool.get('tool_info') or {}).get('label') or tool.get('node_type')}"- for tool in delegates+ if (+ tool_info["node_type"] == "taskManager"+ and isinstance(tool_result, dict)+ and isinstance(tool_result.get("delegation_request"), dict)+ ):+ if call_index in task_manager_delegation_tasks:+ # Await in original tool-call order; every+ # child was already started above, so slow+ # earlier siblings do not prevent later work.+ # pop: a finished task must not block the+ # rollover guard for the rest of the turn.+ delegated = await task_manager_delegation_tasks.pop(call_index)+ else:+ delegated = await _run_task_manager_delegation(+ tool_result["delegation_request"], call_index, call)- messages.append(_native_message(- role="system",- content="Updated connected teammates (assignee_node_id: label/type):\n" + roster,- ))- workflow.logger.info(- "AgentWorkflow rebound %d tool(s) after canvas mutation (total bound=%d)",- len(added_tools),- len(tools),- )- except asyncio.CancelledError:- await _cleanup_cancelled_delegations()- raise- except Exception as e: # noqa: BLE001 — Temporal handles retries- # After all retries exhausted, surface the error to- # the LLM (per user decision: LLM sees error and- # continues — matches the in-process agent loop).- workflow.logger.warning(f"AgentWorkflow tool {tool_info['node_type']!r} failed: {e}")- team_id = str(payload.get("team_id") or context.get("team_id") or "")- if is_delegation and team_id:- task_id = (- f"task-{root_execution_id}-{agent_node_id}-"- f"{iteration + 1}-{call_index + 1}"- )- try:- await workflow.execute_activity(- "agent.finish_delegation",- args=[{- "team_id": team_id,- "team_task_id": task_id,- "parent_agent_node_id": agent_node_id,- "child_agent_node_id": tool_info["tool_node_id"],- "child_agent_name": str(- (tool_info.get("tool_info") or {}).get(- "label"- )- or tool_info.get("node_type")- or "agent"- ),- "workflow_id": payload.get("workflow_id"),- "parent_agent_workflow_id": (- workflow.info().workflow_id- ),- "root_execution_id": root_execution_id,- "trace_id": str(call.get("id", "") or ""),- "success": False,- "error": f"{type(e).__name__}: {e}",- "terminal_event_id": f"{task_id}:terminal",- }],- activity_id=(- "finish-delegation-"- f"{iteration + 1}-{call_index + 1}"- ),- start_to_close_timeout=PERSIST_TURN_TIMEOUT,- retry_policy=AGENT_ACTIVITY_RETRY,- )- delegation_lifecycles.pop(- call_index,- None,+ tool_result = {+ **tool_result,+ "delegation_status": delegated["status"],+ "delegation_result": delegated["result"],+ "delegation_usage": delegated.get("usage"),+ }+ if (+ isinstance(tool_result, dict)+ and tool_result.get("error")+ and tool_result.get("requires_user_action") is True+ and workflow.patched("agent-user-action-error-v1")+ ):+ blocked_error = blocked_error or {+ "error": tool_result["error"],+ "error_type": "NodeUserError",+ "hint": tool_result.get("hint"),+ "requires_user_action": True,+ "retryable": False,+ }+ tool_content = _serialise_tool_result(tool_result)+ if (+ tool_output_limit+ and not is_delegation+ and tool_output_is_capped(tool_info["node_type"])+ ):+ # Only the model's copy is cut; the tool node keeps+ # its whole result.+ tool_content = bound_tool_output(+ tool_content, tool_output_limit)- except asyncio.CancelledError:- await _cleanup_cancelled_delegations()- raise- tool_content = f'{{"error": "{type(e).__name__}: {e}"}}'- try:await self._emit_phase(agent_node_id,agent_workflow_id,@@ -2231,26 +2211,216 @@ class AgentWorkflow:"tool_name": call.get("name", ""),"tool_node_id": tool_info["tool_node_id"],"tool_call_id": str(- call.get("id")- or f"{iteration + 1}:{call_index + 1}"+ call.get("id") or f"{iteration + 1}:{call_index + 1}"),- # Do not put raw failures into public status- # events. This safe flag is enough for the- # broadcaster to retain ``tool <name>`` with a- # failed capability state.- "tool_failed": True,},)++ # Hot-rebind: if the tool returned ``operations`` (canvas+ # mutation), schedule ``agent.refresh_tools.v1`` to build+ # new tool_payload entries from the ops and splice them+ # into the workflow's live ``tools`` / ``tool_index``.+ # The next ``execute_llm_step`` invocation rebuilds the+ # bound LLM surface from this updated list, so the new+ # tool is callable in the very next iteration without a+ # Run-stop-Run cycle.+ auto_rebind_enabled = bool(payload.get("auto_rebind_tools", True))+ if self._execution_control.enabled and call_index not in recorded_tool_results:+ _append_tool_result_message(messages, content=tool_content,+ tool_call_id=call.get("id", ""), name=call.get("name", ""))+ recorded_tool_results.add(call_index)+ if tool_action_active:+ self._execution_control.end_action()+ tool_action_active = False+ if auto_rebind_enabled and isinstance(tool_result, dict):+ ops_from_tool = tool_result.get("operations") or []+ if ops_from_tool:+ # Deliberately multi-attempt (unlike the LLM+ # step's one-shot LLM_STEP_RETRY): rebuilding+ # the tool surface from canvas state is fully+ # idempotent, so retries are free.+ refresh_activity_id = _refresh_tools_activity_id(+ tool_info["tool_node_id"],+ iteration,+ call_index,+ )+ refresh_payload = {+ "operations": ops_from_tool,+ "agent_node_type": payload.get("node_type") or context.get("node_type"),+ **call_metadata,+ **({"execution_control_version": 1} if self._execution_control.enabled else {}),+ }+ if binding_refresh_v2:+ refresh_payload["bound_node_ids"] = [tool.get("tool_node_id") for tool in tools if tool.get("tool_node_id")]+ refresh_payload["graph_snapshot"] = {"nodes": context.get("nodes") or [], "edges": context.get("edges") or []}+ refresh_payload["parameter_snapshot"] = context.get("parameter_snapshot") or {}+ await self._wait_until_resumed()+ async with self._execution_control.action():+ refresh_result = await workflow.execute_activity(+ "agent.refresh_tools",+ args=[refresh_payload],+ activity_id=refresh_activity_id,+ start_to_close_timeout=timedelta(seconds=30),+ retry_policy=AGENT_ACTIVITY_RETRY,+ )+ if binding_refresh_v2 and refresh_result.get("graph_snapshot"):+ refreshed_graph = refresh_result["graph_snapshot"]+ context["nodes"] = refreshed_graph.get("nodes") or context.get("nodes") or []+ context["edges"] = refreshed_graph.get("edges") or context.get("edges") or []+ context["parameter_snapshot"] = {**(context.get("parameter_snapshot") or {}), **(refresh_result.get("parameter_updates") or {})}+ added_tools = refresh_result.get("tools") or []+ if binding_refresh_v2:+ added_tools = unique_node_bindings(+ added_tools, id_key="tool_node_id",+ bound=[tool.get("tool_node_id") for tool in tools if tool.get("tool_node_id")],+ )+ refresh_duplicate_error = _duplicate_visible_tool_name_error(+ [*tools, *added_tools]+ )+ refresh_duplicate_conflicts = (+ _duplicate_visible_tool_name_conflicts([*tools, *added_tools])+ if refresh_duplicate_error+ else {}+ )+ if refresh_duplicate_error:+ workflow.logger.warning(+ "AgentWorkflow rejected hot-rebound tools: %s",+ refresh_duplicate_error,+ )+ tool_content = _serialise_tool_result(+ {+ "error_type": DUPLICATE_TOOL_NAME_ERROR_TYPE,+ "error": refresh_duplicate_error,+ "conflicts": refresh_duplicate_conflicts,+ }+ )+ await self._emit_phase(+ agent_node_id,+ agent_workflow_id,+ iteration,+ max_iterations,+ phase="tool_error",+ extra={+ "error_type": DUPLICATE_TOOL_NAME_ERROR_TYPE,+ "error": refresh_duplicate_error,+ "conflicts": refresh_duplicate_conflicts,+ **call_metadata,+ },+ )+ added_tools = []+ for new_tool in added_tools:+ tools.append(new_tool)+ tool_index[new_tool["name"]] = new_tool+ if added_tools:+ delegates = [tool for tool in tools if tool.get("name", "").startswith("delegate_to_")]+ if delegates and binding_refresh_v2:+ roster = "\n".join(+ f"{tool.get('tool_node_id')}: {(tool.get('tool_info') or {}).get('label') or tool.get('node_type')}"+ for tool in delegates+ )+ messages.append(_native_message(+ role="system",+ content="Updated connected teammates (assignee_node_id: label/type):\n" + roster,+ ))+ workflow.logger.info(+ "AgentWorkflow rebound %d tool(s) after canvas mutation (total bound=%d)",+ len(added_tools),+ len(tools),+ )except asyncio.CancelledError:await _cleanup_cancelled_delegations()raise+ except Exception as e: # noqa: BLE001 — Temporal handles retries+ # After all retries exhausted, surface the error to+ # the LLM (per user decision: LLM sees error and+ # continues — matches the in-process agent loop).+ workflow.logger.warning(f"AgentWorkflow tool {tool_info['node_type']!r} failed: {e}")+ team_id = str(payload.get("team_id") or context.get("team_id") or "")+ if is_delegation and team_id:+ task_id = (+ f"task-{root_execution_id}-{agent_node_id}-"+ f"{iteration + 1}-{call_index + 1}"+ )+ try:+ await workflow.execute_activity(+ "agent.finish_delegation",+ args=[{+ "team_id": team_id,+ "team_task_id": task_id,+ "parent_agent_node_id": agent_node_id,+ "child_agent_node_id": tool_info["tool_node_id"],+ "child_agent_name": str(+ (tool_info.get("tool_info") or {}).get(+ "label"+ )+ or tool_info.get("node_type")+ or "agent"+ ),+ "workflow_id": payload.get("workflow_id"),+ "parent_agent_workflow_id": (+ workflow.info().workflow_id+ ),+ "root_execution_id": root_execution_id,+ "trace_id": str(call.get("id", "") or ""),+ "success": False,+ "error": f"{type(e).__name__}: {e}",+ "terminal_event_id": f"{task_id}:terminal",+ }],+ activity_id=(+ "finish-delegation-"+ f"{iteration + 1}-{call_index + 1}"+ ),+ start_to_close_timeout=PERSIST_TURN_TIMEOUT,+ retry_policy=AGENT_ACTIVITY_RETRY,+ )+ delegation_lifecycles.pop(+ call_index,+ None,+ )+ except asyncio.CancelledError:+ await _cleanup_cancelled_delegations()+ raise+ tool_content = f'{{"error": "{type(e).__name__}: {e}"}}'+ try:+ await self._emit_phase(+ agent_node_id,+ agent_workflow_id,+ iteration,+ max_iterations,+ phase="tool_completed",+ extra={+ "tool_name": call.get("name", ""),+ "tool_node_id": tool_info["tool_node_id"],+ "tool_call_id": str(+ call.get("id")+ or f"{iteration + 1}:{call_index + 1}"+ ),+ # Do not put raw failures into public status+ # events. This safe flag is enough for the+ # broadcaster to retain ``tool <name>`` with a+ # failed capability state.+ "tool_failed": True,+ },+ )+ except asyncio.CancelledError:+ await _cleanup_cancelled_delegations()+ raise- _append_tool_result_message(- messages,- content=tool_content,- tool_call_id=call.get("id", ""),- name=call.get("name", ""),- )+ if call_index not in recorded_tool_results:+ _append_tool_result_message(+ messages,+ content=tool_content,+ tool_call_id=call.get("id", ""),+ name=call.get("name", ""),+ )++ finally:+ if tool_action_active:+ self._execution_control.end_action()++ if self._execution_control.enabled and is_delegation and next_delegation_to_start < len(delegation_call_indices):+ await _start_delegation(delegation_call_indices[next_delegation_to_start])+ next_delegation_to_start += 1if yielded_own_permit:await self._wait_until_resumed()@@ -2398,127 +2568,128 @@ class AgentWorkflow:"model": payload["model"],}await self._wait_until_resumed()- try:- compact_result = await workflow.execute_activity(- "agent.compact_context",- args=[compact_payload],- activity_id=f"compact-context-{iteration + 1}",- start_to_close_timeout=COMPACT_MEMORY_TIMEOUT,- retry_policy=AGENT_ACTIVITY_RETRY,- )- except Exception as compact_error:- # Compaction is the run's pressure-relief valve. If it- # fails even after the activity policy's retries, the- # transcript can only grow until the provider rejects- # it — fail the run loudly NOW, at the moment the- # cause is clear, instead of later with a confusing- # context-overflow error.- cause = getattr(compact_error, "cause", None)- compact_detail = str(- getattr(cause, "message", "") or cause or compact_error- )- workflow.logger.error(- f"AgentWorkflow compaction failed terminally "- f"(iteration {iteration + 1}): {compact_detail}"- )- # Same terminal-cleanup contract as the LLM-failure- # path: clear turn-scoped skill badges before the- # error status, or the canvas stays stuck on the last- # capability after the workflow has ended.- await workflow.execute_activity(- "agent.skill.clear",- args=[{- "workflow_id": payload.get("workflow_id"),- "execution_id": task_scope_execution_id,- "agent_node_id": agent_node_id,- }],- activity_id="clear-active-skills-compaction-failed",- start_to_close_timeout=PERSIST_TURN_TIMEOUT,- retry_policy=AGENT_ACTIVITY_RETRY,- )- await self._emit_phase(- agent_node_id,- agent_workflow_id,- iteration,- max_iterations,- phase="failed",- status="error",+ async with self._execution_control.action():+ try:+ compact_result = await workflow.execute_activity(+ "agent.compact_context",+ args=[compact_payload],+ activity_id=f"compact-context-{iteration + 1}",+ start_to_close_timeout=COMPACT_MEMORY_TIMEOUT,+ retry_policy=AGENT_ACTIVITY_RETRY,+ )+ except Exception as compact_error:+ # Compaction is the run's pressure-relief valve. If it+ # fails even after the activity policy's retries, the+ # transcript can only grow until the provider rejects+ # it — fail the run loudly NOW, at the moment the+ # cause is clear, instead of later with a confusing+ # context-overflow error.+ cause = getattr(compact_error, "cause", None)+ compact_detail = str(+ getattr(cause, "message", "") or cause or compact_error+ )+ workflow.logger.error(+ f"AgentWorkflow compaction failed terminally "+ f"(iteration {iteration + 1}): {compact_detail}"+ )+ # Same terminal-cleanup contract as the LLM-failure+ # path: clear turn-scoped skill badges before the+ # error status, or the canvas stays stuck on the last+ # capability after the workflow has ended.+ await workflow.execute_activity(+ "agent.skill.clear",+ args=[{+ "workflow_id": payload.get("workflow_id"),+ "execution_id": task_scope_execution_id,+ "agent_node_id": agent_node_id,+ }],+ activity_id="clear-active-skills-compaction-failed",+ start_to_close_timeout=PERSIST_TURN_TIMEOUT,+ retry_policy=AGENT_ACTIVITY_RETRY,+ )+ await self._emit_phase(+ agent_node_id,+ agent_workflow_id,+ iteration,+ max_iterations,+ phase="failed",+ status="error",+ )+ return {+ "success": False,+ "error": f"Compaction failed: {compact_detail}",+ "error_type": "CompactionError",+ "result": {+ "iterations": iteration + 1,+ "usage": usage_total,+ },+ }+ # The summarizer is another billed model call. Include it in+ # execution-wide usage, but never in the active-context+ # counter.+ for key, value in (+ compact_result.get("usage") or {}+ ).items():+ if isinstance(value, int):+ usage_total[key] = (+ usage_total.get(key, 0) + value+ )+ # The activity raises on any failure, so a result here+ # always carries a non-empty summary.+ summary = compact_result.get("summary", "")+ dropped_count = len(messages) - len(kept_turn)+ # Rebuild as: the ORIGINAL system prompt, verbatim, plus ONE+ # user message carrying the summary and the live request.+ #+ # The system prompt must never be modified or duplicated by+ # compaction: (1) it is the agent's contract (personality ++ # tool/delegation guidance) and must stay byte-stable for+ # provider prompt caching and behavioral consistency; (2) the+ # next firing's seeding drops stored system messages so+ # policy changes take effect — anything compaction stores+ # under the system role silently vanishes on the next+ # firing, which is how a summary once survived only until+ # the next chat message while the noisy tool tail outlived+ # it.+ #+ # The summary rides a USER message for the same reason: user+ # wires persist through seeding, so the compacted knowledge+ # crosses firings with the conversation it summarizes.+ compacted_content = (+ "## Compacted conversation summary\n"+ "The conversation so far was compacted to stay within "+ "the model's context window. Treat this summary as the "+ "authoritative record of prior work; do not repeat "+ "completed steps.\n\n"+ f"{summary}")- return {- "success": False,- "error": f"Compaction failed: {compact_detail}",- "error_type": "CompactionError",- "result": {- "iterations": iteration + 1,- "usage": usage_total,- },- }- # The summarizer is another billed model call. Include it in- # execution-wide usage, but never in the active-context- # counter.- for key, value in (- compact_result.get("usage") or {}- ).items():- if isinstance(value, int):- usage_total[key] = (- usage_total.get(key, 0) + value+ if user_prompt:+ compacted_content += (+ f"\n\n## Current request\n{user_prompt}")- # The activity raises on any failure, so a result here- # always carries a non-empty summary.- summary = compact_result.get("summary", "")- dropped_count = len(messages) - len(kept_turn)- # Rebuild as: the ORIGINAL system prompt, verbatim, plus ONE- # user message carrying the summary and the live request.- #- # The system prompt must never be modified or duplicated by- # compaction: (1) it is the agent's contract (personality +- # tool/delegation guidance) and must stay byte-stable for- # provider prompt caching and behavioral consistency; (2) the- # next firing's seeding drops stored system messages so- # policy changes take effect — anything compaction stores- # under the system role silently vanishes on the next- # firing, which is how a summary once survived only until- # the next chat message while the noisy tool tail outlived- # it.- #- # The summary rides a USER message for the same reason: user- # wires persist through seeding, so the compacted knowledge- # crosses firings with the conversation it summarizes.- compacted_content = (- "## Compacted conversation summary\n"- "The conversation so far was compacted to stay within "- "the model's context window. Treat this summary as the "- "authoritative record of prior work; do not repeat "- "completed steps.\n\n"- f"{summary}"- )- if user_prompt:- compacted_content += (- f"\n\n## Current request\n{user_prompt}"+ messages = [+ _native_message(+ role="system",+ content=system,+ ),+ _native_message(+ role="user",+ content=compacted_content,+ ),+ # Version 1 keeps this turn verbatim, because its tool+ # results are still unread. Empty on the original path.+ *kept_turn,+ ]+ context_usage_total = {}+ turn_start = len(messages) - len(kept_turn)+ workflow.logger.info(+ f"AgentWorkflow compaction applied at iteration "+ f"{iteration + 1}: {dropped_count} messages -> "+ f"{len(messages)} (summary {len(summary)} chars; "+ f"{len(kept_turn)} message(s) of this turn kept verbatim; "+ "earlier tool calls/results now live only inside the "+ "summary; system prompt preserved verbatim)")- messages = [- _native_message(- role="system",- content=system,- ),- _native_message(- role="user",- content=compacted_content,- ),- # Version 1 keeps this turn verbatim, because its tool- # results are still unread. Empty on the original path.- *kept_turn,- ]- context_usage_total = {}- turn_start = len(messages) - len(kept_turn)- workflow.logger.info(- f"AgentWorkflow compaction applied at iteration "- f"{iteration + 1}: {dropped_count} messages -> "- f"{len(messages)} (summary {len(summary)} chars; "- f"{len(kept_turn)} message(s) of this turn kept verbatim; "- "earlier tool calls/results now live only inside the "- "summary; system prompt preserved verbatim)"- )if pressure_version >= 1:# Last resort, against whatever room the earlier turns now@@ -2567,6 +2738,36 @@ class AgentWorkflow:delegation_handles or task_manager_delegation_tasks)if not delegations_live:+ if self._execution_control.enabled:+ # A stopped workflow remains in this run. No live child+ # handle or unfinished Update crosses the boundary.+ await self._wait_until_resumed()+ await workflow.wait_condition(workflow.all_handlers_finished)+ next_context = {+ **context,+ **self._execution_control.carry(),+ _RESUME_MARKER: {+ "iteration": iteration + 1,+ "execution_id": execution_id,+ "transcript": messages,+ "usage": usage_total,+ "context_usage": context_usage_total,+ "prepared_payload": payload,+ "tools": tools,+ "thinking": thinking_accumulated,+ },+ }+ # Measure the converter's complete argument, including+ # graph, bindings and preparation, not just messages.+ encoded = workflow.payload_converter().to_payloads([next_context])+ input_bytes = sum(item.ByteSize() for item in encoded)+ if input_bytes > _CAN_INPUT_MAX_BYTES:+ from temporalio.exceptions import ApplicationError+ raise ApplicationError(+ f"Agent continuation requires {input_bytes} bytes; limit is {_CAN_INPUT_MAX_BYTES}. Reduce tool output or enable context compaction.",+ type="AgentContinuationTooLarge", non_retryable=True,+ )+ workflow.continue_as_new(args=[next_context])# The live transcript crosses the boundary directly.# Compaction keeps it token-bounded; the byte guard# below keeps a pathological transcript away from@@ -2651,10 +2852,11 @@ class AgentWorkflow:retry_policy=AGENT_ACTIVITY_RETRY,)if payload.get("employee_runtime_delivery") and workflow.patched("employee-task-manager-runtime-v2"):- await workflow.execute_activity("employee.job.deliver", {**context, "team_id": result_payload.get("team_id"),- "outputs": {**(context.get("outputs") or {}), agent_node_id: result_payload}},- activity_id="employee-reviewed-delivery", start_to_close_timeout=timedelta(days=31),- heartbeat_timeout=timedelta(minutes=2), retry_policy=AGENT_ACTIVITY_RETRY)+ async with self._execution_control.action():+ await workflow.execute_activity("employee.job.deliver", {**context, "team_id": result_payload.get("team_id"),+ "outputs": {**(context.get("outputs") or {}), agent_node_id: result_payload}},+ activity_id="employee-reviewed-delivery", start_to_close_timeout=timedelta(days=31),+ heartbeat_timeout=timedelta(minutes=2), retry_policy=AGENT_ACTIVITY_RETRY)# Final lifecycle broadcast — canvas glow goes green + FE# consumers of com.opencompany.agent.progress see phase="completed".diff --git a/server/services/temporal/employee_job_workflow.py b/server/services/temporal/employee_job_workflow.pyindex 3e5f408e..25140454 100644--- a/server/services/temporal/employee_job_workflow.py+++ b/server/services/temporal/employee_job_workflow.py@@ -5,6 +5,7 @@ from typing import Anyfrom temporalio import workflow, activityfrom temporalio.common import RetryPolicyfrom temporalio.exceptions import is_cancelled_exception+from services.temporal.execution_control import ExecutionControl@activity.defn(name="employee.job.deliver")@@ -45,10 +46,53 @@ async def fail_employee_job(context: dict[str, Any]) -> dict:@workflow.defn(name="EmployeeJobWorkflow", sandboxed=False)class EmployeeJobWorkflow:+ @workflow.init+ def __init__(self, context: dict[str, Any] | None = None):+ self._execution_control = ExecutionControl()+ if context is not None:+ self._execution_control.bind(context)++ @workflow.update+ async def set_control_state(self, payload: dict) -> dict:+ return await self._execution_control.set_control_state(payload)++ @set_control_state.validator+ def validate_set_control_state(self, payload: dict) -> None:+ self._execution_control.validate_control(payload)++ @workflow.update+ async def wait_for_checkpoint(self, payload: dict) -> dict:+ return await self._execution_control.wait_for_checkpoint(payload)++ @wait_for_checkpoint.validator+ def validate_wait_for_checkpoint(self, payload: dict) -> None:+ self._execution_control.validate_control(payload)++ @workflow.query+ def execution_control_status(self) -> dict:+ return self._execution_control.status()+@workflow.run- async def run(self, context: dict[str, Any]) -> dict:+ async def run(self, context: dict[str, Any] | None = None) -> dict:+ context = context or {}+ self._execution_control.bind(context)+ if not self._execution_control.enabled:+ return await self._run_job(context)+ await self._execution_control.register_root()+ try:+ return await self._run_job(context)+ finally:+ await self._execution_control.unregister_root()+ await workflow.wait_condition(workflow.all_handlers_finished)++ async def _run_job(self, context: dict[str, Any]) -> dict:try:- result = await workflow.execute_child_workflow("AgentWorkflow", context, id=workflow.info().workflow_id + ":lead")+ if self._execution_control.enabled:+ async with self._execution_control.child_start():+ handle = await workflow.start_child_workflow("AgentWorkflow", context, id=workflow.info().workflow_id + ":lead")+ result = await handle+ else:+ result = await workflow.execute_child_workflow("AgentWorkflow", context, id=workflow.info().workflow_id + ":lead")except BaseException as exc:# Cancellation must leave a durable terminal job record too.# Shield this short cleanup from the parent's cancellation.@@ -60,6 +104,11 @@ class EmployeeJobWorkflow:start_to_close_timeout=timedelta(seconds=60), retry_policy=RetryPolicy(maximum_attempts=3))return resultdelivery = {**context, "outputs": {**context.get("outputs", {}), context["node_id"]: result.get("result", {})}}+ if self._execution_control.enabled:+ async with self._execution_control.action():+ return await workflow.execute_activity("employee.job.deliver", delivery,+ start_to_close_timeout=timedelta(days=31), heartbeat_timeout=timedelta(minutes=2),+ retry_policy=RetryPolicy(maximum_attempts=3))return await workflow.execute_activity("employee.job.deliver", delivery,start_to_close_timeout=timedelta(days=31), heartbeat_timeout=timedelta(minutes=2),retry_policy=RetryPolicy(maximum_attempts=3))diff --git a/server/services/temporal/execution_control.py b/server/services/temporal/execution_control.pynew file mode 100644index 00000000..424c7ddd--- /dev/null+++ b/server/services/temporal/execution_control.py@@ -0,0 +1,229 @@+"""Cooperative admission and checkpoint state held in Temporal history.++Only versioned generations use this protocol. The workflow that owns this+helper exposes the Update/Query methods; the helper never installs handlers or+intercepts SDK commands. An action includes consumption of its recorded result,+while a child start includes only its start acknowledgement, never child result.+"""++from contextlib import asynccontextmanager+from datetime import timedelta+from typing import Any++from temporalio import workflow+from temporalio.exceptions import ApplicationError++from services.temporal._retry_policies import DEFAULT_ACTIVITY_RETRY+++class ExecutionControl:+ def __init__(self) -> None:+ self.version = 0+ self.state = "running"+ self.revision = -1+ self.producers_held = False+ self.generation: str | None = None+ self._generation_value: Any = None+ self.controller_workflow_id: str | None = None+ self.root_run_id: str | None = None+ self._actions = 0+ self._child_starts = 0+ self._bound = False+ self._control_initialized = False+ self._released_revision = -1++ @property+ def enabled(self) -> bool:+ return self.version == 1++ def bind(self, data: dict[str, Any]) -> None:+ if self._bound:+ return+ self._bound = True+ self.version = int(data.get("execution_control_version") or 0)+ if not self.enabled:+ return+ self.controller_workflow_id = data.get("controller_workflow_id")+ generation = data.get("generation")+ self._generation_value = generation+ self.generation = str(generation) if generation is not None else None+ self.root_run_id = data.get("execution_control_root_run_id")+ self._released_revision = int(data.get("execution_control_released_revision", -1))+ self._apply({+ "state": data.get("execution_control_state", "running"),+ "revision": data.get("execution_control_revision", -1),+ "producers_held": data.get("execution_control_producers_held", False),+ })++ def validate_control(self, payload: dict[str, Any] | None) -> None:+ """Reject malformed control messages without changing Workflow state.++ Update validators use this before acceptance. Handlers also call it+ because validators do not execute during replay or direct helper calls.+ An omitted checkpoint payload means only wait for the existing intent.+ """+ if not self.enabled or payload is None:+ return+ if not isinstance(payload, dict):+ raise ApplicationError("Control request must be a mapping", type="InvalidWorkflowControlRequest", non_retryable=True)+ generation = payload.get("generation")+ if generation is not None and self.generation != str(generation):+ raise ApplicationError("Execution generation does not match", type="StaleExecutionGeneration", non_retryable=True)+ first_run_id = payload.get("first_execution_run_id")+ if first_run_id is not None and str(first_run_id) != workflow.info().first_execution_run_id:+ raise ApplicationError("Execution chain identity has changed", type="ExecutionChainConflict", non_retryable=True)+ if payload.get("state") not in ("paused", "running"):+ raise ApplicationError("Control state must be paused or running", type="InvalidWorkflowControlState", non_retryable=True)+ try:+ int(payload.get("revision", -1))+ except (TypeError, ValueError, OverflowError) as exc:+ raise ApplicationError("Control revision must be an integer", type="InvalidWorkflowControlRevision", non_retryable=True) from exc+ if "producers_held" in payload and not isinstance(payload["producers_held"], bool):+ raise ApplicationError("Producer admission hold must be a boolean", type="InvalidWorkflowControlRequest", non_retryable=True)++ def _apply(self, payload: dict[str, Any], *, explicit_release: bool = False) -> None:+ if not self.enabled:+ return+ if payload is None:+ raise ApplicationError("Control request must be a mapping", type="InvalidWorkflowControlRequest", non_retryable=True)+ self.validate_control(payload)+ state = payload["state"]+ revision = int(payload.get("revision", -1))+ # The initial input may use -1. Once a revision is observed, delayed+ # registration responses and duplicate messages cannot undo it.+ if not self._control_initialized or revision > self.revision:+ self.state, self.revision = state, revision+ self.producers_held = bool(payload.get("producers_held", False))+ self._control_initialized = True+ elif revision == self.revision and state == self.state and "producers_held" in payload:+ # Only an acknowledged release finishes this revision's phase.+ # A new root may inherit unheld input while the controller is held;+ # enrollment must still hold it before admitting business work.+ held = bool(payload["producers_held"])+ if not held or self._released_revision < revision:+ self.producers_held = held+ if explicit_release and revision == self.revision and state == self.state == "running" and payload.get("producers_held") is False:+ self._released_revision = max(self._released_revision, revision)++ async def set_control_state(self, payload: dict[str, Any], *, explicit_release: bool = True) -> dict[str, Any]:+ self._apply(payload, explicit_release=explicit_release)+ if self.enabled and (self.state == "paused" or self.producers_held) and self._child_starts:+ await workflow.wait_condition(lambda: self._child_starts == 0 or (self.state == "running" and not self.producers_held))+ return self.status()++ async def wait_for_checkpoint(self, payload: dict[str, Any] | None = None) -> dict[str, Any]:+ if payload is not None:+ self._apply(payload)+ if self.enabled and (self._actions or self._child_starts):+ await workflow.wait_condition(lambda: (self._actions == 0 and self._child_starts == 0) or (self.state == "running" and not self.producers_held))+ return self.status()++ async def wait_until_running(self) -> None:+ if self.enabled and (self.state != "running" or self.producers_held):+ await workflow.wait_condition(lambda: self.state == "running" and not self.producers_held)++ async def begin_action(self) -> None:+ await self.wait_until_running()+ if self.enabled:+ self._actions += 1++ def begin_bookkeeping(self) -> None:+ """Consume completed child results while admission remains closed."""+ if self.enabled:+ self._actions += 1++ def end_action(self) -> None:+ if self.enabled:+ self._actions -= 1+ if self._actions < 0:+ raise RuntimeError("Unbalanced execution action accounting")++ async def begin_child_start(self) -> None:+ await self.wait_until_running()+ if self.enabled:+ self._child_starts += 1++ def end_child_start(self) -> None:+ if self.enabled:+ self._child_starts -= 1+ if self._child_starts < 0:+ raise RuntimeError("Unbalanced child-start accounting")++ @asynccontextmanager+ async def action(self):+ await self.begin_action()+ try:+ yield+ finally:+ self.end_action()++ @asynccontextmanager+ async def child_start(self):+ await self.begin_child_start()+ try:+ yield+ finally:+ self.end_child_start()++ def status(self) -> dict[str, Any]:+ return {+ "execution_control_version": self.version,+ "generation": self._generation_value,+ "state": self.state,+ "revision": self.revision,+ "producers_held": self.producers_held,+ "active_actions": self._actions,+ "pending_child_starts": self._child_starts,+ "checkpoint": self._actions == 0 and self._child_starts == 0,+ }++ def carry(self) -> dict[str, Any]:+ if not self.enabled:+ return {}+ return {+ "execution_control_version": self.version,+ "controller_workflow_id": self.controller_workflow_id,+ "generation": self._generation_value,+ "execution_control_state": self.state,+ "execution_control_revision": self.revision,+ "execution_control_producers_held": self.producers_held,+ "execution_control_root_run_id": self.root_run_id,+ "execution_control_released_revision": self._released_revision,+ }++ def _registration(self) -> dict[str, Any]:+ if not self.controller_workflow_id or self.generation is None:+ raise ApplicationError("Controlled execution requires controller identity and generation", type="MissingExecutionController", non_retryable=True)+ info = workflow.info()+ self.root_run_id = self.root_run_id or info.first_execution_run_id+ return {+ "controller_workflow_id": self.controller_workflow_id,+ "generation": self._generation_value,+ "workflow_id": info.workflow_id,+ "first_execution_run_id": self.root_run_id,+ }++ async def register_root(self) -> None:+ if not self.enabled or self.root_run_id is not None:+ return+ payload = self._registration()+ result = await workflow.execute_activity(+ "execution_control.register", payload,+ start_to_close_timeout=timedelta(seconds=60),+ retry_policy=DEFAULT_ACTIVITY_RETRY,+ )+ await self.set_control_state({+ "state": result["participant_state"],+ "revision": result["participant_revision"],+ "generation": self.generation,+ "producers_held": result.get("producers_held", False),+ }, explicit_release=False)++ async def unregister_root(self) -> None:+ if not self.enabled:+ return+ await workflow.execute_activity(+ "execution_control.unregister", self._registration(),+ start_to_close_timeout=timedelta(seconds=60),+ retry_policy=DEFAULT_ACTIVITY_RETRY,+ )diff --git a/server/services/temporal/execution_control_activities.py b/server/services/temporal/execution_control_activities.pynew file mode 100644index 00000000..7e095fc5--- /dev/null+++ b/server/services/temporal/execution_control_activities.py@@ -0,0 +1,30 @@+"""Client-side bridges for acknowledged execution-root enrollment.++The worker injects its existing Temporal client. Activities perform no provider+work, and retries are safe because enrollment is identified by execution chain.+"""++from typing import Any++from temporalio import activity+from temporalio.client import Client+++class ExecutionControlActivities:+ def __init__(self, client: Client) -> None:+ self.client = client++ @activity.defn(name="execution_control.register")+ async def register(self, payload: dict[str, Any]) -> dict[str, Any]:+ return await self.client.get_workflow_handle(payload["controller_workflow_id"]).execute_update(+ "register_execution", payload,+ )++ @activity.defn(name="execution_control.unregister")+ async def unregister(self, payload: dict[str, Any]) -> dict[str, Any]:+ return await self.client.get_workflow_handle(payload["controller_workflow_id"]).execute_update(+ "unregister_execution", payload,+ )++ def activities(self) -> list:+ return [self.register, self.unregister]diff --git a/server/services/temporal/executor.py b/server/services/temporal/executor.pyindex 42407105..d632666a 100644--- a/server/services/temporal/executor.py+++ b/server/services/temporal/executor.py@@ -142,6 +142,15 @@ class TemporalExecutor:"root_execution_id": execution_id,})+ from core.container import container+ control = await container.database().get_latest_workflow_control(workflow_id)+ if control is not None and control.generation == int(generation):+ workflow_payload.update({+ "root_execution_id": control.root_execution_id,+ "data_scope_id": control.data_scope_id or control.execution_id,+ "controller_workflow_id": control.controller_workflow_id,+ "execution_control_version": int((control.resource_manifest or {}).get("execution_control_version", 0)),+ })result = await self.client.execute_workflow(MachinaWorkflow.run,diff --git a/server/services/temporal/trigger_listener_workflow.py b/server/services/temporal/trigger_listener_workflow.pyindex f6006af0..1b586978 100644--- a/server/services/temporal/trigger_listener_workflow.py+++ b/server/services/temporal/trigger_listener_workflow.py@@ -306,6 +306,7 @@ class TriggerListenerWorkflow:listener_data: Dict[str, Any],admission_check=None,search_attributes: Optional[TypedSearchAttributes] = None,+ child_start_control: Any = None,) -> None:"""Build the filtered downstream graph + start a child MachinaWorkflow.@@ -446,6 +447,11 @@ class TriggerListenerWorkflow:"root_execution_id": listener_data.get("root_execution_id"),"data_scope_id": listener_data.get("data_scope_id"),}+ if listener_data.get("execution_control_version") == 1:+ child_payload.update({key: listener_data[key] for key in (+ "execution_control_version", "controller_workflow_id", "execution_control_state",+ "execution_control_revision", "execution_control_producers_held",+ ) if key in listener_data})frozen_routing = listener_data.get(TEMPORAL_ROUTING_INPUT_KEY)if isinstance(frozen_routing, dict):child_payload[TEMPORAL_ROUTING_INPUT_KEY] = dict(@@ -472,11 +478,11 @@ class TriggerListenerWorkflow:if listener_data.get("parameter_snapshot"):child_payload["parameter_snapshot"] = listener_data["parameter_snapshot"]- await workflow.start_child_workflow(- "MachinaWorkflow",- args=[child_payload],- **child_options,- )+ if child_start_control is not None:+ async with child_start_control.child_start():+ await workflow.start_child_workflow("MachinaWorkflow", args=[child_payload], **child_options)+ else:+ await workflow.start_child_workflow("MachinaWorkflow", args=[child_payload], **child_options)workflow.logger.info(f"TriggerListener spawned child run: child_id={child_id} " f"event.id={event.get('id')} event.type={event.get('type')}"diff --git a/server/services/temporal/worker.py b/server/services/temporal/worker.pyindex efba8606..8c2a4d0f 100644--- a/server/services/temporal/worker.py+++ b/server/services/temporal/worker.py@@ -37,6 +37,7 @@ from .activities import (NodeExecutionActivities,create_shared_session,)+from .execution_control_activities import ExecutionControlActivities@@ -283,6 +284,7 @@ class TemporalWorkerManager:workflows=_framework_workflows(),activities=[self._activities.execute_node_activity,+ *ExecutionControlActivities(self.client).activities(),broadcast_trigger_status_activity,evaluate_trigger_filter_activity,load_persisted_workflow_graph_activity,@@ -777,6 +779,7 @@ async def run_standalone_worker(workflows=_framework_workflows(),activities=[activities.execute_node_activity,+ *ExecutionControlActivities(client).activities(),broadcast_trigger_status_activity,evaluate_trigger_filter_activity,load_persisted_workflow_graph_activity,@@ -864,6 +867,7 @@ async def create_worker(workflows=_framework_workflows(),activities=[activities.execute_node_activity,+ *ExecutionControlActivities(client).activities(),broadcast_trigger_status_activity,evaluate_trigger_filter_activity,load_persisted_workflow_graph_activity,diff --git a/server/services/temporal/workflow.py b/server/services/temporal/workflow.pyindex 6b1e4019..364a009d 100644--- a/server/services/temporal/workflow.py+++ b/server/services/temporal/workflow.py@@ -17,7 +17,7 @@ from typing import Any, Dict, List, Optional, Setfrom temporalio import workflowfrom ._retry_policies import DEFAULT_ACTIVITY_RETRY, QUICK_ACTIVITY_RETRY-from services.workflow_naming import node_label_slug+from .execution_control import ExecutionControl# ``conditions`` is pure -- ``re`` + comparisons, no IO, no clock, no# randomness -- so it is safe to evaluate inside a workflow.@@ -215,7 +215,8 @@ class MachinaWorkflow:deterministically from Event History on replay)."""- def __init__(self) -> None:+ @workflow.init+ def __init__(self, workflow_data: Optional[Dict[str, Any]] = None) -> None:# Wave 12 A7: per-run event-framework state. Populated by the# ``on_event`` signal handler; consumed by Phase C1's# ``wait_condition`` predicate in the trigger-waiter rewrite.@@ -224,17 +225,48 @@ class MachinaWorkflow:self._seen_event_ids: Set[str] = set()self._matched_events: List[Dict[str, Any]] = []self._control_paused = False+ self._execution_control = ExecutionControl()+ if workflow_data is not None:+ self._execution_control.bind(workflow_data)+ self._action_contexts: Dict[str, Any] = {}@workflow.signalasync def pause(self) -> None:"""Cooperatively gate new node/activity scheduling."""+ if self._execution_control.enabled:+ returnself._control_paused = True@workflow.signalasync def resume(self) -> None:+ if self._execution_control.enabled:+ returnself._control_paused = False+ @workflow.update+ async def set_control_state(self, payload: Dict[str, Any]) -> Dict[str, Any]:+ return await self._execution_control.set_control_state(payload)++ @set_control_state.validator+ def validate_set_control_state(self, payload: Dict[str, Any]) -> None:+ self._execution_control.validate_control(payload)++ @workflow.update+ async def wait_for_checkpoint(self, payload: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:+ return await self._execution_control.wait_for_checkpoint(payload)++ @wait_for_checkpoint.validator+ def validate_wait_for_checkpoint(self, payload: Optional[Dict[str, Any]] = None) -> None:+ self._execution_control.validate_control(payload)++ @workflow.query+ def execution_control_status(self) -> Dict[str, Any]:+ return self._execution_control.status()+async def _wait_until_resumed(self) -> None:+ if self._execution_control.enabled:+ await self._execution_control.wait_until_running()+ returnif self._control_paused:await workflow.wait_condition(lambda: not self._control_paused)@@ -462,7 +494,22 @@ class MachinaWorkflow:return None@workflow.run- async def run(self, workflow_data: Dict[str, Any]) -> Dict[str, Any]:+ async def run(self, workflow_data: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:+ workflow_data = workflow_data or {}+ self._execution_control.bind(workflow_data)+ if not self._execution_control.enabled:+ return await self._run_graph(workflow_data)+ await self._execution_control.register_root()+ try:+ return await self._run_graph(workflow_data)+ finally:+ for admission in self._action_contexts.values():+ await admission.__aexit__(None, None, None)+ self._action_contexts.clear()+ await self._execution_control.unregister_root()+ await workflow.wait_condition(workflow.all_handlers_finished)++ async def _run_graph(self, workflow_data: Dict[str, Any]) -> Dict[str, Any]:"""Execute workflow by orchestrating node activities.Args:@@ -597,10 +644,18 @@ class MachinaWorkflow:# Avoid emitting a redundant wait command on the normal running# path. Once paused, the condition is durable and replay-safe;# the resume signal flips the flag and releases scheduling.- if self._control_paused:+ if self._control_paused and not self._execution_control.enabled:await workflow.wait_condition(lambda: not self._control_paused)# Find ready nodes (all deps completed, not running/completed)ready = self._find_ready_nodes(deps, completed, running, node_map)+ if self._execution_control.enabled and errors:+ ready = []+ if self._execution_control.enabled and self._execution_control.status()["state"] != "running" and running:+ # Results and graph bookkeeping settle even when admission is+ # closed. Waiting for Resume here would deadlock checkpoint.+ ready = []+ elif self._execution_control.enabled and self._execution_control.status().get("producers_held") and running:+ ready = []workflow.logger.debug(f"Loop {loop_count}: ready={len(ready)}, running={len(running)}, completed={len(completed)}")# Triggers auto-completed by the skip branch below unblock their@@ -662,6 +717,9 @@ class MachinaWorkflow:"pre_executed": node.get("_pre_executed", False),"trigger_output": node.get("_trigger_output"),}+ if self._execution_control.enabled:+ context.update({key: value for key, value in self._execution_control.carry().items()+ if key not in {"execution_control_root_run_id", "execution_control_released_revision"}})if (graph_version >= AGENT_CONTEXT_GRAPH_VERSIONand generation > 0@@ -714,6 +772,10 @@ class MachinaWorkflow:# A preceding child start yields to the workflow event loop,# so a pause signal may have landed since ``ready`` was# computed. Re-admit every command in the batch.+ if self._execution_control.enabled and running and (+ self._execution_control.state != "running" or self._execution_control.producers_held+ ):+ breakawait self._wait_until_resumed()if dispatch["kind"] == "child_workflow":# ``workflow.start_child_workflow`` is ``async def`` —@@ -738,10 +800,11 @@ class MachinaWorkflow:args=[context],id=child_workflow_id,)- handle = await workflow.start_child_workflow(- dispatch["name"],- **child_start_kwargs,- )+ if self._execution_control.enabled:+ async with self._execution_control.child_start():+ handle = await workflow.start_child_workflow(dispatch["name"], **child_start_kwargs)+ else:+ handle = await workflow.start_child_workflow(dispatch["name"], **child_start_kwargs)running[node_id] = handleworkflow.logger.info(f"Scheduled child workflow for node: {node_id} " f"(workflow={dispatch['name']})")else:@@ -778,6 +841,10 @@ class MachinaWorkflow:if dispatch.get("queue") is not None:start_kwargs["task_queue"] = dispatch["queue"]+ if self._execution_control.enabled:+ admission = self._execution_control.action()+ await admission.__aenter__()+ self._action_contexts[node_id] = admissionhandle = workflow.start_activity(dispatch["name"], **start_kwargs)running[node_id] = handleworkflow.logger.info(@@ -811,10 +878,16 @@ class MachinaWorkflow:}errors.append(error_info)workflow.logger.error(f"Node failed: {done_id} - {error_info['error']}")+ if done_id in self._action_contexts:+ await self._action_contexts.pop(done_id).__aexit__(None, None, None)# Stop workflow on failure# TODO: Could add option to continue with partial results- break+ if not self._execution_control.enabled:+ break++ if done_id in self._action_contexts:+ await self._action_contexts.pop(done_id).__aexit__(None, None, None)# Build final resultsuccess = len(errors) == 0 and len(completed) == len(node_map)diff --git a/server/services/temporal/workflow_control_workflow.py b/server/services/temporal/workflow_control_workflow.pyindex b8fa3f59..f3482344 100644--- a/server/services/temporal/workflow_control_workflow.py+++ b/server/services/temporal/workflow_control_workflow.py@@ -2,13 +2,14 @@import asynciofrom datetime import timedelta-from typing import Any, Dict, List, Optional, Tuple+from typing import Any, Dict, Optionalfrom temporalio import workflowfrom temporalio.common import SearchAttributeKey, SearchAttributePair-from temporalio.exceptions import ApplicationError+from temporalio.exceptions import ApplicationError, WorkflowAlreadyStartedErrorfrom services.temporal._retry_policies import DEFAULT_ACTIVITY_RETRY+from services.temporal.execution_control import ExecutionControl# The controller is the ONE workflow expected to live for months, and it@@ -34,6 +35,11 @@ _MAX_CARRIED_SEEN_IDS = 4_096# (PollingTriggerNode.poll_interval_clamp) which the legacy asyncio path# applies but workflow payloads historically did not._MIN_POLL_INTERVAL_S = 30+_CAN_INPUT_MAX_BYTES = 1_900_000+++class _AdmissionDeferred(Exception):+ """An accepted event remains queued while controller admission is closed."""@workflow.defn(name="WorkflowControlWorkflow", sandboxed=False)@@ -53,7 +59,8 @@ class WorkflowControlWorkflow:run id."""- def __init__(self) -> None:+ @workflow.init+ def __init__(self, control_data: Dict[str, Any] = None) -> None:self._state = "running"self._revision = 0self._closed = False@@ -62,6 +69,13 @@ class WorkflowControlWorkflow:# Insertion-ordered so the carry across continue-as-new can keep# the newest entries when trimming to _MAX_CARRIED_SEEN_IDS.self._seen_event_ids: dict[str, None] = {}+ # Pending items (including the one whose child start is in progress)+ # must remain deduplicated even after the bounded recent-ID cache ages+ # out. This index is rebuilt from the carried queue and durable pages;+ # it never adds an unbounded list of IDs to continuation inputs.+ self._pending_event_ids: set[str] = set()+ self._event_accumulator_v1: Optional[bool] = None+ self._messaging_v1: Optional[bool] = Noneself._poll_tasks: dict[str, asyncio.Task] = {}self._can_requested = Falseself._overflow = False@@ -69,23 +83,52 @@ class WorkflowControlWorkflow:self._review_overflow_checked = Falseself._queue_reads = 0self._drain_tasks = False+ self._execution_control = ExecutionControl()+ self._live_roots: dict[str, Dict[str, Any]] = {}+ self._membership_epoch = 0+ self._carried_state_seeded = False+ if control_data is not None and int(control_data.get("execution_control_version") or 0) == 1:+ self._state = control_data.get("state", "running")+ self._seed_carried_state(control_data, start_runtime=False)@workflow.signalasync def pause(self) -> None:+ if self._execution_control.enabled:+ returnself._apply_control_state("paused")@workflow.signalasync def resume(self) -> None:+ if self._execution_control.enabled:+ returnself._apply_control_state("running")@workflow.update- async def set_control_state(self, requested_state: str) -> Dict[str, Any]:+ async def set_control_state(self, requested_state: str | Dict[str, Any]) -> Dict[str, Any]:"""Idempotently apply and acknowledge a pause/resume transition.Updates give callers a durable acknowledgement that the controllerprocessed the request. The legacy signals above remain registered forolder callers and histories."""+ if isinstance(requested_state, dict):+ self.validate_control_state(requested_state)+ expected_epoch = self._expected_membership_epoch(requested_state)+ if requested_state.get("producers_held") is False and expected_epoch is not None and expected_epoch != self._membership_epoch:+ self._maybe_request_rollover_from_update()+ return self.status()+ previous_revision = self._execution_control.revision+ if "hold_admissions" in requested_state and "producers_held" not in requested_state:+ requested_state = {**requested_state, "producers_held": requested_state["hold_admissions"]}+ self._execution_control._apply(requested_state)+ if int(requested_state.get("revision", -1)) >= previous_revision:+ target = self._execution_control.state+ self._state = "paused" if self._execution_control.producers_held else target+ self._revision = self._execution_control.revision+ self._drain_tasks = False+ await self._execution_control.set_control_state(requested_state)+ self._maybe_request_rollover_from_update()+ return self.status()normalized = str(requested_state).strip().lower()aliases = {"pause": "paused",@@ -100,14 +143,124 @@ class WorkflowControlWorkflow:type="InvalidWorkflowControlState",non_retryable=True,)+ if self._messaging_enabled() and target_state == "running" and (+ self._closed or self._execution_control.state != "running" or self._execution_control.producers_held+ ):+ # safe_apply restores producer admission with this legacy string+ # Update. It cannot release a whole-generation Stop or the held+ # phase of Resume, whose revisioned Update owns that transition.+ self._apply_control_state("paused")+ self._drain_tasks = False+ self._maybe_request_rollover_from_update()+ return self.status()self._apply_control_state(target_state)self._drain_tasks = False+ self._maybe_request_rollover_from_update()+ return self.status()++ @set_control_state.validator+ def validate_control_state(self, requested_state: str | Dict[str, Any]) -> None:+ if isinstance(requested_state, dict):+ if not self._execution_control.enabled:+ raise ApplicationError("Controller does not support execution control v1", type="UnsupportedExecutionControlVersion", non_retryable=True)+ if "hold_admissions" in requested_state and "producers_held" not in requested_state:+ requested_state = {**requested_state, "producers_held": requested_state["hold_admissions"]}+ self._execution_control.validate_control(requested_state)+ self._expected_membership_epoch(requested_state)+ elif str(requested_state).strip().lower() not in {"pause", "paused", "resume", "running"}:+ raise ApplicationError(+ "Control state must be one of: pause, paused, resume, running",+ type="InvalidWorkflowControlState", non_retryable=True,+ )++ @staticmethod+ def _expected_membership_epoch(payload: Dict[str, Any]) -> Optional[int]:+ value = payload.get("expected_membership_epoch")+ if value is None:+ return None+ try:+ return int(value)+ except (TypeError, ValueError, OverflowError) as exc:+ raise ApplicationError(+ "Expected membership epoch must be an integer",+ type="InvalidExecutionMembershipEpoch", non_retryable=True,+ ) from exc++ @workflow.update+ async def wait_for_checkpoint(self, payload: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:+ self._execution_control.validate_control(payload)+ await self._execution_control.wait_for_checkpoint(payload)+ self._maybe_request_rollover_from_update()+ return self.status()++ @wait_for_checkpoint.validator+ def validate_checkpoint(self, payload: Optional[Dict[str, Any]] = None) -> None:+ self._execution_control.validate_control(payload)++ @workflow.query+ def execution_control_status(self) -> Dict[str, Any]:+ return self._execution_control.status()++ def _check_registration(self, registration: Dict[str, Any]) -> tuple[str, str]:+ if not self._execution_control.enabled:+ raise ApplicationError("Controller does not support execution control v1", type="UnsupportedExecutionControlVersion", non_retryable=True)+ if self._closed:+ raise ApplicationError("Execution controller is closing", type="ExecutionControllerClosed", non_retryable=True)+ if not isinstance(registration, dict):+ raise ApplicationError("Execution registration must be an object", type="InvalidExecutionRegistration", non_retryable=True)+ if str(registration.get("generation")) != self._execution_control.generation:+ raise ApplicationError("Execution generation does not match", type="StaleExecutionGeneration", non_retryable=True)+ workflow_id = str(registration.get("workflow_id") or "")+ first_run_id = str(registration.get("first_execution_run_id") or "")+ if not workflow_id or not first_run_id:+ raise ApplicationError("Execution chain identity is required", type="InvalidExecutionRegistration", non_retryable=True)+ return workflow_id, first_run_id++ @workflow.update+ async def register_execution(self, registration: Dict[str, Any]) -> Dict[str, Any]:+ workflow_id, first_run_id = self._check_registration(registration)+ existing = self._live_roots.get(workflow_id)+ if existing and existing["first_execution_run_id"] != first_run_id:+ raise ApplicationError("Execution root identity has changed", type="ExecutionChainConflict", non_retryable=True)+ if not existing:+ self._live_roots[workflow_id] = {+ "workflow_id": workflow_id,+ "first_execution_run_id": first_run_id,+ }+ self._membership_epoch += 1+ self._maybe_request_rollover_from_update()+ return self.status()++ @register_execution.validator+ def validate_registration(self, registration: Dict[str, Any]) -> None:+ workflow_id, first_run_id = self._check_registration(registration)+ existing = self._live_roots.get(workflow_id)+ if existing and existing["first_execution_run_id"] != first_run_id:+ raise ApplicationError("Execution root identity has changed", type="ExecutionChainConflict", non_retryable=True)++ @workflow.update+ async def unregister_execution(self, registration: Dict[str, Any]) -> Dict[str, Any]:+ workflow_id, first_run_id = self._check_registration(registration)+ existing = self._live_roots.get(workflow_id)+ if existing and existing["first_execution_run_id"] == first_run_id:+ del self._live_roots[workflow_id]+ self._membership_epoch += 1+ self._maybe_request_rollover_from_update()return self.status()+ @unregister_execution.validator+ def validate_unregistration(self, registration: Dict[str, Any]) -> None:+ self._check_registration(registration)+@workflow.updateasync def pause_admissions(self, drain_tasks: bool = True) -> Dict[str, Any]:self._apply_control_state("paused")self._drain_tasks = drain_tasks+ if self._messaging_enabled() and (+ self._closed or self._execution_control.state != "running" or self._execution_control.producers_held+ ):+ self._drain_tasks = False+ self._maybe_request_rollover_from_update()return self.status()def _apply_control_state(self, target_state: str) -> None:@@ -133,8 +286,7 @@ class WorkflowControlWorkflow:Existing run children already hold their own snapshot. Pausing onlyadmissions lets those children finish before this update is called."""- if self._state != "paused":- raise ApplicationError("Pause admissions before applying changes", non_retryable=True)+ self.validate_graph(graph)node_ids = {node["id"] for node in graph["nodes"]}for listener_id, spec in self._triggers.items():listener = spec.get("listener_args", {})@@ -153,8 +305,23 @@ class WorkflowControlWorkflow:listener["parameter_snapshot"] = graph.get("parameters") or {}listener["graphVersion"] = graph.get("graphVersion", 2)self._revision += 1+ self._maybe_request_rollover_from_update()return {**self.status(), "applied": True}+ @replace_graph.validator+ def validate_graph(self, graph: Dict[str, Any]) -> None:+ if self._state != "paused":+ raise ApplicationError("Pause admissions before applying changes", non_retryable=True)+ if not isinstance(graph, dict) or not isinstance(graph.get("nodes"), list) or not isinstance(graph.get("edges"), list):+ raise ApplicationError("Graph snapshot requires nodes and edges lists", type="InvalidWorkflowGraphSnapshot", non_retryable=True)+ for node in graph["nodes"]:+ if not isinstance(node, dict) or "id" not in node:+ raise ApplicationError("Each graph node requires an id", type="InvalidWorkflowGraphSnapshot", non_retryable=True)+ try:+ hash(node["id"])+ except TypeError as exc:+ raise ApplicationError("Graph node ids must be hashable", type="InvalidWorkflowGraphSnapshot", non_retryable=True) from exc+@workflow.signalasync def register_trigger(self, spec: Dict[str, Any]) -> None:listener_id = str(spec["listener_id"])@@ -180,9 +347,11 @@ class WorkflowControlWorkflow:if event_type not in set(spec.get("event_types") or [spec.get("event_type")]):continuededup_key = f"{listener_id}:{event_id}"- if dedup_key not in self._seen_event_ids:+ if not self._event_already_accepted(dedup_key):self._remember_event_id(dedup_key)self._events.append((listener_id, event))+ if self._event_accumulator_v1:+ self._pending_event_ids.add(dedup_key)# A paused controller still receives matching signals; without this# check its history could overflow mid-pause with the run loop# parked. The flag wakes the run loop, which rolls over carrying@@ -199,20 +368,33 @@ class WorkflowControlWorkflow:@workflow.querydef status(self) -> Dict[str, Any]:- return {+ result = {"state": self._state, "revision": self._revision,"triggers": {key: value["trigger_node_id"] for key, value in self._triggers.items()},"queued_events": len(self._events),}+ if self._execution_control.enabled:+ result.update({+ "execution_control_version": 1,+ "participant_state": self._execution_control.state,+ "participant_revision": self._execution_control.revision,+ "producers_held": self._execution_control.producers_held,+ "live_roots": {key: dict(value) for key, value in self._live_roots.items()},+ "membership_epoch": self._membership_epoch,+ "active_actions": self._execution_control.status()["active_actions"],+ "pending_child_starts": self._execution_control.status()["pending_child_starts"],+ "checkpoint": self._execution_control.status()["checkpoint"],+ })+ return result@workflow.run- async def run(self, control_data: Dict[str, Any]) -> Dict[str, Any]:- self._state = control_data.get("state", "running")+ async def run(self, control_data: Dict[str, Any] = None) -> Dict[str, Any]:+ control_data = control_data or {}+ if not self._carried_state_seeded:+ self._state = control_data.get("state", "running")self._seed_carried_state(control_data)- self._overflow = bool(control_data.get("overflow"))- self._durable_prefix = int(control_data.get("durable_prefix", len(self._events)))queue_v2 = workflow.patched("controller-durable-queue-v2")- self._drain_tasks = bool(control_data.get("drain_tasks"))+ accumulator_v1 = self._accumulator_enabled()while not self._closed:read_reviews = queue_v2 and self._state == "paused" and self._drain_tasks and self._next_event() is None and not self._review_overflow_checkedif self._overflow and ((queue_v2 and self._state == "running" and self._durable_prefix == 0) or read_reviews or (not queue_v2 and not self._events)):@@ -224,6 +406,21 @@ class WorkflowControlWorkflow:start_to_close_timeout=timedelta(seconds=60), retry_policy=DEFAULT_ACTIVITY_RETRY)incoming = [(str(item[0]), item[1]) for item in page["events"]]if queue_v2:+ if accumulator_v1:+ # A retry whose recent key aged out may already be in+ # the live tail. Keep the older durable item's FIFO+ # position rather than dispatching both copies.+ incoming_ids = {self._event_key(*item) for item in incoming}+ removed_prefix = sum(+ self._event_key(*item) in incoming_ids+ for item in self._events[:self._durable_prefix]+ )+ self._events = [+ item for item in self._events+ if self._event_key(*item) not in incoming_ids+ ]+ self._durable_prefix -= removed_prefix+ self._pending_event_ids.update(incoming_ids)self._events[0:0] = incomingself._durable_prefix += len(incoming)if read_reviews:@@ -245,53 +442,87 @@ class WorkflowControlWorkflow:# Unreachable in a real run; direct unit invocation returns.breakselected_index = self._next_event() or 0+ was_durable = queue_v2 and selected_index < self._durable_prefixlistener_id, event = self._events.pop(selected_index)- if queue_v2 and selected_index < self._durable_prefix:+ if was_durable:self._durable_prefix -= 1spec = self._triggers.get(listener_id)if spec is None:+ if accumulator_v1:+ self._finish_pending_event(listener_id, event)continue+ accepted = Falsetry:await self._spawn_push_run(event, spec)+ accepted = True+ except _AdmissionDeferred:+ self._events.insert(selected_index, (listener_id, event))+ if was_durable:+ self._durable_prefix += 1except asyncio.CancelledError:raiseexcept Exception as exc: # noqa: BLE001 — per-event isolation# One bad event (duplicate child id, exhausted broadcast# retries) must not fail the deployment's entire control# plane. Same isolation contract as the listener classes.- workflow.logger.error(- f"Controller push spawn failed for event.id={event.get('id')}: {exc}"- )- if queue_v2:+ if accumulator_v1 and isinstance(exc, WorkflowAlreadyStartedError):+ # The deterministic child ID is the durable dispatch key.+ # Temporal has already accepted this event's child start;+ # retrying it forever would block all following events.+ accepted = True+ else:+ workflow.logger.error(+ f"Controller push spawn failed for event.id={event.get('id')}: {exc}"+ )+ if queue_v2 and not accepted:self._events.insert(0, (listener_id, event))self._durable_prefix += 1await workflow.sleep(timedelta(seconds=5))+ if accumulator_v1 and accepted:+ self._finish_pending_event(listener_id, event)self._maybe_request_rollover()return {"state": self._state, "generation": control_data.get("generation")}# ---- continue-as-new machinery ------------------------------------- def _seed_carried_state(self, control_data: Dict[str, Any]) -> None:- """Rehydrate rollover-carried state; no-op for first-run payloads.+ def _seed_carried_state(self, control_data: Dict[str, Any], *, start_runtime: bool = True) -> None:+ """Restore snapshots once, then start runtime pollers from the run body.- Everything here is command-free, so pre-patch histories (whose- payloads lack the carried keys) replay identically.+ Versioned initialization uses only the command-free snapshot portion;+ legacy runs retain their original startup command order."""- self._revision = int(control_data.get("revision") or 0)- for seen_id in control_data.get("seen_event_ids") or []:- self._seen_event_ids[str(seen_id)] = None- for pending in control_data.get("pending_events") or []:- try:- listener_id, event = pending[0], pending[1]- except (IndexError, TypeError, KeyError):- continue- self._events.append((str(listener_id), event))- for listener_id, spec in (control_data.get("triggers") or {}).items():- listener_id = str(listener_id)- if listener_id in self._triggers:- continue- self._triggers[listener_id] = spec- if spec.get("workflow_type") == "PollingTriggerWorkflow":+ if not self._carried_state_seeded:+ self._revision = int(control_data.get("revision") or 0)+ self._execution_control.bind({+ **control_data,+ "execution_control_state": control_data.get("participant_state", control_data.get("state", "running")),+ "execution_control_revision": control_data.get("participant_revision", self._revision),+ })+ self._live_roots = {key: dict(value) for key, value in (control_data.get("live_roots") or {}).items()}+ self._membership_epoch = int(control_data.get("membership_epoch") or 0)+ for seen_id in control_data.get("seen_event_ids") or []:+ self._seen_event_ids[str(seen_id)] = None+ for pending in control_data.get("pending_events") or []:+ try:+ listener_id, event = pending[0], pending[1]+ except (IndexError, TypeError, KeyError):+ continue+ self._events.append((str(listener_id), event))+ # Initialization must remain command-free. Provisionally+ # index v1 carries, then discard the index if the run's patch+ # decision restores its pre-accumulator command path.+ if self._execution_control.enabled and self._event_accumulator_v1 is not False:+ self._pending_event_ids.add(self._event_key(str(listener_id), event))+ for key, value in (control_data.get("triggers") or {}).items():+ self._triggers.setdefault(str(key), value)+ self._overflow = bool(control_data.get("overflow"))+ self._durable_prefix = int(control_data.get("durable_prefix", len(self._events)))+ self._drain_tasks = bool(control_data.get("drain_tasks"))+ self._carried_state_seeded = True+ if not start_runtime:+ return+ for listener_id, spec in self._triggers.items():+ if spec.get("workflow_type") == "PollingTriggerWorkflow" and listener_id not in self._poll_tasks:self._poll_tasks[listener_id] = asyncio.create_task(self._poll_trigger(listener_id, spec))@@ -307,6 +538,33 @@ class WorkflowControlWorkflow:oldest = next(iter(self._seen_event_ids))del self._seen_event_ids[oldest]+ @staticmethod+ def _event_key(listener_id: str, event: Dict[str, Any]) -> str:+ return f"{listener_id}:{event.get('id') or ''}"++ def _accumulator_enabled(self) -> bool:+ if self._event_accumulator_v1 is None:+ # The new branch must not change legacy generations or decisions+ # already recorded by an unpatched v1 controller run.+ self._event_accumulator_v1 = self._execution_control.enabled and workflow.patched(+ "controller-event-accumulator-v1"+ )+ if not self._event_accumulator_v1:+ self._pending_event_ids.clear()+ return self._event_accumulator_v1++ def _event_already_accepted(self, dedup_key: str) -> bool:+ return dedup_key in self._seen_event_ids or (+ self._accumulator_enabled() and dedup_key in self._pending_event_ids+ )++ def _finish_pending_event(self, listener_id: str, event: Dict[str, Any]) -> None:+ dedup_key = self._event_key(listener_id, event)+ self._pending_event_ids.discard(dedup_key)+ # Retain the most recently completed work as well as recent arrivals.+ self._seen_event_ids.pop(dedup_key, None)+ self._remember_event_id(dedup_key)+def _history_pressure(self) -> bool:from services.temporal.trigger_listener_workflow import _history_pressure@@ -317,6 +575,23 @@ class WorkflowControlWorkflow:if self._history_pressure():self._can_requested = True+ def _messaging_enabled(self) -> bool:+ if self._messaging_v1 is None:+ # Evaluate only from runtime handlers. @workflow.init must remain+ # command-free, and old generations and pre-marker v1 histories+ # retain their previous control and rollover decisions.+ self._messaging_v1 = self._execution_control.enabled and workflow.patched(+ "controller-messaging-v1"+ )+ return self._messaging_v1++ def _maybe_request_rollover_from_update(self) -> None:+ # Updates add history even when there are no Signals or pollers. Wake+ # the main run loop, which alone owns CAN and the handler completion+ # fence, rather than continuing as new from an Update handler.+ if self._messaging_enabled():+ self._maybe_request_rollover()+async def _continue_as_new(self, control_data: Dict[str, Any]) -> None:"""Roll the run over, carrying everything a controller must keep.@@ -326,6 +601,14 @@ class WorkflowControlWorkflow:the loops in the new run without re-emitting old items. Workswhile paused too: the paused state (and queued events) carry."""+ accumulator_v1 = self._accumulator_enabled()+ if self._execution_control.enabled:+ # Stop and rollover do not cancel an admitted provider poll. Let its+ # result update the seen-id baseline and durable pending queue.+ await workflow.wait_condition(lambda: self._execution_control.status()["checkpoint"])+ await workflow.wait_condition(workflow.all_handlers_finished)+ if accumulator_v1 and self._closed:+ returnfor task in self._poll_tasks.values():task.cancel()if self._poll_tasks:@@ -359,9 +642,31 @@ class WorkflowControlWorkflow:for batch in reversed(chunks[1:] if carry else chunks):await spill(batch, prepend=True)self._overflow = True- for batch in event_batches(self._events[prefix_count:]):- await spill(batch)- self._overflow = True+ if accumulator_v1 and self._closed:+ return+ if accumulator_v1:+ # Every spill yields to Signals. Re-snapshot the newly appended+ # tail until it is empty, then recheck after waiting for message+ # handlers. There is no await between this final fence and CAN.+ cursor = prefix_count+ while True:+ while cursor < len(self._events):+ tail_end = len(self._events)+ for batch in event_batches(self._events[cursor:tail_end]):+ await spill(batch)+ self._overflow = True+ if self._closed:+ return+ cursor = tail_end+ await workflow.wait_condition(workflow.all_handlers_finished)+ if self._closed:+ return+ if cursor == len(self._events):+ break+ else:+ for batch in event_batches(self._events[prefix_count:]):+ await spill(batch)+ self._overflow = Trueelse:durable_queue = workflow.patched("controller-durable-overflow-v1")carry = self._events[-_MAX_CARRIED_EVENTS:]@@ -374,6 +679,8 @@ class WorkflowControlWorkflow:start_to_close_timeout=timedelta(seconds=60), retry_policy=DEFAULT_ACTIVITY_RETRY)self._overflow = Truecarry = self._events[:_MAX_CARRIED_EVENTS]+ if self._execution_control.enabled and not accumulator_v1:+ await workflow.wait_condition(workflow.all_handlers_finished)carried: Dict[str, Any] = {**control_data,"state": self._state,@@ -387,6 +694,21 @@ class WorkflowControlWorkflow:"drain_tasks": self._drain_tasks,"seen_event_ids": list(self._seen_event_ids)[-_MAX_CARRIED_SEEN_IDS:],}+ if self._execution_control.enabled:+ carried.update({+ **self._execution_control.carry(),+ "participant_state": self._execution_control.state,+ "participant_revision": self._execution_control.revision,+ "live_roots": self._live_roots,+ "membership_epoch": self._membership_epoch,+ })+ encoded = workflow.payload_converter().to_payloads([carried])+ input_bytes = sum(item.ByteSize() for item in encoded)+ if input_bytes > _CAN_INPUT_MAX_BYTES:+ raise ApplicationError(+ f"Controller continuation requires {input_bytes} bytes; limit is {_CAN_INPUT_MAX_BYTES}.",+ type="ControllerContinuationTooLarge", non_retryable=True,+ )workflow.logger.info(f"Controller continue_as_new: triggers={len(self._triggers)} "f"pending={len(carried['pending_events'])} state={self._state}"@@ -438,6 +760,8 @@ class WorkflowControlWorkflow:listener = TriggerListenerWorkflow()listener_args = spec["listener_args"]+ if self._execution_control.enabled:+ listener_args = {**listener_args, **self._execution_control.carry()}await listener._spawn_child_run(event,listener_args,@@ -447,6 +771,7 @@ class WorkflowControlWorkflow:listener_args.get("workflow_id"))),+ **({"child_start_control": self._execution_control} if self._execution_control.enabled else {}),)async def _wait_until_review(self) -> None:@@ -456,6 +781,8 @@ class WorkflowControlWorkflow:raise asyncio.CancelledErrorasync def _wait_until_running(self) -> None:+ if self._execution_control.enabled and (self._state != "running" or self._can_requested):+ raise _AdmissionDeferredif self._state != "running":await workflow.wait_condition(lambda: self._closed or self._state == "running")if self._closed:@@ -500,6 +827,30 @@ class WorkflowControlWorkflow:"node_id": listener_data["trigger_node_id"], "params": params,"seen_ids": list(seen_ids), "baseline_only": baseline_only,}+ if self._execution_control.enabled:+ try:+ async with self._execution_control.action():+ result = await workflow.execute_activity(+ activity_name, payload, activity_id=listener_data["trigger_node_id"],+ start_to_close_timeout=timedelta(seconds=activity_timeout_s),+ retry_policy=DEFAULT_ACTIVITY_RETRY,+ )+ seen_ids = set(result.get("seen_ids") or [])+ listener_data["seen_ids"] = list(seen_ids)+ for event in result.get("events") or []:+ event_id = str(event.get("id") or "")+ dedup_key = f"{listener_id}:{event_id}"+ if event_id and not self._event_already_accepted(dedup_key):+ self._remember_event_id(dedup_key)+ self._events.append((listener_id, event))+ if self._event_accumulator_v1:+ self._pending_event_ids.add(dedup_key)+ except asyncio.CancelledError:+ raise+ except Exception as exc: # per-poll isolation+ workflow.logger.error(f"Controlled polling trigger failed: {exc}")+ self._maybe_request_rollover()+ continuetry:result = await workflow.execute_activity(activity_name, payload, activity_id=listener_data["trigger_node_id"],diff --git a/server/tests/nodes/test_discord.py b/server/tests/nodes/test_discord.pyindex 4ab3e30c..12d34b40 100644--- a/server/tests/nodes/test_discord.py+++ b/server/tests/nodes/test_discord.py@@ -438,7 +438,27 @@ class TestTriggerIsDeployable:class TestEventShaping:- """_dispatch.shape_message feeds _filters; the two move together."""+ """Provider identities and shaped payloads feed trigger filtering."""++ @pytest.mark.parametrize(+ "factory_name, identity_field", [("discord_message_received", "message_id"), ("discord_interaction_created", "interaction_id")]+ )+ def test_provider_redelivery_keeps_a_stable_event_identity(self, factory_name, identity_field):+ from nodes.discord import _events++ factory = getattr(_events, factory_name)+ original = factory({identity_field: "123", "channel_id": "456"})+ redelivery = factory({identity_field: "123", "channel_id": "456", "timestamp": "later"})+ another = factory({identity_field: "124", "channel_id": "456"})++ assert original.id == redelivery.id+ assert original.id != another.id+ assert factory({"channel_id": "456"}).id != factory({"channel_id": "456"}).id++ def test_message_and_interaction_identities_do_not_collide(self):+ from nodes.discord._events import discord_interaction_created, discord_message_received++ assert discord_message_received({"message_id": "123"}).id != discord_interaction_created({"interaction_id": "123"}).iddef _message(self, **overrides):base = SimpleNamespace(diff --git a/server/tests/services/chat/test_handlers.py b/server/tests/services/chat/test_handlers.pyindex 182725ca..a1402f66 100644--- a/server/tests/services/chat/test_handlers.py+++ b/server/tests/services/chat/test_handlers.py@@ -110,6 +110,20 @@ async def test_no_run_when_no_deployed_chat_trigger_answers_the_session(chat):assert len(await chat.database.read_chat_messages("wf")) == 1+async def test_untracked_chat_redelivery_uses_the_saved_message_identity(chat):+ await chat.handlers._dispatch(+ "default", None, message_uid="message-1", prompt="hello", run_id=None, timestamp="first"+ )+ await chat.handlers._dispatch(+ "default", None, message_uid="message-1", prompt="hello", run_id=None, timestamp="later"+ )+ await chat.handlers._dispatch(+ "default", None, message_uid="message-2", prompt="hello", run_id=None, timestamp="later"+ )++ assert [event["event_id"] for event in chat.dispatched] == ["message-1", "message-1", "message-2"]++async def test_the_owner_writes_only_their_own_messages(chat):await talking(chat.database)assert (await send(chat, role="assistant"))["error"] == "invalid_request"@@ -136,7 +150,7 @@ async def test_reading_the_thread_returns_the_v2_shape(chat):await talking(chat.database)sent = await send(chat, client_message_id="c-7")result = await chat.handlers.handle_get_chat_messages({"session_id": "wf", "all_generations": True}, None)- assert result["success"] is True and result["protocol_version"] == 1+ assert result["success"] is True and result["protocol_version"] == 2[message] = result["messages"]assert message["id"] == sent["message_id"] and message["run_id"] == sent["run_id"]assert (message["text"], message["message"], message["kind"], message["status"]) == ("Book Saturday",) * 2 + ("text", "complete")diff --git a/server/tests/services/chat/test_stop.py b/server/tests/services/chat/test_stop.pyindex 6d25dc80..67769a73 100644--- a/server/tests/services/chat/test_stop.py+++ b/server/tests/services/chat/test_stop.py@@ -9,10 +9,12 @@ from __future__ import annotationsimport asynciofrom datetime import datetime, timedelta, timezone+from unittest.mock import AsyncMockfrom sqlalchemy import updatefrom models.chat import ChatRun+from models.database import WorkflowControlExecutionfrom services.chat import ledgerfrom tests.services.chat._helpers import FakeSocket, talking@@ -219,3 +221,78 @@ async def test_only_the_workflows_owner_may_stop_its_runs(chat):for socket in (FakeSocket(user_id="mallory"), FakeSocket(path="/ws/internal")):assert (await chat.handlers.handle_stop_chat_run({"run_id": sent["run_id"]}, socket))["error"] == "access_denied"assert (await ledger.get_run(chat.database, sent["run_id"])).state == "pending"+++async def version_control(database, *, status="running", resumed=None):+ manifest = {"execution_control_version": 1}+ if resumed:+ manifest["last_resumed_at"] = resumed.isoformat()+ async with database.get_session() as session:+ await session.execute(update(WorkflowControlExecution).where(WorkflowControlExecution.id == "wf-1").values(+ resource_manifest=manifest, status=status, revision=3,+ ))+ await session.commit()+++async def test_controlled_stop_keeps_the_chat_run_reply_and_lane(chat, monkeypatch):+ from services.deployment import handlers as deployment_handlers++ await talking(chat.database)+ await version_control(chat.database)+ run = await running(chat.database)+ await ledger.post_reply(chat.database, run=run, node_id="n", text="A partial answer", execution_id="gen-1")+ pause = AsyncMock(return_value={"success": True, "state": "paused", "revision": 5})+ monkeypatch.setattr(deployment_handlers, "handle_pause_workflow", pause)+ reply = await chat.handlers.handle_stop_chat_run({+ "run_id": run.run_id, "expected_revision": 3, "idempotency_key": "stop-1",+ }, None)+ assert reply["resumable"] is True and reply["state"] == "paused"+ pause.assert_awaited_once_with({+ "workflow_id": "wf", "expected_root_execution_id": "gen-1", "expected_revision": 3,+ "idempotency_key": "stop-1",+ }, None)+ saved = await ledger.get_run(chat.database, run.run_id)+ assert saved.state == "running" and saved.stop_requested_at is None and saved.finished_at is None+ assert (await ledger.lane_run(chat.database, "wf")).run_id == run.run_id+ assert (await ledger.saved_message(chat.database, saved.reply_message_uid)).message == "A partial answer"+++async def test_controlled_chat_reload_uses_its_owning_generation(chat):+ await talking(chat.database)+ await version_control(chat.database, status="paused")+ run = await running(chat.database)+ # A newer generation must not masquerade as this suspended run's owner.+ await talking(chat.database, generation=2)+ [snapshot] = await chat.handlers.run_snapshots(chat.database, "wf")+ assert snapshot["run_id"] == run.run_id and snapshot["state"] == "running"+ assert snapshot["workflow_control"]["root_execution_id"] == "gen-1"+ assert snapshot["workflow_control"]["state"] == "paused"+++async def test_controlled_stop_requires_an_idempotency_key(chat):+ await talking(chat.database)+ await version_control(chat.database)+ run = await running(chat.database)+ result = await chat.handlers.handle_stop_chat_run({"run_id": run.run_id, "expected_revision": 3}, None)+ assert result == {"success": False, "error": "idempotency_key_required"}+ assert (await ledger.get_run(chat.database, run.run_id)).state == "running"+++async def test_long_controlled_pause_does_not_expire_and_resume_grants_a_fresh_window(database, hub):+ from services.chat.config import runs_setting++ await talking(database)+ run = await running(database)+ longest = timedelta(seconds=runs_setting("max_running_s"))+ async with database.get_session() as session:+ await session.execute(update(ChatRun).where(ChatRun.run_id == run.run_id).values(+ started_at=NOW - timedelta(days=30), created_at=NOW - timedelta(days=30),+ ))+ await session.commit()+ for state in ("pausing", "paused", "resuming"):+ await version_control(database, status=state)+ assert await ledger.sweep(database, now=NOW, process_started=NOW - timedelta(days=30)) == []+ await version_control(database, status="running", resumed=NOW)+ assert await ledger.sweep(database, now=NOW + longest - timedelta(seconds=1), process_started=NOW) == []+ assert await ledger.sweep(database, now=NOW + longest + timedelta(seconds=1), process_started=NOW) == [run.run_id]+ assert (await ledger.get_run(database, run.run_id)).error_code == "timed_out"diff --git a/server/tests/services/chat/test_watchdog.py b/server/tests/services/chat/test_watchdog.pyindex c0bfc331..862b9803 100644--- a/server/tests/services/chat/test_watchdog.py+++ b/server/tests/services/chat/test_watchdog.py@@ -96,7 +96,33 @@ async def test_a_workflow_is_closed_once_it_stops_running(monkeypatch, status, c_container(monkeypatch, Client())assert await watchdog.temporal_workflow_closed("w", "r") is closed- assert asked == [("w", "r")]+ assert asked == [("w", None)]++[email protected](("current_first", "closed"), [("first", False), ("replacement", True)])+async def test_continue_as_new_tracks_the_claimants_execution_chain(monkeypatch, current_first, closed):+ from temporalio.client import WorkflowExecutionStatus++ descriptions = {+ None: SimpleNamespace(run_id="next", status=WorkflowExecutionStatus.RUNNING,+ raw_info=SimpleNamespace(first_run_id=current_first)),+ "first": SimpleNamespace(run_id="first", status=WorkflowExecutionStatus.CONTINUED_AS_NEW,+ raw_info=SimpleNamespace(first_run_id="first")),+ }++ class Handle:+ def __init__(self, run_id):+ self.run_id = run_id++ async def describe(self):+ return descriptions[self.run_id]++ class Client:+ def get_workflow_handle(self, workflow_id, run_id=None):+ return Handle(run_id)++ _container(monkeypatch, Client())+ assert await watchdog.temporal_workflow_closed("w", "first") is closedasync def test_a_workflow_temporal_no_longer_knows_is_closed(monkeypatch):@@ -122,6 +148,29 @@ async def test_a_workflow_temporal_no_longer_knows_is_closed(monkeypatch):assert await watchdog.temporal_workflow_closed("w", "r") is None+async def test_a_running_chain_does_not_end_when_an_older_run_aged_out(monkeypatch):+ from temporalio.client import WorkflowExecutionStatus+ from temporalio.service import RPCError, RPCStatusCode++ class Handle:+ def __init__(self, run_id):+ self.run_id = run_id++ async def describe(self):+ if self.run_id:+ raise RPCError("aged out", RPCStatusCode.NOT_FOUND, b"")+ return SimpleNamespace(run_id="new", status=WorkflowExecutionStatus.RUNNING,+ raw_info=SimpleNamespace(first_run_id="first"))++ class Client:+ def get_workflow_handle(self, workflow_id, run_id=None):+ return Handle(run_id)++ _container(monkeypatch, Client())+ assert await watchdog.temporal_workflow_closed("w", "first") is False+ assert await watchdog.temporal_workflow_closed("w", "intermediate") is None++async def test_cancelling_a_stopped_runs_workflow(monkeypatch):from temporalio.service import RPCError, RPCStatusCodediff --git a/server/tests/services/test_versioned_workflow_control.py b/server/tests/services/test_versioned_workflow_control.pynew file mode 100644index 00000000..2c91fc41--- /dev/null+++ b/server/tests/services/test_versioned_workflow_control.py@@ -0,0 +1,91 @@+"""Durable transition intent and compatibility of generation-level control."""+from types import SimpleNamespace+from unittest.mock import AsyncMock++import pytest++from services.deployment import handlers+from services.deployment import execution_control+++def control(state="pausing"):+ return SimpleNamespace(id="control:1", workflow_id="canvas", generation=1, revision=3,+ root_execution_id="generation-root", controller_workflow_id="controller", controller_run_id="first",+ status=state, resource_manifest={"execution_control_version": 1})+++async def test_stop_timeout_keeps_intent_and_closed_producers(monkeypatch):+ from core.container import container+ row = control()+ runtime = SimpleNamespace(pause_deployment=lambda wid: closed.append(wid),+ update_trigger_pause_status=AsyncMock(), resume_deployment=AsyncMock())+ closed = []+ monkeypatch.setattr(container, "workflow_service", lambda: runtime)+ monkeypatch.setattr(container, "temporal_client", lambda: SimpleNamespace(client=object()))+ schedules = AsyncMock()+ monkeypatch.setattr(handlers, "_set_cron_pause", schedules)+ transition = AsyncMock(side_effect=TimeoutError("request timeout; tool still running"))+ monkeypatch.setattr(execution_control, "transition_generation", transition)+ service = SimpleNamespace(transition=AsyncMock())+ with pytest.raises(TimeoutError):+ await handlers._finish_versioned_transition(service, row)+ assert row.status == "pausing"+ assert closed == ["canvas"]+ schedules.assert_awaited_once_with("canvas", paused=True, strict=True)+ runtime.resume_deployment.assert_not_awaited()+ service.transition.assert_not_awaited()+++async def test_resume_releases_continuations_before_producers_and_saves_timeout_window(monkeypatch):+ from core.container import container+ row = control("resuming")+ order = []+ async def release(*args, **kwargs):+ order.append("continuations")+ return {"controller_status": {}, "controlled_executions": 2}+ async def schedules(*args, **kwargs):+ order.append("schedules")+ return 1+ async def local(*args, **kwargs):+ order.append("local")+ return 0+ async def stable(target, **kwargs):+ assert kwargs["values"]["resource_manifest"]["last_resumed_at"]+ assert kwargs["values"]["resource_manifest"]["execution_control_version"] == 1+ order.append("running")+ target.status = kwargs["status"]+ return target+ monkeypatch.setattr(container, "workflow_service", lambda: SimpleNamespace(pause_deployment=lambda _: None,+ update_trigger_pause_status=AsyncMock(), resume_deployment=local))+ monkeypatch.setattr(container, "temporal_client", lambda: SimpleNamespace(client=object()))+ monkeypatch.setattr(execution_control, "transition_generation", release)+ monkeypatch.setattr(handlers, "_set_cron_pause", schedules)+ monkeypatch.setattr(handlers, "_clear_failure_streak", AsyncMock())+ monkeypatch.setattr(handlers, "_broadcast_control", AsyncMock(return_value={}))+ from services import status_broadcaster+ monkeypatch.setattr(status_broadcaster, "get_status_broadcaster", lambda: AsyncMock())+ result, _ = await handlers._finish_versioned_transition(SimpleNamespace(transition=stable, database=object()), row)+ assert result.status == "running"+ assert order == ["continuations", "schedules", "local", "running"]+++async def test_missing_versioned_controller_cannot_rebuild(monkeypatch):+ start = AsyncMock()+ monkeypatch.setattr(handlers, "_start_controller", start)+ with pytest.raises(handlers.ControllerExecutionMissing):+ await handlers._rebuild_missing_controller(object(), control("paused"))+ start.assert_not_awaited()+++async def test_chat_generation_guard_precedes_reconciliation(monkeypatch):+ service = SimpleNamespace(database=SimpleNamespace(get_latest_workflow_control=AsyncMock(return_value=control("running"))))+ monkeypatch.setattr(handlers, "_control_service", lambda: service)+ reconcile = AsyncMock()+ monkeypatch.setattr(handlers, "_reconcile_control", reconcile)+ result = await handlers.handle_pause_workflow({"workflow_id": "canvas", "expected_root_execution_id": "other"}, None)+ assert result["error"] == "control_generation_conflict"+ reconcile.assert_not_awaited()diff --git a/server/tests/temporal/test_agent_execution_control_integration.py b/server/tests/temporal/test_agent_execution_control_integration.pynew file mode 100644index 00000000..797316eb--- /dev/null+++ b/server/tests/temporal/test_agent_execution_control_integration.py@@ -0,0 +1,372 @@+"""Real SDK checkpoint, restart, rollover and replay gates with stub tools."""++from __future__ import annotations++import asyncio+import subprocess+import sys+from collections import Counter+from datetime import timedelta+from pathlib import Path+from uuid import uuid4++from temporalio import activity+from temporalio.api.enums.v1 import EventType+from temporalio.client import WorkflowFailureError, WorkflowHistory+from temporalio.testing import WorkflowEnvironment+from temporalio.worker import Replayer, Worker++from services.llm.protocol import Message, ToolCall, message_to_wire+from services.temporal import agent_workflow as module+from services.temporal.agent_workflow import AgentWorkflow, DelegatedTaskWorkflow+++QUEUE = "agent-control-sdk-gate"+COUNTS: Counter = Counter()+STARTED: dict[str, asyncio.Event] = {}+RELEASE: dict[str, asyncio.Event] = {}+LLM_INPUTS: list[dict] = []+++def _event(name: str, events: dict[str, asyncio.Event]) -> asyncio.Event:+ return events.setdefault(name, asyncio.Event())+++def _tool(name: str, node_type: str = "control_test_tool") -> dict:+ return {+ "name": name, "node_type": node_type, "version": 1,+ "tool_node_id": name, "parameters": {}, "tool_info": {},+ "definition": {"name": name, "description": name, "parameters": {"type": "object", "properties": {}}},+ "activity_policy": {+ "start_to_close_seconds": 60, "heartbeat_seconds": 10,+ "task_queue": QUEUE,+ "retry_policy": {"initial_interval_seconds": 1, "maximum_interval_seconds": 5,+ "backoff_coefficient": 2, "maximum_attempts": 1, "non_retryable_error_types": []},+ },+ }++[email protected](name="agent.prepare_payload")+async def _prepare(context: dict) -> dict:+ COUNTS["prepare"] += 1+ mode = context.get("mode", "tools")+ return {+ "node_id": "agent", "node_type": "aiAgent", "provider": "openai",+ "model": "stub", "workflow_id": "graph", "session_id": "session",+ "system_message": ("s" * 450_000 if mode == "capacity" else "Keep working"), "user_prompt": mode,+ "tools": ([_tool("A", "taskManager"), _tool("B", "taskManager")] if mode == "parallel"+ else [] if mode == "delegate-child" else [_tool("A"), _tool("B")]),+ "max_iterations": 4, "memory_node_id": "", "compaction_threshold": None,+ }++[email protected](name="agent.execute_llm_step")+async def _llm(payload: dict) -> dict:+ mode = next(message["content"] for message in payload["messages"] if message["role"] == "user")+ iteration = payload["iteration"]+ COUNTS["llm"] += 1+ LLM_INPUTS.append(payload)+ if mode == "llm-stop" and iteration == 0:+ _event("llm", STARTED).set()+ await _event("llm", RELEASE).wait()+ if mode == "rollover" and iteration == 1:+ _event("llm-after-rollover", STARTED).set()+ await _event("llm-after-rollover", RELEASE).wait()+ if mode == "delegate-child" or iteration >= (2 if mode == "rollover" else 1):+ return {"kind": "final", "content": "done", "thinking": f"thought-{iteration}",+ "assistant_message": message_to_wire(Message(role="assistant", content="done")),+ "usage": {"total_tokens": 3}}+ names = (["A"] if iteration == 0 else ["B"]) if mode == "rollover" else ["A", "B"]+ calls = [{"id": f"{iteration}-{name}", "name": name,+ "args": {"operation": "assign_task"} if mode == "parallel" else {}} for name in names]+ return {"kind": "tool_calls", "calls": calls, "thinking": f"thought-{iteration}",+ "assistant_message": message_to_wire(Message(role="assistant", tool_calls=[ToolCall(**call) for call in calls])),+ "usage": {"total_tokens": 3}}++[email protected](name="node.control_test_tool.v1")+async def _tool_activity(payload: dict) -> dict:+ name = payload["node_id"]+ COUNTS[name] += 1+ assert activity.info().heartbeat_timeout == timedelta(seconds=10)+ _event(name, STARTED).set()+ if not any(mode in activity.info().workflow_id for mode in ("rollover", "capacity")):+ await _event(name, RELEASE).wait()+ return {"result": name, **({"operations": [{"type": "add_node"}]} if name == "A" else {})}++[email protected](name="node.taskManager.v1")+async def _preflight(payload: dict) -> dict:+ name = payload["node_id"]+ COUNTS[name] += 1+ _event(name, STARTED).set()+ await _event(name, RELEASE).wait()+ return {"status": "queued", "result": name}++[email protected](name="agent.refresh_tools")+async def _refresh(payload: dict) -> dict:+ COUNTS["refresh"] += 1+ return {"tools": []}++[email protected](name="agent.broadcast_progress")+async def _broadcast(payload: dict) -> dict:+ return {}++[email protected](name="agent.store_output")+async def _store(payload: dict) -> dict:+ return {}++[email protected](name="agent.skill.clear")+async def _clear(payload: dict) -> dict:+ return {}++[email protected](name="execution_control.register")+async def _register(payload: dict) -> dict:+ COUNTS["register"] += 1+ return {"participant_state": "running", "participant_revision": 0}++[email protected](name="execution_control.unregister")+async def _unregister(payload: dict) -> dict:+ COUNTS["unregister"] += 1+ return {}++[email protected](name="agent.register_task_execution")+async def _register_task(payload: dict) -> dict:+ return {}++[email protected](name="agent.acquire_subagent_permit")+async def _acquire(payload: dict) -> dict:+ _event("permit", STARTED).set()+ await _event("permit", RELEASE).wait()+ return {"lease_id": "permit"}++[email protected](name="agent.begin_delegation")+async def _begin(payload: dict) -> dict:+ COUNTS["begin"] += 1+ return {}++[email protected](name="agent.finish_delegation")+async def _finish(payload: dict) -> dict:+ return {"success": True}++[email protected](name="agent.release_subagent_permit")+async def _release_permit(payload: dict) -> dict:+ COUNTS["release-permit"] += 1+ return {}+++ACTIVITIES = [_prepare, _llm, _tool_activity, _preflight, _refresh, _broadcast, _store, _clear,+ _register, _unregister, _register_task, _acquire, _begin, _finish, _release_permit]+WORKFLOWS = [AgentWorkflow, DelegatedTaskWorkflow]+++def _input(mode: str) -> dict:+ return {"node_id": "agent", "mode": mode, "execution_control_version": 1,+ "controller_workflow_id": "controller", "generation": 1, "execution_control_revision": 0}+++def _reset() -> None:+ COUNTS.clear()+ STARTED.clear()+ RELEASE.clear()+ LLM_INPUTS.clear()+++async def _replay(history: WorkflowHistory) -> None:+ for event in history.events:+ if event.event_type == EventType.EVENT_TYPE_ACTIVITY_TASK_SCHEDULED:+ scheduled = event.activity_task_scheduled_event_attributes+ if scheduled.activity_type.name == "node.control_test_tool.v1":+ assert scheduled.start_to_close_timeout.ToTimedelta() == timedelta(seconds=60)+ assert scheduled.retry_policy.maximum_attempts == 1+ result = await Replayer(workflows=WORKFLOWS).replay_workflow(+ WorkflowHistory.from_json(history.workflow_id, history.to_json()))+ assert result.replay_failure is None+++async def _stop_tools(environment: WorkflowEnvironment, *, mode: str, restart: bool = False) -> None:+ _reset()+ async with Worker(environment.client, task_queue=QUEUE, workflows=WORKFLOWS, activities=ACTIVITIES, max_cached_workflows=0):+ handle = await environment.client.start_workflow("AgentWorkflow", _input(mode), id=f"{mode}-{uuid4()}", task_queue=QUEUE)+ await asyncio.wait_for(_event("llm" if mode == "llm-stop" else "A", STARTED).wait(), 10)+ if mode == "parallel":+ await asyncio.wait_for(_event("B", STARTED).wait(), 10)+ stop = {"state": "paused", "revision": 1, "generation": 1}+ admission = await handle.execute_update("set_control_state", stop)+ assert admission["state"] == "paused"+ checkpoint = asyncio.create_task(handle.execute_update("wait_for_checkpoint", stop))+ await asyncio.sleep(0.05)+ assert not checkpoint.done()+ if mode == "llm-stop":+ _event("llm", RELEASE).set()+ else:+ _event("A", RELEASE).set()+ if mode == "parallel":+ await asyncio.sleep(0.05)+ assert not checkpoint.done() # B is independently in flight.+ _event("B", RELEASE).set()+ drained = await asyncio.wait_for(checkpoint, 10)+ assert drained["checkpoint"] and drained["active_actions"] == 0+ assert COUNTS["llm"] == 1 and COUNTS["refresh"] == 0+ assert COUNTS["B"] == (1 if mode == "parallel" else 0)+ await handle.execute_update("set_control_state", {"state": "running", "revision": 0, "generation": 1})+ assert (await handle.query("execution_control_status"))["state"] == "paused"+ if not restart:+ await handle.execute_update("set_control_state", {"state": "running", "revision": 2, "generation": 1, "producers_held": False})+ late = await handle.execute_update("set_control_state", {"state": "running", "revision": 2, "generation": 1, "producers_held": True})+ assert late["producers_held"] is False # A delayed admission cannot undo release.+ _event("A", RELEASE).set()+ _event("B", RELEASE).set()+ result = await asyncio.wait_for(handle.result(), 10)+ history = await handle.fetch_history()+ if restart:+ async with Worker(environment.client, task_queue=QUEUE, workflows=WORKFLOWS, activities=ACTIVITIES, max_cached_workflows=0):+ assert (await handle.query("execution_control_status"))["state"] == "paused"+ await handle.execute_update("set_control_state", {"state": "running", "revision": 2, "generation": 1, "producers_held": False})+ late = await handle.execute_update("set_control_state", {"state": "running", "revision": 2, "generation": 1, "producers_held": True})+ assert late["producers_held"] is False+ _event("B", RELEASE).set()+ result = await asyncio.wait_for(handle.result(), 10)+ history = await handle.fetch_history()+ assert result["success"] is True+ assert COUNTS["A"] == COUNTS["B"] == 1 and COUNTS["llm"] == 2+ assert COUNTS["prepare"] == 1+ tool_messages = [message for message in LLM_INPUTS[-1]["messages"] if message["role"] == "tool"]+ assert [message["name"] for message in tool_messages] == ["A", "B"]+ await _replay(history)+++async def _permit_checkpoint(environment: WorkflowEnvironment) -> None:+ _reset()+ request = {"lifecycle": {"team_task_id": "task", "root_execution_id": "root"},+ "child_context": _input("delegate-child"), "child_workflow_id": "delegate-" + uuid4().hex}+ async with Worker(environment.client, task_queue=QUEUE, workflows=WORKFLOWS, activities=ACTIVITIES, max_cached_workflows=0):+ handle = await environment.client.start_workflow("DelegatedTaskWorkflow", request, id="runner-" + uuid4().hex, task_queue=QUEUE)+ await asyncio.wait_for(_event("permit", STARTED).wait(), 10)+ stop = {"state": "paused", "revision": 1, "generation": 1}+ await handle.execute_update("set_control_state", stop)+ drained = await asyncio.wait_for(handle.execute_update("wait_for_checkpoint", stop), 10)+ assert drained["checkpoint"]+ _event("permit", RELEASE).set()+ await asyncio.sleep(0.1)+ assert COUNTS["begin"] == 0+ await handle.execute_update("set_control_state", {"state": "running", "revision": 2, "generation": 1, "producers_held": False})+ assert (await asyncio.wait_for(handle.result(), 10))["success"]+ history = await handle.fetch_history()+ assert COUNTS["register"] == COUNTS["unregister"] == COUNTS["release-permit"] == 1+ await _replay(history)+++async def _stop_before_first_workflow_task(environment: WorkflowEnvironment) -> None:+ """An Update delivered before run initialization must fence preparation."""+ _reset()+ handle = await environment.client.start_workflow("AgentWorkflow", _input("tools"), id="cold-stop-" + uuid4().hex, task_queue=QUEUE)+ pending_stop = asyncio.create_task(handle.execute_update("set_control_state", {"state": "paused", "revision": 1, "generation": 1}))+ await asyncio.sleep(0.05) # Submit control while no worker can execute the run.+ async with Worker(environment.client, task_queue=QUEUE, workflows=WORKFLOWS, activities=ACTIVITIES, max_cached_workflows=0):+ assert (await asyncio.wait_for(pending_stop, 10))["state"] == "paused"+ await handle.execute_update("wait_for_checkpoint", {"state": "paused", "revision": 1, "generation": 1})+ assert COUNTS["prepare"] == COUNTS["llm"] == COUNTS["A"] == COUNTS["B"] == 0+ _event("A", RELEASE).set()+ _event("B", RELEASE).set()+ await handle.execute_update("set_control_state", {"state": "running", "revision": 2, "generation": 1, "producers_held": False})+ assert (await asyncio.wait_for(handle.result(), 10))["success"]+ history = await handle.fetch_history()+ assert COUNTS["prepare"] == COUNTS["A"] == COUNTS["B"] == 1+ await _replay(history)+++async def _rollover(environment: WorkflowEnvironment) -> None:+ _reset()+ original_cap = module._AGENT_HISTORY_SOFT_CAP+ module._AGENT_HISTORY_SOFT_CAP = 1+ try:+ async with Worker(environment.client, task_queue=QUEUE, workflows=WORKFLOWS, activities=ACTIVITIES, max_cached_workflows=0):+ handle = await environment.client.start_workflow("AgentWorkflow", _input("rollover"), id="rollover-" + uuid4().hex, task_queue=QUEUE)+ await asyncio.wait_for(_event("llm-after-rollover", STARTED).wait(), 10)+ stop = {"state": "paused", "revision": 1, "generation": 1}+ await handle.execute_update("set_control_state", stop)+ checkpoint = asyncio.create_task(handle.execute_update("wait_for_checkpoint", stop))+ _event("llm-after-rollover", RELEASE).set()+ assert (await asyncio.wait_for(checkpoint, 10))["checkpoint"]+ assert COUNTS["prepare"] == COUNTS["A"] == 1 and COUNTS["B"] == 0+ await handle.execute_update("set_control_state", {"state": "running", "revision": 2, "generation": 1, "producers_held": False})+ late = await handle.execute_update("set_control_state", {"state": "running", "revision": 2, "generation": 1, "producers_held": True})+ assert late["producers_held"] is False+ result = await asyncio.wait_for(handle.result(), 15)+ run_id = handle.first_execution_run_id+ histories = []+ while run_id:+ history = await environment.client.get_workflow_handle(handle.id, run_id=run_id).fetch_history()+ histories.append(history)+ next_events = [event for event in history.events if event.event_type == EventType.EVENT_TYPE_WORKFLOW_EXECUTION_CONTINUED_AS_NEW]+ run_id = next_events[0].workflow_execution_continued_as_new_event_attributes.new_execution_run_id if next_events else None+ assert len(histories) == 3 # Two actual server-side continue-as-new transitions.+ [final_input] = await environment.client.data_converter.decode(histories[-1].events[0].workflow_execution_started_event_attributes.input.payloads)+ assert final_input["execution_control_revision"] == 2+ assert COUNTS["prepare"] == COUNTS["A"] == COUNTS["B"] == 1+ assert COUNTS["llm"] == 3+ assert result["result"]["usage"]["total_tokens"] == 9+ assert all(f"thought-{index}" in result["result"]["thinking"] for index in range(3))+ assert [message["name"] for message in LLM_INPUTS[-1]["messages"] if message["role"] == "tool"] == ["A", "B"]+ for history in histories:+ await _replay(history)+ finally:+ module._AGENT_HISTORY_SOFT_CAP = original_cap+++async def _capacity(environment: WorkflowEnvironment) -> None:+ """A small transcript plus large graph/bindings must not escape the guard."""+ _reset()+ original_cap = module._AGENT_HISTORY_SOFT_CAP+ module._AGENT_HISTORY_SOFT_CAP = 1+ try:+ async with Worker(environment.client, task_queue=QUEUE, workflows=WORKFLOWS, activities=ACTIVITIES, max_cached_workflows=0):+ handle = await environment.client.start_workflow("AgentWorkflow", {**_input("capacity"), "graph_data": "g" * 1_100_000},+ id="capacity-" + uuid4().hex, task_queue=QUEUE)+ try:+ await asyncio.wait_for(handle.result(), 15)+ except WorkflowFailureError as error:+ assert error.cause.type == "AgentContinuationTooLarge"+ else:+ raise AssertionError("An oversized continuation must fail explicitly")+ history = await handle.fetch_history()+ assert COUNTS["llm"] == COUNTS["prepare"] == COUNTS["A"] == COUNTS["B"] == 1+ assert not any(event.event_type == EventType.EVENT_TYPE_WORKFLOW_EXECUTION_CONTINUED_AS_NEW for event in history.events)+ await _replay(history)+ finally:+ module._AGENT_HISTORY_SOFT_CAP = original_cap+++async def _main() -> None:+ async with await WorkflowEnvironment.start_time_skipping() as environment:+ with environment.auto_time_skipping_disabled():+ await _stop_tools(environment, mode="tools", restart=True)+ await _stop_tools(environment, mode="llm-stop")+ await _stop_tools(environment, mode="parallel")+ await _permit_checkpoint(environment)+ await _stop_before_first_workflow_task(environment)+ await _rollover(environment)+ await _capacity(environment)+++def test_real_sdk_agent_stop_resume_restart_rollover_and_replay() -> None:+ result = subprocess.run([sys.executable, "-m", "tests.temporal.test_agent_execution_control_integration"],+ cwd=Path(__file__).parents[2], stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,+ stderr=subprocess.STDOUT, timeout=120, check=False)+ assert result.returncode == 0, result.stdout.decode(errors="replace")[-12000:]+++if __name__ == "__main__":+ asyncio.run(_main())diff --git a/server/tests/temporal/test_agent_workflow.py b/server/tests/temporal/test_agent_workflow.pyindex 5d09e5f8..8d2d4eb7 100644--- a/server/tests/temporal/test_agent_workflow.py+++ b/server/tests/temporal/test_agent_workflow.py@@ -32,6 +32,22 @@ class TestAgentWorkflowDefinition:assert defn is not None, "AgentWorkflow missing @workflow.defn"assert defn.name == "AgentWorkflow"+ def test_new_child_does_not_inherit_ancestor_release_or_chain_identity(self):+ from services.temporal.agent_workflow import _child_control_scope+ from services.temporal.execution_control import ExecutionControl++ control = ExecutionControl()+ control.bind({"execution_control_version": 1, "generation": 1,+ "controller_workflow_id": "controller", "execution_control_revision": 2,+ "execution_control_state": "running", "execution_control_producers_held": False,+ "execution_control_root_run_id": "ancestor-run", "execution_control_released_revision": 2})+ assert control.carry()["execution_control_released_revision"] == 2+ child_scope = _child_control_scope(control)+ assert "execution_control_released_revision" not in child_scope+ assert "execution_control_root_run_id" not in child_scope+ assert child_scope["execution_control_revision"] == 2+ assert child_scope["execution_control_state"] == "running"+def test_detached_delegation_runner_is_workflow_defn(self):from services.temporal.agent_workflow import DelegatedTaskWorkflow@@ -230,17 +246,17 @@ class TestConversationIdentity:def test_rollover_guards_transcript_size(self):"""The CAN argument shares Temporal's 2 MiB payload error limit.- An oversized transcript must degrade to the opening prompt with a- warning instead of failing the rollover itself (which would kill- the run at exactly the moment it tried to survive).+ Controlled generations fail explicitly rather than repeat the+ opening prompt, and measure the complete converter-encoded input."""import inspectfrom services.temporal.agent_workflow import AgentWorkflowsource = inspect.getsource(AgentWorkflow._run_impl)- assert "_CAN_TRANSCRIPT_MAX_BYTES" in source- guard_at = source.index("_CAN_TRANSCRIPT_MAX_BYTES")+ assert "AgentContinuationTooLarge" in source+ assert "workflow.payload_converter().to_payloads([next_context])" in source+ guard_at = source.index("_CAN_INPUT_MAX_BYTES")can_at = source.index("workflow.continue_as_new(")assert guard_at < can_at, ("the size guard must run before continue_as_new is issued"@@ -910,7 +926,7 @@ class TestExecutionIdPropagation:from services.temporal.workflow import MachinaWorkflow- src = inspect.getsource(MachinaWorkflow.run)+ src = inspect.getsource(MachinaWorkflow.run) + inspect.getsource(MachinaWorkflow._run_graph)assert '"execution_id"' in src, ("MachinaWorkflow per-node context must carry execution_id.")diff --git a/server/tests/temporal/test_controller_accumulator.py b/server/tests/temporal/test_controller_accumulator.pynew file mode 100644index 00000000..023874a3--- /dev/null+++ b/server/tests/temporal/test_controller_accumulator.py@@ -0,0 +1,289 @@+"""Versioned accumulator durability at dispatch and rollover boundaries."""++from types import SimpleNamespace+from unittest.mock import MagicMock++import pytest+from temporalio.converter import DefaultPayloadConverter+from temporalio.exceptions import WorkflowAlreadyStartedError++from services.temporal import workflow_control_workflow as module+from services.temporal.workflow_control_workflow import WorkflowControlWorkflow+++def control_data(**extra):+ return {+ "execution_control_version": 1,+ "generation": 7,+ "controller_workflow_id": "controller",+ **extra,+ }+++def push_spec():+ return {+ "trigger_node_id": "trigger",+ "workflow_type": "TriggerListenerWorkflow",+ "event_types": ["mail"],+ "listener_args": {"node_type": "gmailTrigger"},+ }+++def wf(monkeypatch):+ async def wait(predicate):+ assert predicate(), "An accumulator fence unexpectedly blocked"++ async def sleep(_delay):+ pass++ monkeypatch.setattr(module.workflow, "logger", MagicMock())+ monkeypatch.setattr(module.workflow, "patched", lambda _: True)+ monkeypatch.setattr(module.workflow, "info", lambda: SimpleNamespace(workflow_id="controller", run_id="run"))+ monkeypatch.setattr(module.workflow, "wait_condition", wait)+ monkeypatch.setattr(module.workflow, "sleep", sleep)+ monkeypatch.setattr(module.workflow, "all_handlers_finished", lambda: True)+ monkeypatch.setattr(module.workflow, "payload_converter", lambda: DefaultPayloadConverter())+ monkeypatch.setattr(WorkflowControlWorkflow, "_upsert_event_types_attribute", lambda _: None)+ monkeypatch.setattr(WorkflowControlWorkflow, "_history_pressure", lambda _: False)+ return module.workflow+++async def test_signal_during_final_spill_is_carried_or_spilled_in_fifo_order(wf, monkeypatch):+ monkeypatch.setattr(module, "_MAX_CARRIED_EVENTS", 2)+ controller = WorkflowControlWorkflow(control_data(state="paused", triggers={"mail": push_spec()}))+ for index in range(5):+ await controller.on_event({"id": str(index), "type": "mail"})+ spilled, carried = [], {}++ async def spill(name, payload, **_kwargs):+ assert name == "controller.queue.spill"+ items = payload["events"]+ if payload["prepend"]:+ spilled[0:0] = items+ else:+ spilled.extend(items)+ if [item[1]["id"] for item in items] == ["2", "3"]:+ await controller.on_event({"id": "late", "type": "mail"})+ await controller.on_event({"id": "late", "type": "mail"})++ monkeypatch.setattr(wf, "execute_activity", spill)+ monkeypatch.setattr(wf, "continue_as_new", lambda *, args: carried.update(args[0]))+ await controller._continue_as_new(control_data())+ assert [item[1]["id"] for item in carried["pending_events"] + spilled] == ["0", "1", "2", "3", "4", "late"]+ assert "mail:late" in carried["seen_event_ids"]+ assert carried["state"] == "paused"+++async def test_signal_while_handlers_finish_is_rechecked_before_rollover(wf, monkeypatch):+ controller = WorkflowControlWorkflow(control_data(state="paused", triggers={"mail": push_spec()}))+ await controller.on_event({"id": "first", "type": "mail"})+ waits, spilled, carried = 0, [], {}++ async def wait(predicate):+ nonlocal waits+ waits += 1+ if waits == 3:+ await controller.on_event({"id": "last-handler", "type": "mail"})+ assert predicate()++ async def spill(_name, payload, **_kwargs):+ spilled.extend(payload["events"])++ monkeypatch.setattr(wf, "wait_condition", wait)+ monkeypatch.setattr(wf, "execute_activity", spill)+ monkeypatch.setattr(wf, "continue_as_new", lambda *, args: carried.update(args[0]))+ await controller._continue_as_new(control_data())+ assert [item[1]["id"] for item in carried["pending_events"] + spilled] == ["first", "last-handler"]+++async def test_pending_retry_survives_recent_id_cache_eviction(wf):+ controller = WorkflowControlWorkflow(control_data(state="paused", triggers={"mail": push_spec()}))+ for index in range(module._MAX_CARRIED_SEEN_IDS + 10):+ await controller.on_event({"id": str(index), "type": "mail"})+ assert "mail:0" not in controller._seen_event_ids+ await controller.on_event({"id": "0", "type": "mail"})+ assert sum(event["id"] == "0" for _, event in controller._events) == 1+ assert len(controller._seen_event_ids) == module._MAX_CARRIED_SEEN_IDS+++async def test_pending_key_is_rebuilt_from_continuation(wf):+ controller = WorkflowControlWorkflow(control_data(+ state="paused", triggers={"mail": push_spec()},+ pending_events=[["mail", {"id": "old", "type": "mail"}]],+ seen_event_ids=[f"mail:recent-{index}" for index in range(module._MAX_CARRIED_SEEN_IDS)],+ ))+ await controller.on_event({"id": "old", "type": "mail"})+ assert len(controller._events) == 1+++async def test_dispatching_key_survives_arrival_cache_eviction(wf, monkeypatch):+ controller = WorkflowControlWorkflow()+ seen_queue = []++ async def spawn(event, _spec):+ assert event["id"] == "active"+ for index in range(module._MAX_CARRIED_SEEN_IDS + 1):+ await controller.on_event({"id": str(index), "type": "mail"})+ assert "mail:active" not in controller._seen_event_ids+ await controller.on_event({"id": "active", "type": "mail"})+ seen_queue.extend(item[1]["id"] for item in controller._events)+ controller._closed = True++ monkeypatch.setattr(controller, "_spawn_push_run", spawn)+ await controller.run(control_data(+ pending_events=[["mail", {"id": "active", "type": "mail"}]],+ triggers={"mail": push_spec()},+ ))+ assert "active" not in seen_queue+ assert "mail:active" not in controller._pending_event_ids+ assert "mail:active" in controller._seen_event_ids+++async def test_page_restores_pending_keys_and_retains_older_fifo_position(wf, monkeypatch):+ controller = WorkflowControlWorkflow()+ dispatched = []++ async def read(_name, _payload, **_kwargs):+ return {"events": [["mail", {"id": "durable", "type": "mail"}]], "more": False}++ async def spawn(event, _spec):+ dispatched.append(event["id"])+ if event["id"] == "carried":+ await controller.on_event({"id": "durable", "type": "mail"})+ await controller.on_event({"id": "new", "type": "mail"})+ if event["id"] == "durable":+ await controller.on_event({"id": "durable", "type": "mail"})+ if event["id"] == "new":+ controller._closed = True++ monkeypatch.setattr(wf, "execute_activity", read)+ monkeypatch.setattr(controller, "_spawn_push_run", spawn)+ await controller.run(control_data(+ overflow=True, durable_prefix=1,+ pending_events=[["mail", {"id": "carried", "type": "mail"}]],+ triggers={"mail": push_spec()},+ ))+ assert dispatched == ["carried", "durable", "new"]+ assert not controller._events+++async def test_duplicate_child_start_does_not_block_next_event(wf, monkeypatch):+ controller = WorkflowControlWorkflow()+ calls = []++ async def spawn(event, _spec):+ calls.append(event["id"])+ if event["id"] == "duplicate":+ raise WorkflowAlreadyStartedError("duplicate-child", "MachinaWorkflow")+ controller._closed = True++ monkeypatch.setattr(controller, "_spawn_push_run", spawn)+ await controller.run(control_data(+ pending_events=[["mail", {"id": value, "type": "mail"}] for value in ("duplicate", "next")],+ triggers={"mail": push_spec()},+ ))+ assert calls == ["duplicate", "next"]+ assert not controller._pending_event_ids+++async def test_transient_failure_keeps_same_event_pending_until_success(wf, monkeypatch):+ controller = WorkflowControlWorkflow()+ calls = []++ async def spawn(event, _spec):+ calls.append(event["id"])+ assert "mail:retry" in controller._pending_event_ids+ if len(calls) == 1:+ await controller.on_event(event)+ raise RuntimeError("temporary dispatch failure")+ controller._closed = True++ monkeypatch.setattr(controller, "_spawn_push_run", spawn)+ await controller.run(control_data(+ pending_events=[["mail", {"id": "retry", "type": "mail"}]],+ triggers={"mail": push_spec()},+ ))+ assert calls == ["retry", "retry"]+ assert not controller._events+ assert not controller._pending_event_ids+++async def test_reset_during_spill_does_not_resurrect_controller(wf, monkeypatch):+ monkeypatch.setattr(module, "_MAX_CARRIED_EVENTS", 1)+ controller = WorkflowControlWorkflow(control_data(state="paused", triggers={"mail": push_spec()}))+ for index in range(2):+ await controller.on_event({"id": str(index), "type": "mail"})+ continued = []++ async def spill(_name, _payload, **_kwargs):+ await controller.reset()++ monkeypatch.setattr(wf, "execute_activity", spill)+ monkeypatch.setattr(wf, "continue_as_new", lambda *, args: continued.append(args[0]))+ await controller._continue_as_new(control_data())+ assert not continued+ assert controller._closed is True+ assert controller._state == "resetting"+++async def test_patch_off_preserves_prior_v1_spill_commands(wf, monkeypatch):+ monkeypatch.setattr(module, "_MAX_CARRIED_EVENTS", 1)+ monkeypatch.setattr(wf, "patched", lambda name: name != "controller-event-accumulator-v1")+ controller = WorkflowControlWorkflow(control_data(state="paused", triggers={"mail": push_spec()}))+ for index in range(2):+ await controller.on_event({"id": str(index), "type": "mail"})+ commands, carried = [], {}++ async def spill(name, payload, **_kwargs):+ commands.append((name, payload["events"]))+ await controller.on_event({"id": "late", "type": "mail"})++ monkeypatch.setattr(wf, "execute_activity", spill)+ monkeypatch.setattr(wf, "continue_as_new", lambda *, args: carried.update(args[0]))+ await controller._continue_as_new(control_data())+ # Existing pre-marker histories scheduled one prefix and one tail spill.+ assert [item[1]["id"] for _, batch in commands for item in batch] == ["1", "late"]+ assert carried["pending_events"][0][1]["id"] == "0"+ assert controller._event_accumulator_v1 is False+ assert not controller._pending_event_ids+++async def test_patch_off_discards_provisional_index_without_changing_recent_ids(wf, monkeypatch):+ patches, dispatched = [], []++ def patched(name):+ patches.append(name)+ return name != "controller-event-accumulator-v1"++ monkeypatch.setattr(wf, "patched", patched)+ data = control_data(+ pending_events=[["mail", {"id": value, "type": "mail"}] for value in ("old", "next")],+ seen_event_ids=["mail:old", "mail:prior", "mail:next"],+ triggers={"mail": push_spec()},+ )+ controller = WorkflowControlWorkflow(data)+ assert not patches, "Initializing a carried controller must issue no patch commands"+ assert controller._pending_event_ids == {"mail:old", "mail:next"}++ async def spawn(event, _spec):+ assert not controller._pending_event_ids+ dispatched.append(event["id"])+ if event["id"] == "next":+ controller._closed = True++ monkeypatch.setattr(controller, "_spawn_push_run", spawn)+ await controller.run(data)+ assert dispatched == ["old", "next"]+ assert not controller._pending_event_ids+ assert list(controller._seen_event_ids) == ["mail:old", "mail:prior", "mail:next"]+++async def test_legacy_controller_does_not_record_accumulator_marker(wf, monkeypatch):+ patches = []+ monkeypatch.setattr(wf, "patched", lambda name: patches.append(name) or True)+ controller = WorkflowControlWorkflow()+ controller._triggers["mail"] = push_spec()+ await controller.on_event({"id": "legacy", "type": "mail"})+ assert "controller-event-accumulator-v1" not in patches+ assert not controller._pending_event_idsdiff --git a/server/tests/temporal/test_controller_accumulator_integration.py b/server/tests/temporal/test_controller_accumulator_integration.pynew file mode 100644index 00000000..4121f7d2--- /dev/null+++ b/server/tests/temporal/test_controller_accumulator_integration.py@@ -0,0 +1,354 @@+"""Native controller accumulation, spill handoff, restart, and patch replay."""++from __future__ import annotations++import asyncio+import subprocess+import sys+from collections import Counter+from datetime import timedelta+from pathlib import Path+from uuid import uuid4++from temporalio import activity, workflow+from temporalio.api.enums.v1 import EventType, IndexedValueType+from temporalio.api.operatorservice.v1 import AddSearchAttributesRequest+from temporalio.testing import WorkflowEnvironment+from temporalio.worker import Replayer, Worker++from services.temporal.execution_control import ExecutionControl+from services.temporal.execution_control_activities import ExecutionControlActivities+from services.temporal.workflow_control_workflow import WorkflowControlWorkflow+++_PATCH = "controller-event-accumulator-v1"+_spill_started: asyncio.Event+_spill_release: asyncio.Event+_spill_blocked: bool+_spilled: list[list]+_spill_keys: set[tuple[str, str]]+_read_receipts: dict[str, dict]+_recorded: list[str]++[email protected](name="controller.queue.spill")+async def _spill(payload: dict) -> dict:+ global _spill_blocked+ # The first tail spill is the last Activity in the original rollover+ # snapshot. Signals delivered while it is in flight must be carried or+ # spilled too, rather than disappearing when that snapshot continues.+ if not _spill_blocked:+ _spill_blocked = True+ _spill_started.set()+ await _spill_release.wait()+ incoming = []+ for listener_id, event in payload["events"]:+ key = (listener_id, event["id"])+ if key not in _spill_keys:+ _spill_keys.add(key)+ incoming.append([listener_id, event])+ if payload.get("prepend"):+ _spilled[0:0] = incoming+ else:+ _spilled.extend(incoming)+ return {"spilled": len(incoming)}++[email protected](name="controller.queue.read")+async def _read(payload: dict) -> dict:+ # Model the production Activity's mutation receipt: an Activity retry+ # receives the same page, even after its rows have been removed.+ request_id = payload["request_id"]+ if request_id not in _read_receipts:+ page = _spilled[:100]+ del _spilled[:100]+ _read_receipts[request_id] = {"events": page, "more": bool(_spilled)}+ return _read_receipts[request_id]++[email protected](name="broadcast_trigger_status_activity")+async def _broadcast(_payload: dict) -> None:+ return None++[email protected](name="evaluate_trigger_filter_activity")+async def _filter(_payload: dict) -> bool:+ return True++[email protected](name="accumulator.record")+async def _record(payload: dict) -> str:+ event_id = payload["event_id"]+ _recorded.append(event_id)+ return event_id++[email protected](name="MachinaWorkflow", sandboxed=False)+class AccumulatorGraphProbe:+ """A short real child, with the same independent-root admission contract."""++ @workflow.init+ def __init__(self, payload: dict | None = None):+ self.control = ExecutionControl()+ if payload is not None:+ self.control.bind(payload)++ @workflow.update+ async def set_control_state(self, payload: dict) -> dict:+ return await self.control.set_control_state(payload)++ @workflow.update+ async def wait_for_checkpoint(self, payload: dict | None = None) -> dict:+ return await self.control.wait_for_checkpoint(payload)++ @workflow.query+ def execution_control_status(self) -> dict:+ return self.control.status()++ @workflow.run+ async def run(self, payload: dict | None = None) -> str:+ payload = payload or {}+ self.control.bind(payload)+ await self.control.register_root()+ event_id = payload.get("probe_event")+ if event_id is None:+ trigger = next(node for node in payload["nodes"] if node.get("_pre_executed"))+ event_id = trigger["_trigger_output"]["_event_envelope"]["id"]+ async with self.control.action():+ result = await workflow.execute_activity("accumulator.record", {"event_id": event_id},+ start_to_close_timeout=timedelta(seconds=30))+ await self.control.unregister_root()+ await workflow.wait_condition(workflow.all_handlers_finished)+ return result+++def _trigger(slug: str) -> dict:+ return {"listener_id": "push", "trigger_node_id": "trigger", "event_type": "probe.event",+ "workflow_type": "TriggerListenerWorkflow", "listener_args": {+ "trigger_node_id": "trigger", "trigger_label": "trigger", "node_type": "webhookTrigger",+ "workflow_slug": slug, "workflow_id": None, "graphVersion": 2, "generation": 7,+ "data_scope_id": "generation-7", "filter_params": {"test": True},+ "nodes": [{"id": "trigger", "type": "webhookTrigger", "data": {}}], "edges": [],+ }}+++def _event(event_id: str) -> dict:+ return {"id": event_id, "type": "probe.event", "data": {"value": event_id}}+++async def _wait_status(handle, predicate):+ for _ in range(300):+ state = await handle.query("status")+ if predicate(state):+ return state+ await asyncio.sleep(0.02)+ raise AssertionError(f"Controller did not reach expected state: {state}")+++async def _wait_recorded(event_ids):+ for _ in range(500):+ if all(event_id in _recorded for event_id in event_ids):+ return+ await asyncio.sleep(0.02)+ raise AssertionError(f"Expected graph runs {event_ids}; recorded {_recorded}")+++async def _chain_histories(client, workflow_id: str, first_run_id: str):+ histories = []+ run_id = first_run_id+ while True:+ history = await client.get_workflow_handle(workflow_id, run_id=run_id).fetch_history()+ histories.append(history)+ continued = next((event.workflow_execution_continued_as_new_event_attributes+ for event in history.events+ if event.event_type == EventType.EVENT_TYPE_WORKFLOW_EXECUTION_CONTINUED_AS_NEW), None)+ if continued is None:+ return histories+ run_id = continued.new_execution_run_id+++async def _register_search_attributes(client):+ await client.operator_service.add_search_attributes(AddSearchAttributesRequest(+ namespace=client.namespace,+ search_attributes={"ControlEventTypes": IndexedValueType.INDEXED_VALUE_TYPE_KEYWORD_LIST},+ ))+++def _has_accumulator_patch(history) -> bool:+ return any(_PATCH.encode() in payload.data+ for event in history.events+ if event.event_type == EventType.EVENT_TYPE_MARKER_RECORDED+ for payloads in event.marker_recorded_event_attributes.details.values()+ for payload in payloads.payloads)+++async def _run_accumulator_gate():+ global _spill_started, _spill_release, _spill_blocked, _spilled, _spill_keys, _read_receipts, _recorded+ _spill_started, _spill_release = asyncio.Event(), asyncio.Event()+ _spill_blocked, _spilled, _spill_keys, _read_receipts, _recorded = False, [], set(), {}, []+ from services.temporal import workflow_control_workflow as controller_module++ original_pressure = WorkflowControlWorkflow._history_pressure+ original_carry_limit = controller_module._MAX_CARRIED_EVENTS+ original_seen_limit = controller_module._MAX_CARRIED_SEEN_IDS+ # Small, deterministic pressure avoids thousands of Signals. The same+ # production method override is used for execution and history replay.+ WorkflowControlWorkflow._history_pressure = lambda self: len(self._events) >= 3 and not self._overflow+ controller_module._MAX_CARRIED_EVENTS = 2+ controller_module._MAX_CARRIED_SEEN_IDS = 2+ definitions = [WorkflowControlWorkflow, AccumulatorGraphProbe]+ try:+ async with await WorkflowEnvironment.start_time_skipping() as environment:+ client = environment.client+ await _register_search_attributes(client)+ queue = "controller-accumulator-" + uuid4().hex+ slug = "accumulator-" + uuid4().hex+ controller_id = "controller-" + uuid4().hex+ activities = [_spill, _read, _broadcast, _filter, _record,+ *ExecutionControlActivities(client).activities()]+ async with Worker(client, task_queue=queue, workflows=definitions, activities=activities,+ max_cached_workflows=0, identity="accumulator-before-restart"):+ controller = await client.start_workflow("WorkflowControlWorkflow", {+ "execution_control_version": 1, "generation": 7, "state": "paused",+ "revision": 1, "participant_state": "paused", "participant_revision": 1,+ "controller_workflow_id": controller_id,+ }, id=controller_id, task_queue=queue)+ await controller.signal("register_trigger", _trigger(slug))+ await _wait_status(controller, lambda state: "push" in state["triggers"])+ for event_id in ("A", "A", "B", "B"):+ await controller.signal("on_event", _event(event_id))+ await _wait_status(controller, lambda state: state["queued_events"] == 2)+ assert _recorded == []+ await controller.signal("on_event", _event("C"))+ await asyncio.wait_for(_spill_started.wait(), timeout=15)+ await controller.signal("on_event", _event("D"))+ await controller.signal("on_event", _event("D"))+ await controller.signal("on_event", _event("A"))+ # An acknowledged Update after the Signals ensures their+ # handlers ran before completing the last original spill.+ status = await controller.execute_update("set_control_state", {+ "state": "paused", "revision": 1, "generation": 7,+ })+ assert status["queued_events"] == 4+ _spill_release.set()+ for _ in range(300):+ description = await controller.describe()+ if description.run_id != controller.first_execution_run_id:+ break+ await asyncio.sleep(0.02)+ assert description.run_id != controller.first_execution_run_id+ stopped = await controller.query("status")+ assert stopped["state"] == "paused"+ assert stopped["queued_events"] == 2+ assert [item[1]["id"] for item in _spilled] == ["C", "D"]+ assert _recorded == []++ async with Worker(client, task_queue=queue, workflows=definitions, activities=activities,+ max_cached_workflows=0, identity="accumulator-after-restart"):+ stopped = await controller.execute_update("set_control_state", {+ "state": "paused", "revision": 1, "generation": 7,+ })+ assert stopped["state"] == "paused"+ # Dedup applies to events held in memory and the spillway.+ for event_id in ("A", "C", "D"):+ await controller.signal("on_event", _event(event_id))+ await controller.execute_update("set_control_state", {+ "state": "running", "revision": 2, "generation": 7, "producers_held": False,+ })+ await _wait_recorded("ABCD")+ with environment.auto_time_skipping_disabled():+ results = await asyncio.wait_for(asyncio.gather(*(client.get_workflow_handle(+ f"{slug}-trigger-{event_id}").result() for event_id in ("A", "B", "C", "D"))), timeout=20)+ assert results == ["A", "B", "C", "D"]+ assert Counter(_recorded) == Counter({"A": 1, "B": 1, "C": 1, "D": 1})+ assert _spilled == []++ # A stable event-derived child ID may already have completed+ # even after the controller's dedup window has expired. Native+ # duplicate-start rejection must not poison the queue head.+ existing = await client.start_workflow("MachinaWorkflow", {"probe_event": "E"},+ id=f"{slug}-trigger-E", task_queue=queue)+ assert await existing.result() == "E"+ await controller.signal("on_event", _event("E"))+ await controller.signal("on_event", _event("F"))+ await _wait_recorded("F")+ with environment.auto_time_skipping_disabled():+ assert await asyncio.wait_for(client.get_workflow_handle(f"{slug}-trigger-F").result(), timeout=20) == "F"+ await _wait_status(controller, lambda state: state["queued_events"] == 0 and not state["live_roots"])+ assert Counter(_recorded) == Counter({event_id: 1 for event_id in "ABCDEF"})+ await controller.signal("reset")+ await controller.result()+ histories = await _chain_histories(client, controller_id, controller.first_execution_run_id)+ histories.extend([await client.get_workflow_handle(f"{slug}-trigger-{event_id}").fetch_history()+ for event_id in "ABCDEF"])++ controller_histories = [history for history in histories if history.workflow_id == controller_id]+ assert len(controller_histories) == 2+ assert all(_has_accumulator_patch(history) for history in controller_histories)+ child_starts = [event.start_child_workflow_execution_initiated_event_attributes.workflow_id+ for history in controller_histories for event in history.events+ if event.event_type == EventType.EVENT_TYPE_START_CHILD_WORKFLOW_EXECUTION_INITIATED]+ assert child_starts == [f"{slug}-trigger-{event_id}" for event_id in "ABCDEF"]+ for history in histories:+ replay = await Replayer(workflows=definitions).replay_workflow(history)+ assert replay.replay_failure is None+ finally:+ WorkflowControlWorkflow._history_pressure = original_pressure+ controller_module._MAX_CARRIED_EVENTS = original_carry_limit+ controller_module._MAX_CARRIED_SEEN_IDS = original_seen_limit+++async def _run_pre_marker_replay():+ global _recorded+ _recorded = []+ original_patched = workflow.patched+ # Record the actual prior command path with the new marker absent, then+ # replay with the normal SDK patched() implementation. No edited history+ # is substituted for a genuine pre-marker execution.+ workflow.patched = lambda patch_id: False if patch_id == _PATCH else original_patched(patch_id)+ definitions = [WorkflowControlWorkflow, AccumulatorGraphProbe]+ histories = []+ try:+ async with await WorkflowEnvironment.start_time_skipping() as environment:+ client = environment.client+ await _register_search_attributes(client)+ queue = "controller-accumulator-legacy-" + uuid4().hex+ activities = [_broadcast, _filter, _record, *ExecutionControlActivities(client).activities()]+ async with Worker(client, task_queue=queue, workflows=definitions, activities=activities,+ max_cached_workflows=0):+ for version in (0, 1):+ slug = "pre-marker-" + uuid4().hex+ controller_id = "controller-" + uuid4().hex+ controller = await client.start_workflow("WorkflowControlWorkflow", {+ "execution_control_version": version, "generation": 7,+ "controller_workflow_id": controller_id,+ }, id=controller_id, task_queue=queue)+ await controller.signal("register_trigger", _trigger(slug))+ await _wait_status(controller, lambda state: "push" in state["triggers"])+ await controller.signal("on_event", _event(str(version)))+ await _wait_recorded([str(version)])+ child = client.get_workflow_handle(f"{slug}-trigger-{version}")+ with environment.auto_time_skipping_disabled():+ assert await asyncio.wait_for(child.result(), timeout=15) == str(version)+ await controller.signal("reset")+ await controller.result()+ histories.extend([await controller.fetch_history(), await child.fetch_history()])+ workflow.patched = original_patched+ assert not any(_has_accumulator_patch(history) for history in histories)+ for history in histories:+ replay = await Replayer(workflows=definitions).replay_workflow(history)+ assert replay.replay_failure is None+ finally:+ workflow.patched = original_patched+++def test_controller_accumulation_spill_signal_restart_and_replay():+ completed = subprocess.run([sys.executable, "-m", "tests.temporal.test_controller_accumulator_integration"],+ cwd=Path(__file__).parents[2], stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,+ stderr=subprocess.STDOUT, timeout=150, check=False)+ assert completed.returncode == 0, completed.stdout.decode(errors="replace")[-12000:]+++if __name__ == "__main__":+ asyncio.run(_run_accumulator_gate())+ asyncio.run(_run_pre_marker_replay())diff --git a/server/tests/temporal/test_controller_messaging.py b/server/tests/temporal/test_controller_messaging.pynew file mode 100644index 00000000..3f24c6f6--- /dev/null+++ b/server/tests/temporal/test_controller_messaging.py@@ -0,0 +1,286 @@+"""Acknowledged controller messages respect suspension and history rollover."""++import asyncio+from copy import deepcopy+from types import SimpleNamespace+from unittest.mock import MagicMock++import pytest+from temporalio.converter import DefaultPayloadConverter+from temporalio.exceptions import ApplicationError++from services.temporal import workflow_control_workflow as module+from services.temporal.workflow_control_workflow import WorkflowControlWorkflow+++def control_data(**extra):+ return {+ "execution_control_version": 1,+ "generation": 7,+ "controller_workflow_id": "controller",+ **extra,+ }+++def registration():+ return {"generation": 7, "workflow_id": "root", "first_execution_run_id": "first"}+++def runtime(monkeypatch):+ patches = []++ def patched(name):+ patches.append(name)+ return True++ async def wait(predicate):+ while not predicate():+ await asyncio.sleep(0)++ monkeypatch.setattr(module.workflow, "patched", patched)+ monkeypatch.setattr(module.workflow, "info", lambda: SimpleNamespace(+ workflow_id="controller", run_id="run", first_execution_run_id="controller-first",+ ))+ monkeypatch.setattr(module.workflow, "wait_condition", wait)+ monkeypatch.setattr(module.workflow, "all_handlers_finished", lambda: True)+ monkeypatch.setattr(module.workflow, "payload_converter", DefaultPayloadConverter)+ monkeypatch.setattr(module.workflow, "logger", MagicMock())+ monkeypatch.setattr(WorkflowControlWorkflow, "_history_pressure", lambda _: False)+ return patches++[email protected]("message", ["state", "legacy", "checkpoint", "register", "unregister", "pause", "replace"])+async def test_successful_updates_request_main_thread_rollover(runtime, monkeypatch, message):+ controller = WorkflowControlWorkflow(control_data())+ assert not runtime, "Workflow initialization must not issue patch commands"+ monkeypatch.setattr(controller, "_history_pressure", lambda: True)+ continue_as_new = MagicMock()+ monkeypatch.setattr(module.workflow, "continue_as_new", continue_as_new)++ if message == "state":+ await controller.set_control_state({"state": "paused", "revision": 1})+ elif message == "legacy":+ await controller.set_control_state("paused")+ elif message == "checkpoint":+ await controller.wait_for_checkpoint()+ elif message == "register":+ await controller.register_execution(registration())+ elif message == "unregister":+ await controller.unregister_execution(registration())+ elif message == "pause":+ await controller.pause_admissions()+ else:+ controller._state = "paused"+ await controller.replace_graph({"nodes": [], "edges": []})++ assert controller._can_requested is True+ assert runtime == ["controller-messaging-v1"]+ continue_as_new.assert_not_called()+++async def test_registration_alone_wakes_an_idle_controller(runtime, monkeypatch):+ data = control_data()+ controller = WorkflowControlWorkflow(data)+ waiting = asyncio.Event()++ async def wait(predicate):+ if not predicate():+ waiting.set()+ while not predicate():+ await asyncio.sleep(0)++ class Continued(BaseException):+ pass++ carried = {}++ def continue_as_new(*, args):+ carried.update(args[0])+ raise Continued()++ monkeypatch.setattr(module.workflow, "wait_condition", wait)+ monkeypatch.setattr(module.workflow, "continue_as_new", continue_as_new)+ run = asyncio.create_task(controller.run(data))+ await asyncio.wait_for(waiting.wait(), 1)+ monkeypatch.setattr(controller, "_history_pressure", lambda: True)+ await controller.register_execution(registration())+ with pytest.raises(Continued):+ await asyncio.wait_for(run, 1)+ assert carried["live_roots"]["root"]["first_execution_run_id"] == "first"+ assert carried["membership_epoch"] == 1++[email protected]("state,held", [("paused", False), ("running", True)])+async def test_legacy_producer_resume_cannot_release_generation_suspension(runtime, state, held):+ controller = WorkflowControlWorkflow(control_data())+ await controller.set_control_state({"state": state, "revision": 4, "producers_held": held})+ before = controller._execution_control.carry()++ result = await controller.set_control_state("running")++ assert result["state"] == "paused"+ assert result["participant_state"] == state+ assert result["producers_held"] is held+ assert controller._execution_control.carry() == before++[email protected]("state,held", [("paused", False), ("running", True)])+async def test_producer_pause_cannot_drain_reviews_during_generation_suspension(runtime, state, held):+ controller = WorkflowControlWorkflow(control_data())+ controller._triggers["review"] = {+ "trigger_node_id": "review-node", "listener_args": {"node_type": "taskTrigger"},+ }+ controller._events.append(("review", {"id": "review-event"}))+ await controller.set_control_state({"state": state, "revision": 4, "producers_held": held})++ result = await controller.pause_admissions(drain_tasks=True)++ assert result["state"] == "paused"+ assert controller._drain_tasks is False+ assert controller._next_event() is None+++async def test_producer_pause_keeps_normal_safe_apply_review_drain_and_restore(runtime):+ controller = WorkflowControlWorkflow(control_data())+ controller._triggers.update({+ "ordinary": {"trigger_node_id": "ordinary-node", "listener_args": {"node_type": "mailTrigger"}},+ "review": {"trigger_node_id": "review-node", "listener_args": {"node_type": "taskTrigger"}},+ })+ controller._events.extend([+ ("ordinary", {"id": "ordinary-event"}), ("review", {"id": "review-event"}),+ ])++ await controller.pause_admissions(drain_tasks=True)+ assert controller._next_event() == 1+ assert controller._execution_control.state == "running"+ assert controller._execution_control.producers_held is False+ await controller.set_control_state("running")+ assert controller.status()["state"] == "running"+ assert controller._next_event() == 0+ assert controller._drain_tasks is False+++async def test_legacy_restore_cannot_reopen_a_closed_controller(runtime):+ controller = WorkflowControlWorkflow(control_data())+ await controller.reset()+ result = await controller.set_control_state("running")+ assert result["state"] == "resetting"+ assert controller._closed is True+ assert controller._can_requested is False+++async def test_pre_marker_v1_retains_prior_update_and_producer_commands(runtime, monkeypatch):+ monkeypatch.setattr(module.workflow, "patched", lambda name: name != "controller-messaging-v1")+ controller = WorkflowControlWorkflow(control_data())+ monkeypatch.setattr(controller, "_history_pressure", lambda: True)+ await controller.set_control_state({"state": "paused", "revision": 4})+ await controller.pause_admissions(drain_tasks=True)+ assert controller._drain_tasks is True+ result = await controller.set_control_state("running")+ assert result["state"] == "running"+ assert result["participant_state"] == "paused"+ assert controller._can_requested is False+ assert controller._messaging_v1 is False+++async def test_legacy_generation_updates_do_not_record_the_new_patch(runtime, monkeypatch):+ controller = WorkflowControlWorkflow()+ monkeypatch.setattr(controller, "_history_pressure", lambda: True)+ await controller.pause_admissions(drain_tasks=True)+ await controller.set_control_state("running")+ await controller.wait_for_checkpoint()+ assert not runtime+ assert controller._can_requested is False++[email protected]("payload,error_type", [+ ({"state": "paused", "revision": []}, "InvalidWorkflowControlRevision"),+ ({"state": "paused", "revision": 1, "expected_membership_epoch": []}, "InvalidExecutionMembershipEpoch"),+ ({"state": "paused", "revision": 1, "hold_admissions": "yes"}, "InvalidWorkflowControlRequest"),+])+async def test_invalid_control_messages_rejected_without_mutation(runtime, payload, error_type):+ controller = WorkflowControlWorkflow(control_data())+ before = controller.status()+ for apply in (controller.validate_control_state, controller.set_control_state):+ with pytest.raises(ApplicationError) as exc:+ result = apply(payload)+ if asyncio.iscoroutine(result):+ await result+ assert exc.value.type == error_type+ assert controller.status() == before+ assert not runtime, "A rejected validator must issue no patch commands"+++def test_checkpoint_validator_is_read_only_and_accepts_omitted_payload(runtime):+ controller = WorkflowControlWorkflow(control_data())+ controller.validate_checkpoint()+ before = controller.status()+ controller.validate_checkpoint({"state": "paused", "revision": 8})+ assert controller.status() == before+ with pytest.raises(ApplicationError):+ controller.validate_checkpoint({"state": [], "revision": 8})+ assert not runtime+++async def test_registration_shape_is_rejected_in_validator_and_handler(runtime):+ controller = WorkflowControlWorkflow(control_data())+ for validate, handler in (+ (controller.validate_registration, controller.register_execution),+ (controller.validate_unregistration, controller.unregister_execution),+ ):+ with pytest.raises(ApplicationError) as exc:+ validate([])+ assert exc.value.type == "InvalidExecutionRegistration"+ with pytest.raises(ApplicationError):+ await handler([])+ assert controller.status()["live_roots"] == {}+ assert controller.status()["membership_epoch"] == 0+ assert not runtime++[email protected]("graph", [+ None,+ {},+ {"nodes": []},+ {"nodes": {}, "edges": []},+ {"nodes": [{}], "edges": []},+ {"nodes": [{"id": []}], "edges": []},+ {"nodes": [{"id": "trigger"}], "edges": None},+])+async def test_invalid_graph_update_preserves_revision_and_existing_snapshots(runtime, graph):+ controller = WorkflowControlWorkflow(control_data(state="paused", revision=3, triggers={+ "listener": {+ "trigger_node_id": "trigger", "listener_args": {+ "nodes": [{"id": "trigger"}], "edges": [], "parameter_snapshot": {"old": True},+ },+ },+ }))+ before = controller.status()+ triggers = deepcopy(controller._triggers)+ for apply in (controller.validate_graph, controller.replace_graph):+ with pytest.raises(ApplicationError) as exc:+ result = apply(graph)+ if asyncio.iscoroutine(result):+ await result+ assert exc.value.type == "InvalidWorkflowGraphSnapshot"+ assert controller.status() == before+ assert controller._triggers == triggers+ assert not runtime+++async def test_graph_message_preserves_existing_hashable_id_contract(runtime):+ controller = WorkflowControlWorkflow(control_data(state="paused", triggers={+ "listener": {"trigger_node_id": 7, "listener_args": {}},+ }))+ graph = {"nodes": [{"id": 7}], "edges": []}+ controller.validate_graph(graph)+ result = await controller.replace_graph(graph)+ assert result["applied"] is True+ assert controller._triggers["listener"]["listener_args"]["nodes"] == graph["nodes"]+++def test_controller_registers_read_only_control_and_membership_validators():+ definitions = WorkflowControlWorkflow.__temporal_workflow_definition.updates+ for name in ("set_control_state", "wait_for_checkpoint", "register_execution", "unregister_execution", "replace_graph"):+ assert definitions[name].validator is not Nonediff --git a/server/tests/temporal/test_controller_messaging_integration.py b/server/tests/temporal/test_controller_messaging_integration.pynew file mode 100644index 00000000..bb00ac9e--- /dev/null+++ b/server/tests/temporal/test_controller_messaging_integration.py@@ -0,0 +1,257 @@+"""Native Update validation, Update-only rollover, and messaging patch replay."""++from __future__ import annotations++import asyncio+import subprocess+import sys+from pathlib import Path+from uuid import uuid4++from temporalio import workflow+from temporalio.api.enums.v1 import EventType+from temporalio.client import WorkflowUpdateFailedError+from temporalio.exceptions import ApplicationError+from temporalio.testing import WorkflowEnvironment+from temporalio.worker import Replayer, Worker++from services.temporal.workflow_control_workflow import WorkflowControlWorkflow+++_PATCH = "controller-messaging-v1"+++def _history_pressure(controller: WorkflowControlWorkflow) -> bool:+ # The time-skipping test server does not expose production dynamic config+ # for a low accepted-Update limit. Use a deterministic pressure hook for+ # the original run, and keep the same hook installed during replay. All+ # messages, their acknowledgements, and Continue-As-New still use the SDK.+ return not workflow.info().continued_run_id and controller._membership_epoch >= 3+++def _has_patch(history) -> bool:+ return any(_PATCH.encode() in payload.data+ for event in history.events+ if event.event_type == EventType.EVENT_TYPE_MARKER_RECORDED+ for payloads in event.marker_recorded_event_attributes.details.values()+ for payload in payloads.payloads)+++def _accepted_ids(histories) -> set[str]:+ return {event.workflow_execution_update_accepted_event_attributes.accepted_request.meta.update_id+ for history in histories for event in history.events+ if event.event_type == EventType.EVENT_TYPE_WORKFLOW_EXECUTION_UPDATE_ACCEPTED}+++def _completed_ids(histories) -> set[str]:+ return {event.workflow_execution_update_completed_event_attributes.meta.update_id+ for history in histories for event in history.events+ if event.event_type == EventType.EVENT_TYPE_WORKFLOW_EXECUTION_UPDATE_COMPLETED}+++async def _chain_histories(client, handle):+ histories = []+ run_id = handle.first_execution_run_id+ while True:+ history = await client.get_workflow_handle(handle.id, run_id=run_id).fetch_history()+ histories.append(history)+ continued = next((event.workflow_execution_continued_as_new_event_attributes+ for event in history.events+ if event.event_type == EventType.EVENT_TYPE_WORKFLOW_EXECUTION_CONTINUED_AS_NEW), None)+ if continued is None:+ return histories+ run_id = continued.new_execution_run_id+++async def _reject_update(handle, method: str, payload: dict, update_id: str, error_type: str):+ try:+ await asyncio.wait_for(handle.execute_update(method, payload, id=update_id), timeout=15)+ except WorkflowUpdateFailedError as exc:+ assert isinstance(exc.cause, ApplicationError)+ assert exc.cause.type == error_type, repr(exc.cause)+ else:+ raise AssertionError(f"Malformed Update {update_id} was accepted")+++async def _wait_for_rollover(handle):+ for _ in range(300):+ description = await handle.describe()+ if description.run_id != handle.first_execution_run_id:+ return description.run_id+ await asyncio.sleep(0.02)+ raise AssertionError("Register/unregister Update traffic did not roll the controller over")+++def _registration(workflow_id: str) -> dict:+ return {"generation": 7, "workflow_id": workflow_id,+ "first_execution_run_id": str(uuid4())}+++async def _run_messaging_gate():+ original_pressure = WorkflowControlWorkflow._history_pressure+ WorkflowControlWorkflow._history_pressure = _history_pressure+ try:+ async with await WorkflowEnvironment.start_time_skipping() as environment:+ client = environment.client+ queue = "controller-messaging-" + uuid4().hex+ controller_id = "controller-" + uuid4().hex+ successful_ids: set[str] = set()+ rejected_ids = {"bad-revision", "bad-epoch", "bad-checkpoint"}+ async with Worker(client, task_queue=queue, workflows=[WorkflowControlWorkflow],+ max_cached_workflows=0, identity="messaging-before-rollover"):+ handle = await client.start_workflow("WorkflowControlWorkflow", {+ "execution_control_version": 1, "generation": 7,+ "controller_workflow_id": controller_id,+ "state": "paused", "revision": 1,+ "participant_state": "paused", "participant_revision": 1,+ }, id=controller_id, task_queue=queue)+ await _reject_update(handle, "set_control_state", {+ "state": "running", "revision": "malformed", "generation": 7,+ }, "bad-revision", "InvalidWorkflowControlRevision")+ await _reject_update(handle, "set_control_state", {+ "state": "running", "revision": 2, "generation": 7,+ "producers_held": False, "expected_membership_epoch": "malformed",+ }, "bad-epoch", "InvalidExecutionMembershipEpoch")+ await _reject_update(handle, "wait_for_checkpoint", {+ "state": "paused", "revision": "malformed", "generation": 7,+ }, "bad-checkpoint", "InvalidWorkflowControlRevision")++ # A rejected validator leaves the execution usable and its+ # acknowledged state untouched. Enrollment alone then wakes+ # an otherwise idle controller's actual rollover path.+ stopped = await handle.execute_update("set_control_state", {+ "state": "paused", "revision": 1, "generation": 7,+ }, id="valid-after-rejection")+ successful_ids.add("valid-after-rejection")+ assert stopped["state"] == "paused" and stopped["revision"] == 1+ keeper = _registration("keeper-" + uuid4().hex)+ temporary = _registration("temporary-" + uuid4().hex)+ registered = await handle.execute_update("register_execution", keeper, id="register-keeper")+ successful_ids.add("register-keeper")+ assert registered["membership_epoch"] == 1+ registered = await handle.execute_update("register_execution", temporary, id="register-temporary")+ successful_ids.add("register-temporary")+ assert registered["membership_epoch"] == 2+ unregistered = await handle.execute_update("unregister_execution", temporary, id="unregister-temporary")+ successful_ids.add("unregister-temporary")+ assert unregistered["membership_epoch"] == 3+ assert unregistered["live_roots"] == {keeper["workflow_id"]: {+ "workflow_id": keeper["workflow_id"],+ "first_execution_run_id": keeper["first_execution_run_id"],+ }}+ await _wait_for_rollover(handle)++ # A new worker reconstructs the carried entity state from history;+ # no Signal, polling trigger, or business Activity caused rollover.+ async with Worker(client, task_queue=queue, workflows=[WorkflowControlWorkflow],+ max_cached_workflows=0, identity="messaging-after-rollover"):+ carried = await handle.execute_update("wait_for_checkpoint", None, id="carried-checkpoint")+ successful_ids.add("carried-checkpoint")+ assert carried["state"] == "paused" and carried["revision"] == 1+ assert carried["membership_epoch"] == 3+ assert carried["live_roots"] == unregistered["live_roots"]+ assert carried["checkpoint"] is True+ assert (await handle.execute_update("set_control_state", "resume", id="paused-legacy-resume"))["state"] == "paused"+ successful_ids.add("paused-legacy-resume")+ held = await handle.execute_update("set_control_state", {+ "state": "running", "revision": 2, "generation": 7,+ "producers_held": True,+ }, id="hold-resume")+ successful_ids.add("hold-resume")+ assert held["state"] == "paused" and held["participant_state"] == "running"+ legacy = await handle.execute_update("set_control_state", "resume", id="held-legacy-resume")+ successful_ids.add("held-legacy-resume")+ assert legacy["state"] == "paused" and legacy["producers_held"] is True+ assert legacy["participant_revision"] == 2+ review = await handle.execute_update("pause_admissions", True, id="held-review-drain")+ successful_ids.add("held-review-drain")+ assert review["state"] == "paused" and review["producers_held"] is True+ released = await handle.execute_update("set_control_state", {+ "state": "running", "revision": 2, "generation": 7,+ "producers_held": False, "expected_membership_epoch": 3,+ }, id="release-resume")+ successful_ids.add("release-resume")+ assert released["state"] == "running" and released["producers_held"] is False+ await handle.signal("reset")+ await handle.result()+ histories = await _chain_histories(client, handle)++ assert len(histories) == 2+ assert all(_has_patch(history) for history in histories)+ assert not (_accepted_ids(histories) & rejected_ids)+ assert _accepted_ids(histories) == successful_ids+ assert _completed_ids(histories) == successful_ids+ assert not any(event.event_type == EventType.EVENT_TYPE_WORKFLOW_TASK_FAILED+ for history in histories for event in history.events)+ assert not any(event.event_type in {+ EventType.EVENT_TYPE_ACTIVITY_TASK_SCHEDULED,+ EventType.EVENT_TYPE_START_CHILD_WORKFLOW_EXECUTION_INITIATED,+ } for history in histories for event in history.events)+ # The only Signal is the terminal cleanup after both runs finish+ # their messaging checks; Update-only traffic drove continuation.+ assert sum(event.event_type == EventType.EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED+ for history in histories for event in history.events) == 1+ for history in histories:+ replay = await Replayer(workflows=[WorkflowControlWorkflow]).replay_workflow(history)+ assert replay.replay_failure is None+ finally:+ WorkflowControlWorkflow._history_pressure = original_pressure+++async def _run_pre_marker_replay():+ original_pressure = WorkflowControlWorkflow._history_pressure+ original_patched = workflow.patched+ WorkflowControlWorkflow._history_pressure = _history_pressure+ workflow.patched = lambda patch_id: False if patch_id == _PATCH else original_patched(patch_id)+ histories = []+ try:+ async with await WorkflowEnvironment.start_time_skipping() as environment:+ client = environment.client+ queue = "controller-messaging-legacy-" + uuid4().hex+ async with Worker(client, task_queue=queue, workflows=[WorkflowControlWorkflow],+ max_cached_workflows=0, identity="messaging-pre-marker"):+ for version in (0, 1):+ controller_id = "controller-" + uuid4().hex+ handle = await client.start_workflow("WorkflowControlWorkflow", {+ "execution_control_version": version, "generation": 7,+ "controller_workflow_id": controller_id,+ "state": "paused", "revision": 1,+ "participant_state": "paused", "participant_revision": 1,+ }, id=controller_id, task_queue=queue)+ if version == 1:+ keeper = _registration("legacy-keeper-" + uuid4().hex)+ temporary = _registration("legacy-temporary-" + uuid4().hex)+ await handle.execute_update("register_execution", keeper)+ await handle.execute_update("register_execution", temporary)+ status = await handle.execute_update("unregister_execution", temporary)+ assert status["membership_epoch"] == 3+ # Pre-marker v1 retains its recorded legacy command path.+ # Fresh v1's stronger producer guard applies only when+ # the new marker has been recorded, as tested above.+ resumed = await handle.execute_update("set_control_state", "resume")+ assert resumed["state"] == "running"+ await handle.execute_update("pause_admissions", False)+ assert (await handle.describe()).run_id == handle.first_execution_run_id+ await handle.signal("reset")+ await handle.result()+ histories.append(await handle.fetch_history())+ workflow.patched = original_patched+ assert not any(_has_patch(history) for history in histories)+ for history in histories:+ replay = await Replayer(workflows=[WorkflowControlWorkflow]).replay_workflow(history)+ assert replay.replay_failure is None+ finally:+ WorkflowControlWorkflow._history_pressure = original_pressure+ workflow.patched = original_patched+++def test_controller_native_messaging_validation_rollover_and_replay():+ completed = subprocess.run([sys.executable, "-m", "tests.temporal.test_controller_messaging_integration"],+ cwd=Path(__file__).parents[2], stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,+ stderr=subprocess.STDOUT, timeout=150, check=False)+ assert completed.returncode == 0, completed.stdout.decode(errors="replace")[-12000:]+++if __name__ == "__main__":+ asyncio.run(_run_messaging_gate())+ asyncio.run(_run_pre_marker_replay())diff --git a/server/tests/temporal/test_execution_control.py b/server/tests/temporal/test_execution_control.pynew file mode 100644index 00000000..c3a16005--- /dev/null+++ b/server/tests/temporal/test_execution_control.py@@ -0,0 +1,392 @@+"""Versioned control admission, result checkpoints and controller membership."""++import asyncio+from types import SimpleNamespace+from unittest.mock import AsyncMock, MagicMock++import pytest+from temporalio.exceptions import ApplicationError+from temporalio.converter import DefaultPayloadConverter++from services.temporal import execution_control as control_module+from services.temporal.execution_control import ExecutionControl+from services.temporal.execution_control_activities import ExecutionControlActivities+from services.temporal.workflow_control_workflow import WorkflowControlWorkflow+++async def _wait(predicate):+ while not predicate():+ await asyncio.sleep(0)++[email protected](autouse=True)+def controller_runtime(monkeypatch):+ monkeypatch.setattr(control_module.workflow, "patched", lambda _: True)+ monkeypatch.setattr(WorkflowControlWorkflow, "_history_pressure", lambda _: False)+++def control(monkeypatch):+ monkeypatch.setattr(control_module.workflow, "wait_condition", _wait)+ monkeypatch.setattr(control_module.workflow, "info", lambda: SimpleNamespace(workflow_id="root", first_execution_run_id="first"))+ instance = ExecutionControl()+ instance.bind({"execution_control_version": 1, "generation": 7, "controller_workflow_id": "controller"})+ return instance+++async def test_stop_retains_current_result_and_blocks_next_action(control):+ tool_started, tool_finished = asyncio.Event(), asyncio.Event()+ actions, results = [], []++ async def run():+ async with control.action():+ actions.append("A")+ tool_started.set()+ await tool_finished.wait()+ results.append("A-result")+ async with control.action():+ actions.append("B")+ results.append("B-result")++ task = asyncio.create_task(run())+ await tool_started.wait()+ admission = await control.set_control_state({"state": "paused", "revision": 1})+ assert admission["active_actions"] == 1+ checkpoint = asyncio.create_task(control.wait_for_checkpoint())+ await asyncio.sleep(0)+ assert not checkpoint.done()+ tool_finished.set()+ assert (await checkpoint)["checkpoint"] is True+ assert actions == ["A"]+ assert results == ["A-result"]+ await control.set_control_state({"state": "running", "revision": 2})+ await task+ assert actions == ["A", "B"]+ assert results == ["A-result", "B-result"]+++async def test_parallel_actions_drain_without_serializing(control):+ releases = [asyncio.Event(), asyncio.Event()]+ started = []++ async def action(index):+ async with control.action():+ started.append(index)+ await releases[index].wait()++ tasks = [asyncio.create_task(action(index)) for index in range(2)]+ while len(started) != 2:+ await asyncio.sleep(0)+ await control.set_control_state({"state": "paused", "revision": 1})+ checkpoint = asyncio.create_task(control.wait_for_checkpoint())+ releases[0].set()+ await tasks[0]+ assert not checkpoint.done()+ releases[1].set()+ await tasks[1]+ assert (await checkpoint)["active_actions"] == 0+++async def test_setter_waits_child_start_ack_not_child_result(control):+ await control.begin_child_start()+ setter = asyncio.create_task(control.set_control_state({"state": "paused", "revision": 1}))+ await asyncio.sleep(0)+ assert control.state == "paused"+ assert not setter.done()+ control.end_child_start()+ assert (await setter)["pending_child_starts"] == 0+ assert (await control.wait_for_checkpoint())["checkpoint"] is True+++async def test_bookkeeping_runs_while_stopped_and_drains(control):+ await control.set_control_state({"state": "paused", "revision": 1})+ control.begin_bookkeeping()+ checkpoint = asyncio.create_task(control.wait_for_checkpoint())+ await asyncio.sleep(0)+ assert not checkpoint.done()+ control.end_action()+ assert (await checkpoint)["checkpoint"] is True+++async def test_stale_or_conflicting_revision_cannot_override_resume(control):+ await control.set_control_state({"state": "running", "revision": 5})+ await control.set_control_state({"state": "paused", "revision": 4})+ await control.set_control_state({"state": "paused", "revision": 5})+ assert control.state == "running"+ assert control.revision == 5+++async def test_resume_hold_prevents_new_admission_until_released(control):+ await control.set_control_state({"state": "running", "revision": 5, "producers_held": True})+ gate = asyncio.create_task(control.wait_until_running())+ await asyncio.sleep(0)+ assert not gate.done()+ await control.set_control_state({"state": "running", "revision": 5, "producers_held": False})+ await gate+++async def test_old_same_revision_admission_cannot_undo_release(control):+ await control.set_control_state({"state": "running", "revision": 5, "producers_held": True})+ await control.set_control_state({"state": "running", "revision": 5, "producers_held": False})+ await control.set_control_state({"state": "running", "revision": 5, "producers_held": True})+ assert control.producers_held is False+++async def test_delayed_same_revision_registration_cannot_undo_release(control, monkeypatch):+ async def registration(*args, **kwargs):+ await control.set_control_state({"state": "running", "revision": 5, "producers_held": True})+ await control.set_control_state({"state": "running", "revision": 5, "producers_held": False})+ return {"participant_state": "running", "participant_revision": 5, "producers_held": True}+ monkeypatch.setattr(control_module.workflow, "execute_activity", registration)+ await control.register_root()+ assert control.producers_held is False+++async def test_new_root_inheriting_unheld_input_still_obeys_same_revision_registration(monkeypatch):+ control = ExecutionControl()+ control.bind({"execution_control_version": 1, "generation": 7, "controller_workflow_id": "controller",+ "execution_control_revision": 5, "execution_control_state": "running", "execution_control_producers_held": False})+ monkeypatch.setattr(control_module.workflow, "info", lambda: SimpleNamespace(workflow_id="root", first_execution_run_id="first"))+ monkeypatch.setattr(control_module.workflow, "execute_activity", AsyncMock(return_value={+ "participant_state": "running", "participant_revision": 5, "producers_held": True}))+ await control.register_root()+ assert control.producers_held is True+++async def test_release_phase_survives_same_workflow_rollover(control):+ await control.set_control_state({"state": "running", "revision": 5, "producers_held": False})+ restored = ExecutionControl()+ restored.bind(control.carry())+ await restored.set_control_state({"state": "running", "revision": 5, "producers_held": True})+ assert restored.producers_held is False+++async def test_repeated_bind_cannot_overwrite_pre_run_update(control):+ await control.set_control_state({"state": "paused", "revision": 5})+ control.bind({"execution_control_version": 1, "generation": 7, "execution_control_state": "running", "execution_control_revision": 0})+ assert control.state == "paused"+ assert control.revision == 5+++def test_first_bind_retains_initial_hold_without_revision():+ control = ExecutionControl()+ control.bind({"execution_control_version": 1, "execution_control_producers_held": True})+ assert control.producers_held is True+++async def test_mismatched_generation_and_execution_chain_rejected(control):+ for payload in ({"generation": 8}, {"first_execution_run_id": "replacement"}):+ with pytest.raises(ApplicationError):+ await control.set_control_state({"state": "paused", "revision": 1, **payload})+ assert control.state == "running"+++async def test_carry_preserves_control_and_root_identity(control):+ control.root_run_id = "first"+ await control.set_control_state({"state": "paused", "revision": 9, "producers_held": True})+ restored = ExecutionControl()+ restored.bind(control.carry())+ assert restored.status() == control.status()+ assert restored.root_run_id == "first"+ assert restored.carry()["generation"] == 7+ assert isinstance(restored.carry()["generation"], int)+ assert restored._registration()["generation"] == 7+++ "extra, error_type",+ [+ ({"revision": None}, "InvalidWorkflowControlRevision"),+ ({"revision": "bad"}, "InvalidWorkflowControlRevision"),+ ({"revision": []}, "InvalidWorkflowControlRevision"),+ ({"revision": float("inf")}, "InvalidWorkflowControlRevision"),+ ({"state": []}, "InvalidWorkflowControlState"),+ ({"producers_held": "false"}, "InvalidWorkflowControlRequest"),+ ],+)+async def test_malformed_control_rejects_without_mutating_intent(control, extra, error_type):+ before = control.status()+ payload = {"state": "paused", "revision": 1, **extra}+ with pytest.raises(ApplicationError) as validator_failure:+ control.validate_control(payload)+ assert validator_failure.value.type == error_type+ with pytest.raises(ApplicationError) as handler_failure:+ await control.set_control_state(payload)+ assert handler_failure.value.type == error_type+ assert control.status() == before+ assert (await control.set_control_state({"state": "paused", "revision": 1}))["state"] == "paused"+++def test_control_validator_is_read_only_and_allows_existing_checkpoint(control):+ before = control.status()+ control.validate_control({"state": "paused", "revision": 1, "generation": 7, "first_execution_run_id": "first"})+ control.validate_control(None)+ assert control.status() == before+++def test_carry_preserves_string_generation_input():+ control = ExecutionControl()+ control.bind({"execution_control_version": 1, "generation": "7"})+ assert control.carry()["generation"] == "7"+++async def test_root_registration_response_cannot_override_newer_resume(control, monkeypatch):+ async def delayed_registration(*args, **kwargs):+ await control.set_control_state({"state": "running", "revision": 8})+ return {"participant_state": "paused", "participant_revision": 7}++ monkeypatch.setattr(control_module.workflow, "execute_activity", delayed_registration)+ await control.register_root()+ assert control.state == "running"+ assert control.revision == 8+ assert control.root_run_id == "first"+++async def test_late_registered_root_receives_held_state(control, monkeypatch):+ activity = AsyncMock(return_value={"participant_state": "running", "participant_revision": 7, "producers_held": True})+ monkeypatch.setattr(control_module.workflow, "execute_activity", activity)+ await control.register_root()+ assert control.producers_held is True+ assert activity.call_args.args[0] == "execution_control.register"+++async def test_carried_root_keeps_membership_without_reregistering(control, monkeypatch):+ activity = AsyncMock()+ monkeypatch.setattr(control_module.workflow, "execute_activity", activity)+ control.root_run_id = "first"+ await control.register_root()+ activity.assert_not_awaited()+++async def test_legacy_helper_emits_no_temporal_commands(monkeypatch):+ execute = AsyncMock()+ wait = AsyncMock()+ monkeypatch.setattr(control_module.workflow, "execute_activity", execute)+ monkeypatch.setattr(control_module.workflow, "wait_condition", wait)+ legacy = ExecutionControl()+ legacy.bind({})+ async with legacy.action():+ pass+ async with legacy.child_start():+ pass+ await legacy.register_root()+ await legacy.unregister_root()+ await legacy.wait_for_checkpoint()+ assert legacy.carry() == {}+ execute.assert_not_awaited()+ wait.assert_not_awaited()+++def _controller():+ controller = WorkflowControlWorkflow()+ controller._seed_carried_state({"execution_control_version": 1, "generation": 7})+ return controller+++async def test_root_membership_is_idempotent_and_identity_checked():+ controller = _controller()+ registration = {"workflow_id": "root", "first_execution_run_id": "first", "generation": 7}+ first = await controller.register_execution(registration)+ duplicate = await controller.register_execution(registration)+ assert first["membership_epoch"] == duplicate["membership_epoch"] == 1+ with pytest.raises(ApplicationError, match="identity"):+ await controller.register_execution({**registration, "first_execution_run_id": "replacement"})+ with pytest.raises(ApplicationError, match="generation"):+ await controller.register_execution({**registration, "generation": 8})+ stale_remove = await controller.unregister_execution({**registration, "first_execution_run_id": "replacement"})+ assert stale_remove["membership_epoch"] == 1+ removed = await controller.unregister_execution(registration)+ assert removed["membership_epoch"] == 2+ assert removed["live_roots"] == {}+++async def test_controller_holds_producers_through_resume_and_late_enrollment():+ controller = _controller()+ await controller.set_control_state({"state": "paused", "revision": 2, "generation": 7})+ held = await controller.set_control_state({"state": "running", "revision": 3, "producers_held": True})+ assert held["state"] == "paused"+ assert held["participant_state"] == "running"+ assert held["producers_held"] is True+ root = await controller.register_execution({"workflow_id": "late", "first_execution_run_id": "first", "generation": 7})+ assert root["participant_revision"] == 3+ assert root["producers_held"] is True+ released = await controller.set_control_state({"state": "running", "revision": 3, "producers_held": False})+ assert released["state"] == "running"+++async def test_controller_final_release_checks_membership_atomically():+ controller = _controller()+ await controller.set_control_state({"state": "running", "revision": 3, "producers_held": True})+ await controller.register_execution({"workflow_id": "late", "first_execution_run_id": "first", "generation": 7})+ retry = await controller.set_control_state({"state": "running", "revision": 3, "producers_held": False, "expected_membership_epoch": 0})+ assert retry["producers_held"] is True+ done = await controller.set_control_state({"state": "running", "revision": 3, "producers_held": False, "expected_membership_epoch": 1})+ assert done["producers_held"] is False+++async def test_controller_rollover_carries_roots_target_and_revision(monkeypatch):+ from services.temporal import workflow_control_workflow as module+ monkeypatch.setattr(module.workflow, "logger", MagicMock())+ monkeypatch.setattr(module.workflow, "patched", lambda _: True)+ monkeypatch.setattr(module.workflow, "wait_condition", _wait)+ monkeypatch.setattr(module.workflow, "all_handlers_finished", lambda: True)+ monkeypatch.setattr(module.workflow, "payload_converter", DefaultPayloadConverter)+ controller = _controller()+ await controller.register_execution({"workflow_id": "root", "first_execution_run_id": "first", "generation": 7})+ await controller.set_control_state({"state": "running", "revision": 4, "producers_held": True})+ carried = {}+ monkeypatch.setattr(module.workflow, "continue_as_new", lambda *, args: carried.update(args[0]))+ await controller._continue_as_new({"execution_control_version": 1, "generation": 7})+ restored = WorkflowControlWorkflow()+ restored._state = carried["state"]+ restored._seed_carried_state(carried)+ assert restored.status() == controller.status()+++async def test_controller_init_does_not_reseed_over_pre_run_updates(monkeypatch):+ from services.temporal import workflow_control_workflow as module+ monkeypatch.setattr(module.workflow, "logger", MagicMock())+ original = {"execution_control_version": 1, "generation": 7, "revision": 3,+ "state": "paused", "participant_state": "running", "participant_revision": 3,+ "execution_control_producers_held": True}+ controller = WorkflowControlWorkflow(original)+ await controller.set_control_state({"state": "running", "revision": 3, "producers_held": False})+ await controller.register_execution({"workflow_id": "new", "first_execution_run_id": "first", "generation": 7})+ controller._seed_carried_state(original)+ assert controller.status()["state"] == "running"+ assert controller.status()["live_roots"]["new"]["first_execution_run_id"] == "first"+++async def test_controller_rollover_rejects_complete_oversized_input(monkeypatch):+ from services.temporal import workflow_control_workflow as module+ monkeypatch.setattr(module.workflow, "logger", MagicMock())+ monkeypatch.setattr(module.workflow, "patched", lambda _: True)+ monkeypatch.setattr(module.workflow, "wait_condition", _wait)+ monkeypatch.setattr(module.workflow, "all_handlers_finished", lambda: True)+ monkeypatch.setattr(module.workflow, "payload_converter", DefaultPayloadConverter)+ can = MagicMock()+ monkeypatch.setattr(module.workflow, "continue_as_new", can)+ controller = _controller()+ with pytest.raises(ApplicationError, match="continuation requires") as exc:+ await controller._continue_as_new({"execution_control_version": 1, "generation": 7, "graph": "x" * 1_900_001})+ assert exc.value.type == "ControllerContinuationTooLarge"+ assert exc.value.non_retryable is True+ can.assert_not_called()+++async def test_control_activity_bridges_use_injected_client_and_acknowledged_updates():+ handle = SimpleNamespace(execute_update=AsyncMock(return_value={"ok": True}))+ client = SimpleNamespace(get_workflow_handle=MagicMock(return_value=handle))+ activities = ExecutionControlActivities(client)+ payload = {"controller_workflow_id": "controller", "generation": 7}+ assert await activities.register(payload) == {"ok": True}+ assert await activities.unregister(payload) == {"ok": True}+ assert [call.args[0] for call in handle.execute_update.await_args_list] == ["register_execution", "unregister_execution"]+++async def test_paused_push_admission_defers_for_rollover(control):+ from services.temporal.workflow_control_workflow import _AdmissionDeferred+ controller = _controller()+ await controller.set_control_state({"state": "paused", "revision": 1})+ with pytest.raises(_AdmissionDeferred):+ await controller._wait_until_running()diff --git a/server/tests/temporal/test_execution_control_replay.py b/server/tests/temporal/test_execution_control_replay.pynew file mode 100644index 00000000..3de21825--- /dev/null+++ b/server/tests/temporal/test_execution_control_replay.py@@ -0,0 +1,195 @@+"""Real SDK execution, paused worker restart, rollover and replay control gate."""++from __future__ import annotations++import asyncio+import subprocess+import sys+from datetime import timedelta+from pathlib import Path+from uuid import uuid4++from temporalio import activity, workflow+from temporalio.api.enums.v1 import EventType+from temporalio.testing import WorkflowEnvironment+from temporalio.worker import Replayer, Worker++from services.temporal.execution_control import ExecutionControl+from services.temporal.execution_control_activities import ExecutionControlActivities+from services.temporal.workflow_control_workflow import WorkflowControlWorkflow+++_started: asyncio.Event+_release: asyncio.Event+_calls: list[str]++[email protected](name="control.probe.tool")+async def _tool(payload: dict) -> str:+ name = payload["name"]+ _calls.append(name)+ if name == "A":+ _started.set()+ await _release.wait()+ return f"{name}-result"++[email protected](sandboxed=False)+class ExecutionControlProbeWorkflow:+ def __init__(self):+ self.control = ExecutionControl()+ self.results: list[str] = []++ @workflow.update+ async def set_control_state(self, payload: dict) -> dict:+ return await self.control.set_control_state(payload)++ @workflow.update+ async def wait_for_checkpoint(self, payload: dict | None = None) -> dict:+ return await self.control.wait_for_checkpoint(payload)++ @workflow.query+ def execution_control_status(self) -> dict:+ return {**self.control.status(), "results": self.results}++ @workflow.run+ async def run(self, payload: dict) -> list[str]:+ self.control.bind(payload)+ self.results = list(payload.get("results") or [])+ await self.control.register_root()+ phase = int(payload.get("phase") or 0)+ async with self.control.action():+ result = await workflow.execute_activity(+ "control.probe.tool", {"name": "ABC"[phase]},+ start_to_close_timeout=timedelta(minutes=2),+ )+ self.results.append(result)+ await self.control.wait_until_running()+ if phase < 2:+ await workflow.wait_condition(workflow.all_handlers_finished)+ workflow.continue_as_new({**payload, **self.control.carry(), "phase": phase + 1, "results": self.results})+ await self.control.unregister_root()+ return self.results+++async def _histories(client, workflow_id, first_run_id):+ histories = []+ run_id = first_run_id+ while True:+ history = await client.get_workflow_handle(workflow_id, run_id=run_id).fetch_history()+ histories.append(history)+ continued = next((event.workflow_execution_continued_as_new_event_attributes for event in history.events+ if event.event_type == EventType.EVENT_TYPE_WORKFLOW_EXECUTION_CONTINUED_AS_NEW), None)+ if continued is None:+ return histories+ run_id = continued.new_execution_run_id+++def _paused_controller_history_pressure(controller: WorkflowControlWorkflow) -> bool:+ # A tiny global event limit would now roll on every successful Update.+ # Pressure the intended paused phase once, and retain this deterministic+ # hook while replaying the history that records the actual continuation.+ return (not workflow.info().continued_run_id+ and controller._execution_control.state == "paused"+ and bool(controller._live_roots))+++async def _run_gate():+ original_pressure = WorkflowControlWorkflow._history_pressure+ WorkflowControlWorkflow._history_pressure = _paused_controller_history_pressure+ try:+ await _run_gate_with_pressure()+ finally:+ WorkflowControlWorkflow._history_pressure = original_pressure+++async def _run_gate_with_pressure():+ global _started, _release, _calls+ _started, _release, _calls = asyncio.Event(), asyncio.Event(), []+ queue = "control-native-" + uuid4().hex+ async with await WorkflowEnvironment.start_time_skipping() as environment:+ client = environment.client+ definitions = [ExecutionControlProbeWorkflow, WorkflowControlWorkflow]+ activities = [_tool, *ExecutionControlActivities(client).activities()]+ controller_id = "controller-" + uuid4().hex+ root_id = "root-" + uuid4().hex+ async with Worker(client, task_queue=queue, workflows=definitions, activities=activities, max_cached_workflows=0,+ identity="control-probe-before-restart"):+ controller = await client.start_workflow("WorkflowControlWorkflow", {+ "execution_control_version": 1, "generation": 7,+ }, id=controller_id, task_queue=queue)+ root = await client.start_workflow("ExecutionControlProbeWorkflow", {+ "execution_control_version": 1, "generation": 7,+ "controller_workflow_id": controller_id,+ }, id=root_id, task_queue=queue)+ await asyncio.wait_for(_started.wait(), timeout=15)+ pause = {"state": "paused", "revision": 1, "generation": 7}+ await controller.execute_update("set_control_state", pause)+ admitted = await root.execute_update("set_control_state", {**pause, "first_execution_run_id": root.first_execution_run_id})+ assert admitted["active_actions"] == 1+ checkpoint = asyncio.create_task(root.execute_update("wait_for_checkpoint", pause))+ await asyncio.sleep(0.05)+ assert not checkpoint.done()+ _release.set()+ assert (await asyncio.wait_for(checkpoint, timeout=15))["checkpoint"] is True+ stopped = await root.query("execution_control_status")+ assert stopped["results"] == ["A-result"]+ assert _calls == ["A"]+ # The acknowledged pause Update pressures this original run.+ # Its chain roster and state survive while the root is stopped.+ for _ in range(100):+ description = await controller.describe()+ if description.run_id != controller.first_execution_run_id:+ break+ await asyncio.sleep(0.02)+ assert description.run_id != controller.first_execution_run_id+ status = await controller.query("status")+ assert status["live_roots"][root_id]["first_execution_run_id"] == root.first_execution_run_id+ assert status["participant_state"] == "paused"++ # A new SDK Worker rebuilds all Python workflow state from history.+ async with Worker(client, task_queue=queue, workflows=definitions, activities=activities, max_cached_workflows=0,+ identity="control-probe-after-restart"):+ # Synchronize the restarted worker with an idempotent Update before+ # querying the workflow state reconstructed by its first task.+ await root.execute_update("set_control_state", {"state": "paused", "revision": 1, "generation": 7})+ stopped = await root.query("execution_control_status")+ assert stopped["results"] == ["A-result"]+ resume = {"state": "running", "revision": 2, "generation": 7, "producers_held": True}+ await controller.execute_update("set_control_state", resume)+ await root.execute_update("set_control_state", resume)+ assert (await root.query("execution_control_status"))["results"] == ["A-result"]+ assert _calls == ["A"]+ await root.execute_update("set_control_state", {**resume, "producers_held": False})+ await controller.execute_update("set_control_state", {**resume, "producers_held": False})+ assert await root.result() == ["A-result", "B-result", "C-result"]+ assert _calls == ["A", "B", "C"]+ assert (await controller.query("status"))["live_roots"] == {}+ await controller.signal("reset")+ await controller.result()+ legacy = await client.start_workflow("ExecutionControlProbeWorkflow", {"phase": 2},+ id="legacy-" + uuid4().hex, task_queue=queue)+ assert await legacy.result() == ["C-result"]+ legacy_history = await legacy.fetch_history()++ roots = await _histories(client, root_id, root.first_execution_run_id)+ controllers = await _histories(client, controller_id, controller.first_execution_run_id)+ assert len(roots) == 3+ assert len(controllers) == 2+ for history in [*roots, *controllers, legacy_history]:+ result = await Replayer(workflows=definitions).replay_workflow(history)+ assert result.replay_failure is None+ legacy_activities = [event.activity_task_scheduled_event_attributes.activity_type.name+ for event in legacy_history.events if event.event_type == EventType.EVENT_TYPE_ACTIVITY_TASK_SCHEDULED]+ assert legacy_activities == ["control.probe.tool"]+++def test_control_restart_rollover_and_histories_replay():+ completed = subprocess.run([sys.executable, "-m", "tests.temporal.test_execution_control_replay"],+ cwd=Path(__file__).parents[2], stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,+ stderr=subprocess.STDOUT, timeout=120, check=False)+ assert completed.returncode == 0, completed.stdout.decode(errors="replace")[-8000:]+++if __name__ == "__main__":+ asyncio.run(_run_gate())diff --git a/server/tests/temporal/test_generation_execution_control_integration.py b/server/tests/temporal/test_generation_execution_control_integration.pynew file mode 100644index 00000000..d316390a--- /dev/null+++ b/server/tests/temporal/test_generation_execution_control_integration.py@@ -0,0 +1,332 @@+"""Native graph/tree control with late roots, parallel drain and history replay."""++from __future__ import annotations++import asyncio+import importlib.util+import subprocess+import sys+from datetime import timedelta+from pathlib import Path+from types import SimpleNamespace+from uuid import uuid4++from temporalio import activity, workflow+from temporalio.api.enums.v1 import EventType+from temporalio.testing import WorkflowEnvironment+from temporalio.worker import Replayer, Worker++from services.deployment.execution_control import transition_generation+from services.temporal.execution_control import ExecutionControl+from services.temporal.execution_control_activities import ExecutionControlActivities+from services.temporal.workflow import MachinaWorkflow+from services.temporal.workflow_control_workflow import WorkflowControlWorkflow+from services.temporal.employee_job_workflow import EmployeeJobWorkflow++# Load the plugin-owned workflow itself without booting every plugin/provider+# as a side effect of importing nodes/__init__.py in this clean replay process.+_cron_spec = importlib.util.spec_from_file_location("execution_control_cron_test",+ Path(__file__).parents[2] / "nodes/scheduler/cron_scheduler/_workflow.py")+_cron_module = importlib.util.module_from_spec(_cron_spec)+sys.modules[_cron_spec.name] = _cron_module+_cron_spec.loader.exec_module(_cron_module)+CronTriggerWorkflow = _cron_module.CronTriggerWorkflow+++_started: dict[str, asyncio.Event]+_release: dict[str, asyncio.Event]+_calls: list[str]+_contexts: dict[str, dict]+_deliveries: list[dict]++[email protected](name="execute_node_activity")+async def _execute_node(context: dict) -> dict:+ node_id = context["node_id"]+ _calls.append(node_id)+ _contexts[node_id] = context+ if node_id in _release:+ _started[node_id].set()+ await _release[node_id].wait()+ return {"success": True, "node_id": node_id, "result": {"completed": node_id}}++[email protected](name="store_node_output_activity")+async def _store_output(_payload: dict) -> dict:+ return {"stored": True}++[email protected](name="workflow_runs.record_completion")+async def _record_completion(_payload: dict) -> dict:+ return {"recorded": True}++[email protected](name="employee.job.deliver")+async def _deliver(context: dict) -> dict:+ _deliveries.append(context)+ return {"delivered": True, "state": "delivered"}++[email protected](name="employee.job.failed")+async def _fail_job(_context: dict) -> dict:+ raise AssertionError("A successful controlled job must not fail")++[email protected](name="AgentWorkflow", sandboxed=False)+class ControlledChildProbe:+ """An attached child uses the shared contract without root enrollment."""++ @workflow.init+ def __init__(self, context: dict | None = None):+ self.control = ExecutionControl()+ self.results = []+ if context is not None:+ self.control.bind(context)++ @workflow.update+ async def set_control_state(self, payload: dict) -> dict:+ return await self.control.set_control_state(payload)++ @workflow.update+ async def wait_for_checkpoint(self, payload: dict | None = None) -> dict:+ return await self.control.wait_for_checkpoint(payload)++ @workflow.query+ def execution_control_status(self) -> dict:+ return {**self.control.status(), "results": self.results}++ @workflow.run+ async def run(self, context: dict | None = None) -> dict:+ context = context or {}+ self.control.bind(context)+ if (context.get("node_data") or {}).get("probe_rollover") and not context.get("_probe_after_rollover"):+ await workflow.wait_condition(workflow.all_handlers_finished)+ workflow.continue_as_new({**context, **self.control.carry(), "_probe_after_rollover": True})+ for tool_id in context.get("probe_tool_ids", ("D", "E")):+ async with self.control.action():+ result = await workflow.execute_activity("execute_node_activity", {**context, "node_id": tool_id},+ start_to_close_timeout=timedelta(minutes=2))+ self.results.append(result["result"])+ await workflow.wait_condition(workflow.all_handlers_finished)+ return {"success": True, "node_id": context["node_id"], "result": {"tools": self.results}}+++async def _wait_status(handle, predicate):+ for _ in range(200):+ status = await handle.query("status")+ if predicate(status):+ return status+ await asyncio.sleep(0.01)+ raise AssertionError("Expected controller state was not observed")+++async def _chain_histories(client, workflow_id, first_run_id):+ histories = []+ run_id = first_run_id+ while True:+ history = await client.get_workflow_handle(workflow_id, run_id=run_id).fetch_history()+ histories.append(history)+ continued = next((event.workflow_execution_continued_as_new_event_attributes for event in history.events+ if event.event_type == EventType.EVENT_TYPE_WORKFLOW_EXECUTION_CONTINUED_AS_NEW), None)+ if continued is None:+ return histories+ run_id = continued.new_execution_run_id+++def _input(controller_id: str, *, late: bool = False) -> dict:+ nodes = [{"id": "late", "type": "controlProbe", "data": {}}] if late else [+ {"id": "A", "type": "controlProbe", "data": {}},+ {"id": "C", "type": "controlProbe", "data": {}},+ {"id": "agent", "type": "aiAgent", "data": {"probe_rollover": True}},+ {"id": "B", "type": "controlProbe", "data": {}},+ ]+ edges = [] if late else [{"source": source, "target": "B"} for source in ("A", "C", "agent")]+ return {"nodes": nodes, "edges": edges, "workflow_id": "graph", "workflow_slug": "graph",+ "execution_control_version": 1, "generation": 7, "graphVersion": 2,+ "controller_workflow_id": controller_id,+ "_temporal_routing_v1": {"version": 1, "agent_workflow_enabled": True,+ "per_type_dispatch_enabled": False, "worker_pool_enabled": False}}+++async def _run_gate():+ global _started, _release, _calls, _contexts+ _started = {key: asyncio.Event() for key in ("A", "C", "D")}+ _release = {key: asyncio.Event() for key in _started}+ _calls, _contexts = [], {}+ queue = "generation-control-" + uuid4().hex+ definitions = [MachinaWorkflow, WorkflowControlWorkflow, ControlledChildProbe]+ async with await WorkflowEnvironment.start_time_skipping() as environment:+ client = environment.client+ activities = [_execute_node, *ExecutionControlActivities(client).activities()]+ async with Worker(client, task_queue=queue, workflows=definitions, activities=activities, max_cached_workflows=0):+ controller_id = "controller-" + uuid4().hex+ controller = await client.start_workflow("WorkflowControlWorkflow", {+ "execution_control_version": 1, "generation": 7,+ }, id=controller_id, task_queue=queue)+ root_id = "root-" + uuid4().hex+ root = await client.start_workflow("MachinaWorkflow", _input(controller_id), id=root_id, task_queue=queue)+ await asyncio.wait_for(asyncio.gather(*(event.wait() for event in _started.values())), timeout=15)+ child = client.get_workflow_handle(root_id + "-agent-agent")+ parent_description = await root.describe()+ pending = parent_description.raw_description.pending_children[0]+ child_description = await child.describe()+ assert pending.run_id != child_description.run_id+ assert pending.run_id == child_description.raw_description.workflow_execution_info.first_run_id+ scope = SimpleNamespace(controller_workflow_id=controller_id, controller_run_id=controller.first_execution_run_id,+ workflow_id="graph", generation=7, revision=1)+ stop = asyncio.create_task(transition_generation(client, scope, paused=True))+ await _wait_status(controller, lambda status: status["participant_state"] == "paused")+ late_id = "late-root-" + uuid4().hex+ late = await client.start_workflow("MachinaWorkflow", _input(controller_id, late=True), id=late_id, task_queue=queue)+ await _wait_status(controller, lambda status: late_id in status["live_roots"])+ assert "late" not in _calls+ _release["A"].set()+ await asyncio.sleep(0.03)+ assert not stop.done()+ _release["C"].set()+ await asyncio.sleep(0.03)+ assert not stop.done()+ _release["D"].set()+ stopped = await asyncio.wait_for(stop, timeout=15)+ assert stopped["controlled_executions"] == 4+ assert sorted(_calls) == ["A", "C", "D"]+ child_status = await child.query("execution_control_status")+ assert child_status["state"] == "paused"+ assert child_status["results"] == [{"completed": "D"}]+ assert (await root.query("execution_control_status"))["active_actions"] == 0+ assert (await late.query("execution_control_status"))["state"] == "paused"+ scope.revision = 2+ resumed = await asyncio.wait_for(transition_generation(client, scope, paused=False), timeout=15)+ assert resumed["controller_status"]["state"] == "running"+ # Concurrent SDK result interceptors both unlock test-server time.+ # These activities are ready immediately, so use wall-clock waits.+ with environment.auto_time_skipping_disabled():+ root_result, late_result = await asyncio.gather(root.result(), late.result())+ assert root_result["success"] is True+ assert late_result["success"] is True+ assert sorted(_calls) == ["A", "B", "C", "D", "E", "late"]+ assert _contexts["B"]["inputs"] == {"A": {"completed": "A"}, "C": {"completed": "C"},+ "agent": {"tools": [{"completed": "D"}, {"completed": "E"}]}}+ await controller.signal("reset")+ await controller.result()+ legacy_input = _input(controller_id, late=True)+ legacy_input.pop("execution_control_version")+ legacy_input["nodes"][0]["id"] = "legacy"+ legacy = await client.start_workflow("MachinaWorkflow", legacy_input, id="legacy-" + uuid4().hex, task_queue=queue)+ assert (await legacy.result())["success"] is True+ histories = [await handle.fetch_history() for handle in (root, late, legacy, controller)]+ histories.extend(await _chain_histories(client, child.id, pending.run_id))+ for history in histories:+ replay = await Replayer(workflows=definitions).replay_workflow(history)+ assert replay.replay_failure is None+++def _cron_job_history_pressure(controller: WorkflowControlWorkflow) -> bool:+ # An explicit acknowledged pause revision follows the complete drain.+ # Pressure only that original-run phase with both roots enrolled, using+ # the same hook in replay, rather than churning after every small Update.+ return (not workflow.info().continued_run_id+ and controller._execution_control.state == "paused"+ and controller._execution_control.revision == 2+ and len(controller._live_roots) >= 2)+++async def _run_cron_job_gate():+ global _started, _release, _calls, _contexts, _deliveries+ _started, _release = {"cron-A": asyncio.Event()}, {"cron-A": asyncio.Event()}+ _calls, _contexts, _deliveries = [], {}, []+ queue = "generation-cron-job-" + uuid4().hex+ definitions = [MachinaWorkflow, WorkflowControlWorkflow, ControlledChildProbe, CronTriggerWorkflow, EmployeeJobWorkflow]+ original_pressure = WorkflowControlWorkflow._history_pressure+ WorkflowControlWorkflow._history_pressure = _cron_job_history_pressure+ try:+ async with await WorkflowEnvironment.start_time_skipping() as environment:+ client = environment.client+ activities = [_execute_node, _store_output, _record_completion, _deliver, _fail_job,+ *ExecutionControlActivities(client).activities()]+ async with Worker(client, task_queue=queue, workflows=definitions, activities=activities, max_cached_workflows=0):+ controller_id = "controller-" + uuid4().hex+ controller = await client.start_workflow("WorkflowControlWorkflow", {+ "execution_control_version": 1, "generation": 7,+ }, id=controller_id, task_queue=queue)+ cron_id = "cron-parent-" + uuid4().hex+ cron_input = {"execution_control_version": 1, "controller_workflow_id": controller_id,+ "generation": 7, "graphVersion": 2, "workflow_id": None, "workflow_slug": "cron-probe",+ "trigger_node_id": "cron", "nodes": [+ {"id": "cron", "type": "cronScheduler", "data": {}},+ {"id": "cron-A", "type": "controlProbe", "data": {}},+ {"id": "cron-B", "type": "controlProbe", "data": {}},+ ], "edges": [{"source": "cron", "target": "cron-A"}, {"source": "cron-A", "target": "cron-B"}],+ "_temporal_routing_v1": {"version": 1, "agent_workflow_enabled": True,+ "per_type_dispatch_enabled": False, "worker_pool_enabled": False}}+ cron = await client.start_workflow("CronTriggerWorkflow", cron_input, id=cron_id, task_queue=queue)+ with environment.auto_time_skipping_disabled():+ firing = await cron.result()+ graph_id = firing["spawned_child_id"]+ graph = client.get_workflow_handle(graph_id)+ await asyncio.wait_for(_started["cron-A"].wait(), timeout=15)+ enrolled = await _wait_status(controller, lambda status: graph_id in status["live_roots"] and cron_id not in status["live_roots"])+ graph_chain = enrolled["live_roots"][graph_id]["first_execution_run_id"]+ assert enrolled["live_roots"][graph_id]["workflow_id"] == graph_id+ scope = SimpleNamespace(controller_workflow_id=controller_id, controller_run_id=controller.first_execution_run_id,+ workflow_id="graph", generation=7, revision=1)+ stop = asyncio.create_task(transition_generation(client, scope, paused=True))+ await _wait_status(controller, lambda status: status["participant_state"] == "paused")+ job_id = "job-root-" + uuid4().hex+ job = await client.start_workflow("EmployeeJobWorkflow", {+ "execution_control_version": 1, "controller_workflow_id": controller_id,+ "generation": 7, "workflow_id": "graph", "node_id": "lead", "probe_tool_ids": ["job-tool"],+ }, id=job_id, task_queue=queue)+ await _wait_status(controller, lambda status: job_id in status["live_roots"])+ assert (await job.describe()).raw_description.pending_children == []+ assert "job-tool" not in _calls+ assert _deliveries == []+ _release["cron-A"].set()+ await asyncio.wait_for(stop, timeout=15)+ assert _calls == ["cron-A"]+ # Pressure the stable stopped phase after enrollment and+ # drain, keeping the detached graph and late job in the carry.+ await controller.execute_update("set_control_state", {+ "state": "paused", "revision": 2, "generation": 7,+ })+ for _ in range(100):+ description = await controller.describe()+ if description.run_id != controller.first_execution_run_id:+ break+ await asyncio.sleep(0.01)+ assert description.run_id != controller.first_execution_run_id+ carried = await controller.query("status")+ assert carried["live_roots"][graph_id]["first_execution_run_id"] == graph_chain+ assert job_id in carried["live_roots"]+ scope.revision = 3+ await asyncio.wait_for(transition_generation(client, scope, paused=False), timeout=15)+ with environment.auto_time_skipping_disabled():+ graph_result, delivered = await asyncio.gather(graph.result(), job.result())+ assert graph_result["success"] is True+ assert delivered["delivered"] is True+ assert sorted(_calls) == ["cron-A", "cron-B", "job-tool"]+ assert len(_deliveries) == 1+ lead = client.get_workflow_handle(job_id + ":lead")+ await controller.signal("reset")+ await controller.result()+ histories = [await handle.fetch_history() for handle in (cron, graph, job, lead)]+ controller_histories = await _chain_histories(client, controller_id, controller.first_execution_run_id)+ assert len(controller_histories) == 2+ histories.extend(controller_histories)+ for history in histories:+ replay = await Replayer(workflows=definitions).replay_workflow(history)+ assert replay.replay_failure is None+ finally:+ WorkflowControlWorkflow._history_pressure = original_pressure+++def test_generation_stop_drains_tree_late_roots_and_resume_replays():+ completed = subprocess.run([sys.executable, "-m", "tests.temporal.test_generation_execution_control_integration"],+ cwd=Path(__file__).parents[2], stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,+ stderr=subprocess.STDOUT, timeout=120, check=False)+ assert completed.returncode == 0, completed.stdout.decode(errors="replace")[-10000:]+++if __name__ == "__main__":+ asyncio.run(_run_gate())+ asyncio.run(_run_cron_job_gate())diff --git a/server/tests/test_telegram_trigger_canary_producer.py b/server/tests/test_telegram_trigger_canary_producer.pyindex 4a97fce5..9df80428 100644--- a/server/tests/test_telegram_trigger_canary_producer.py+++ b/server/tests/test_telegram_trigger_canary_producer.py@@ -39,6 +39,23 @@ _EVENTS_EMIT_PATTERN = re.compile(r"\bemit\s*\(")class TestTelegramProducerCanaryEmit:"""Producer wrapper emits via the canary CloudEvents path only."""+ def test_redelivery_keeps_the_message_id_scoped_to_its_chat(self):+ from nodes.telegram._events import telegram_message_received++ original = telegram_message_received({"chat_id": 123, "message_id": 7, "text": "hello"})+ redelivery = telegram_message_received({"chat_id": 123, "message_id": 7, "text": "hello", "timestamp": "later"})+ another_chat = telegram_message_received({"chat_id": 456, "message_id": 7, "text": "hello"})+ another_message = telegram_message_received({"chat_id": 123, "message_id": 8, "text": "hello"})++ assert original.id == redelivery.id+ assert len({original.id, another_chat.id, another_message.id}) == 3++ @pytest.mark.parametrize("payload", [{"chat_id": 123}, {"message_id": 7}, {}])+ def test_missing_provider_identity_does_not_deduplicate_unrelated_messages(self, payload):+ from nodes.telegram._events import telegram_message_received++ assert telegram_message_received(payload).id != telegram_message_received(payload).id+def test_dispatcher_is_async(self):from nodes.telegram._events import dispatch_telegram_message_receiveddiff --git a/server/tests/test_whatsapp_trigger_canary_producer.py b/server/tests/test_whatsapp_trigger_canary_producer.pyindex f29e3980..854cfb9a 100644--- a/server/tests/test_whatsapp_trigger_canary_producer.py+++ b/server/tests/test_whatsapp_trigger_canary_producer.py@@ -41,6 +41,25 @@ _DIRECTION_GUARD_PATTERN = re.compile(r"direction\s*==\s*[\"']received[\"']")class TestWhatsappProducerDualEmit:"""Producer wrapper fans out received messages to the canary path."""+ def test_redelivery_keeps_the_id_scoped_to_chat_and_direction(self):+ from nodes.whatsapp._events import whatsapp_message_event++ payload = {"chat_id": "[email protected]", "message_id": "abc", "text": "hello"}+ original = whatsapp_message_event("received", payload)+ redelivery = whatsapp_message_event("received", {**payload, "timestamp": "later"})+ another_chat = whatsapp_message_event("received", {**payload, "chat_id": "[email protected]"})+ another_message = whatsapp_message_event("received", {**payload, "message_id": "def"})+ sent = whatsapp_message_event("sent", payload)++ assert original.id == redelivery.id+ assert len({original.id, another_chat.id, another_message.id, sent.id}) == 4++ @pytest.mark.parametrize("payload", [{"chat_id": "[email protected]"}, {"message_id": "abc"}, {}])+ def test_missing_provider_identity_does_not_deduplicate_unrelated_messages(self, payload):+ from nodes.whatsapp._events import whatsapp_message_event++ assert whatsapp_message_event("received", payload).id != whatsapp_message_event("received", payload).id+def test_broadcaster_is_async(self):from nodes.whatsapp._events import broadcast_whatsapp_message