Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
fix(control): sync canvas locking with workflow execution via a server-owned can_edit capability
trohitg45a7aa27e234
diff --git a/CLAUDE.md b/CLAUDE.mdindex c1ee5851..00988b18 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -742,7 +742,7 @@ Node groups (palette categories): agent, model, skill, tool, trigger, workflow,Workflows execute via Temporal for durability and horizontal scaling, gated by `TEMPORAL_ENABLED`. Three dispatch paths exist — legacy single `execute_node_activity` (fallback), per-type `node.{type}.v{version}` activities (F4.A, `TEMPORAL_PER_TYPE_DISPATCH=true`), and Agent-as-child-workflow (F4.B, `TEMPORAL_AGENT_WORKFLOW_ENABLED=true`) — with `rlm_agent`/`claude_code_agent` always bypassing AgentWorkflow. **Per-queue activity routing is production-default (Wave 16)**: a `TemporalWorkerPool` runs one activity-only worker per plugin-declared `cls.task_queue` (ai-heavy / browser / code-exec / rest-api / messaging / ...) with per-queue concurrency + rate limits + resource-based tuning for ai-heavy/browser; rollback via `TEMPORAL_WORKER_POOL_ENABLED=false`. Resilience knobs (Waves 17-18): `DEPLOYMENT_MODE` (local halves concurrency; sizes the sticky cache), one-shot retry on `agent.execute_llm_step.v1` (LLM calls are not idempotent), observability interceptors (`activity_retry` WARN on re-dispatch), 30s periodic heartbeat during long activity bodies, cron `catchup_window=24h`, poller autoscaling. Execution routing falls back Temporal → sequential; the Temporal dev server + embedded worker are managed in-process under `server/services/temporal/` (pooch-downloaded CLI, single `temporal server start-dev` process, gRPC `TEMPORAL_FRONTEND_GRPC_PORT` + Web UI `TEMPORAL_UI_PORT`, SQLite at `~/.opencompany/temporal.db`). Full architecture, dispatch matrix, per-node + agent-loop lifecycle, all 16 F4.B `agent.*.v1` activities, heartbeat semantics, delegation input contract, the worker tuning recipe, and all `.env` tunables live in [docs-internal/TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md).-**Months-long durability contract (July 2026).** Running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. Load-bearing invariants: (1) `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` stays `false` (debug-only sweep; even enabled, any control row in the shared `WORKFLOW_CONTROL_ACTIVE_STATES` — including `resetting` — vetoes it). (2) New child workflow starts carry **no** `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so lifetime caps silently terminated paused runs; liveness comes from activity heartbeats (30s beat / 2min timeout), never from workflow lifetime caps. Removing/adding start options or retry policies on recorded commands requires a `workflow.patched(...)` guard (see the `*-unbounded-*` / `*-history-bounded-can-*` markers). (3) Every long-lived workflow loop must continue-as-new under history pressure (`is_continue_as_new_suggested()` + a soft cap) — Temporal hard-terminates around ~51,200 history events; `WorkflowControlWorkflow` carries triggers/queued events/seen-ids/control state across rollovers, so controller handles are addressed **by workflow id only, never run_id-pinned**. (4) `dispatch.emit` skips controllers whose `ControlEventTypes` Search Attribute doesn't match the event (absent attribute = legacy match-all). (5) The Temporal lifecycle is owned by [services/temporal/lifecycle.py](./server/services/temporal/lifecycle.py) (`main.py` only schedules it): dev-server supervision + resident watchdog (`TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS`), worker crash-restarts that REBUILD the single-use SDK `Worker` per attempt (manager + every pool queue worker), and the boot-time `reconcile_active_controls_on_boot` pass (converges crash-stranded `starting` rows, re-arms running/paused generations from the persisted graph snapshot). (6) **Recovery policies** (env-driven, defaults in `.env.template`; full semantics in [temporal-workflow-control.md → Recovery policies](./docs-internal/temporal-workflow-control.md)): `WORKFLOW_CONTROL_CRASH_RECOVERY=pause` — after a kill/crash (dirty-bit marker cleared only by graceful teardown) boot recovers running generations as **paused** so the user consciously resumes; `WORKFLOW_CONTROL_MISSING_CONTROLLER=pause` — a killed/vanished controller converges the generation to paused and **Resume rebuilds the controller** (`WorkflowIdConflictPolicy.USE_EXISTING`, re-arm from snapshot) instead of forcing a state-archiving Reset; `WORKFLOW_CONTROL_PAUSE_ON_FAILURE=true` — repeatedly-failing trigger-spawned runs pause their deployment via the `workflow_control.pause_on_failure.v1` activity (circuit breaker; trips only after `WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD` (default 3) failures inside a rolling `_WINDOW_SECONDS` (default 600) so one node hiccup never pauses anything; Resume resets the streak; manual canvas runs never qualify). Temporal's native Pause/Unpause (server 1.28+) is deliberately NOT used — no Python SDK surface, and it halts workflow-task dispatch so a natively-paused controller couldn't process the resume Update. Locked by `tests/temporal/test_durability_hardening.py` + `test_worker_restart.py`.+**Months-long durability contract (July 2026).** Running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. Load-bearing invariants: (1) `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` stays `false` (debug-only sweep; even enabled, any control row in the shared `WORKFLOW_CONTROL_ACTIVE_STATES` — including `resetting` — vetoes it). (2) New child workflow starts carry **no** `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so lifetime caps silently terminated paused runs; liveness comes from activity heartbeats (30s beat / 2min timeout), never from workflow lifetime caps. Removing/adding start options or retry policies on recorded commands requires a `workflow.patched(...)` guard (see the `*-unbounded-*` / `*-history-bounded-can-*` markers). (3) Every long-lived workflow loop must continue-as-new under history pressure (`is_continue_as_new_suggested()` + a soft cap) — Temporal hard-terminates around ~51,200 history events; `WorkflowControlWorkflow` carries triggers/queued events/seen-ids/control state across rollovers, so controller handles are addressed **by workflow id only, never run_id-pinned**. (4) `dispatch.emit` skips controllers whose `ControlEventTypes` Search Attribute doesn't match the event (absent attribute = legacy match-all). (5) The Temporal lifecycle is owned by [services/temporal/lifecycle.py](./server/services/temporal/lifecycle.py) (`main.py` only schedules it): dev-server supervision + resident watchdog (`TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS`), worker crash-restarts that REBUILD the single-use SDK `Worker` per attempt (manager + every pool queue worker), and the boot-time `reconcile_active_controls_on_boot` pass (converges crash-stranded `starting` rows, re-arms running/paused generations from the persisted graph snapshot). (6) **Recovery policies** (env-driven, defaults in `.env.template`; full semantics in [temporal-workflow-control.md → Recovery policies](./docs-internal/temporal-workflow-control.md)): `WORKFLOW_CONTROL_CRASH_RECOVERY=pause` — after a kill/crash (dirty-bit marker cleared only by graceful teardown) boot recovers running generations as **paused** so the user consciously resumes; `WORKFLOW_CONTROL_MISSING_CONTROLLER=pause` — a killed/vanished controller converges the generation to paused and **Resume rebuilds the controller** (`WorkflowIdConflictPolicy.USE_EXISTING`, re-arm from snapshot) instead of forcing a state-archiving Reset; `WORKFLOW_CONTROL_PAUSE_ON_FAILURE=true` — repeatedly-failing trigger-spawned runs pause their deployment via the `workflow_control.pause_on_failure.v1` activity (circuit breaker; trips only after `WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD` (default 3) failures inside a rolling `_WINDOW_SECONDS` (default 600) so one node hiccup never pauses anything; Resume resets the streak; manual canvas runs never qualify). Temporal's native Pause/Unpause (server 1.28+) is deliberately NOT used — no Python SDK surface, and it halts workflow-task dispatch so a natively-paused controller couldn't process the resume Update. (7) **Workflow-scoped event delivery**: an envelope whose `workflow_id` is set by its producer's call site reaches only that workflow's consumers (`dispatch.emit` narrows both Visibility clauses by `EventWorkflowId`); chat is the first scoped producer — the decision lives in `routers/websocket.py`, the narrowing in core dispatch, and plugin `_events.py` factories only plumb the field (see [event_framework.md](./docs-internal/event_framework.md)). (8) **Canvas editability is the server-owned `can_edit` capability** emitted by `serialize_control` (paused = editable; starting/running/transitional = locked); the FE renders it via `lib/canvasLock.ts` + Dashboard's `guardCanvasEdit` and never re-derives the rule from state strings (see [temporal-workflow-control.md → Canvas editability](./docs-internal/temporal-workflow-control.md)). Locked by `tests/temporal/test_durability_hardening.py` + `test_worker_restart.py`.**Temporal tracing ownership is strict.** Register the SDK `TracingInterceptor` exactly once on the shared `Client.connect(...)`; the Python SDK automatically prepends compatible client interceptors to every worker. Never repeat `TracingInterceptor` in `Worker(..., interceptors=...)`. Worker lists contain the distinct `ObservabilityWorkerInterceptor` and plugin-specific worker interceptors only. Exact paired `StartActivity` / `RunActivity` / `CompleteWorkflow` console spans with identical Temporal identities indicate duplicate instrumentation, not duplicate execution; use Temporal Event History and the `node.<type>.execute` application span to verify actual attempts. See the tracing and debugging sections in [TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md) and [errors.md](./docs-internal/errors.md).diff --git a/client/src/Dashboard.tsx b/client/src/Dashboard.tsxindex 3c425668..0512edfa 100644--- a/client/src/Dashboard.tsx+++ b/client/src/Dashboard.tsx@@ -15,6 +15,7 @@ import {Edge,} from 'reactflow';import { featureFlags } from './lib/featureFlags';+import { deriveCanvasLock } from './lib/canvasLock';import { prefetchAllNodeSpecs, listCachedNodeSpecs, cachedNodeSpecTypesKey } from './lib/nodeSpec';import AIAgentNode from './components/AIAgentNode';import SquareNode from './components/SquareNode';@@ -240,17 +241,46 @@ const DashboardContent: React.FC = () => {workflow_id: currentWorkflow.id, state: 'never_started' as const, revision: 0,active_count: 0, in_flight_count: 0, queued_count: 0,// Status is unresolved until the backend snapshot arrives. Keep every- // lifecycle action disabled rather than briefly exposing a stale Start.+ // lifecycle action disabled rather than briefly exposing a stale Start;+ // the canvas stays editable until the server says otherwise.can_start: false, can_pause: false, can_resume: false, can_reset: false,+ can_edit: true,}- : { state: 'never_started' as const, revision: 0, active_count: 0, in_flight_count: 0, queued_count: 0, can_start: false, can_pause: false, can_resume: false, can_reset: false };+ : { state: 'never_started' as const, revision: 0, active_count: 0, in_flight_count: 0, queued_count: 0, can_start: false, can_pause: false, can_resume: false, can_reset: false, can_edit: true };const workflowControlPendingMutation = currentWorkflow?.id? workflowControlPending[currentWorkflow.id]: undefined;- const isCurrentWorkflowLocked = workflowLock.locked &&- workflowLock.workflow_id === currentWorkflow?.id;+ // Canvas edit lock — the server-owned `can_edit` capability from the+ // durable control plane is authoritative (backend SSOT; the FE only+ // renders it). The legacy broadcaster lock remains a secondary input+ // for deployments driven outside the control plane.+ const canvasLock = deriveCanvasLock({+ control: workflowControl,+ legacyLock: workflowLock,+ workflowId: currentWorkflow?.id ?? null,+ });+ const isCurrentWorkflowLocked = canvasLock.locked;+ // Single guard for every canvas mutation the React Flow props can't+ // reach (palette drop, paste, context-menu delete/rename, node+ // disable, import, parameter saves). Toasts the reason so a blocked+ // edit never reads as a dead UI.+ const guardCanvasEdit = React.useCallback((): boolean => {+ if (!canvasLock.locked) return true;+ toast.warning(canvasLock.reason ?? 'Workflow is running — pause it to edit the canvas');+ return false;+ }, [canvasLock.locked, canvasLock.reason]);const [globalModelDefaults, setGlobalModelDefaults] = React.useState<{ provider: string; model: string } | null>(null);const { onDragOver, onDrop, handleComponentDragStart } = useDragAndDrop({ nodes, setNodes, saveNodeParameters, globalModelDefaults, workflowId: currentWorkflow?.id ?? 'new' });+ // Palette drop creates nodes AND persists their default parameters —+ // an HTML5 drop that React Flow's nodesDraggable/Connectable cannot+ // block, so it goes through the shared guard.+ const handleGuardedDrop = React.useCallback((event: React.DragEvent) => {+ if (!guardCanvasEdit()) {+ event.preventDefault();+ return;+ }+ onDrop(event);+ }, [guardCanvasEdit, onDrop]);const { onConnect: baseOnConnect, onNodesDelete, onEdgesDelete: baseOnEdgesDelete } = useReactFlowNodes({ setNodes, setEdges, clearNodeStatus });const { onConnect, onEdgesDelete } = useAutoSkillEdges({baseOnConnect,@@ -1168,11 +1198,11 @@ const DashboardContent: React.FC = () => {// Context menu actionsconst handleContextMenuRename = React.useCallback(() => {- if (contextMenu) {+ if (contextMenu && guardCanvasEdit()) {setRenamingNodeId(contextMenu.nodeId);}closeContextMenu();- }, [contextMenu, setRenamingNodeId, closeContextMenu]);+ }, [contextMenu, guardCanvasEdit, setRenamingNodeId, closeContextMenu]);const handleContextMenuCopy = React.useCallback(() => {if (contextMenu) {@@ -1188,11 +1218,13 @@ const DashboardContent: React.FC = () => {}, [contextMenu, nodes, setNodes, copySelectedNodes, closeContextMenu]);const handleContextMenuDelete = React.useCallback(() => {- if (contextMenu) {+ // The emptied deleteKeyCode only blocks the keyboard path — this+ // menu action bypassed the lock entirely before the shared guard.+ if (contextMenu && guardCanvasEdit()) {onNodesDelete([nodes.find(n => n.id === contextMenu.nodeId)].filter(Boolean) as Node[]);}closeContextMenu();- }, [contextMenu, nodes, onNodesDelete, closeContextMenu]);+ }, [contextMenu, guardCanvasEdit, nodes, onNodesDelete, closeContextMenu]);// Keyboard shortcut handler for workflow operationsuseEffect(() => {@@ -1211,7 +1243,9 @@ const DashboardContent: React.FC = () => {// F2 to rename selected nodeif (event.key === 'F2' && selectedNode) {event.preventDefault();- setRenamingNodeId(selectedNode.id);+ if (guardCanvasEdit()) {+ setRenamingNodeId(selectedNode.id);+ }return;}@@ -1229,7 +1263,11 @@ const DashboardContent: React.FC = () => {break;case 'v':event.preventDefault();- pasteNodes();+ // Paste creates nodes AND persists their parameters to the+ // DB — canvas-mutation guard applies.+ if (guardCanvasEdit()) {+ pasteNodes();+ }break;}} else {@@ -1238,7 +1276,9 @@ const DashboardContent: React.FC = () => {case 'd':// Toggle disable on selected nodesevent.preventDefault();- toggleDisableSelected();+ if (guardCanvasEdit()) {+ toggleDisableSelected();+ }break;}}@@ -1249,7 +1289,7 @@ const DashboardContent: React.FC = () => {return () => {document.removeEventListener('keydown', handleKeyDown);};- }, [handleSave, copySelectedNodes, pasteNodes, toggleDisableSelected, selectedNode, renamingNodeId, setRenamingNodeId]);+ }, [handleSave, copySelectedNodes, pasteNodes, toggleDisableSelected, selectedNode, renamingNodeId, setRenamingNodeId, guardCanvasEdit]);return (<>@@ -1364,7 +1404,7 @@ const DashboardContent: React.FC = () => {onEdgesDelete={onEdgesDelete}onConnect={onConnect}onDragOver={onDragOver}- onDrop={onDrop}+ onDrop={handleGuardedDrop}onNodeContextMenu={onNodeContextMenu}nodeTypes={nodeTypes}edgeTypes={edgeTypes}diff --git a/client/src/ParameterPanel.tsx b/client/src/ParameterPanel.tsxindex 0af5f029..370cc867 100644--- a/client/src/ParameterPanel.tsx+++ b/client/src/ParameterPanel.tsx@@ -8,6 +8,7 @@ import { ExecutionService, ExecutionResult } from './services/executionService';import { ActionButton } from './components/ui/action-button';import { NodeIcon } from './assets/icons';import { nodeExecutionReducer } from './utils/parameterPanelExecutionState';+import { deriveCanvasLock } from './lib/canvasLock';const ParameterPanel: React.FC = () => {const {@@ -22,7 +23,16 @@ const ParameterPanel: React.FC = () => {} = useParameterPanel();const currentWorkflow = useAppStore((s) => s.currentWorkflow);- const { executeNode, getNodeParameters, clearNodeStatus, cancelEventWait, getNodeStatus } = useWebSocket();+ const { executeNode, getNodeParameters, clearNodeStatus, cancelEventWait, getNodeStatus, workflowControlStatuses, workflowLock } = useWebSocket();++ // Server-owned canvas-edit capability: parameter saves mutate the+ // workflow's persisted configuration, so they follow the same+ // `can_edit` capability the canvas gates on.+ const canvasLock = deriveCanvasLock({+ control: currentWorkflow?.id ? workflowControlStatuses[currentWorkflow.id] : null,+ legacyLock: workflowLock,+ workflowId: currentWorkflow?.id ?? null,+ });// Get current node status to check if waitingconst nodeStatus = selectedNode ? getNodeStatus(selectedNode.id) : null;@@ -170,7 +180,12 @@ const ParameterPanel: React.FC = () => {{selectedExecution.running ? 'Running...' : 'Run'}</ActionButton>)}- <ActionButton intent="tools" onClick={handleSave} disabled={!hasUnsavedChanges}>+ <ActionButton+ intent="tools"+ onClick={handleSave}+ disabled={!hasUnsavedChanges || canvasLock.locked}+ title={canvasLock.locked ? (canvasLock.reason ?? undefined) : undefined}+ ><svg width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2.5" strokeLinecap="round" strokeLinejoin="round"><path d="M19 21H5a2 2 0 0 1-2-2V5a2 2 0 0 1 2-2h11l5 5v11a2 2 0 0 1-2 2z" /><polyline points="17 21 17 13 7 13 7 21" />diff --git a/client/src/components/ui/TopToolbar.tsx b/client/src/components/ui/TopToolbar.tsxindex 7eb84551..b7c7a7a9 100644--- a/client/src/components/ui/TopToolbar.tsx+++ b/client/src/components/ui/TopToolbar.tsx@@ -23,7 +23,9 @@ import {Repeat,Clock,Zap,+ Lock,} from 'lucide-react';+import { Badge } from '@/components/ui/badge';import {Select,SelectContent,@@ -485,6 +487,19 @@ const TopToolbar: React.FC<TopToolbarProps> = ({</ActionButton>)}+ {/* Canvas-lock indicator — renders the server-owned `can_edit`+ capability so a silently-dead drag never reads as broken UI. */}+ {workflowControl.can_edit === false && (+ <Badge+ variant="outline"+ className="gap-1 text-muted-foreground"+ title="Canvas locked while the workflow runs — pause it to edit"+ >+ <Lock className="h-3 w-3" />+ Locked+ </Badge>+ )}+<AlertDialog open={resetOpen} onOpenChange={setResetOpen}><AlertDialogContent><AlertDialogHeader><AlertDialogTitle>Reset workflow execution?</AlertDialogTitle><AlertDialogDescription>This immediately terminates active work and archives the current generation. Task history remains available. The workflow will remain stopped until you press Start.</AlertDialogDescription></AlertDialogHeader><AlertDialogFooter><AlertDialogCancel>Keep current execution</AlertDialogCancel><AlertDialogAction disabled={hasPendingControlMutation} onClick={onResetWorkflow}>{isRetryingReset ? 'Retry reset' : 'Reset workflow'}</AlertDialogAction></AlertDialogFooter></AlertDialogContent></AlertDialog><ActionButtondiff --git a/client/src/components/ui/__tests__/CommandPaletteHost.workflowControl.test.ts b/client/src/components/ui/__tests__/CommandPaletteHost.workflowControl.test.tsindex 74582e20..7a9d7780 100644--- a/client/src/components/ui/__tests__/CommandPaletteHost.workflowControl.test.ts+++ b/client/src/components/ui/__tests__/CommandPaletteHost.workflowControl.test.ts@@ -19,6 +19,7 @@ const control = (can_pause: false,can_resume: false,can_reset: false,+ can_edit: false,...overrides,});diff --git a/client/src/contexts/WebSocketContext.tsx b/client/src/contexts/WebSocketContext.tsxindex 058b2024..7f184a50 100644--- a/client/src/contexts/WebSocketContext.tsx+++ b/client/src/contexts/WebSocketContext.tsx@@ -132,7 +132,7 @@ export interface WorkflowStatus {export interface DeploymentStatus {isRunning: boolean;activeRuns: number;- status: 'idle' | 'starting' | 'running' | 'stopped' | 'cancelled' | 'error';+ status: 'idle' | 'starting' | 'running' | 'paused' | 'stopped' | 'cancelled' | 'error';workflow_id?: string | null; // Which workflow is deployed (for scoping)totalTime?: number;error?: string;@@ -159,6 +159,9 @@ export interface WorkflowControlStatus {can_pause: boolean;can_resume: boolean;can_reset: boolean;+ /** Server-owned canvas-editability capability (backend SSOT — the FE+ * renders it, never re-derives it from state strings). */+ can_edit: boolean;started_at?: string | null;updated_at?: string | null;error?: string | null;@@ -560,6 +563,7 @@ export const emptyWorkflowControlStatus = (workflowId?: string): WorkflowControlcan_pause: false,can_resume: false,can_reset: false,+ can_edit: true,});export const normalizeWorkflowControlStatus = (value: any, workflowId?: string): WorkflowControlStatus => {@@ -579,6 +583,10 @@ export const normalizeWorkflowControlStatus = (value: any, workflowId?: string):can_pause: source.can_pause ?? state === 'running',can_resume: source.can_resume ?? state === 'paused',can_reset: source.can_reset ?? state !== 'never_started',+ // Pre-capability payloads (older backend) fall back to a state-derived+ // approximation of the server rule; new payloads carry it verbatim.+ can_edit: source.can_edit+ ?? !['starting', 'running', 'pausing', 'resuming', 'resetting'].includes(state),};};@@ -950,7 +958,11 @@ export const WebSocketProvider: React.FC<{ children: React.ReactNode }> = ({ chisetDeploymentStatus({isRunning: data.deployment.isRunning || false,activeRuns: data.deployment.activeRuns || 0,- status: data.deployment.status || 'idle'+ status: data.deployment.status || 'idle',+ // The backend slot carries the owning workflow — dropping+ // it left isCurrentWorkflowDeployed false after reconnect+ // (and mis-attributed running state across workflows).+ workflow_id: data.deployment.workflow_id ?? null,});}// Catalogue 'stored' flags derive from api_keys + oauth state;@@ -1508,9 +1520,13 @@ export const WebSocketProvider: React.FC<{ children: React.ReactNode }> = ({ chi// stayed showing "Stop" forever.const envelope = data as WorkflowEvent<{running_workflow_ids?: string[];+ paused_workflow_ids?: string[];}> | undefined;const runningIds = envelope?.data?.running_workflow_ids ?? [];const runningSet = new Set(runningIds);+ // Armed-but-paused deployments: deployed (in running_workflow_ids)+ // but must not render as executing.+ const pausedSet = new Set(envelope?.data?.paused_workflow_ids ?? []);const store = useAppStore.getState();const currentId = store.currentWorkflow?.id;@@ -1518,15 +1534,17 @@ export const WebSocketProvider: React.FC<{ children: React.ReactNode }> = ({ chi// Anything currently flagged isExecuting=true that isn't in// the snapshot's running set gets cleared (the load-bearing// reset for stale state after backend restart). Anything in- // the snapshot gets flagged true.+ // the snapshot gets flagged true — unless it is paused.const existingStates = store.workflowUIStates ?? {};for (const [wid, ui] of Object.entries(existingStates)) {- if (ui?.isExecuting && !runningSet.has(wid)) {+ if (ui?.isExecuting && (!runningSet.has(wid) || pausedSet.has(wid))) {store.setWorkflowExecuting(wid, false);}}for (const wid of runningIds) {- store.setWorkflowExecuting(wid, true);+ if (!pausedSet.has(wid)) {+ store.setWorkflowExecuting(wid, true);+ }}// Reconcile the toolbar `deploymentStatus` for the active workflow@@ -1534,7 +1552,7 @@ export const WebSocketProvider: React.FC<{ children: React.ReactNode }> = ({ chiconst next: DeploymentStatus = { ...prev };if (currentId && runningSet.has(currentId)) {next.isRunning = true;- next.status = 'running';+ next.status = pausedSet.has(currentId) ? 'paused' : 'running';next.workflow_id = currentId;} else if (currentId && !runningSet.has(currentId) && prev.workflow_id === currentId) {// Previously thought current workflow was deployed; backend says no.@@ -1623,8 +1641,18 @@ export const WebSocketProvider: React.FC<{ children: React.ReactNode }> = ({ chinewStatus.activeRuns = message.data?.active_runs || prev.activeRuns + 1;break;case 'run_complete':+ // The deployment manager emits `run_completed` /+ // `run_failed` (see manager status callbacks); without+ // these cases activeRuns only ever incremented.+ case 'run_completed':+ case 'run_failed':newStatus.activeRuns = Math.max(0, message.data?.active_runs || prev.activeRuns - 1);break;+ case 'paused':+ // Deployment stays armed; nothing new executes.+ newStatus.isRunning = true;+ newStatus.status = 'paused';+ break;case 'stopped':// Only clear if this was our workflow or no workflow was trackedif (!prev.workflow_id || prev.workflow_id === deploymentWorkflowId) {@@ -1661,7 +1689,9 @@ export const WebSocketProvider: React.FC<{ children: React.ReactNode }> = ({ chiif (deploymentWorkflowId) {const { setWorkflowExecuting } = useAppStore.getState();const isRunning = ['starting', 'running', 'started', 'run_started'].includes(message.status);- const isStopped = ['stopped', 'cancelled', 'error'].includes(message.status);+ // `paused` keeps the deployment armed but stops execution+ // visuals (edge animation, toolbar indicator).+ const isStopped = ['stopped', 'cancelled', 'error', 'paused'].includes(message.status);if (isRunning || isStopped) {setWorkflowExecuting(deploymentWorkflowId, isRunning);}@@ -1744,27 +1774,34 @@ export const WebSocketProvider: React.FC<{ children: React.ReactNode }> = ({ chicase 'workflow_lock':// Handle workflow lock status updates (per-workflow locking - n8n pattern)- // Only update lock state if it's for the current workflow or if unlockingif (data) {const lockWorkflowId = message.workflow_id || data.workflow_id;const activeWorkflowId = useAppStore.getState().currentWorkflow?.id;- // Apply lock update if:- // 1. It's for the current workflow, OR- // 2. We're unlocking (locked=false), OR- // 3. No specific workflow context (backward compatibility)- const shouldApplyLock = !lockWorkflowId ||- lockWorkflowId === activeWorkflowId ||- !data.locked;-- if (shouldApplyLock) {- setWorkflowLock({- locked: data.locked || false,- workflow_id: data.workflow_id || null,- locked_at: data.locked_at || null,- reason: data.reason || null- });- }+ setWorkflowLock(prev => {+ if (data.locked) {+ // Apply a lock when it targets the current workflow, or+ // carries no workflow context (backward compatibility).+ if (!lockWorkflowId || lockWorkflowId === activeWorkflowId) {+ return {+ locked: true,+ workflow_id: data.workflow_id || null,+ locked_at: data.locked_at || null,+ reason: data.reason || null,+ };+ }+ return prev;+ }+ // Unlock ONLY when it releases the lock we're holding —+ // an unscoped unlock or one for the held workflow. An+ // unlock for a DIFFERENT workflow (e.g. resetting workflow+ // B while viewing locked workflow A) previously clobbered+ // the singleton and unlocked A's canvas mid-run.+ if (!prev.locked || !lockWorkflowId || prev.workflow_id === lockWorkflowId || !prev.workflow_id) {+ return { locked: false, workflow_id: null, locked_at: null, reason: null };+ }+ return prev;+ });}break;diff --git a/client/src/contexts/__tests__/workflowControlStatus.test.ts b/client/src/contexts/__tests__/workflowControlStatus.test.tsindex a63c222b..5c250f0c 100644--- a/client/src/contexts/__tests__/workflowControlStatus.test.ts+++ b/client/src/contexts/__tests__/workflowControlStatus.test.ts@@ -26,6 +26,7 @@ const status = (can_pause: true,can_resume: false,can_reset: true,+ can_edit: false,...overrides,});diff --git a/client/src/lib/__tests__/canvasLock.test.ts b/client/src/lib/__tests__/canvasLock.test.tsnew file mode 100644index 00000000..53905d47--- /dev/null+++ b/client/src/lib/__tests__/canvasLock.test.ts@@ -0,0 +1,84 @@+import { describe, expect, it } from 'vitest';+import { deriveCanvasLock } from '../canvasLock';++const legacyUnlocked = { locked: false, workflow_id: null };++describe('deriveCanvasLock', () => {+ it('renders the server-owned can_edit capability verbatim', () => {+ expect(+ deriveCanvasLock({+ control: { state: 'running', can_edit: false },+ legacyLock: legacyUnlocked,+ workflowId: 'wf-1',+ }).locked,+ ).toBe(true);++ // paused is editable — controlled generations execute their+ // immutable admitted snapshot, so the server grants can_edit.+ expect(+ deriveCanvasLock({+ control: { state: 'paused', can_edit: true },+ legacyLock: legacyUnlocked,+ workflowId: 'wf-1',+ }).locked,+ ).toBe(false);+ });++ it('never re-derives the rule from state strings alone', () => {+ // A running state WITH can_edit granted stays editable — the FE+ // must not override the backend's decision.+ expect(+ deriveCanvasLock({+ control: { state: 'running', can_edit: true },+ legacyLock: legacyUnlocked,+ workflowId: 'wf-1',+ }).locked,+ ).toBe(false);+ });++ it('gives transitional states a distinct reason', () => {+ const transition = deriveCanvasLock({+ control: { state: 'pausing', can_edit: false },+ legacyLock: legacyUnlocked,+ workflowId: 'wf-1',+ });+ expect(transition.locked).toBe(true);+ expect(transition.reason).toContain('transition');++ const running = deriveCanvasLock({+ control: { state: 'running', can_edit: false },+ legacyLock: legacyUnlocked,+ workflowId: 'wf-1',+ });+ expect(running.reason).toContain('pause it to edit');+ });++ it('falls back to the legacy broadcaster lock for the same workflow only', () => {+ const legacyLocked = { locked: true, workflow_id: 'wf-1' };+ expect(+ deriveCanvasLock({+ control: { state: 'never_started', can_edit: true },+ legacyLock: legacyLocked,+ workflowId: 'wf-1',+ }).locked,+ ).toBe(true);++ // A lock held by ANOTHER workflow never locks this canvas.+ expect(+ deriveCanvasLock({+ control: { state: 'never_started', can_edit: true },+ legacyLock: legacyLocked,+ workflowId: 'wf-2',+ }).locked,+ ).toBe(false);+ });++ it('is unlocked with no control status and no legacy lock', () => {+ const result = deriveCanvasLock({+ control: null,+ legacyLock: legacyUnlocked,+ workflowId: 'wf-1',+ });+ expect(result).toEqual({ locked: false, reason: null });+ });+});diff --git a/client/src/lib/canvasLock.ts b/client/src/lib/canvasLock.tsnew file mode 100644index 00000000..f5d705c5--- /dev/null+++ b/client/src/lib/canvasLock.ts@@ -0,0 +1,58 @@+/**+ * Canvas edit-lock presentation helper.+ *+ * The DECISION of whether a workflow's canvas may be edited is+ * server-owned: `serialize_control` (backend) emits the `can_edit`+ * capability alongside `can_start` / `can_pause` / …, and the frontend+ * renders it — it never re-derives the rule from state strings. This+ * module only maps the capability (plus the legacy broadcaster lock for+ * deployments driven outside the control plane) to a boolean + a+ * human-readable reason, which are pure UI concerns.+ */++export interface CanvasLockControlInput {+ state?: string | null;+ can_edit?: boolean;+}++export interface CanvasLockInput {+ /** Durable control-plane status for the workflow (server-normalized). */+ control: CanvasLockControlInput | null | undefined;+ /** Legacy broadcaster lock singleton (`workflowLock` from WebSocketContext). */+ legacyLock: { locked: boolean; workflow_id: string | null } | null | undefined;+ /** The workflow currently open on the canvas. */+ workflowId: string | null | undefined;+}++export interface CanvasLockResult {+ locked: boolean;+ /** Human-readable cause for the indicator / blocked-edit toast. */+ reason: string | null;+}++const TRANSITION_STATES = new Set(['pausing', 'resuming', 'resetting', 'starting']);++export function deriveCanvasLock({+ control,+ legacyLock,+ workflowId,+}: CanvasLockInput): CanvasLockResult {+ if (control && control.can_edit === false) {+ const state = control.state ?? null;+ return {+ locked: true,+ reason:+ state && TRANSITION_STATES.has(state) && state !== 'running'+ ? 'Workflow is changing state — wait for the transition to finish'+ : 'Workflow is running — pause it to edit the canvas',+ };+ }+ if (+ legacyLock?.locked &&+ !!workflowId &&+ legacyLock.workflow_id === workflowId+ ) {+ return { locked: true, reason: 'Workflow is deployed — stop it to edit the canvas' };+ }+ return { locked: false, reason: null };+}diff --git a/docs-internal/temporal-workflow-control.md b/docs-internal/temporal-workflow-control.mdindex 403e68b0..91ae3af9 100644--- a/docs-internal/temporal-workflow-control.md+++ b/docs-internal/temporal-workflow-control.md@@ -214,6 +214,30 @@ that Resume relies on.every knob is evaluated on the activity side so flipping config nevertouches recorded workflow commands.+## Canvas editability (`can_edit` capability)++Whether a workflow's canvas may be edited is a **server-owned capability**,+emitted by `serialize_control` alongside `can_start` / `can_pause` / … —+the frontend renders it and never re-derives the rule from state strings.+Editable while nothing is armed (`ready` / `failed` / `never_started`) and+while **paused** (controlled generations execute their immutable admitted+snapshot, so edits cannot corrupt in-flight runs — paused is exactly the+"fix it, then resume" posture the recovery policies produce); locked while+`starting` / `running` / transitional. On the client,+[`lib/canvasLock.ts`](../client/src/lib/canvasLock.ts) maps the capability+(plus the legacy broadcaster lock for deployments driven outside the+control plane) to a boolean + reason; `Dashboard` feeds it to the React+Flow interaction props AND a shared `guardCanvasEdit` toast-guard covering+the paths those props cannot reach (palette drop, paste, context-menu+delete/rename, node disable, parameter saves); `TopToolbar` shows a+`Locked` badge so a blocked drag never reads as dead UI.++The pause/resume/cancel/re-arm paths also emit the UI-facing+`workflow_status(executing=…)` + `deployment_status` broadcasts (status+`paused` keeps `isRunning=true` — armed but not executing), and the+connect-time deployment snapshot carries `paused_workflow_ids` so an+armed-but-paused generation doesn't animate as running after a reconnect.+## Delegated-task tracesEach task attempt stores the actual parent, detached runner, and child Temporaldiff --git a/server/services/deployment/control.py b/server/services/deployment/control.pyindex 3d3c795b..dd18c758 100644--- a/server/services/deployment/control.py+++ b/server/services/deployment/control.py@@ -21,11 +21,22 @@ def _graph_hash(nodes: list[dict], edges: list[dict]) -> str:return hashlib.sha256(value.encode()).hexdigest()+# Canvas editability is a server-owned capability, like can_start/can_pause:+# the frontend renders it, never re-derives it from state strings. Editable+# while nothing is armed (ready / failed) and while PAUSED — controlled+# generations execute their immutable admitted snapshot, so edits cannot+# corrupt in-flight runs, and paused is exactly the "fix it, then resume"+# posture the recovery policies produce. Locked while starting / running /+# transitional so a live deployment cannot be rewired mid-flight.+_EDITABLE_STATES = frozenset({"ready", "paused", "failed"})++def serialize_control(control: Optional[WorkflowControlExecution]) -> Dict[str, Any]:if control is None:return {"state": "never_started", "revision": 0, "generation": 0,"can_start": True, "can_pause": False, "can_resume": False, "can_reset": False,+ "can_edit": True,}state = "ready" if control.status == "reset" else control.statusreturn {@@ -42,6 +53,7 @@ def serialize_control(control: Optional[WorkflowControlExecution]) -> Dict[str,"can_pause": state == "running","can_resume": state == "paused","can_reset": state != "ready",+ "can_edit": state in _EDITABLE_STATES,"created_at": control.created_at.isoformat() if control.created_at else None,"updated_at": control.updated_at.isoformat() if control.updated_at else None,"terminal_reason": control.terminal_reason,diff --git a/server/services/deployment/handlers.py b/server/services/deployment/handlers.pyindex 6f495a05..37788595 100644--- a/server/services/deployment/handlers.py+++ b/server/services/deployment/handlers.py@@ -209,6 +209,16 @@ async def handle_deploy_workflow(data: Dict[str, Any], websocket: WebSocket) ->if not result.get("success"):logger.error("Deployment setup failed", error=result.get("error"), workflow_id=workflow_id)+ # Clear BOTH UI-facing flags: the deploy handler already+ # broadcast executing=True, and deployments never touch+ # the run counter, so nothing else would ever emit+ # executing=False for this workflow (toolbar stuck).+ await broadcaster.update_workflow_status(+ executing=False,+ current_node=None,+ progress=0,+ workflow_id=workflow_id,+ )await broadcaster.update_deployment_status(is_running=False,status="error",@@ -238,8 +248,22 @@ async def handle_deploy_workflow(data: Dict[str, Any], websocket: WebSocket) ->)return result+ except asyncio.CancelledError:+ # CancelledError is BaseException — the branch below cannot+ # catch it, and a deploy task cancelled outside+ # handle_cancel_deployment (which unlocks itself) previously+ # left the workflow lock dangling forever.+ await broadcaster.unlock_workflow(workflow_id)+ _deployment_tasks.pop(workflow_id, None)+ raiseexcept Exception as e:logger.error("Deployment task error", workflow_id=workflow_id, error=str(e))+ await broadcaster.update_workflow_status(+ executing=False,+ current_node=None,+ progress=0,+ workflow_id=workflow_id,+ )await broadcaster.update_deployment_status(is_running=False,status="error",@@ -308,21 +332,26 @@ async def handle_cancel_deployment(data: Dict[str, Any], websocket: WebSocket) -for node_id in result.get("cancelled_listener_node_ids", []):await broadcaster.clear_node_status(node_id)- await broadcaster.update_workflow_status(- executing=False,- current_node=None,- progress=0,- workflow_id=workflow_id,- )- await broadcaster.update_deployment_status(- is_running=False,- status="cancelled",- active_runs=0,- workflow_id=workflow_id,- data={- "iterations_completed": result.get("iterations_completed", 0),- },- )+ # Terminal UI-facing state is emitted UNCONDITIONALLY: cancelling a+ # workflow that is not locally deployed (post-restart before re-arm,+ # or a Reset after a crash) previously emitted nothing, leaving the+ # FE showing executing/running forever. The idempotent terminal+ # broadcast is harmless when nothing was running.+ await broadcaster.update_workflow_status(+ executing=False,+ current_node=None,+ progress=0,+ workflow_id=workflow_id,+ )+ await broadcaster.update_deployment_status(+ is_running=False,+ status="cancelled",+ active_runs=0,+ workflow_id=workflow_id,+ data={+ "iterations_completed": result.get("iterations_completed", 0),+ },+ )return {"success": result.get("success", False),@@ -1315,6 +1344,20 @@ async def _rearm_generation(control) -> None:paused=True,)await _set_cron_pause(control.workflow_id, paused=True, strict=False)+ # The deploy handler just broadcast executing=true / status=running+ # on the wire; counter-broadcast the paused posture so a connected+ # client doesn't watch the workflow flip to running and never back.+ from services.status_broadcaster import get_status_broadcaster++ broadcaster = get_status_broadcaster()+ await broadcaster.update_workflow_status(+ executing=False, current_node=None, progress=0,+ workflow_id=control.workflow_id,+ )+ await broadcaster.update_deployment_status(+ is_running=True, status="paused", active_runs=0,+ workflow_id=control.workflow_id,+ )logger.info("Re-armed durable generation after restart",workflow_id=control.workflow_id,@@ -1542,6 +1585,19 @@ async def handle_start_workflow(data: Dict[str, Any], websocket: WebSocket) -> Derror=str(reset_exc),)await _terminate_generation_workflows(control)+ # Tear down the local half too: the deploy may have armed+ # listeners and taken the workflow lock, and leaving them meant a+ # locked, armed, invisible deployment that blocked every future+ # Start AND the boot re-arm for this workflow. Cancel unlocks and+ # emits the terminal UI-facing state; "not deployed" is benign.+ try:+ await handle_cancel_deployment({"workflow_id": workflow_id}, None)+ except Exception as cancel_exc: # noqa: BLE001 — best-effort teardown+ logger.warning(+ "Local teardown after failed start did not complete",+ workflow_id=workflow_id,+ error=str(cancel_exc),+ )await _broadcast_control(control)raise@@ -1619,6 +1675,20 @@ async def handle_pause_workflow(data: Dict[str, Any], websocket: WebSocket) -> Dstrict=True,)control = await service.transition(control, expected_revision=control.revision, from_statuses={"pausing"}, status="paused")+ # UI-facing execution flags follow the pause: the deployment stays+ # armed but nothing new executes, so the canvas edge animation and+ # toolbar executing indicator must stop. The pause family previously+ # emitted only workflow_control_status, leaving `executing=true` /+ # `status=running` stale on every connected client.+ from services.status_broadcaster import get_status_broadcaster++ broadcaster = get_status_broadcaster()+ await broadcaster.update_workflow_status(+ executing=False, current_node=None, progress=0, workflow_id=workflow_id,+ )+ await broadcaster.update_deployment_status(+ is_running=True, status="paused", active_runs=0, workflow_id=workflow_id,+ )payload = await _broadcast_control(control, controller_status=controller_status, extra={"signalled_executions": signalled,"paused_schedules": paused_schedules,@@ -1716,6 +1786,17 @@ async def handle_resume_workflow(data: Dict[str, Any], websocket: WebSocket) -># Operator intervention resets the circuit-breaker streak — the next# failure after a resume starts a fresh count, not a near-tripped one.await _clear_failure_streak(service.database, control)+ # Mirror of the pause-side emission: flip the UI-facing execution+ # flags back so the canvas and toolbar re-animate on every client.+ from services.status_broadcaster import get_status_broadcaster++ broadcaster = get_status_broadcaster()+ await broadcaster.update_workflow_status(+ executing=True, current_node=None, progress=0, workflow_id=workflow_id,+ )+ await broadcaster.update_deployment_status(+ is_running=True, status="running", active_runs=0, workflow_id=workflow_id,+ )payload = await _broadcast_control(control, controller_status=controller_status, extra={"resumed_queued_events": queued,"signalled_executions": signalled,diff --git a/server/services/deployment/manager.py b/server/services/deployment/manager.pyindex 864d23bf..1b901b34 100644--- a/server/services/deployment/manager.py+++ b/server/services/deployment/manager.py@@ -100,6 +100,16 @@ class DeploymentManager:"""Get list of deployed workflow IDs."""return [wid for wid, state in self._deployments.items() if state.is_running]+ def get_paused_workflows(self) -> List[str]:+ """Deployed workflow IDs whose admission is cooperatively paused.++ A paused deployment stays in ``get_deployed_workflows`` (it is+ armed), so consumers that render execution state need this set to+ avoid animating a paused workflow as running (the deployment+ snapshot carries both).+ """+ return [wid for wid in self._paused_workflows if self.is_workflow_deployed(wid)]+def pause(self, workflow_id: str) -> bool:"""Cooperatively stop admitting new runs for a deployment."""if not self.is_workflow_deployed(workflow_id):diff --git a/server/services/events/envelope.py b/server/services/events/envelope.pyindex dff818e3..a3764b3e 100644--- a/server/services/events/envelope.py+++ b/server/services/events/envelope.py@@ -404,6 +404,7 @@ class WorkflowEvent(BaseModel):def deployment_snapshot(cls,running_workflow_ids: list[str],+ paused_workflow_ids: Optional[list[str]] = None,) -> "WorkflowEvent":"""Push-on-connect snapshot of every currently-running deployment.@@ -424,7 +425,13 @@ class WorkflowEvent(BaseModel):return cls(source="opencompany://services/workflow",type=f"{_TYPE_PREFIX}workflow.deployment.snapshot",- data={"running_workflow_ids": list(running_workflow_ids)},+ data={+ "running_workflow_ids": list(running_workflow_ids),+ # Armed-but-paused deployments: still deployed (they stay+ # in running_workflow_ids) but must not render as+ # executing on the client.+ "paused_workflow_ids": list(paused_workflow_ids or []),+ },)@classmethoddiff --git a/server/services/status_broadcaster.py b/server/services/status_broadcaster.pyindex 05980e8d..2ac0233d 100644--- a/server/services/status_broadcaster.py+++ b/server/services/status_broadcaster.py@@ -156,6 +156,7 @@ class StatusBroadcaster:workflow_service = container.workflow_service()dm = workflow_service._get_deployment_manager()running_ids = dm.get_deployed_workflows()+ paused_ids = dm.get_paused_workflows()except Exception as e:# Backend startup race or DI not wired yet -- skip the snapshot# rather than fail the connect entirely. The empty-list path@@ -165,7 +166,7 @@ class StatusBroadcaster:)return- event = WorkflowEvent.deployment_snapshot(running_ids)+ event = WorkflowEvent.deployment_snapshot(running_ids, paused_ids)await websocket.send_json({"type": "deployment_snapshot",diff --git a/server/tests/temporal/test_durability_hardening.py b/server/tests/temporal/test_durability_hardening.pyindex 52c02e69..2a4610f6 100644--- a/server/tests/temporal/test_durability_hardening.py+++ b/server/tests/temporal/test_durability_hardening.py@@ -512,6 +512,92 @@ def test_control_event_types_search_attribute_is_registered():assert "ControlEventTypes" in names+class _EmptyWorkflowList:+ def __aiter__(self):+ return self++ async def __anext__(self):+ raise StopAsyncIteration+++ ("workflow_id", "expect_scoped"),+ [+ (None, False), # broadcast semantics (telegram/webhook/gmail)+ ("wf-1", True), # workflow-scoped (chat panel)+ ],+)+async def test_dispatch_narrows_visibility_query_for_scoped_envelopes(+ monkeypatch, workflow_id, expect_scoped+):+ """A workflow-scoped envelope must reach ONLY its own workflow's+ consumers — without the EventWorkflowId narrowing, one workflow's+ chat message fired every deployed workflow's chatTrigger."""+ from core.container import container+ from services.events.dispatch import _signal_running_consumers+ from services.events.envelope import WorkflowEvent++ captured: dict = {}++ def list_workflows(query):+ captured["query"] = query+ return _EmptyWorkflowList()++ client = MagicMock()+ client.list_workflows = list_workflows+ monkeypatch.setattr(+ container,+ "temporal_client",+ MagicMock(return_value=SimpleNamespace(client=client)),+ )++ event = WorkflowEvent(+ source="opencompany://nodes/chat_trigger",+ type="com.opencompany.chat.message.received",+ workflow_id=workflow_id,+ )+ await _signal_running_consumers(event)++ query = captured["query"]+ assert "EventType='com.opencompany.chat.message.received'" in query+ assert "WorkflowType='WorkflowControlWorkflow'" in query+ if expect_scoped:+ # Both the listener clause and the controller clause are scoped.+ assert query.count("AND EventWorkflowId='wf-1'") == 2+ else:+ assert "EventWorkflowId" not in query+++async def test_dispatch_scope_escapes_visibility_literals(monkeypatch):+ from core.container import container+ from services.events.dispatch import _signal_running_consumers+ from services.events.envelope import WorkflowEvent++ captured: dict = {}++ def list_workflows(query):+ captured["query"] = query+ return _EmptyWorkflowList()++ client = MagicMock()+ client.list_workflows = list_workflows+ monkeypatch.setattr(+ container,+ "temporal_client",+ MagicMock(return_value=SimpleNamespace(client=client)),+ )++ event = WorkflowEvent(+ source="opencompany://nodes/chat_trigger",+ type="com.opencompany.chat.message.received",+ workflow_id="wf'1",+ )+ await _signal_running_consumers(event)+ assert "EventWorkflowId='wf''1'" in captured["query"]++# ---------------------------------------------------------------------------# Terminate-sweep guard + controller addressing.# ---------------------------------------------------------------------------@@ -1186,6 +1272,42 @@ async def test_machina_failure_schedules_the_circuit_breaker_activity(monkeypatcassert captured == {}+def test_can_edit_capability_is_server_owned():+ """Canvas editability is a capability the backend computes — the FE+ renders it and never re-derives the rule from state strings. Paused+ is editable (controlled generations execute an immutable snapshot);+ starting/running/transitional states are not."""+ from services.deployment.control import serialize_control++ assert serialize_control(None)["can_edit"] is True++ expectations = {+ "starting": False,+ "running": False,+ "pausing": False,+ "resuming": False,+ "resetting": False,+ "paused": True,+ "failed": True,+ "reset": True, # serialized as "ready"+ }+ for status, expected in expectations.items():+ control = _boot_control(status)+ assert serialize_control(control)["can_edit"] is expected, status+++def test_deployment_snapshot_carries_paused_ids():+ from services.events.envelope import WorkflowEvent++ event = WorkflowEvent.deployment_snapshot(["wf-1", "wf-2"], ["wf-2"])+ assert event.data["running_workflow_ids"] == ["wf-1", "wf-2"]+ assert event.data["paused_workflow_ids"] == ["wf-2"]++ # Compatibility: the paused list defaults to empty, never absent.+ legacy = WorkflowEvent.deployment_snapshot(["wf-1"])+ assert legacy.data["paused_workflow_ids"] == []++def test_recovery_policies_normalize_unknown_values(monkeypatch):from core.container import containerfrom services.deployment import handlers