Skip to content

Self-improving AI Employees for everyone

fix(control): sync canvas locking with workflow execution via a server-owned can_edit capability

trohitg45a7aa27e234
diff --git a/CLAUDE.md b/CLAUDE.md
index c1ee5851..00988b18 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -742,7 +742,7 @@ Node groups (palette categories): agent, model, skill, tool, trigger, workflow,
Workflows execute via Temporal for durability and horizontal scaling, gated by `TEMPORAL_ENABLED`. Three dispatch paths exist — legacy single `execute_node_activity` (fallback), per-type `node.{type}.v{version}` activities (F4.A, `TEMPORAL_PER_TYPE_DISPATCH=true`), and Agent-as-child-workflow (F4.B, `TEMPORAL_AGENT_WORKFLOW_ENABLED=true`) — with `rlm_agent`/`claude_code_agent` always bypassing AgentWorkflow. **Per-queue activity routing is production-default (Wave 16)**: a `TemporalWorkerPool` runs one activity-only worker per plugin-declared `cls.task_queue` (ai-heavy / browser / code-exec / rest-api / messaging / ...) with per-queue concurrency + rate limits + resource-based tuning for ai-heavy/browser; rollback via `TEMPORAL_WORKER_POOL_ENABLED=false`. Resilience knobs (Waves 17-18): `DEPLOYMENT_MODE` (local halves concurrency; sizes the sticky cache), one-shot retry on `agent.execute_llm_step.v1` (LLM calls are not idempotent), observability interceptors (`activity_retry` WARN on re-dispatch), 30s periodic heartbeat during long activity bodies, cron `catchup_window=24h`, poller autoscaling. Execution routing falls back Temporal → sequential; the Temporal dev server + embedded worker are managed in-process under `server/services/temporal/` (pooch-downloaded CLI, single `temporal server start-dev` process, gRPC `TEMPORAL_FRONTEND_GRPC_PORT` + Web UI `TEMPORAL_UI_PORT`, SQLite at `~/.opencompany/temporal.db`). Full architecture, dispatch matrix, per-node + agent-loop lifecycle, all 16 F4.B `agent.*.v1` activities, heartbeat semantics, delegation input contract, the worker tuning recipe, and all `.env` tunables live in [docs-internal/TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md).
-**Months-long durability contract (July 2026).** Running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. Load-bearing invariants: (1) `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` stays `false` (debug-only sweep; even enabled, any control row in the shared `WORKFLOW_CONTROL_ACTIVE_STATES` — including `resetting` — vetoes it). (2) New child workflow starts carry **no** `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so lifetime caps silently terminated paused runs; liveness comes from activity heartbeats (30s beat / 2min timeout), never from workflow lifetime caps. Removing/adding start options or retry policies on recorded commands requires a `workflow.patched(...)` guard (see the `*-unbounded-*` / `*-history-bounded-can-*` markers). (3) Every long-lived workflow loop must continue-as-new under history pressure (`is_continue_as_new_suggested()` + a soft cap) — Temporal hard-terminates around ~51,200 history events; `WorkflowControlWorkflow` carries triggers/queued events/seen-ids/control state across rollovers, so controller handles are addressed **by workflow id only, never run_id-pinned**. (4) `dispatch.emit` skips controllers whose `ControlEventTypes` Search Attribute doesn't match the event (absent attribute = legacy match-all). (5) The Temporal lifecycle is owned by [services/temporal/lifecycle.py](./server/services/temporal/lifecycle.py) (`main.py` only schedules it): dev-server supervision + resident watchdog (`TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS`), worker crash-restarts that REBUILD the single-use SDK `Worker` per attempt (manager + every pool queue worker), and the boot-time `reconcile_active_controls_on_boot` pass (converges crash-stranded `starting` rows, re-arms running/paused generations from the persisted graph snapshot). (6) **Recovery policies** (env-driven, defaults in `.env.template`; full semantics in [temporal-workflow-control.md → Recovery policies](./docs-internal/temporal-workflow-control.md)): `WORKFLOW_CONTROL_CRASH_RECOVERY=pause` — after a kill/crash (dirty-bit marker cleared only by graceful teardown) boot recovers running generations as **paused** so the user consciously resumes; `WORKFLOW_CONTROL_MISSING_CONTROLLER=pause` — a killed/vanished controller converges the generation to paused and **Resume rebuilds the controller** (`WorkflowIdConflictPolicy.USE_EXISTING`, re-arm from snapshot) instead of forcing a state-archiving Reset; `WORKFLOW_CONTROL_PAUSE_ON_FAILURE=true` — repeatedly-failing trigger-spawned runs pause their deployment via the `workflow_control.pause_on_failure.v1` activity (circuit breaker; trips only after `WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD` (default 3) failures inside a rolling `_WINDOW_SECONDS` (default 600) so one node hiccup never pauses anything; Resume resets the streak; manual canvas runs never qualify). Temporal's native Pause/Unpause (server 1.28+) is deliberately NOT used — no Python SDK surface, and it halts workflow-task dispatch so a natively-paused controller couldn't process the resume Update. Locked by `tests/temporal/test_durability_hardening.py` + `test_worker_restart.py`.
+**Months-long durability contract (July 2026).** Running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. Load-bearing invariants: (1) `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` stays `false` (debug-only sweep; even enabled, any control row in the shared `WORKFLOW_CONTROL_ACTIVE_STATES` — including `resetting` — vetoes it). (2) New child workflow starts carry **no** `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so lifetime caps silently terminated paused runs; liveness comes from activity heartbeats (30s beat / 2min timeout), never from workflow lifetime caps. Removing/adding start options or retry policies on recorded commands requires a `workflow.patched(...)` guard (see the `*-unbounded-*` / `*-history-bounded-can-*` markers). (3) Every long-lived workflow loop must continue-as-new under history pressure (`is_continue_as_new_suggested()` + a soft cap) — Temporal hard-terminates around ~51,200 history events; `WorkflowControlWorkflow` carries triggers/queued events/seen-ids/control state across rollovers, so controller handles are addressed **by workflow id only, never run_id-pinned**. (4) `dispatch.emit` skips controllers whose `ControlEventTypes` Search Attribute doesn't match the event (absent attribute = legacy match-all). (5) The Temporal lifecycle is owned by [services/temporal/lifecycle.py](./server/services/temporal/lifecycle.py) (`main.py` only schedules it): dev-server supervision + resident watchdog (`TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS`), worker crash-restarts that REBUILD the single-use SDK `Worker` per attempt (manager + every pool queue worker), and the boot-time `reconcile_active_controls_on_boot` pass (converges crash-stranded `starting` rows, re-arms running/paused generations from the persisted graph snapshot). (6) **Recovery policies** (env-driven, defaults in `.env.template`; full semantics in [temporal-workflow-control.md → Recovery policies](./docs-internal/temporal-workflow-control.md)): `WORKFLOW_CONTROL_CRASH_RECOVERY=pause` — after a kill/crash (dirty-bit marker cleared only by graceful teardown) boot recovers running generations as **paused** so the user consciously resumes; `WORKFLOW_CONTROL_MISSING_CONTROLLER=pause` — a killed/vanished controller converges the generation to paused and **Resume rebuilds the controller** (`WorkflowIdConflictPolicy.USE_EXISTING`, re-arm from snapshot) instead of forcing a state-archiving Reset; `WORKFLOW_CONTROL_PAUSE_ON_FAILURE=true` — repeatedly-failing trigger-spawned runs pause their deployment via the `workflow_control.pause_on_failure.v1` activity (circuit breaker; trips only after `WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD` (default 3) failures inside a rolling `_WINDOW_SECONDS` (default 600) so one node hiccup never pauses anything; Resume resets the streak; manual canvas runs never qualify). Temporal's native Pause/Unpause (server 1.28+) is deliberately NOT used — no Python SDK surface, and it halts workflow-task dispatch so a natively-paused controller couldn't process the resume Update. (7) **Workflow-scoped event delivery**: an envelope whose `workflow_id` is set by its producer's call site reaches only that workflow's consumers (`dispatch.emit` narrows both Visibility clauses by `EventWorkflowId`); chat is the first scoped producer — the decision lives in `routers/websocket.py`, the narrowing in core dispatch, and plugin `_events.py` factories only plumb the field (see [event_framework.md](./docs-internal/event_framework.md)). (8) **Canvas editability is the server-owned `can_edit` capability** emitted by `serialize_control` (paused = editable; starting/running/transitional = locked); the FE renders it via `lib/canvasLock.ts` + Dashboard's `guardCanvasEdit` and never re-derives the rule from state strings (see [temporal-workflow-control.md → Canvas editability](./docs-internal/temporal-workflow-control.md)). Locked by `tests/temporal/test_durability_hardening.py` + `test_worker_restart.py`.
**Temporal tracing ownership is strict.** Register the SDK `TracingInterceptor` exactly once on the shared `Client.connect(...)`; the Python SDK automatically prepends compatible client interceptors to every worker. Never repeat `TracingInterceptor` in `Worker(..., interceptors=...)`. Worker lists contain the distinct `ObservabilityWorkerInterceptor` and plugin-specific worker interceptors only. Exact paired `StartActivity` / `RunActivity` / `CompleteWorkflow` console spans with identical Temporal identities indicate duplicate instrumentation, not duplicate execution; use Temporal Event History and the `node.<type>.execute` application span to verify actual attempts. See the tracing and debugging sections in [TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md) and [errors.md](./docs-internal/errors.md).
diff --git a/client/src/Dashboard.tsx b/client/src/Dashboard.tsx
index 3c425668..0512edfa 100644
--- a/client/src/Dashboard.tsx
+++ b/client/src/Dashboard.tsx
@@ -15,6 +15,7 @@ import {
Edge,
} from 'reactflow';
import { featureFlags } from './lib/featureFlags';
+import { deriveCanvasLock } from './lib/canvasLock';
import { prefetchAllNodeSpecs, listCachedNodeSpecs, cachedNodeSpecTypesKey } from './lib/nodeSpec';
import AIAgentNode from './components/AIAgentNode';
import SquareNode from './components/SquareNode';
@@ -240,17 +241,46 @@ const DashboardContent: React.FC = () => {
workflow_id: currentWorkflow.id, state: 'never_started' as const, revision: 0,
active_count: 0, in_flight_count: 0, queued_count: 0,
// Status is unresolved until the backend snapshot arrives. Keep every
- // lifecycle action disabled rather than briefly exposing a stale Start.
+ // lifecycle action disabled rather than briefly exposing a stale Start;
+ // the canvas stays editable until the server says otherwise.
can_start: false, can_pause: false, can_resume: false, can_reset: false,
+ can_edit: true,
}
- : { state: 'never_started' as const, revision: 0, active_count: 0, in_flight_count: 0, queued_count: 0, can_start: false, can_pause: false, can_resume: false, can_reset: false };
+ : { state: 'never_started' as const, revision: 0, active_count: 0, in_flight_count: 0, queued_count: 0, can_start: false, can_pause: false, can_resume: false, can_reset: false, can_edit: true };
const workflowControlPendingMutation = currentWorkflow?.id
? workflowControlPending[currentWorkflow.id]
: undefined;
- const isCurrentWorkflowLocked = workflowLock.locked &&
- workflowLock.workflow_id === currentWorkflow?.id;
+ // Canvas edit lock — the server-owned `can_edit` capability from the
+ // durable control plane is authoritative (backend SSOT; the FE only
+ // renders it). The legacy broadcaster lock remains a secondary input
+ // for deployments driven outside the control plane.
+ const canvasLock = deriveCanvasLock({
+ control: workflowControl,
+ legacyLock: workflowLock,
+ workflowId: currentWorkflow?.id ?? null,
+ });
+ const isCurrentWorkflowLocked = canvasLock.locked;
+ // Single guard for every canvas mutation the React Flow props can't
+ // reach (palette drop, paste, context-menu delete/rename, node
+ // disable, import, parameter saves). Toasts the reason so a blocked
+ // edit never reads as a dead UI.
+ const guardCanvasEdit = React.useCallback((): boolean => {
+ if (!canvasLock.locked) return true;
+ toast.warning(canvasLock.reason ?? 'Workflow is running — pause it to edit the canvas');
+ return false;
+ }, [canvasLock.locked, canvasLock.reason]);
const [globalModelDefaults, setGlobalModelDefaults] = React.useState<{ provider: string; model: string } | null>(null);
const { onDragOver, onDrop, handleComponentDragStart } = useDragAndDrop({ nodes, setNodes, saveNodeParameters, globalModelDefaults, workflowId: currentWorkflow?.id ?? 'new' });
+ // Palette drop creates nodes AND persists their default parameters —
+ // an HTML5 drop that React Flow's nodesDraggable/Connectable cannot
+ // block, so it goes through the shared guard.
+ const handleGuardedDrop = React.useCallback((event: React.DragEvent) => {
+ if (!guardCanvasEdit()) {
+ event.preventDefault();
+ return;
+ }
+ onDrop(event);
+ }, [guardCanvasEdit, onDrop]);
const { onConnect: baseOnConnect, onNodesDelete, onEdgesDelete: baseOnEdgesDelete } = useReactFlowNodes({ setNodes, setEdges, clearNodeStatus });
const { onConnect, onEdgesDelete } = useAutoSkillEdges({
baseOnConnect,
@@ -1168,11 +1198,11 @@ const DashboardContent: React.FC = () => {
// Context menu actions
const handleContextMenuRename = React.useCallback(() => {
- if (contextMenu) {
+ if (contextMenu && guardCanvasEdit()) {
setRenamingNodeId(contextMenu.nodeId);
}
closeContextMenu();
- }, [contextMenu, setRenamingNodeId, closeContextMenu]);
+ }, [contextMenu, guardCanvasEdit, setRenamingNodeId, closeContextMenu]);
const handleContextMenuCopy = React.useCallback(() => {
if (contextMenu) {
@@ -1188,11 +1218,13 @@ const DashboardContent: React.FC = () => {
}, [contextMenu, nodes, setNodes, copySelectedNodes, closeContextMenu]);
const handleContextMenuDelete = React.useCallback(() => {
- if (contextMenu) {
+ // The emptied deleteKeyCode only blocks the keyboard path — this
+ // menu action bypassed the lock entirely before the shared guard.
+ if (contextMenu && guardCanvasEdit()) {
onNodesDelete([nodes.find(n => n.id === contextMenu.nodeId)].filter(Boolean) as Node[]);
}
closeContextMenu();
- }, [contextMenu, nodes, onNodesDelete, closeContextMenu]);
+ }, [contextMenu, guardCanvasEdit, nodes, onNodesDelete, closeContextMenu]);
// Keyboard shortcut handler for workflow operations
useEffect(() => {
@@ -1211,7 +1243,9 @@ const DashboardContent: React.FC = () => {
// F2 to rename selected node
if (event.key === 'F2' && selectedNode) {
event.preventDefault();
- setRenamingNodeId(selectedNode.id);
+ if (guardCanvasEdit()) {
+ setRenamingNodeId(selectedNode.id);
+ }
return;
}
@@ -1229,7 +1263,11 @@ const DashboardContent: React.FC = () => {
break;
case 'v':
event.preventDefault();
- pasteNodes();
+ // Paste creates nodes AND persists their parameters to the
+ // DB — canvas-mutation guard applies.
+ if (guardCanvasEdit()) {
+ pasteNodes();
+ }
break;
}
} else {
@@ -1238,7 +1276,9 @@ const DashboardContent: React.FC = () => {
case 'd':
// Toggle disable on selected nodes
event.preventDefault();
- toggleDisableSelected();
+ if (guardCanvasEdit()) {
+ toggleDisableSelected();
+ }
break;
}
}
@@ -1249,7 +1289,7 @@ const DashboardContent: React.FC = () => {
return () => {
document.removeEventListener('keydown', handleKeyDown);
};
- }, [handleSave, copySelectedNodes, pasteNodes, toggleDisableSelected, selectedNode, renamingNodeId, setRenamingNodeId]);
+ }, [handleSave, copySelectedNodes, pasteNodes, toggleDisableSelected, selectedNode, renamingNodeId, setRenamingNodeId, guardCanvasEdit]);
return (
<>
@@ -1364,7 +1404,7 @@ const DashboardContent: React.FC = () => {
onEdgesDelete={onEdgesDelete}
onConnect={onConnect}
onDragOver={onDragOver}
- onDrop={onDrop}
+ onDrop={handleGuardedDrop}
onNodeContextMenu={onNodeContextMenu}
nodeTypes={nodeTypes}
edgeTypes={edgeTypes}
diff --git a/client/src/ParameterPanel.tsx b/client/src/ParameterPanel.tsx
index 0af5f029..370cc867 100644
--- a/client/src/ParameterPanel.tsx
+++ b/client/src/ParameterPanel.tsx
@@ -8,6 +8,7 @@ import { ExecutionService, ExecutionResult } from './services/executionService';
import { ActionButton } from './components/ui/action-button';
import { NodeIcon } from './assets/icons';
import { nodeExecutionReducer } from './utils/parameterPanelExecutionState';
+import { deriveCanvasLock } from './lib/canvasLock';
const ParameterPanel: React.FC = () => {
const {
@@ -22,7 +23,16 @@ const ParameterPanel: React.FC = () => {
} = useParameterPanel();
const currentWorkflow = useAppStore((s) => s.currentWorkflow);
- const { executeNode, getNodeParameters, clearNodeStatus, cancelEventWait, getNodeStatus } = useWebSocket();
+ const { executeNode, getNodeParameters, clearNodeStatus, cancelEventWait, getNodeStatus, workflowControlStatuses, workflowLock } = useWebSocket();
+
+ // Server-owned canvas-edit capability: parameter saves mutate the
+ // workflow's persisted configuration, so they follow the same
+ // `can_edit` capability the canvas gates on.
+ const canvasLock = deriveCanvasLock({
+ control: currentWorkflow?.id ? workflowControlStatuses[currentWorkflow.id] : null,
+ legacyLock: workflowLock,
+ workflowId: currentWorkflow?.id ?? null,
+ });
// Get current node status to check if waiting
const nodeStatus = selectedNode ? getNodeStatus(selectedNode.id) : null;
@@ -170,7 +180,12 @@ const ParameterPanel: React.FC = () => {
{selectedExecution.running ? 'Running...' : 'Run'}
</ActionButton>
)}
- <ActionButton intent="tools" onClick={handleSave} disabled={!hasUnsavedChanges}>
+ <ActionButton
+ intent="tools"
+ onClick={handleSave}
+ disabled={!hasUnsavedChanges || canvasLock.locked}
+ title={canvasLock.locked ? (canvasLock.reason ?? undefined) : undefined}
+ >
<svg width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2.5" strokeLinecap="round" strokeLinejoin="round">
<path d="M19 21H5a2 2 0 0 1-2-2V5a2 2 0 0 1 2-2h11l5 5v11a2 2 0 0 1-2 2z" />
<polyline points="17 21 17 13 7 13 7 21" />
diff --git a/client/src/components/ui/TopToolbar.tsx b/client/src/components/ui/TopToolbar.tsx
index 7eb84551..b7c7a7a9 100644
--- a/client/src/components/ui/TopToolbar.tsx
+++ b/client/src/components/ui/TopToolbar.tsx
@@ -23,7 +23,9 @@ import {
Repeat,
Clock,
Zap,
+ Lock,
} from 'lucide-react';
+import { Badge } from '@/components/ui/badge';
import {
Select,
SelectContent,
@@ -485,6 +487,19 @@ const TopToolbar: React.FC<TopToolbarProps> = ({
</ActionButton>
)}
+ {/* Canvas-lock indicator — renders the server-owned `can_edit`
+ capability so a silently-dead drag never reads as broken UI. */}
+ {workflowControl.can_edit === false && (
+ <Badge
+ variant="outline"
+ className="gap-1 text-muted-foreground"
+ title="Canvas locked while the workflow runs — pause it to edit"
+ >
+ <Lock className="h-3 w-3" />
+ Locked
+ </Badge>
+ )}
+
<AlertDialog open={resetOpen} onOpenChange={setResetOpen}><AlertDialogContent><AlertDialogHeader><AlertDialogTitle>Reset workflow execution?</AlertDialogTitle><AlertDialogDescription>This immediately terminates active work and archives the current generation. Task history remains available. The workflow will remain stopped until you press Start.</AlertDialogDescription></AlertDialogHeader><AlertDialogFooter><AlertDialogCancel>Keep current execution</AlertDialogCancel><AlertDialogAction disabled={hasPendingControlMutation} onClick={onResetWorkflow}>{isRetryingReset ? 'Retry reset' : 'Reset workflow'}</AlertDialogAction></AlertDialogFooter></AlertDialogContent></AlertDialog>
<ActionButton
diff --git a/client/src/components/ui/__tests__/CommandPaletteHost.workflowControl.test.ts b/client/src/components/ui/__tests__/CommandPaletteHost.workflowControl.test.ts
index 74582e20..7a9d7780 100644
--- a/client/src/components/ui/__tests__/CommandPaletteHost.workflowControl.test.ts
+++ b/client/src/components/ui/__tests__/CommandPaletteHost.workflowControl.test.ts
@@ -19,6 +19,7 @@ const control = (
can_pause: false,
can_resume: false,
can_reset: false,
+ can_edit: false,
...overrides,
});
diff --git a/client/src/contexts/WebSocketContext.tsx b/client/src/contexts/WebSocketContext.tsx
index 058b2024..7f184a50 100644
--- a/client/src/contexts/WebSocketContext.tsx
+++ b/client/src/contexts/WebSocketContext.tsx
@@ -132,7 +132,7 @@ export interface WorkflowStatus {
export interface DeploymentStatus {
isRunning: boolean;
activeRuns: number;
- status: 'idle' | 'starting' | 'running' | 'stopped' | 'cancelled' | 'error';
+ status: 'idle' | 'starting' | 'running' | 'paused' | 'stopped' | 'cancelled' | 'error';
workflow_id?: string | null; // Which workflow is deployed (for scoping)
totalTime?: number;
error?: string;
@@ -159,6 +159,9 @@ export interface WorkflowControlStatus {
can_pause: boolean;
can_resume: boolean;
can_reset: boolean;
+ /** Server-owned canvas-editability capability (backend SSOT — the FE
+ * renders it, never re-derives it from state strings). */
+ can_edit: boolean;
started_at?: string | null;
updated_at?: string | null;
error?: string | null;
@@ -560,6 +563,7 @@ export const emptyWorkflowControlStatus = (workflowId?: string): WorkflowControl
can_pause: false,
can_resume: false,
can_reset: false,
+ can_edit: true,
});
export const normalizeWorkflowControlStatus = (value: any, workflowId?: string): WorkflowControlStatus => {
@@ -579,6 +583,10 @@ export const normalizeWorkflowControlStatus = (value: any, workflowId?: string):
can_pause: source.can_pause ?? state === 'running',
can_resume: source.can_resume ?? state === 'paused',
can_reset: source.can_reset ?? state !== 'never_started',
+ // Pre-capability payloads (older backend) fall back to a state-derived
+ // approximation of the server rule; new payloads carry it verbatim.
+ can_edit: source.can_edit
+ ?? !['starting', 'running', 'pausing', 'resuming', 'resetting'].includes(state),
};
};
@@ -950,7 +958,11 @@ export const WebSocketProvider: React.FC<{ children: React.ReactNode }> = ({ chi
setDeploymentStatus({
isRunning: data.deployment.isRunning || false,
activeRuns: data.deployment.activeRuns || 0,
- status: data.deployment.status || 'idle'
+ status: data.deployment.status || 'idle',
+ // The backend slot carries the owning workflow — dropping
+ // it left isCurrentWorkflowDeployed false after reconnect
+ // (and mis-attributed running state across workflows).
+ workflow_id: data.deployment.workflow_id ?? null,
});
}
// Catalogue 'stored' flags derive from api_keys + oauth state;
@@ -1508,9 +1520,13 @@ export const WebSocketProvider: React.FC<{ children: React.ReactNode }> = ({ chi
// stayed showing "Stop" forever.
const envelope = data as WorkflowEvent<{
running_workflow_ids?: string[];
+ paused_workflow_ids?: string[];
}> | undefined;
const runningIds = envelope?.data?.running_workflow_ids ?? [];
const runningSet = new Set(runningIds);
+ // Armed-but-paused deployments: deployed (in running_workflow_ids)
+ // but must not render as executing.
+ const pausedSet = new Set(envelope?.data?.paused_workflow_ids ?? []);
const store = useAppStore.getState();
const currentId = store.currentWorkflow?.id;
@@ -1518,15 +1534,17 @@ export const WebSocketProvider: React.FC<{ children: React.ReactNode }> = ({ chi
// Anything currently flagged isExecuting=true that isn't in
// the snapshot's running set gets cleared (the load-bearing
// reset for stale state after backend restart). Anything in
- // the snapshot gets flagged true.
+ // the snapshot gets flagged true — unless it is paused.
const existingStates = store.workflowUIStates ?? {};
for (const [wid, ui] of Object.entries(existingStates)) {
- if (ui?.isExecuting && !runningSet.has(wid)) {
+ if (ui?.isExecuting && (!runningSet.has(wid) || pausedSet.has(wid))) {
store.setWorkflowExecuting(wid, false);
}
}
for (const wid of runningIds) {
- store.setWorkflowExecuting(wid, true);
+ if (!pausedSet.has(wid)) {
+ store.setWorkflowExecuting(wid, true);
+ }
}
// Reconcile the toolbar `deploymentStatus` for the active workflow
@@ -1534,7 +1552,7 @@ export const WebSocketProvider: React.FC<{ children: React.ReactNode }> = ({ chi
const next: DeploymentStatus = { ...prev };
if (currentId && runningSet.has(currentId)) {
next.isRunning = true;
- next.status = 'running';
+ next.status = pausedSet.has(currentId) ? 'paused' : 'running';
next.workflow_id = currentId;
} else if (currentId && !runningSet.has(currentId) && prev.workflow_id === currentId) {
// Previously thought current workflow was deployed; backend says no.
@@ -1623,8 +1641,18 @@ export const WebSocketProvider: React.FC<{ children: React.ReactNode }> = ({ chi
newStatus.activeRuns = message.data?.active_runs || prev.activeRuns + 1;
break;
case 'run_complete':
+ // The deployment manager emits `run_completed` /
+ // `run_failed` (see manager status callbacks); without
+ // these cases activeRuns only ever incremented.
+ case 'run_completed':
+ case 'run_failed':
newStatus.activeRuns = Math.max(0, message.data?.active_runs || prev.activeRuns - 1);
break;
+ case 'paused':
+ // Deployment stays armed; nothing new executes.
+ newStatus.isRunning = true;
+ newStatus.status = 'paused';
+ break;
case 'stopped':
// Only clear if this was our workflow or no workflow was tracked
if (!prev.workflow_id || prev.workflow_id === deploymentWorkflowId) {
@@ -1661,7 +1689,9 @@ export const WebSocketProvider: React.FC<{ children: React.ReactNode }> = ({ chi
if (deploymentWorkflowId) {
const { setWorkflowExecuting } = useAppStore.getState();
const isRunning = ['starting', 'running', 'started', 'run_started'].includes(message.status);
- const isStopped = ['stopped', 'cancelled', 'error'].includes(message.status);
+ // `paused` keeps the deployment armed but stops execution
+ // visuals (edge animation, toolbar indicator).
+ const isStopped = ['stopped', 'cancelled', 'error', 'paused'].includes(message.status);
if (isRunning || isStopped) {
setWorkflowExecuting(deploymentWorkflowId, isRunning);
}
@@ -1744,27 +1774,34 @@ export const WebSocketProvider: React.FC<{ children: React.ReactNode }> = ({ chi
case 'workflow_lock':
// Handle workflow lock status updates (per-workflow locking - n8n pattern)
- // Only update lock state if it's for the current workflow or if unlocking
if (data) {
const lockWorkflowId = message.workflow_id || data.workflow_id;
const activeWorkflowId = useAppStore.getState().currentWorkflow?.id;
- // Apply lock update if:
- // 1. It's for the current workflow, OR
- // 2. We're unlocking (locked=false), OR
- // 3. No specific workflow context (backward compatibility)
- const shouldApplyLock = !lockWorkflowId ||
- lockWorkflowId === activeWorkflowId ||
- !data.locked;
-
- if (shouldApplyLock) {
- setWorkflowLock({
- locked: data.locked || false,
- workflow_id: data.workflow_id || null,
- locked_at: data.locked_at || null,
- reason: data.reason || null
- });
- }
+ setWorkflowLock(prev => {
+ if (data.locked) {
+ // Apply a lock when it targets the current workflow, or
+ // carries no workflow context (backward compatibility).
+ if (!lockWorkflowId || lockWorkflowId === activeWorkflowId) {
+ return {
+ locked: true,
+ workflow_id: data.workflow_id || null,
+ locked_at: data.locked_at || null,
+ reason: data.reason || null,
+ };
+ }
+ return prev;
+ }
+ // Unlock ONLY when it releases the lock we're holding —
+ // an unscoped unlock or one for the held workflow. An
+ // unlock for a DIFFERENT workflow (e.g. resetting workflow
+ // B while viewing locked workflow A) previously clobbered
+ // the singleton and unlocked A's canvas mid-run.
+ if (!prev.locked || !lockWorkflowId || prev.workflow_id === lockWorkflowId || !prev.workflow_id) {
+ return { locked: false, workflow_id: null, locked_at: null, reason: null };
+ }
+ return prev;
+ });
}
break;
diff --git a/client/src/contexts/__tests__/workflowControlStatus.test.ts b/client/src/contexts/__tests__/workflowControlStatus.test.ts
index a63c222b..5c250f0c 100644
--- a/client/src/contexts/__tests__/workflowControlStatus.test.ts
+++ b/client/src/contexts/__tests__/workflowControlStatus.test.ts
@@ -26,6 +26,7 @@ const status = (
can_pause: true,
can_resume: false,
can_reset: true,
+ can_edit: false,
...overrides,
});
diff --git a/client/src/lib/__tests__/canvasLock.test.ts b/client/src/lib/__tests__/canvasLock.test.ts
new file mode 100644
index 00000000..53905d47
--- /dev/null
+++ b/client/src/lib/__tests__/canvasLock.test.ts
@@ -0,0 +1,84 @@
+import { describe, expect, it } from 'vitest';
+import { deriveCanvasLock } from '../canvasLock';
+
+const legacyUnlocked = { locked: false, workflow_id: null };
+
+describe('deriveCanvasLock', () => {
+ it('renders the server-owned can_edit capability verbatim', () => {
+ expect(
+ deriveCanvasLock({
+ control: { state: 'running', can_edit: false },
+ legacyLock: legacyUnlocked,
+ workflowId: 'wf-1',
+ }).locked,
+ ).toBe(true);
+
+ // paused is editable — controlled generations execute their
+ // immutable admitted snapshot, so the server grants can_edit.
+ expect(
+ deriveCanvasLock({
+ control: { state: 'paused', can_edit: true },
+ legacyLock: legacyUnlocked,
+ workflowId: 'wf-1',
+ }).locked,
+ ).toBe(false);
+ });
+
+ it('never re-derives the rule from state strings alone', () => {
+ // A running state WITH can_edit granted stays editable — the FE
+ // must not override the backend's decision.
+ expect(
+ deriveCanvasLock({
+ control: { state: 'running', can_edit: true },
+ legacyLock: legacyUnlocked,
+ workflowId: 'wf-1',
+ }).locked,
+ ).toBe(false);
+ });
+
+ it('gives transitional states a distinct reason', () => {
+ const transition = deriveCanvasLock({
+ control: { state: 'pausing', can_edit: false },
+ legacyLock: legacyUnlocked,
+ workflowId: 'wf-1',
+ });
+ expect(transition.locked).toBe(true);
+ expect(transition.reason).toContain('transition');
+
+ const running = deriveCanvasLock({
+ control: { state: 'running', can_edit: false },
+ legacyLock: legacyUnlocked,
+ workflowId: 'wf-1',
+ });
+ expect(running.reason).toContain('pause it to edit');
+ });
+
+ it('falls back to the legacy broadcaster lock for the same workflow only', () => {
+ const legacyLocked = { locked: true, workflow_id: 'wf-1' };
+ expect(
+ deriveCanvasLock({
+ control: { state: 'never_started', can_edit: true },
+ legacyLock: legacyLocked,
+ workflowId: 'wf-1',
+ }).locked,
+ ).toBe(true);
+
+ // A lock held by ANOTHER workflow never locks this canvas.
+ expect(
+ deriveCanvasLock({
+ control: { state: 'never_started', can_edit: true },
+ legacyLock: legacyLocked,
+ workflowId: 'wf-2',
+ }).locked,
+ ).toBe(false);
+ });
+
+ it('is unlocked with no control status and no legacy lock', () => {
+ const result = deriveCanvasLock({
+ control: null,
+ legacyLock: legacyUnlocked,
+ workflowId: 'wf-1',
+ });
+ expect(result).toEqual({ locked: false, reason: null });
+ });
+});
diff --git a/client/src/lib/canvasLock.ts b/client/src/lib/canvasLock.ts
new file mode 100644
index 00000000..f5d705c5
--- /dev/null
+++ b/client/src/lib/canvasLock.ts
@@ -0,0 +1,58 @@
+/**
+ * Canvas edit-lock presentation helper.
+ *
+ * The DECISION of whether a workflow's canvas may be edited is
+ * server-owned: `serialize_control` (backend) emits the `can_edit`
+ * capability alongside `can_start` / `can_pause` / …, and the frontend
+ * renders it — it never re-derives the rule from state strings. This
+ * module only maps the capability (plus the legacy broadcaster lock for
+ * deployments driven outside the control plane) to a boolean + a
+ * human-readable reason, which are pure UI concerns.
+ */
+
+export interface CanvasLockControlInput {
+ state?: string | null;
+ can_edit?: boolean;
+}
+
+export interface CanvasLockInput {
+ /** Durable control-plane status for the workflow (server-normalized). */
+ control: CanvasLockControlInput | null | undefined;
+ /** Legacy broadcaster lock singleton (`workflowLock` from WebSocketContext). */
+ legacyLock: { locked: boolean; workflow_id: string | null } | null | undefined;
+ /** The workflow currently open on the canvas. */
+ workflowId: string | null | undefined;
+}
+
+export interface CanvasLockResult {
+ locked: boolean;
+ /** Human-readable cause for the indicator / blocked-edit toast. */
+ reason: string | null;
+}
+
+const TRANSITION_STATES = new Set(['pausing', 'resuming', 'resetting', 'starting']);
+
+export function deriveCanvasLock({
+ control,
+ legacyLock,
+ workflowId,
+}: CanvasLockInput): CanvasLockResult {
+ if (control && control.can_edit === false) {
+ const state = control.state ?? null;
+ return {
+ locked: true,
+ reason:
+ state && TRANSITION_STATES.has(state) && state !== 'running'
+ ? 'Workflow is changing state — wait for the transition to finish'
+ : 'Workflow is running — pause it to edit the canvas',
+ };
+ }
+ if (
+ legacyLock?.locked &&
+ !!workflowId &&
+ legacyLock.workflow_id === workflowId
+ ) {
+ return { locked: true, reason: 'Workflow is deployed — stop it to edit the canvas' };
+ }
+ return { locked: false, reason: null };
+}
diff --git a/docs-internal/temporal-workflow-control.md b/docs-internal/temporal-workflow-control.md
index 403e68b0..91ae3af9 100644
--- a/docs-internal/temporal-workflow-control.md
+++ b/docs-internal/temporal-workflow-control.md
@@ -214,6 +214,30 @@ that Resume relies on.
every knob is evaluated on the activity side so flipping config never
touches recorded workflow commands.
+## Canvas editability (`can_edit` capability)
+
+Whether a workflow's canvas may be edited is a **server-owned capability**,
+emitted by `serialize_control` alongside `can_start` / `can_pause` / … —
+the frontend renders it and never re-derives the rule from state strings.
+Editable while nothing is armed (`ready` / `failed` / `never_started`) and
+while **paused** (controlled generations execute their immutable admitted
+snapshot, so edits cannot corrupt in-flight runs — paused is exactly the
+"fix it, then resume" posture the recovery policies produce); locked while
+`starting` / `running` / transitional. On the client,
+[`lib/canvasLock.ts`](../client/src/lib/canvasLock.ts) maps the capability
+(plus the legacy broadcaster lock for deployments driven outside the
+control plane) to a boolean + reason; `Dashboard` feeds it to the React
+Flow interaction props AND a shared `guardCanvasEdit` toast-guard covering
+the paths those props cannot reach (palette drop, paste, context-menu
+delete/rename, node disable, parameter saves); `TopToolbar` shows a
+`Locked` badge so a blocked drag never reads as dead UI.
+
+The pause/resume/cancel/re-arm paths also emit the UI-facing
+`workflow_status(executing=…)` + `deployment_status` broadcasts (status
+`paused` keeps `isRunning=true` — armed but not executing), and the
+connect-time deployment snapshot carries `paused_workflow_ids` so an
+armed-but-paused generation doesn't animate as running after a reconnect.
+
## Delegated-task traces
Each task attempt stores the actual parent, detached runner, and child Temporal
diff --git a/server/services/deployment/control.py b/server/services/deployment/control.py
index 3d3c795b..dd18c758 100644
--- a/server/services/deployment/control.py
+++ b/server/services/deployment/control.py
@@ -21,11 +21,22 @@ def _graph_hash(nodes: list[dict], edges: list[dict]) -> str:
return hashlib.sha256(value.encode()).hexdigest()
+# Canvas editability is a server-owned capability, like can_start/can_pause:
+# the frontend renders it, never re-derives it from state strings. Editable
+# while nothing is armed (ready / failed) and while PAUSED — controlled
+# generations execute their immutable admitted snapshot, so edits cannot
+# corrupt in-flight runs, and paused is exactly the "fix it, then resume"
+# posture the recovery policies produce. Locked while starting / running /
+# transitional so a live deployment cannot be rewired mid-flight.
+_EDITABLE_STATES = frozenset({"ready", "paused", "failed"})
+
+
def serialize_control(control: Optional[WorkflowControlExecution]) -> Dict[str, Any]:
if control is None:
return {
"state": "never_started", "revision": 0, "generation": 0,
"can_start": True, "can_pause": False, "can_resume": False, "can_reset": False,
+ "can_edit": True,
}
state = "ready" if control.status == "reset" else control.status
return {
@@ -42,6 +53,7 @@ def serialize_control(control: Optional[WorkflowControlExecution]) -> Dict[str,
"can_pause": state == "running",
"can_resume": state == "paused",
"can_reset": state != "ready",
+ "can_edit": state in _EDITABLE_STATES,
"created_at": control.created_at.isoformat() if control.created_at else None,
"updated_at": control.updated_at.isoformat() if control.updated_at else None,
"terminal_reason": control.terminal_reason,
diff --git a/server/services/deployment/handlers.py b/server/services/deployment/handlers.py
index 6f495a05..37788595 100644
--- a/server/services/deployment/handlers.py
+++ b/server/services/deployment/handlers.py
@@ -209,6 +209,16 @@ async def handle_deploy_workflow(data: Dict[str, Any], websocket: WebSocket) ->
if not result.get("success"):
logger.error("Deployment setup failed", error=result.get("error"), workflow_id=workflow_id)
+ # Clear BOTH UI-facing flags: the deploy handler already
+ # broadcast executing=True, and deployments never touch
+ # the run counter, so nothing else would ever emit
+ # executing=False for this workflow (toolbar stuck).
+ await broadcaster.update_workflow_status(
+ executing=False,
+ current_node=None,
+ progress=0,
+ workflow_id=workflow_id,
+ )
await broadcaster.update_deployment_status(
is_running=False,
status="error",
@@ -238,8 +248,22 @@ async def handle_deploy_workflow(data: Dict[str, Any], websocket: WebSocket) ->
)
return result
+ except asyncio.CancelledError:
+ # CancelledError is BaseException — the branch below cannot
+ # catch it, and a deploy task cancelled outside
+ # handle_cancel_deployment (which unlocks itself) previously
+ # left the workflow lock dangling forever.
+ await broadcaster.unlock_workflow(workflow_id)
+ _deployment_tasks.pop(workflow_id, None)
+ raise
except Exception as e:
logger.error("Deployment task error", workflow_id=workflow_id, error=str(e))
+ await broadcaster.update_workflow_status(
+ executing=False,
+ current_node=None,
+ progress=0,
+ workflow_id=workflow_id,
+ )
await broadcaster.update_deployment_status(
is_running=False,
status="error",
@@ -308,21 +332,26 @@ async def handle_cancel_deployment(data: Dict[str, Any], websocket: WebSocket) -
for node_id in result.get("cancelled_listener_node_ids", []):
await broadcaster.clear_node_status(node_id)
- await broadcaster.update_workflow_status(
- executing=False,
- current_node=None,
- progress=0,
- workflow_id=workflow_id,
- )
- await broadcaster.update_deployment_status(
- is_running=False,
- status="cancelled",
- active_runs=0,
- workflow_id=workflow_id,
- data={
- "iterations_completed": result.get("iterations_completed", 0),
- },
- )
+ # Terminal UI-facing state is emitted UNCONDITIONALLY: cancelling a
+ # workflow that is not locally deployed (post-restart before re-arm,
+ # or a Reset after a crash) previously emitted nothing, leaving the
+ # FE showing executing/running forever. The idempotent terminal
+ # broadcast is harmless when nothing was running.
+ await broadcaster.update_workflow_status(
+ executing=False,
+ current_node=None,
+ progress=0,
+ workflow_id=workflow_id,
+ )
+ await broadcaster.update_deployment_status(
+ is_running=False,
+ status="cancelled",
+ active_runs=0,
+ workflow_id=workflow_id,
+ data={
+ "iterations_completed": result.get("iterations_completed", 0),
+ },
+ )
return {
"success": result.get("success", False),
@@ -1315,6 +1344,20 @@ async def _rearm_generation(control) -> None:
paused=True,
)
await _set_cron_pause(control.workflow_id, paused=True, strict=False)
+ # The deploy handler just broadcast executing=true / status=running
+ # on the wire; counter-broadcast the paused posture so a connected
+ # client doesn't watch the workflow flip to running and never back.
+ from services.status_broadcaster import get_status_broadcaster
+
+ broadcaster = get_status_broadcaster()
+ await broadcaster.update_workflow_status(
+ executing=False, current_node=None, progress=0,
+ workflow_id=control.workflow_id,
+ )
+ await broadcaster.update_deployment_status(
+ is_running=True, status="paused", active_runs=0,
+ workflow_id=control.workflow_id,
+ )
logger.info(
"Re-armed durable generation after restart",
workflow_id=control.workflow_id,
@@ -1542,6 +1585,19 @@ async def handle_start_workflow(data: Dict[str, Any], websocket: WebSocket) -> D
error=str(reset_exc),
)
await _terminate_generation_workflows(control)
+ # Tear down the local half too: the deploy may have armed
+ # listeners and taken the workflow lock, and leaving them meant a
+ # locked, armed, invisible deployment that blocked every future
+ # Start AND the boot re-arm for this workflow. Cancel unlocks and
+ # emits the terminal UI-facing state; "not deployed" is benign.
+ try:
+ await handle_cancel_deployment({"workflow_id": workflow_id}, None)
+ except Exception as cancel_exc: # noqa: BLE001 — best-effort teardown
+ logger.warning(
+ "Local teardown after failed start did not complete",
+ workflow_id=workflow_id,
+ error=str(cancel_exc),
+ )
await _broadcast_control(control)
raise
@@ -1619,6 +1675,20 @@ async def handle_pause_workflow(data: Dict[str, Any], websocket: WebSocket) -> D
strict=True,
)
control = await service.transition(control, expected_revision=control.revision, from_statuses={"pausing"}, status="paused")
+ # UI-facing execution flags follow the pause: the deployment stays
+ # armed but nothing new executes, so the canvas edge animation and
+ # toolbar executing indicator must stop. The pause family previously
+ # emitted only workflow_control_status, leaving `executing=true` /
+ # `status=running` stale on every connected client.
+ from services.status_broadcaster import get_status_broadcaster
+
+ broadcaster = get_status_broadcaster()
+ await broadcaster.update_workflow_status(
+ executing=False, current_node=None, progress=0, workflow_id=workflow_id,
+ )
+ await broadcaster.update_deployment_status(
+ is_running=True, status="paused", active_runs=0, workflow_id=workflow_id,
+ )
payload = await _broadcast_control(control, controller_status=controller_status, extra={
"signalled_executions": signalled,
"paused_schedules": paused_schedules,
@@ -1716,6 +1786,17 @@ async def handle_resume_workflow(data: Dict[str, Any], websocket: WebSocket) ->
# Operator intervention resets the circuit-breaker streak — the next
# failure after a resume starts a fresh count, not a near-tripped one.
await _clear_failure_streak(service.database, control)
+ # Mirror of the pause-side emission: flip the UI-facing execution
+ # flags back so the canvas and toolbar re-animate on every client.
+ from services.status_broadcaster import get_status_broadcaster
+
+ broadcaster = get_status_broadcaster()
+ await broadcaster.update_workflow_status(
+ executing=True, current_node=None, progress=0, workflow_id=workflow_id,
+ )
+ await broadcaster.update_deployment_status(
+ is_running=True, status="running", active_runs=0, workflow_id=workflow_id,
+ )
payload = await _broadcast_control(control, controller_status=controller_status, extra={
"resumed_queued_events": queued,
"signalled_executions": signalled,
diff --git a/server/services/deployment/manager.py b/server/services/deployment/manager.py
index 864d23bf..1b901b34 100644
--- a/server/services/deployment/manager.py
+++ b/server/services/deployment/manager.py
@@ -100,6 +100,16 @@ class DeploymentManager:
"""Get list of deployed workflow IDs."""
return [wid for wid, state in self._deployments.items() if state.is_running]
+ def get_paused_workflows(self) -> List[str]:
+ """Deployed workflow IDs whose admission is cooperatively paused.
+
+ A paused deployment stays in ``get_deployed_workflows`` (it is
+ armed), so consumers that render execution state need this set to
+ avoid animating a paused workflow as running (the deployment
+ snapshot carries both).
+ """
+ return [wid for wid in self._paused_workflows if self.is_workflow_deployed(wid)]
+
def pause(self, workflow_id: str) -> bool:
"""Cooperatively stop admitting new runs for a deployment."""
if not self.is_workflow_deployed(workflow_id):
diff --git a/server/services/events/envelope.py b/server/services/events/envelope.py
index dff818e3..a3764b3e 100644
--- a/server/services/events/envelope.py
+++ b/server/services/events/envelope.py
@@ -404,6 +404,7 @@ class WorkflowEvent(BaseModel):
def deployment_snapshot(
cls,
running_workflow_ids: list[str],
+ paused_workflow_ids: Optional[list[str]] = None,
) -> "WorkflowEvent":
"""Push-on-connect snapshot of every currently-running deployment.
@@ -424,7 +425,13 @@ class WorkflowEvent(BaseModel):
return cls(
source="opencompany://services/workflow",
type=f"{_TYPE_PREFIX}workflow.deployment.snapshot",
- data={"running_workflow_ids": list(running_workflow_ids)},
+ data={
+ "running_workflow_ids": list(running_workflow_ids),
+ # Armed-but-paused deployments: still deployed (they stay
+ # in running_workflow_ids) but must not render as
+ # executing on the client.
+ "paused_workflow_ids": list(paused_workflow_ids or []),
+ },
)
@classmethod
diff --git a/server/services/status_broadcaster.py b/server/services/status_broadcaster.py
index 05980e8d..2ac0233d 100644
--- a/server/services/status_broadcaster.py
+++ b/server/services/status_broadcaster.py
@@ -156,6 +156,7 @@ class StatusBroadcaster:
workflow_service = container.workflow_service()
dm = workflow_service._get_deployment_manager()
running_ids = dm.get_deployed_workflows()
+ paused_ids = dm.get_paused_workflows()
except Exception as e:
# Backend startup race or DI not wired yet -- skip the snapshot
# rather than fail the connect entirely. The empty-list path
@@ -165,7 +166,7 @@ class StatusBroadcaster:
)
return
- event = WorkflowEvent.deployment_snapshot(running_ids)
+ event = WorkflowEvent.deployment_snapshot(running_ids, paused_ids)
await websocket.send_json(
{
"type": "deployment_snapshot",
diff --git a/server/tests/temporal/test_durability_hardening.py b/server/tests/temporal/test_durability_hardening.py
index 52c02e69..2a4610f6 100644
--- a/server/tests/temporal/test_durability_hardening.py
+++ b/server/tests/temporal/test_durability_hardening.py
@@ -512,6 +512,92 @@ def test_control_event_types_search_attribute_is_registered():
assert "ControlEventTypes" in names
+class _EmptyWorkflowList:
+ def __aiter__(self):
+ return self
+
+ async def __anext__(self):
+ raise StopAsyncIteration
+
+
+ ("workflow_id", "expect_scoped"),
+ [
+ (None, False), # broadcast semantics (telegram/webhook/gmail)
+ ("wf-1", True), # workflow-scoped (chat panel)
+ ],
+)
+async def test_dispatch_narrows_visibility_query_for_scoped_envelopes(
+ monkeypatch, workflow_id, expect_scoped
+):
+ """A workflow-scoped envelope must reach ONLY its own workflow's
+ consumers — without the EventWorkflowId narrowing, one workflow's
+ chat message fired every deployed workflow's chatTrigger."""
+ from core.container import container
+ from services.events.dispatch import _signal_running_consumers
+ from services.events.envelope import WorkflowEvent
+
+ captured: dict = {}
+
+ def list_workflows(query):
+ captured["query"] = query
+ return _EmptyWorkflowList()
+
+ client = MagicMock()
+ client.list_workflows = list_workflows
+ monkeypatch.setattr(
+ container,
+ "temporal_client",
+ MagicMock(return_value=SimpleNamespace(client=client)),
+ )
+
+ event = WorkflowEvent(
+ source="opencompany://nodes/chat_trigger",
+ type="com.opencompany.chat.message.received",
+ workflow_id=workflow_id,
+ )
+ await _signal_running_consumers(event)
+
+ query = captured["query"]
+ assert "EventType='com.opencompany.chat.message.received'" in query
+ assert "WorkflowType='WorkflowControlWorkflow'" in query
+ if expect_scoped:
+ # Both the listener clause and the controller clause are scoped.
+ assert query.count("AND EventWorkflowId='wf-1'") == 2
+ else:
+ assert "EventWorkflowId" not in query
+
+
+async def test_dispatch_scope_escapes_visibility_literals(monkeypatch):
+ from core.container import container
+ from services.events.dispatch import _signal_running_consumers
+ from services.events.envelope import WorkflowEvent
+
+ captured: dict = {}
+
+ def list_workflows(query):
+ captured["query"] = query
+ return _EmptyWorkflowList()
+
+ client = MagicMock()
+ client.list_workflows = list_workflows
+ monkeypatch.setattr(
+ container,
+ "temporal_client",
+ MagicMock(return_value=SimpleNamespace(client=client)),
+ )
+
+ event = WorkflowEvent(
+ source="opencompany://nodes/chat_trigger",
+ type="com.opencompany.chat.message.received",
+ workflow_id="wf'1",
+ )
+ await _signal_running_consumers(event)
+ assert "EventWorkflowId='wf''1'" in captured["query"]
+
+
# ---------------------------------------------------------------------------
# Terminate-sweep guard + controller addressing.
# ---------------------------------------------------------------------------
@@ -1186,6 +1272,42 @@ async def test_machina_failure_schedules_the_circuit_breaker_activity(monkeypatc
assert captured == {}
+def test_can_edit_capability_is_server_owned():
+ """Canvas editability is a capability the backend computes — the FE
+ renders it and never re-derives the rule from state strings. Paused
+ is editable (controlled generations execute an immutable snapshot);
+ starting/running/transitional states are not."""
+ from services.deployment.control import serialize_control
+
+ assert serialize_control(None)["can_edit"] is True
+
+ expectations = {
+ "starting": False,
+ "running": False,
+ "pausing": False,
+ "resuming": False,
+ "resetting": False,
+ "paused": True,
+ "failed": True,
+ "reset": True, # serialized as "ready"
+ }
+ for status, expected in expectations.items():
+ control = _boot_control(status)
+ assert serialize_control(control)["can_edit"] is expected, status
+
+
+def test_deployment_snapshot_carries_paused_ids():
+ from services.events.envelope import WorkflowEvent
+
+ event = WorkflowEvent.deployment_snapshot(["wf-1", "wf-2"], ["wf-2"])
+ assert event.data["running_workflow_ids"] == ["wf-1", "wf-2"]
+ assert event.data["paused_workflow_ids"] == ["wf-2"]
+
+ # Compatibility: the paused list defaults to empty, never absent.
+ legacy = WorkflowEvent.deployment_snapshot(["wf-1"])
+ assert legacy.data["paused_workflow_ids"] == []
+
+
def test_recovery_policies_normalize_unknown_values(monkeypatch):
from core.container import container
from services.deployment import handlers