Skip to content

Self-improving AI Employees for everyone

docs(temporal): document Workspace tasks and the new WebSocket close rules

trohitgc98ba8438b86
diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.md
index 8cfa0254..86cd7c99 100644
--- a/docs-internal/TEMPORAL_ARCHITECTURE.md
+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md
@@ -514,6 +514,17 @@ the tool-result cap apply whether or not compaction is on (see
References: [Temporal AI Cookbook](https://docs.temporal.io/ai-cookbook), [`temporal-community/temporal-ai-agent`](https://github.com/temporal-community/temporal-ai-agent), [`temporalio.contrib.openai_agents`](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/openai_agents).
+### Direct Workspace tasks
+
+A node class that declares `workspace_task = True` (the phone nodes in `server/nodes/mobile/`) can also run a single task outside any graph run. The Workspace's **Ask AI to use the phone** box posts to `/api/mobile/{workflow_id}/{node_id}/tasks` (`server/nodes/mobile/_router.py`), and `services/node_invocations.py::submit` admits the task.
+
+- **Admission.** Each OpenCompany workflow has one long-lived `WorkspaceTaskControllerWorkflow` (`services/temporal/workspace_tasks_workflow.py`, Temporal id from `node_invocations.controller_id`), created on first use through Update-With-Start. Its `submit` Update checks the admission epoch, bounds the number of active tasks, deduplicates a retried submission by its id and prompt fingerprint, and starts one `NodeInvocationWorkflow` child per task (`services/temporal/node_invocation.py`). The child runs the node's per-type activity once, with no retry, and records the run through `workflow_runs.record_completion`.
+- **Reset.** The controller's `reset` Update first fences new submissions by advancing the epoch. It then cancels every admitted child and waits for each to close, and runs the `workspace_tasks.reset_runtime` activity (`services/temporal/workspace_task_activities.py::reset_workspace_task_runtime`). That activity cancels invocations started before the controller existed and calls each Workspace-task node's `reset_execution_state`. The Update returns only after that cleanup. If cleanup fails, the fence stays in place, and retrying Reset resumes the same request. The workflow toolbar's Reset reaches the controller through `services/deployment/handlers.py::_reset_workspace_tasks`, even for a workflow that was never started; see [temporal-workflow-control.md](./temporal-workflow-control.md#workspace-tasks).
+- **Registration and rollover.** Both workflow classes are in `worker.py::_framework_workflows`, and every framework worker registers `reset_workspace_task_runtime`. The controller continues-as-new under history pressure, carrying its epoch and its bounded submission and reset records.
+- **Inside graph and agent runs.** Two patch markers give Workspace-task nodes their own activity options. `workspace-node-cancellation-v1` (`MachinaWorkflow`) uses the node class's `start_to_close_timeout` and `heartbeat_timeout`, and makes cancellation wait until the activity has finished cleaning up. `workspace-task-activity-policy-v1` (tool calls in `AgentWorkflow`) does the same and also applies the class's `retry_policy` and, with the worker pool on, its `task_queue`.
+
+What the user sees, including what Reset leaves on the phone, is in [docs/mobile-workspace.md](../docs/mobile-workspace.md#resetting-phone-tasks).
+
## Config Node Filtering
Certain nodes provide configuration rather than executing:
@@ -590,6 +601,9 @@ server/services/temporal/
├── workflow_control_workflow.py # WorkflowControlWorkflow (per-generation controller: triggers, signals, polling, pause)
├── trigger_listener_workflow.py # TriggerListenerWorkflow (legacy push-trigger listener)
├── polling_trigger_workflow.py # PollingTriggerWorkflow (legacy polling listener)
+├── workspace_tasks_workflow.py # WorkspaceTaskControllerWorkflow (per-workflow admission + Reset for direct Workspace tasks)
+├── node_invocation.py # NodeInvocationWorkflow (one direct Workspace task)
+├── workspace_task_activities.py # reset_workspace_task_runtime (activity workspace_tasks.reset_runtime)
├── schedules.py # Temporal Schedule creation for cronScheduler
├── search_attributes.py # EVENT_SEARCH_ATTRIBUTES (7 custom SAs, incl. ControlEventTypes)
├── _retry_policies.py # DEFAULT_ACTIVITY_RETRY / QUICK_ACTIVITY_RETRY / LLM_STEP_RETRY / PERMIT_WAIT_RETRY / ...
@@ -664,7 +678,7 @@ The Temporal binary + persistence are managed in-process by the plugin-folder pa
**Months-long durability contract**: running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. None of the mechanisms below carries a `workflow.patched` marker: the lifetime-cap removal and the poll-interval floor apply unconditionally, including when an older history is replayed.
-- **No lifetime caps on new child runs.** Trigger/cron-spawned `MachinaWorkflow` runs, agent children, and delegated-task runners previously carried 1-2h `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so any pause longer than the cap silently terminated the run (and a timed-out delegated runner skipped its compensation: leaked permit + stuck task row). New executions start children unbounded (no patch marker was retained for this change). The live `workflow.patched` markers are `machina-conditional-edges-v1` and `machina-run-record-v1` in `services/temporal/workflow.py`, `machina-trigger-listener-node-filter` in `services/temporal/trigger_listener_workflow.py`, and `workspace-task-activity-policy-v1` in `services/temporal/agent_workflow.py`; the list grows, so check it with `grep -rn "workflow.patched(" server/services/temporal`. `machina-run-record-v1` gates the `workflow_runs.record_completion` activity that trigger-spawned runs schedule when they finish, for Normal mode's "done today"; see [normal_mode.md](./normal_mode.md#done-today)). Liveness is the activity layer's job: node activities heartbeat every 30s against a 2-minute `heartbeat_timeout`; their `start_to_close` is a generous 24h ceiling, not 10 minutes. The subagent-permit wait uses `PERMIT_WAIT_RETRY` (unlimited attempts) so a queued delegation waits as long as admission takes instead of failing after ~3h.
+- **No lifetime caps on new child runs.** Trigger/cron-spawned `MachinaWorkflow` runs, agent children, and delegated-task runners previously carried 1-2h `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so any pause longer than the cap silently terminated the run (and a timed-out delegated runner skipped its compensation: leaked permit + stuck task row). New executions start children unbounded (no patch marker was retained for this change). The live `workflow.patched` markers are `machina-conditional-edges-v1`, `machina-run-record-v1` and `workspace-node-cancellation-v1` in `services/temporal/workflow.py`, `machina-trigger-listener-node-filter` in `services/temporal/trigger_listener_workflow.py`, and `workspace-task-activity-policy-v1` in `services/temporal/agent_workflow.py`; the list grows, so check it with `grep -rn "workflow.patched(" server/services/temporal`. `machina-run-record-v1` gates the `workflow_runs.record_completion` activity that trigger-spawned runs schedule when they finish, for Normal mode's "done today" (see [normal_mode.md](./normal_mode.md#done-today)). The two `workspace-*` markers are described under [Direct Workspace tasks](#direct-workspace-tasks). Liveness is the activity layer's job: node activities heartbeat every 30s against a 2-minute `heartbeat_timeout`; their `start_to_close` is a generous 24h ceiling, not 10 minutes. The subagent-permit wait uses `PERMIT_WAIT_RETRY` (unlimited attempts) so a queued delegation waits as long as admission takes instead of failing after ~3h.
- **History-pressure continue-as-new everywhere.** Temporal terminates any workflow around ~51,200 history events. `WorkflowControlWorkflow` (which multiplexes all of a deployment's triggers into one history) now rolls over on `is_continue_as_new_suggested()` / a 10K-event soft cap, carrying trigger specs, per-trigger provider `seen_ids` (written back into the spec after every poll cycle), queued push events, the bounded dedup baseline, and the control state — a rollover works mid-pause too, since a paused controller still accretes signal history. `TriggerListenerWorkflow`/`PollingTriggerWorkflow` gained the same pressure check (the old `_processed_count >= 16_000` gate was unreachable: real spawns cost ~15-25 events each, and polling counted only emitted events while a quiet mailbox burned ~11 events/cycle — dead in ~3 days at the 60s default). Poll intervals are clamped to a 30s floor (`_MIN_POLL_INTERVAL_S` in `polling_trigger_workflow.py` and `workflow_control_workflow.py`), with no patch gate. Because run ids change on rollover, **controller handles are addressed by workflow id only, never run_id-pinned** (`_controller_handle`, manager `register_trigger`).
- **dispatch.emit controller narrowing.** Controllers advertise their push event types via the `ControlEventTypes` keyword-list Search Attribute (upserted as triggers register); `dispatch.emit` skips controllers with no matching trigger instead of signalling every running controller with every platform event (each unmatched signal was ~4 immutable history events — one busy deployment burned every other controller's rollover budget). Controllers without the attribute (pre-upgrade histories) keep match-all behaviour.
- **Boot-time reconcile** (`reconcile_active_controls_on_boot`, called from the lifecycle module after workers start): runs the lazy `_reconcile_control` over every active control row, converges `starting` rows a crash left behind (controller alive with triggers registered → `running`; alive-but-empty for a graph that declares triggers → `failed` + controller closed; vanished → `failed`), and re-arms the process-local half of running/paused generations from the persisted graph snapshot — DeploymentManager state, in-process collectors for non-canary trigger types, cron pause posture. Idempotent by construction (controller `register_trigger` keyed by listener id, legacy starts use `USE_EXISTING`, cron creation preserves server-owned pause state).
diff --git a/docs-internal/frontend_architecture.md b/docs-internal/frontend_architecture.md
index 06dfbb36..30a1ccdb 100644
--- a/docs-internal/frontend_architecture.md
+++ b/docs-internal/frontend_architecture.md
@@ -199,7 +199,8 @@ client/src/
│ ├── queryConfig.ts / featureFlags.ts
│ ├── queryPersist.ts # localStorage persister + PERSISTED_KEY_PREFIXES whitelist
│ ├── brandStorage.ts # Canonical browser-storage keys + pre-rebrand aliases
-│ ├── connectionConfig.ts # WS reconnect + auth-bootstrap backoff constants
+│ ├── connectionConfig.ts # WS reconnect, heartbeat and close-code constants + auth-bootstrap backoff
+│ ├── webSocketHeartbeat.ts # App-level ping/pong liveness check for one open socket
│ ├── workflowOps.ts # applyOperations for backend workflow-ops batches
│ ├── canvasLock.ts # Server-owned can_edit capability -> canvas lock
│ ├── credentialProviderId.ts # Which catalogue provider a canvas node's status dot reads (plugin credential ids pass through; legacy names mapped)
@@ -455,13 +456,14 @@ See [media_transport.md](./media_transport.md).
## Real-time
-`contexts/WebSocketContext.tsx` is the single connection + event bus. ~125 handlers (see `server/routers/websocket.py`). Handlers follow a request/response pattern via `sendRequest(type, data)` with correlation IDs. Push-only events (node status, workflow progress, token usage, android/whatsapp status) set context state directly; components subscribe via selector hooks (`useAndroidStatus`, `useNodeStatus(nodeId)`).
+`contexts/WebSocketContext.tsx` is the single connection + event bus. The server's handlers are the `MESSAGE_HANDLERS` dict in `server/routers/websocket.py` plus the plugin handlers registered through `services.ws_handler_registry` (live count: `len(MESSAGE_HANDLERS) + len(get_ws_handlers())`). Handlers follow a request/response pattern via `sendRequest(type, data)` with correlation IDs. Push-only events (node status, workflow progress, token usage, android/whatsapp status) set context state directly; components subscribe via selector hooks (`useAndroidStatus`, `useNodeStatus(nodeId)`).
**Rules:**
- `useEffect` fetch-on-mount is banned for anything the backend can push. Subscribe to the context slice instead.
- All modifying operations go through WebSocket — REST is reserved for auth + webhooks.
- No polling. If a component wants fresh data, call `sendRequest` once (or use TanStack Query with `staleTime: Infinity` + manual `invalidate`).
-- **`sendRequest` queues during disconnect with backpressure.** When the socket is not open, the request enqueues with an `AbortController`-backed per-request timeout (default 30s) and replays on reconnect inside `ws.onopen` before `setIsReady(true)`. Queue caps at 200 with FIFO eviction (rejects oldest with `backpressure: too many queued requests`). Intentional close (`event.code === 1000`) drops the queue; transient closes preserve it. Eliminates indefinite spinners during the 3-second reconnect window. Implementation: `pendingSendQueueRef` + `drainPendingSends` in `WebSocketContext.tsx`.
+- **`sendRequest` queues while the socket is down, with backpressure.** When the socket is not open, the request enqueues with an `AbortController`-backed per-request timeout (default 30s). On reconnect, `ws.onopen` sends it before `setIsReady(true)`, with whatever remains of its deadline. The queue caps at 200 with FIFO eviction (the oldest is rejected with `backpressure: too many queued requests`). Any close rejects requests that were already sent, because their outcome is unknown and they are never replayed; queued requests keep waiting. A remote close, even with code 1000, does not stop reconnecting. Only `disposeConnection` (logout or unmount) closes the socket with 1000, which stops PartySocket's retries and rejects everything still queued. Implementation: `pendingSendQueueRef`, `drainPendingSends` and `disposeConnection` in `WebSocketContext.tsx`.
+- **An application heartbeat replaces a dead socket.** While a socket is open and the page is visible, `lib/webSocketHeartbeat.ts` sends `{"type": "ping"}` every `WS_HEARTBEAT.INTERVAL_MS` (30 s). If no `pong` arrives within `WS_HEARTBEAT.TIMEOUT_MS` (10 s), it calls `ws.reconnect` with close code 4000 (`WS_CLOSE.HEARTBEAT_TIMEOUT`). A hidden page sends no pings. When the page becomes visible again, or the browser reports `online`, it probes at once with a fresh deadline. Reconnect delays come from `WS_RECONNECT` in `lib/connectionConfig.ts`.
- **Workflow-control state reconciles on every WS connect.** The toolbar derives
Start/Pause/Resume/Reset availability from `get_workflow_control_status`, whose
persisted generation record is reconciled with the Temporal controller rather
diff --git a/docs-internal/status_broadcaster.md b/docs-internal/status_broadcaster.md
index 83e9fd2f..a0b5da8c 100644
--- a/docs-internal/status_broadcaster.md
+++ b/docs-internal/status_broadcaster.md
@@ -75,7 +75,7 @@ StatusBroadcaster.connect(ws)
Message loop:
receive_json -> dispatch: MESSAGE_HANDLERS.get(type) or get_ws_handlers().get(type)
send_json -> pushed by broadcaster on state change
- ping/pong -> keepalive every 30s from frontend
+ ping/pong -> frontend heartbeat (WS_HEARTBEAT; visible pages only)
|
v
Frontend unmounts or logs out
@@ -84,7 +84,7 @@ Frontend unmounts or logs out
StatusBroadcaster.disconnect(ws) -> remove from _connections
```
-Auto-reconnect is handled by `WebSocketContext.tsx` through PartySocket's `ReconnectingWebSocket` (`partysocket/ws`): jittered exponential backoff (`MIN_DELAY_MS` / `MAX_DELAY_MS` / `GROW_FACTOR` in `client/src/lib/connectionConfig.ts`), message replay, and intentional-close handling (code 1000 stops the reconnect loop). A 100ms mount delay avoids React Strict Mode double-connect in dev.
+Auto-reconnect is handled by `WebSocketContext.tsx` through PartySocket's `ReconnectingWebSocket` (`partysocket/ws`), with jittered exponential backoff (`MIN_DELAY_MS` / `MAX_DELAY_MS` / `GROW_FACTOR` in `client/src/lib/connectionConfig.ts`). Requests queued while disconnected are sent after the next open; requests already in flight are rejected on close and never replayed. A remote close, even with code 1000, keeps the loop retrying; only the client's own `disposeConnection` (logout or unmount) closes with 1000 and stops it. The full client-side rules are in [frontend_architecture.md → Real-time](./frontend_architecture.md#real-time). A 100ms mount delay avoids React Strict Mode double-connect in dev.
## Handler Registry
@@ -219,8 +219,8 @@ Android service nodes (`batteryMonitor`, `wifiAutomation`, etc.) check `androidS
**Frontend** (`WebSocketContext.tsx`):
-- 30-second `setInterval` sends `{"type": "ping"}`.
-- On disconnect: PartySocket's `ReconnectingWebSocket` reconnects with jittered exponential backoff (`client/src/lib/connectionConfig.ts`), replaying queued messages; an intentional close (code 1000) stops the loop.
+- While the socket is open and the page is visible, `startWebSocketHeartbeat` (`client/src/lib/webSocketHeartbeat.ts`) sends `{"type": "ping"}` every 30 s (`WS_HEARTBEAT`). If no `pong` arrives within 10 s, it reconnects with close code 4000 (`WS_CLOSE.HEARTBEAT_TIMEOUT`).
+- On disconnect: PartySocket's `ReconnectingWebSocket` reconnects with jittered exponential backoff (`client/src/lib/connectionConfig.ts`). In-flight requests are rejected, and requests queued while disconnected are sent after the next open. A remote close, including code 1000, does not stop the loop; only `disposeConnection` (logout or unmount) does.
- 100ms mount delay avoids React Strict Mode double-connect in development.
- `isMountedRef` prevents connections after unmount.
- WebSocket is gated on `isAuthenticated` from `AuthContext`: if auth is disabled or not yet loaded, the provider defers connection.
diff --git a/docs-internal/temporal-workflow-control.md b/docs-internal/temporal-workflow-control.md
index e74f9421..d0774945 100644
--- a/docs-internal/temporal-workflow-control.md
+++ b/docs-internal/temporal-workflow-control.md
@@ -18,8 +18,9 @@ is authoritative for execution history.
nodes.
- **Reset** revision-guards the old generation, closes controller and local
admission, removes Temporal cron schedules, cancels local compatibility
- resources, and then performs a final strict execution sweep before archiving
- the old generation. It leaves the control state `ready`; the user must press
+ resources, and then performs a final strict execution sweep and resets the
+ workflow's [Workspace tasks](#workspace-tasks) before archiving the old
+ generation. It leaves the control state `ready`; the user must press
**Start** to create the next generation.
Clients send an expected revision and idempotency key with every mutation.
@@ -132,6 +133,35 @@ prefer termination over durable resumption. The active-state guard reads the
shared `WORKFLOW_CONTROL_ACTIVE_STATES` set (which includes `resetting`), so a
boot mid-reset can never sweep the namespace.
+## Workspace tasks
+
+Phone tasks submitted directly from the Workspace run outside the control
+generation, under a per-workflow `WorkspaceTaskControllerWorkflow` (see
+[TEMPORAL_ARCHITECTURE.md → Direct Workspace tasks](./TEMPORAL_ARCHITECTURE.md#direct-workspace-tasks)).
+They can exist before the first Start and after a Reset. Pause and Resume do
+not gate them; of the control states, only a Reset in progress blocks new
+submissions. The control plane covers them in two places.
+
+- **Reset.** With an active generation, Reset resets the Workspace tasks after
+ its final execution sweep and before archiving
+ (`services/deployment/handlers.py::_reset_workspace_tasks`), and
+ `workflow_runtime_reset` carries `cancelled_workspace_tasks`. With no
+ generation, or after the latest one was reset, Reset still checks the
+ expected revision and then resets only the Workspace tasks. It returns
+ idempotently when the graph has no Workspace-task node and no controller
+ exists; otherwise it broadcasts `workflow_runtime_reset` and
+ `workflow_control_status`.
+- **Status.** Every control status read, mutation response and
+ `workflow_control_status` broadcast passes through `_with_runtime_counts`,
+ which merges `_workspace_runtime_status`. When the saved graph contains a
+ Workspace-task node, `can_reset` is true in every state and the controller's
+ active tasks are added to `active_count` and `in_flight_count`. Once the
+ controller exists, the payload also carries `workspace_epoch`,
+ `workspace_resetting` and `workspace_reset_request_id`; `workspace_available`
+ is false when Temporal or the controller query is unavailable. While the
+ controller is resetting, the payload reports `state: "resetting"` and turns
+ off `can_start`, `can_pause`, `can_resume` and `can_edit`.
+
## Months-long generations
A controlled generation is expected to run — or stay paused — for months, so