Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
docs(temporal): document Workspace tasks and the new WebSocket close rules
trohitgc98ba8438b86
diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.mdindex 8cfa0254..86cd7c99 100644--- a/docs-internal/TEMPORAL_ARCHITECTURE.md+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md@@ -514,6 +514,17 @@ the tool-result cap apply whether or not compaction is on (seeReferences: [Temporal AI Cookbook](https://docs.temporal.io/ai-cookbook), [`temporal-community/temporal-ai-agent`](https://github.com/temporal-community/temporal-ai-agent), [`temporalio.contrib.openai_agents`](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/openai_agents).+### Direct Workspace tasks++A node class that declares `workspace_task = True` (the phone nodes in `server/nodes/mobile/`) can also run a single task outside any graph run. The Workspace's **Ask AI to use the phone** box posts to `/api/mobile/{workflow_id}/{node_id}/tasks` (`server/nodes/mobile/_router.py`), and `services/node_invocations.py::submit` admits the task.++- **Admission.** Each OpenCompany workflow has one long-lived `WorkspaceTaskControllerWorkflow` (`services/temporal/workspace_tasks_workflow.py`, Temporal id from `node_invocations.controller_id`), created on first use through Update-With-Start. Its `submit` Update checks the admission epoch, bounds the number of active tasks, deduplicates a retried submission by its id and prompt fingerprint, and starts one `NodeInvocationWorkflow` child per task (`services/temporal/node_invocation.py`). The child runs the node's per-type activity once, with no retry, and records the run through `workflow_runs.record_completion`.+- **Reset.** The controller's `reset` Update first fences new submissions by advancing the epoch. It then cancels every admitted child and waits for each to close, and runs the `workspace_tasks.reset_runtime` activity (`services/temporal/workspace_task_activities.py::reset_workspace_task_runtime`). That activity cancels invocations started before the controller existed and calls each Workspace-task node's `reset_execution_state`. The Update returns only after that cleanup. If cleanup fails, the fence stays in place, and retrying Reset resumes the same request. The workflow toolbar's Reset reaches the controller through `services/deployment/handlers.py::_reset_workspace_tasks`, even for a workflow that was never started; see [temporal-workflow-control.md](./temporal-workflow-control.md#workspace-tasks).+- **Registration and rollover.** Both workflow classes are in `worker.py::_framework_workflows`, and every framework worker registers `reset_workspace_task_runtime`. The controller continues-as-new under history pressure, carrying its epoch and its bounded submission and reset records.+- **Inside graph and agent runs.** Two patch markers give Workspace-task nodes their own activity options. `workspace-node-cancellation-v1` (`MachinaWorkflow`) uses the node class's `start_to_close_timeout` and `heartbeat_timeout`, and makes cancellation wait until the activity has finished cleaning up. `workspace-task-activity-policy-v1` (tool calls in `AgentWorkflow`) does the same and also applies the class's `retry_policy` and, with the worker pool on, its `task_queue`.++What the user sees, including what Reset leaves on the phone, is in [docs/mobile-workspace.md](../docs/mobile-workspace.md#resetting-phone-tasks).+## Config Node FilteringCertain nodes provide configuration rather than executing:@@ -590,6 +601,9 @@ server/services/temporal/├── workflow_control_workflow.py # WorkflowControlWorkflow (per-generation controller: triggers, signals, polling, pause)├── trigger_listener_workflow.py # TriggerListenerWorkflow (legacy push-trigger listener)├── polling_trigger_workflow.py # PollingTriggerWorkflow (legacy polling listener)+├── workspace_tasks_workflow.py # WorkspaceTaskControllerWorkflow (per-workflow admission + Reset for direct Workspace tasks)+├── node_invocation.py # NodeInvocationWorkflow (one direct Workspace task)+├── workspace_task_activities.py # reset_workspace_task_runtime (activity workspace_tasks.reset_runtime)├── schedules.py # Temporal Schedule creation for cronScheduler├── search_attributes.py # EVENT_SEARCH_ATTRIBUTES (7 custom SAs, incl. ControlEventTypes)├── _retry_policies.py # DEFAULT_ACTIVITY_RETRY / QUICK_ACTIVITY_RETRY / LLM_STEP_RETRY / PERMIT_WAIT_RETRY / ...@@ -664,7 +678,7 @@ The Temporal binary + persistence are managed in-process by the plugin-folder pa**Months-long durability contract**: running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. None of the mechanisms below carries a `workflow.patched` marker: the lifetime-cap removal and the poll-interval floor apply unconditionally, including when an older history is replayed.-- **No lifetime caps on new child runs.** Trigger/cron-spawned `MachinaWorkflow` runs, agent children, and delegated-task runners previously carried 1-2h `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so any pause longer than the cap silently terminated the run (and a timed-out delegated runner skipped its compensation: leaked permit + stuck task row). New executions start children unbounded (no patch marker was retained for this change). The live `workflow.patched` markers are `machina-conditional-edges-v1` and `machina-run-record-v1` in `services/temporal/workflow.py`, `machina-trigger-listener-node-filter` in `services/temporal/trigger_listener_workflow.py`, and `workspace-task-activity-policy-v1` in `services/temporal/agent_workflow.py`; the list grows, so check it with `grep -rn "workflow.patched(" server/services/temporal`. `machina-run-record-v1` gates the `workflow_runs.record_completion` activity that trigger-spawned runs schedule when they finish, for Normal mode's "done today"; see [normal_mode.md](./normal_mode.md#done-today)). Liveness is the activity layer's job: node activities heartbeat every 30s against a 2-minute `heartbeat_timeout`; their `start_to_close` is a generous 24h ceiling, not 10 minutes. The subagent-permit wait uses `PERMIT_WAIT_RETRY` (unlimited attempts) so a queued delegation waits as long as admission takes instead of failing after ~3h.+- **No lifetime caps on new child runs.** Trigger/cron-spawned `MachinaWorkflow` runs, agent children, and delegated-task runners previously carried 1-2h `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so any pause longer than the cap silently terminated the run (and a timed-out delegated runner skipped its compensation: leaked permit + stuck task row). New executions start children unbounded (no patch marker was retained for this change). The live `workflow.patched` markers are `machina-conditional-edges-v1`, `machina-run-record-v1` and `workspace-node-cancellation-v1` in `services/temporal/workflow.py`, `machina-trigger-listener-node-filter` in `services/temporal/trigger_listener_workflow.py`, and `workspace-task-activity-policy-v1` in `services/temporal/agent_workflow.py`; the list grows, so check it with `grep -rn "workflow.patched(" server/services/temporal`. `machina-run-record-v1` gates the `workflow_runs.record_completion` activity that trigger-spawned runs schedule when they finish, for Normal mode's "done today" (see [normal_mode.md](./normal_mode.md#done-today)). The two `workspace-*` markers are described under [Direct Workspace tasks](#direct-workspace-tasks). Liveness is the activity layer's job: node activities heartbeat every 30s against a 2-minute `heartbeat_timeout`; their `start_to_close` is a generous 24h ceiling, not 10 minutes. The subagent-permit wait uses `PERMIT_WAIT_RETRY` (unlimited attempts) so a queued delegation waits as long as admission takes instead of failing after ~3h.- **History-pressure continue-as-new everywhere.** Temporal terminates any workflow around ~51,200 history events. `WorkflowControlWorkflow` (which multiplexes all of a deployment's triggers into one history) now rolls over on `is_continue_as_new_suggested()` / a 10K-event soft cap, carrying trigger specs, per-trigger provider `seen_ids` (written back into the spec after every poll cycle), queued push events, the bounded dedup baseline, and the control state — a rollover works mid-pause too, since a paused controller still accretes signal history. `TriggerListenerWorkflow`/`PollingTriggerWorkflow` gained the same pressure check (the old `_processed_count >= 16_000` gate was unreachable: real spawns cost ~15-25 events each, and polling counted only emitted events while a quiet mailbox burned ~11 events/cycle — dead in ~3 days at the 60s default). Poll intervals are clamped to a 30s floor (`_MIN_POLL_INTERVAL_S` in `polling_trigger_workflow.py` and `workflow_control_workflow.py`), with no patch gate. Because run ids change on rollover, **controller handles are addressed by workflow id only, never run_id-pinned** (`_controller_handle`, manager `register_trigger`).- **dispatch.emit controller narrowing.** Controllers advertise their push event types via the `ControlEventTypes` keyword-list Search Attribute (upserted as triggers register); `dispatch.emit` skips controllers with no matching trigger instead of signalling every running controller with every platform event (each unmatched signal was ~4 immutable history events — one busy deployment burned every other controller's rollover budget). Controllers without the attribute (pre-upgrade histories) keep match-all behaviour.- **Boot-time reconcile** (`reconcile_active_controls_on_boot`, called from the lifecycle module after workers start): runs the lazy `_reconcile_control` over every active control row, converges `starting` rows a crash left behind (controller alive with triggers registered → `running`; alive-but-empty for a graph that declares triggers → `failed` + controller closed; vanished → `failed`), and re-arms the process-local half of running/paused generations from the persisted graph snapshot — DeploymentManager state, in-process collectors for non-canary trigger types, cron pause posture. Idempotent by construction (controller `register_trigger` keyed by listener id, legacy starts use `USE_EXISTING`, cron creation preserves server-owned pause state).diff --git a/docs-internal/frontend_architecture.md b/docs-internal/frontend_architecture.mdindex 06dfbb36..30a1ccdb 100644--- a/docs-internal/frontend_architecture.md+++ b/docs-internal/frontend_architecture.md@@ -199,7 +199,8 @@ client/src/│ ├── queryConfig.ts / featureFlags.ts│ ├── queryPersist.ts # localStorage persister + PERSISTED_KEY_PREFIXES whitelist│ ├── brandStorage.ts # Canonical browser-storage keys + pre-rebrand aliases-│ ├── connectionConfig.ts # WS reconnect + auth-bootstrap backoff constants+│ ├── connectionConfig.ts # WS reconnect, heartbeat and close-code constants + auth-bootstrap backoff+│ ├── webSocketHeartbeat.ts # App-level ping/pong liveness check for one open socket│ ├── workflowOps.ts # applyOperations for backend workflow-ops batches│ ├── canvasLock.ts # Server-owned can_edit capability -> canvas lock│ ├── credentialProviderId.ts # Which catalogue provider a canvas node's status dot reads (plugin credential ids pass through; legacy names mapped)@@ -455,13 +456,14 @@ See [media_transport.md](./media_transport.md).## Real-time-`contexts/WebSocketContext.tsx` is the single connection + event bus. ~125 handlers (see `server/routers/websocket.py`). Handlers follow a request/response pattern via `sendRequest(type, data)` with correlation IDs. Push-only events (node status, workflow progress, token usage, android/whatsapp status) set context state directly; components subscribe via selector hooks (`useAndroidStatus`, `useNodeStatus(nodeId)`).+`contexts/WebSocketContext.tsx` is the single connection + event bus. The server's handlers are the `MESSAGE_HANDLERS` dict in `server/routers/websocket.py` plus the plugin handlers registered through `services.ws_handler_registry` (live count: `len(MESSAGE_HANDLERS) + len(get_ws_handlers())`). Handlers follow a request/response pattern via `sendRequest(type, data)` with correlation IDs. Push-only events (node status, workflow progress, token usage, android/whatsapp status) set context state directly; components subscribe via selector hooks (`useAndroidStatus`, `useNodeStatus(nodeId)`).**Rules:**- `useEffect` fetch-on-mount is banned for anything the backend can push. Subscribe to the context slice instead.- All modifying operations go through WebSocket — REST is reserved for auth + webhooks.- No polling. If a component wants fresh data, call `sendRequest` once (or use TanStack Query with `staleTime: Infinity` + manual `invalidate`).-- **`sendRequest` queues during disconnect with backpressure.** When the socket is not open, the request enqueues with an `AbortController`-backed per-request timeout (default 30s) and replays on reconnect inside `ws.onopen` before `setIsReady(true)`. Queue caps at 200 with FIFO eviction (rejects oldest with `backpressure: too many queued requests`). Intentional close (`event.code === 1000`) drops the queue; transient closes preserve it. Eliminates indefinite spinners during the 3-second reconnect window. Implementation: `pendingSendQueueRef` + `drainPendingSends` in `WebSocketContext.tsx`.+- **`sendRequest` queues while the socket is down, with backpressure.** When the socket is not open, the request enqueues with an `AbortController`-backed per-request timeout (default 30s). On reconnect, `ws.onopen` sends it before `setIsReady(true)`, with whatever remains of its deadline. The queue caps at 200 with FIFO eviction (the oldest is rejected with `backpressure: too many queued requests`). Any close rejects requests that were already sent, because their outcome is unknown and they are never replayed; queued requests keep waiting. A remote close, even with code 1000, does not stop reconnecting. Only `disposeConnection` (logout or unmount) closes the socket with 1000, which stops PartySocket's retries and rejects everything still queued. Implementation: `pendingSendQueueRef`, `drainPendingSends` and `disposeConnection` in `WebSocketContext.tsx`.+- **An application heartbeat replaces a dead socket.** While a socket is open and the page is visible, `lib/webSocketHeartbeat.ts` sends `{"type": "ping"}` every `WS_HEARTBEAT.INTERVAL_MS` (30 s). If no `pong` arrives within `WS_HEARTBEAT.TIMEOUT_MS` (10 s), it calls `ws.reconnect` with close code 4000 (`WS_CLOSE.HEARTBEAT_TIMEOUT`). A hidden page sends no pings. When the page becomes visible again, or the browser reports `online`, it probes at once with a fresh deadline. Reconnect delays come from `WS_RECONNECT` in `lib/connectionConfig.ts`.- **Workflow-control state reconciles on every WS connect.** The toolbar derivesStart/Pause/Resume/Reset availability from `get_workflow_control_status`, whosepersisted generation record is reconciled with the Temporal controller ratherdiff --git a/docs-internal/status_broadcaster.md b/docs-internal/status_broadcaster.mdindex 83e9fd2f..a0b5da8c 100644--- a/docs-internal/status_broadcaster.md+++ b/docs-internal/status_broadcaster.md@@ -75,7 +75,7 @@ StatusBroadcaster.connect(ws)Message loop:receive_json -> dispatch: MESSAGE_HANDLERS.get(type) or get_ws_handlers().get(type)send_json -> pushed by broadcaster on state change- ping/pong -> keepalive every 30s from frontend+ ping/pong -> frontend heartbeat (WS_HEARTBEAT; visible pages only)|vFrontend unmounts or logs out@@ -84,7 +84,7 @@ Frontend unmounts or logs outStatusBroadcaster.disconnect(ws) -> remove from _connections```-Auto-reconnect is handled by `WebSocketContext.tsx` through PartySocket's `ReconnectingWebSocket` (`partysocket/ws`): jittered exponential backoff (`MIN_DELAY_MS` / `MAX_DELAY_MS` / `GROW_FACTOR` in `client/src/lib/connectionConfig.ts`), message replay, and intentional-close handling (code 1000 stops the reconnect loop). A 100ms mount delay avoids React Strict Mode double-connect in dev.+Auto-reconnect is handled by `WebSocketContext.tsx` through PartySocket's `ReconnectingWebSocket` (`partysocket/ws`), with jittered exponential backoff (`MIN_DELAY_MS` / `MAX_DELAY_MS` / `GROW_FACTOR` in `client/src/lib/connectionConfig.ts`). Requests queued while disconnected are sent after the next open; requests already in flight are rejected on close and never replayed. A remote close, even with code 1000, keeps the loop retrying; only the client's own `disposeConnection` (logout or unmount) closes with 1000 and stops it. The full client-side rules are in [frontend_architecture.md → Real-time](./frontend_architecture.md#real-time). A 100ms mount delay avoids React Strict Mode double-connect in dev.## Handler Registry@@ -219,8 +219,8 @@ Android service nodes (`batteryMonitor`, `wifiAutomation`, etc.) check `androidS**Frontend** (`WebSocketContext.tsx`):-- 30-second `setInterval` sends `{"type": "ping"}`.-- On disconnect: PartySocket's `ReconnectingWebSocket` reconnects with jittered exponential backoff (`client/src/lib/connectionConfig.ts`), replaying queued messages; an intentional close (code 1000) stops the loop.+- While the socket is open and the page is visible, `startWebSocketHeartbeat` (`client/src/lib/webSocketHeartbeat.ts`) sends `{"type": "ping"}` every 30 s (`WS_HEARTBEAT`). If no `pong` arrives within 10 s, it reconnects with close code 4000 (`WS_CLOSE.HEARTBEAT_TIMEOUT`).+- On disconnect: PartySocket's `ReconnectingWebSocket` reconnects with jittered exponential backoff (`client/src/lib/connectionConfig.ts`). In-flight requests are rejected, and requests queued while disconnected are sent after the next open. A remote close, including code 1000, does not stop the loop; only `disposeConnection` (logout or unmount) does.- 100ms mount delay avoids React Strict Mode double-connect in development.- `isMountedRef` prevents connections after unmount.- WebSocket is gated on `isAuthenticated` from `AuthContext`: if auth is disabled or not yet loaded, the provider defers connection.diff --git a/docs-internal/temporal-workflow-control.md b/docs-internal/temporal-workflow-control.mdindex e74f9421..d0774945 100644--- a/docs-internal/temporal-workflow-control.md+++ b/docs-internal/temporal-workflow-control.md@@ -18,8 +18,9 @@ is authoritative for execution history.nodes.- **Reset** revision-guards the old generation, closes controller and localadmission, removes Temporal cron schedules, cancels local compatibility- resources, and then performs a final strict execution sweep before archiving- the old generation. It leaves the control state `ready`; the user must press+ resources, and then performs a final strict execution sweep and resets the+ workflow's [Workspace tasks](#workspace-tasks) before archiving the old+ generation. It leaves the control state `ready`; the user must press**Start** to create the next generation.Clients send an expected revision and idempotency key with every mutation.@@ -132,6 +133,35 @@ prefer termination over durable resumption. The active-state guard reads theshared `WORKFLOW_CONTROL_ACTIVE_STATES` set (which includes `resetting`), so aboot mid-reset can never sweep the namespace.+## Workspace tasks++Phone tasks submitted directly from the Workspace run outside the control+generation, under a per-workflow `WorkspaceTaskControllerWorkflow` (see+[TEMPORAL_ARCHITECTURE.md → Direct Workspace tasks](./TEMPORAL_ARCHITECTURE.md#direct-workspace-tasks)).+They can exist before the first Start and after a Reset. Pause and Resume do+not gate them; of the control states, only a Reset in progress blocks new+submissions. The control plane covers them in two places.++- **Reset.** With an active generation, Reset resets the Workspace tasks after+ its final execution sweep and before archiving+ (`services/deployment/handlers.py::_reset_workspace_tasks`), and+ `workflow_runtime_reset` carries `cancelled_workspace_tasks`. With no+ generation, or after the latest one was reset, Reset still checks the+ expected revision and then resets only the Workspace tasks. It returns+ idempotently when the graph has no Workspace-task node and no controller+ exists; otherwise it broadcasts `workflow_runtime_reset` and+ `workflow_control_status`.+- **Status.** Every control status read, mutation response and+ `workflow_control_status` broadcast passes through `_with_runtime_counts`,+ which merges `_workspace_runtime_status`. When the saved graph contains a+ Workspace-task node, `can_reset` is true in every state and the controller's+ active tasks are added to `active_count` and `in_flight_count`. Once the+ controller exists, the payload also carries `workspace_epoch`,+ `workspace_resetting` and `workspace_reset_request_id`; `workspace_available`+ is false when Temporal or the controller query is unavailable. While the+ controller is resetting, the payload reports `state: "resetting"` and turns+ off `can_start`, `can_pause`, `can_resume` and `can_edit`.+## Months-long generationsA controlled generation is expected to run — or stay paused — for months, so