Skip to content

Self-improving AI Employees for everyone

docs: record the months-long durability contract

trohitg14781b843826
diff --git a/CLAUDE.md b/CLAUDE.md
index a2218648..b5dfb804 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -739,7 +739,9 @@ Node groups (palette categories): agent, model, skill, tool, trigger, workflow,
### Temporal Distributed Execution
-Workflows execute via Temporal for durability and horizontal scaling, gated by `TEMPORAL_ENABLED`. Three dispatch paths exist — legacy single `execute_node_activity` (fallback), per-type `node.{type}.v{version}` activities (F4.A, `TEMPORAL_PER_TYPE_DISPATCH=true`), and Agent-as-child-workflow (F4.B, `TEMPORAL_AGENT_WORKFLOW_ENABLED=true`) — with `rlm_agent`/`claude_code_agent` always bypassing AgentWorkflow. **Per-queue activity routing is production-default (Wave 16)**: a `TemporalWorkerPool` runs one activity-only worker per plugin-declared `cls.task_queue` (ai-heavy / browser / code-exec / rest-api / messaging / ...) with per-queue concurrency + rate limits + resource-based tuning for ai-heavy/browser; rollback via `TEMPORAL_WORKER_POOL_ENABLED=false`. Resilience knobs (Waves 17-18): `DEPLOYMENT_MODE` (local halves concurrency; sizes the sticky cache), one-shot retry on `agent.execute_llm_step.v1` (LLM calls are not idempotent), observability interceptors (`activity_retry` WARN on re-dispatch), 30s periodic heartbeat during long activity bodies, cron `catchup_window=24h`, poller autoscaling. Execution routing falls back Temporal → sequential; the Temporal dev server + embedded worker are managed in-process under `server/services/temporal/` (pooch-downloaded CLI, single `temporal server start-dev` process, gRPC `TEMPORAL_FRONTEND_GRPC_PORT` + Web UI `TEMPORAL_UI_PORT`, SQLite at `~/.opencompany/temporal.db`). Full architecture, dispatch matrix, per-node + agent-loop lifecycle, all 16 F4.B `agent.*.v1` activities, heartbeat semantics, delegation input contract, resumption toggle (`TEMPORAL_TERMINATE_RUNNING_ON_STARTUP`), the worker tuning recipe, and all `.env` tunables live in [docs-internal/TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md).
+Workflows execute via Temporal for durability and horizontal scaling, gated by `TEMPORAL_ENABLED`. Three dispatch paths exist — legacy single `execute_node_activity` (fallback), per-type `node.{type}.v{version}` activities (F4.A, `TEMPORAL_PER_TYPE_DISPATCH=true`), and Agent-as-child-workflow (F4.B, `TEMPORAL_AGENT_WORKFLOW_ENABLED=true`) — with `rlm_agent`/`claude_code_agent` always bypassing AgentWorkflow. **Per-queue activity routing is production-default (Wave 16)**: a `TemporalWorkerPool` runs one activity-only worker per plugin-declared `cls.task_queue` (ai-heavy / browser / code-exec / rest-api / messaging / ...) with per-queue concurrency + rate limits + resource-based tuning for ai-heavy/browser; rollback via `TEMPORAL_WORKER_POOL_ENABLED=false`. Resilience knobs (Waves 17-18): `DEPLOYMENT_MODE` (local halves concurrency; sizes the sticky cache), one-shot retry on `agent.execute_llm_step.v1` (LLM calls are not idempotent), observability interceptors (`activity_retry` WARN on re-dispatch), 30s periodic heartbeat during long activity bodies, cron `catchup_window=24h`, poller autoscaling. Execution routing falls back Temporal → sequential; the Temporal dev server + embedded worker are managed in-process under `server/services/temporal/` (pooch-downloaded CLI, single `temporal server start-dev` process, gRPC `TEMPORAL_FRONTEND_GRPC_PORT` + Web UI `TEMPORAL_UI_PORT`, SQLite at `~/.opencompany/temporal.db`). Full architecture, dispatch matrix, per-node + agent-loop lifecycle, all 16 F4.B `agent.*.v1` activities, heartbeat semantics, delegation input contract, the worker tuning recipe, and all `.env` tunables live in [docs-internal/TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md).
+
+**Months-long durability contract (July 2026).** Running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. Load-bearing invariants: (1) `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` stays `false` (debug-only sweep; even enabled, any control row in the shared `WORKFLOW_CONTROL_ACTIVE_STATES` — including `resetting` — vetoes it). (2) New child workflow starts carry **no** `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so lifetime caps silently terminated paused runs; liveness comes from activity heartbeats (30s beat / 2min timeout), never from workflow lifetime caps. Removing/adding start options or retry policies on recorded commands requires a `workflow.patched(...)` guard (see the `*-unbounded-*` / `*-history-bounded-can-*` markers). (3) Every long-lived workflow loop must continue-as-new under history pressure (`is_continue_as_new_suggested()` + a soft cap) — Temporal hard-terminates around ~51,200 history events; `WorkflowControlWorkflow` carries triggers/queued events/seen-ids/control state across rollovers, so controller handles are addressed **by workflow id only, never run_id-pinned**. (4) `dispatch.emit` skips controllers whose `ControlEventTypes` Search Attribute doesn't match the event (absent attribute = legacy match-all). (5) The Temporal lifecycle is owned by [services/temporal/lifecycle.py](./server/services/temporal/lifecycle.py) (`main.py` only schedules it): dev-server supervision + resident watchdog (`TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS`), worker crash-restarts that REBUILD the single-use SDK `Worker` per attempt (manager + every pool queue worker), and the boot-time `reconcile_active_controls_on_boot` pass (converges crash-stranded `starting` rows, re-arms running/paused generations from the persisted graph snapshot). Locked by `tests/temporal/test_durability_hardening.py` + `test_worker_restart.py`.
**Temporal tracing ownership is strict.** Register the SDK `TracingInterceptor` exactly once on the shared `Client.connect(...)`; the Python SDK automatically prepends compatible client interceptors to every worker. Never repeat `TracingInterceptor` in `Worker(..., interceptors=...)`. Worker lists contain the distinct `ObservabilityWorkerInterceptor` and plugin-specific worker interceptors only. Exact paired `StartActivity` / `RunActivity` / `CompleteWorkflow` console spans with identical Temporal identities indicate duplicate instrumentation, not duplicate execution; use Temporal Event History and the `node.<type>.execute` application span to verify actual attempts. See the tracing and debugging sections in [TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md) and [errors.md](./docs-internal/errors.md).
diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.md
index 7eee37f7..910332ff 100644
--- a/docs-internal/TEMPORAL_ARCHITECTURE.md
+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md
@@ -583,11 +583,20 @@ The Temporal binary + persistence are managed in-process by the plugin-folder pa
- **[`pooch`](https://pypi.org/project/pooch/)** — `services/temporal/_install.py` downloads the official `temporal` CLI archive from `https://temporal.download/cli/archive/latest?platform=<os>&arch=<arch>`. Cross-platform (Windows zip / macOS+Linux tar.gz), cached at `<DATA_DIR>/packages/temporal/` via `_cache_dir() = core.paths.package_dir("temporal")` (pooch's `path=` argument). The retrieve call passes an explicit `downloader=pooch.HTTPDownloader(timeout=300, progressbar=True)`: the timeout is per-socket-read (not total transfer), so arbitrarily slow links can finish the ~114 MB fetch — pooch's 30 s default aborted them — and `progressbar` must live on the downloader because `retrieve()` ignores its own kwarg when `downloader=` is explicit. Failed downloads never poison the cache (pooch writes to a temp file and renames atomically on success). Standalone entry: `python -m services.temporal._install` — invoked **fatally** by `company build` step [6/6] (so first `company start` doesn't pay the download; contract locked by `test_temporal_install_is_fatal_on_failure`) and **non-fatally** by npm postinstall (`scripts/install.js` try/catch — `TemporalServerRuntime._pre_spawn()` re-downloads lazily on first `company start`, so a failed eager fetch never fails `npm install -g`). The binary cache survives `company clean` (`packages` is in `_OPENCOMPANY_KEEP`, `cli/commands/clean.py`). Pre-fix this used `pooch.os_cache("opencompany-temporal")` (`~/.cache/OpenCompany/opencompany-temporal/` etc.) — a separate OS-cache namespace operators reported as "not local"; it now sits under DATA_DIR alongside the Stripe binary and the shared npm tree.
- **`BaseProcessSupervisor` + `BaseSupervisor`** (`server/services/_supervisor/`) — the in-house supervisor base classes that `server/nodes/whatsapp/_runtime.py` also uses. Provides cross-platform signal handling (POSIX `setsid` + Windows Job Objects + `CREATE_NEW_PROCESS_GROUP` for graceful `CTRL_BREAK_EVENT` shutdown), restart policy via tenacity, log draining, status snapshots. We subclass both — zero custom supervisor logic.
-**Lifecycle wiring (July 2026)**: backend-owned. `main.py`'s `_init_temporal_background` calls `TemporalServerRuntime.ensure_started()` before each connect attempt — a TCP probe on the gRPC port skips the spawn when a server (previously spawned or externally managed) is already listening, and non-loopback `TEMPORAL_SERVER_ADDRESS` values are never spawned at. Teardown rides the existing `shutdown_all_supervisors()` lifespan hook. The CLI no longer supervises Temporal: `cli/commands/_temporal_specs.py` and the `_supervised_runtime.py` shim were deleted (the shim was a full second server-venv Python process, ~81 MB resident, plus a ~28 MB resident `uv run` parent — pure supervision overhead).
+**Lifecycle wiring (July 2026)**: backend-owned, and modular — the whole Temporal runtime story lives in [`services/temporal/lifecycle.py`](../server/services/temporal/lifecycle.py) (`run_temporal_lifecycle`); `main.py` only schedules it as one background task. The module owns: (1) dev-server supervision via `TemporalServerRuntime.ensure_started()` before each connect attempt — a TCP probe on the gRPC port skips the spawn when a server (previously spawned or externally managed) is already listening, and non-loopback `TEMPORAL_SERVER_ADDRESS` values are never spawned at; (2) the forever-retrying connect loop; (3) the config-gated startup sweep; (4) executor + worker manager + worker pool wiring; (5) the boot-time workflow-control reconcile (`services.deployment.handlers.reconcile_active_controls_on_boot`); and (6) a **resident dev-server watchdog** — after startup the task stays alive, probing every `TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS` and respawning a dev-server child that died (or restarting a supervisor-owned child that wedged: port bound but gRPC not SERVING for 4 consecutive probes). The runtime registers with `services._supervisor.register_supervisor` so `shutdown_all_supervisors()` actually stops it at teardown. The CLI no longer supervises Temporal: `cli/commands/_temporal_specs.py` and the `_supervised_runtime.py` shim were deleted (the shim was a full second server-venv Python process, ~81 MB resident, plus a ~28 MB resident `uv run` parent — pure supervision overhead).
+
+**Worker crash-restart (months-long durability)**: the SDK `Worker` is single-use — a second `run()` on the same instance raises `RuntimeError("Already started")`. Both restart loops therefore REBUILD a fresh worker per attempt: `TemporalWorkerManager._run_worker` via `_build_worker()`, and every `TemporalWorkerPool` queue worker runs under `_run_queue_worker` (previously a bare `worker.run()` task — one crash silently killed the queue's only worker and its activities pended forever). Backoff knobs: `TEMPORAL_WORKER_RESTART_BACKOFF_SECONDS` / `_MAX_SECONDS`. Locked by `tests/temporal/test_worker_restart.py`.
**WS surface**: `_handlers.py` registers `temporal_status` / `temporal_start` / `temporal_stop` via `services.ws_handler_registry.register_ws_handlers`. `_refresh.py` registers a WS-connect callback via `services.status_broadcaster.register_service_refresh` so the FE health indicator stays current.
-**Resumption toggle**: [`TemporalClientWrapper.terminate_running_workflows`](../server/services/temporal/client.py) — runs once at server lifespan startup, between client connect and worker start, gated on `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` (default `true`). Queries Visibility for every `Running` workflow and calls `handle.terminate(reason="OpenCompany startup: auto-resumption disabled")`. **History is preserved** (UI shows workflows as `Terminated`, not deleted); only active execution stops. Disables resumption while `DeploymentManager` has no boot-time reconcile against Temporal Visibility yet — resumed workflows would otherwise keep executing but be invisible to the OpenCompany UI. Flip to `false` once the reconcile lands.
+**Months-long durability contract**: running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. Mechanisms, each replay-patch-guarded where it changes recorded commands:
+
+- **No lifetime caps on new child runs.** Trigger/cron-spawned `MachinaWorkflow` runs, agent children, and delegated-task runners previously carried 1-2h `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so any pause longer than the cap silently terminated the run (and a timed-out delegated runner skipped its compensation: leaked permit + stuck task row). New executions start children unbounded (patches: `*-unbounded-child-runs-v1`, `machina-unbounded-lifetimes-v1`, `agent-unbounded-lifetimes-v1`). Liveness is the activity layer's job: node activities heartbeat every 30s against a 2-minute `heartbeat_timeout`; their `start_to_close` is a generous 24h ceiling, not 10 minutes. The subagent-permit wait uses `PERMIT_WAIT_RETRY` (unlimited attempts) so a queued delegation waits as long as admission takes instead of failing after ~3h.
+- **History-pressure continue-as-new everywhere.** Temporal terminates any workflow around ~51,200 history events. `WorkflowControlWorkflow` (which multiplexes all of a deployment's triggers into one history) now rolls over on `is_continue_as_new_suggested()` / a 10K-event soft cap, carrying trigger specs, per-trigger provider `seen_ids` (written back into the spec after every poll cycle), queued push events, the bounded dedup baseline, and the control state — a rollover works mid-pause too, since a paused controller still accretes signal history. `TriggerListenerWorkflow`/`PollingTriggerWorkflow` gained the same pressure check (the old `_processed_count >= 16_000` gate was unreachable: real spawns cost ~15-25 events each, and polling counted only emitted events while a quiet mailbox burned ~11 events/cycle — dead in ~3 days at the 60s default). Poll intervals are clamped to a 30s floor on the patched path. Because run ids change on rollover, **controller handles are addressed by workflow id only, never run_id-pinned** (`_controller_handle`, manager `register_trigger`).
+- **dispatch.emit controller narrowing.** Controllers advertise their push event types via the `ControlEventTypes` keyword-list Search Attribute (upserted as triggers register); `dispatch.emit` skips controllers with no matching trigger instead of signalling every running controller with every platform event (each unmatched signal was ~4 immutable history events — one busy deployment burned every other controller's rollover budget). Controllers without the attribute (pre-upgrade histories) keep match-all behaviour.
+- **Boot-time reconcile** (`reconcile_active_controls_on_boot`, called from the lifecycle module after workers start): runs the lazy `_reconcile_control` over every active control row, converges `starting` rows a crash left behind (controller alive with triggers registered → `running`; alive-but-empty for a graph that declares triggers → `failed` + controller closed; vanished → `failed`), and re-arms the process-local half of running/paused generations from the persisted graph snapshot — DeploymentManager state, in-process collectors for non-canary trigger types, cron pause posture. Idempotent by construction (controller `register_trigger` keyed by listener id, legacy starts use `USE_EXISTING`, cron creation preserves server-owned pause state).
+
+**Startup sweep (debug-only escape hatch)**: [`TemporalClientWrapper.terminate_running_workflows`](../server/services/temporal/client.py) — gated on `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` (default **`false`**; keep it false — setting `true` converts every boot into a namespace-wide terminate sweep). Even when enabled, any control row in an active state (the shared `WORKFLOW_CONTROL_ACTIVE_STATES`, which includes `resetting`) vetoes the sweep. History is preserved (UI shows workflows as `Terminated`, not deleted); only active execution stops.
**Port management**: Temporal owns ports 5681 (gRPC) + 5680 (Web UI). Both bound by the same `temporal.exe` process; both listed in `cli.config.Config.all_ports` so `company stop`'s port-freeing pre-flight covers them.
@@ -607,11 +616,10 @@ The Temporal binary + persistence are managed in-process by the plugin-folder pa
| `temporal_ui_port` | `TEMPORAL_UI_PORT` | `5680` | Web UI port (`--ui-port`). CLI default is `--port + 1000`; pinned into the serial 5678-block instead. |
| `temporal_sqlite_path` | `TEMPORAL_SQLITE_PATH` | `temporal.db` | SQLite file (`--db-filename`). Resolved relative to `DATA_DIR` (= `~/.opencompany/`) unless absolute — flat under `~/.opencompany/` like `credentials.db` / `workflow.db`. |
| `temporal_graceful_shutdown_seconds` | `TEMPORAL_GRACEFUL_SHUTDOWN_SECONDS` | `30` | `CTRL_BREAK_EVENT` (Windows) / `SIGTERM` (POSIX) → tree-kill grace window. Shared with the embedded worker shutdown. |
-| `temporal_terminate_running_on_startup` | `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` | `true` | Resumption toggle (see above). |
+| `temporal_terminate_running_on_startup` | `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` | `false` | Debug-only startup sweep (see the durability contract above). Keep false so running and paused deployments survive restarts. |
+| `temporal_health_monitor_interval_seconds` | `TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS` | `15` | Resident dev-server watchdog probe cadence (lifecycle module; loopback deployments only). |
-| Env var | `.env.template` default | Purpose |
-|---|---|---|
-| `TEMPORAL_SERVER_READY_TIMEOUT_SECONDS` | `120` | Legacy knob from the CLI-supervised era (readiness wait for the gRPC port). Unused since the backend-owned cutover; retained in `.env.template` for config compatibility. |
+The legacy `TEMPORAL_SERVER_READY_TIMEOUT_SECONDS` knob (CLI-supervised-era readiness wait) was removed — it had no consumer since the backend-owned cutover.
One temporary agent-engine cutover variable is also read directly rather than
through `Settings`:
diff --git a/docs-internal/errors.md b/docs-internal/errors.md
index cc795e28..52199956 100644
--- a/docs-internal/errors.md
+++ b/docs-internal/errors.md
@@ -137,6 +137,44 @@ construction paths.
---
+### 2b. `GetTaskQueue operation failed ... context canceled` After Launch
+
+**Symptom**: some time after launch the aggregated log shows:
+```
+level=ERROR msg="Operation failed with internal error."
+error="GetTaskQueue operation failed. Failed to check if task queue /_sys/code-exec/2 of type Activity existed. Error: context canceled"
+error-type=serviceerror.Unavailable operation=GetTaskQueue
+```
+
+`/_sys/<queue>/<n>` is Temporal's internal partition naming — `code-exec` here
+is one of the worker pool's per-plugin task queues; any low-traffic queue and
+partition can appear.
+
+**Root cause**: the dev server logs ERROR when a poller's long-poll context is
+canceled mid-persistence-read — routine events with this topology (~10 workers
+long-polling a single-writer SQLite dev server): poller autoscaling settling
+after boot, a worker restart, backend shutdown, laptop sleep/wake breaking gRPC
+channels, or the server's own idle background jobs. Staff-confirmed benign
+upstream ([community.temporal.io/t/18635](https://community.temporal.io/t/temporal-server-errors-with-no-app-running/18635),
+[temporalio/cli#633](https://github.com/temporalio/cli/issues/633)): a canceled
+poll never loses a task — tasks are persisted, the SDK re-polls immediately,
+and the server redelivers.
+
+**Status**: benign noise. `TemporalServerRuntime.stderr_log`
+([`services/temporal/_runtime.py`](../server/services/temporal/_runtime.py))
+downgrades lines matching both `context canceled` and the internal-error /
+`GetTaskQueue` markers to DEBUG; genuine dev-server errors keep surfacing at
+ERROR through the supervisor pipe.
+
+**When it is NOT benign**: if activities on that queue actually hang, check
+whether the queue's pool worker died (Temporal Web UI → task queue pollers).
+Pool workers self-restart with a fresh Worker instance per attempt
+(`TemporalWorkerPool._run_queue_worker`); a queue with zero pollers after
+repeated restarts indicates a real crash loop — read the
+`[Pool] Worker for queue ... crashed` log lines for the underlying error.
+
+---
+
## 3. Temporal Activity `CancelledError` on Long-Running Nodes
**Symptom**: Nodes that run for more than ~2 minutes (Coding Agent, browser automation, AI multi-tool loops) fail with:
diff --git a/docs-internal/temporal-workflow-control.md b/docs-internal/temporal-workflow-control.md
index 618f1a2f..75e5d83c 100644
--- a/docs-internal/temporal-workflow-control.md
+++ b/docs-internal/temporal-workflow-control.md
@@ -128,7 +128,44 @@ their external stores. Deployment control never switches on node type.
On server restart, active controlled deployments are excluded from the legacy
startup termination sweep. `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` defaults to
`false`; enabling it is intended only for legacy installations that explicitly
-prefer termination over durable resumption.
+prefer termination over durable resumption. The active-state guard reads the
+shared `WORKFLOW_CONTROL_ACTIVE_STATES` set (which includes `resetting`), so a
+boot mid-reset can never sweep the namespace.
+
+## Months-long generations
+
+A controlled generation is expected to run — or stay paused — for months, so
+the control plane is hardened against Temporal's per-run event-history ceiling
+(~51,200 events) and against backend restarts:
+
+- **Controller continue-as-new.** `WorkflowControlWorkflow` rolls its run over
+ under history pressure (`is_continue_as_new_suggested()` or a 10K-event soft
+ cap), carrying trigger specs, per-trigger provider `seen_ids` (written back
+ into the carried spec after every poll cycle), queued push events, the
+ bounded dedup baseline, and the control state + revision. Rollover works
+ mid-pause; the paused state carries. Because run ids change on rollover,
+ every control surface addresses the controller **by workflow id only** —
+ `controller_run_id` is stored for provenance but never pinned on a handle.
+- **Signal narrowing.** The controller upserts the `ControlEventTypes`
+ keyword-list Search Attribute as push triggers register; `dispatch.emit`
+ skips controllers whose deployment has no matching trigger so other
+ deployments' traffic cannot burn a controller's rollover budget.
+- **Boot-time reconcile** (`reconcile_active_controls_on_boot`, invoked by the
+ Temporal lifecycle task after workers start): runs the lazy reconcile over
+ every active row, converges crash-stranded `starting` rows (controller alive
+ with registered triggers, or a triggerless graph → `running`; alive-but-empty
+ while the graph declares triggers → `failed` with the orphan controller
+ closed), and re-arms the process-local half of running/paused generations
+ from the persisted graph snapshot — DeploymentManager runtime state,
+ in-process collectors for non-canary trigger types, and the paused posture
+ (admission, trigger pause flags, cron schedule pause). Re-arm is idempotent:
+ controller `register_trigger` is keyed by listener id, legacy listener
+ starts use `USE_EXISTING`, and cron schedule creation preserves server-owned
+ pause state.
+- **No lifetime caps.** Spawned graph runs, agent children, and delegated-task
+ runners no longer carry 1-2h execution/run timeouts (Temporal's timers keep
+ ticking through a pause, so the caps silently terminated paused work).
+ Replay-patched; see TEMPORAL_ARCHITECTURE.md for the patch inventory.
## Delegated-task traces