Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
docs: option and retry-policy changes need no patch (RFC-0006 1b.45)
trohitg24c2907f9a46
diff --git a/CLAUDE.md b/CLAUDE.mdindex 12edf38f..bbed78e5 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -756,7 +756,7 @@ Workflows execute via Temporal for durability and horizontal scaling, gated by `Execution routing falls back Temporal → sequential; the Temporal dev server + embedded worker are managed in-process under `server/services/temporal/` (pooch-downloaded CLI, single `temporal server start-dev` process, gRPC `TEMPORAL_FRONTEND_GRPC_PORT` + Web UI `TEMPORAL_UI_PORT`, SQLite at `~/.opencompany/temporal.db`). Full architecture, dispatch matrix, per-node + agent-loop lifecycle, the F4.B `agent.*` activities (unsuffixed since the AgentWorkflow fold; read the live set from `collect_agent_activities()` rather than a hardcoded count), heartbeat semantics, delegation input contract, the worker tuning recipe, and all `.env` tunables live in [docs-internal/TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md).-**Months-long durability contract (July 2026).** Running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. Load-bearing invariants: (1) `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` stays `false` (debug-only sweep; even enabled, any control row in the shared `WORKFLOW_CONTROL_ACTIVE_STATES` — including `resetting` — vetoes it). (2) New child workflow starts carry **no** `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so lifetime caps silently terminated paused runs; liveness comes from activity heartbeats (30s beat / 2min timeout), never from workflow lifetime caps. Removing/adding start options or retry policies on recorded commands requires a `workflow.patched(...)` guard (list the live markers with `grep -rn "workflow.patched(" server/services/temporal`; [TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md) says what each one gates). (3) Every long-lived workflow loop must continue-as-new under history pressure (`is_continue_as_new_suggested()` + a soft cap) — Temporal hard-terminates around ~51,200 history events; `WorkflowControlWorkflow` carries triggers/queued events/seen-ids/control state across rollovers, so controller handles are addressed **by workflow id only, never run_id-pinned**. (4) `dispatch.emit` skips controllers whose `ControlEventTypes` Search Attribute doesn't match the event (absent attribute = legacy match-all). (5) The Temporal lifecycle is owned by [services/temporal/lifecycle.py](./server/services/temporal/lifecycle.py) (`main.py` only schedules it): dev-server supervision + resident watchdog (`TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS`), worker crash-restarts that REBUILD the single-use SDK `Worker` per attempt (manager + every pool queue worker), and the boot-time `reconcile_active_controls_on_boot` pass (converges crash-stranded `starting` rows, re-arms running/paused generations from the persisted graph snapshot). (6) **Recovery policies** (env-driven, defaults in `.env.template`; full semantics in [temporal-workflow-control.md → Recovery policies](./docs-internal/temporal-workflow-control.md)): `WORKFLOW_CONTROL_CRASH_RECOVERY=pause` — after a kill/crash (dirty-bit marker cleared only by graceful teardown) boot recovers running generations as **paused** so the user consciously resumes; `WORKFLOW_CONTROL_MISSING_CONTROLLER=pause` — a killed/vanished controller converges the generation to paused and **Resume rebuilds the controller** (`WorkflowIdConflictPolicy.USE_EXISTING`, re-arm from snapshot) instead of forcing a state-archiving Reset; `WORKFLOW_CONTROL_PAUSE_ON_FAILURE=true` — repeatedly-failing trigger-spawned runs pause their deployment via the `workflow_control.pause_on_failure` activity (circuit breaker; trips only after `WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD` (default 3) failures inside a rolling `_WINDOW_SECONDS` (default 600) so one node hiccup never pauses anything; Resume resets the streak; manual canvas runs never qualify). Temporal's native Pause/Unpause (server 1.28+) is deliberately NOT used — no Python SDK surface, and it halts workflow-task dispatch so a natively-paused controller couldn't process the resume Update. (7) **Workflow-scoped event delivery**: an envelope whose `workflow_id` is set by its producer's call site reaches only that workflow's consumers (`dispatch.emit` narrows both Visibility clauses by `EventWorkflowId`); chat is the first scoped producer and delegated-task completion (`nodes/agent/_events.py`) the second, whose broadcast frame carries only the task's identity — the decision lives in `routers/websocket.py`, the narrowing in core dispatch, and plugin `_events.py` factories only plumb the field (see [event_framework.md](./docs-internal/event_framework.md)). (8) **Canvas editability is the server-owned `can_edit` capability** emitted by `serialize_control` (paused = editable; starting/running/transitional = locked); the FE renders it via `lib/canvasLock.ts` + Dashboard's `guardCanvasEdit` and never re-derives the rule from state strings (see [temporal-workflow-control.md → Canvas editability](./docs-internal/temporal-workflow-control.md)). Locked by `tests/temporal/test_durability_hardening.py` + `test_worker_restart.py`.+**Months-long durability contract (July 2026).** Running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. Load-bearing invariants: (1) `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` stays `false` (debug-only sweep; even enabled, any control row in the shared `WORKFLOW_CONTROL_ACTIVE_STATES` — including `resetting` — vetoes it). (2) New child workflow starts carry **no** `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so lifetime caps silently terminated paused runs; liveness comes from activity heartbeats (30s beat / 2min timeout), never from workflow lifetime caps. Changing an activity's or child's timeouts, retry policy, task queue or parent close policy needs no patch: on replay Temporal compares only a command's kind, its activity or child type, and its id (RFC-0006 §5 rule 6). A cancellation type is the exception when a recorded history can contain a cancel of that activity or child, and any change to which commands run needs a `workflow.patched(...)` guard (list the live markers with `grep -rn "workflow.patched(" server/services/temporal`; [TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md) says what each one gates). (3) Every long-lived workflow loop must continue-as-new under history pressure (`is_continue_as_new_suggested()` + a soft cap) — Temporal hard-terminates around ~51,200 history events; `WorkflowControlWorkflow` carries triggers/queued events/seen-ids/control state across rollovers, so controller handles are addressed **by workflow id only, never run_id-pinned**. (4) `dispatch.emit` skips controllers whose `ControlEventTypes` Search Attribute doesn't match the event (absent attribute = legacy match-all). (5) The Temporal lifecycle is owned by [services/temporal/lifecycle.py](./server/services/temporal/lifecycle.py) (`main.py` only schedules it): dev-server supervision + resident watchdog (`TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS`), worker crash-restarts that REBUILD the single-use SDK `Worker` per attempt (manager + every pool queue worker), and the boot-time `reconcile_active_controls_on_boot` pass (converges crash-stranded `starting` rows, re-arms running/paused generations from the persisted graph snapshot). (6) **Recovery policies** (env-driven, defaults in `.env.template`; full semantics in [temporal-workflow-control.md → Recovery policies](./docs-internal/temporal-workflow-control.md)): `WORKFLOW_CONTROL_CRASH_RECOVERY=pause` — after a kill/crash (dirty-bit marker cleared only by graceful teardown) boot recovers running generations as **paused** so the user consciously resumes; `WORKFLOW_CONTROL_MISSING_CONTROLLER=pause` — a killed/vanished controller converges the generation to paused and **Resume rebuilds the controller** (`WorkflowIdConflictPolicy.USE_EXISTING`, re-arm from snapshot) instead of forcing a state-archiving Reset; `WORKFLOW_CONTROL_PAUSE_ON_FAILURE=true` — repeatedly-failing trigger-spawned runs pause their deployment via the `workflow_control.pause_on_failure` activity (circuit breaker; trips only after `WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD` (default 3) failures inside a rolling `_WINDOW_SECONDS` (default 600) so one node hiccup never pauses anything; Resume resets the streak; manual canvas runs never qualify). Temporal's native Pause/Unpause (server 1.28+) is deliberately NOT used — no Python SDK surface, and it halts workflow-task dispatch so a natively-paused controller couldn't process the resume Update. (7) **Workflow-scoped event delivery**: an envelope whose `workflow_id` is set by its producer's call site reaches only that workflow's consumers (`dispatch.emit` narrows both Visibility clauses by `EventWorkflowId`); chat is the first scoped producer and delegated-task completion (`nodes/agent/_events.py`) the second, whose broadcast frame carries only the task's identity — the decision lives in `routers/websocket.py`, the narrowing in core dispatch, and plugin `_events.py` factories only plumb the field (see [event_framework.md](./docs-internal/event_framework.md)). (8) **Canvas editability is the server-owned `can_edit` capability** emitted by `serialize_control` (paused = editable; starting/running/transitional = locked); the FE renders it via `lib/canvasLock.ts` + Dashboard's `guardCanvasEdit` and never re-derives the rule from state strings (see [temporal-workflow-control.md → Canvas editability](./docs-internal/temporal-workflow-control.md)). Locked by `tests/temporal/test_durability_hardening.py` + `test_worker_restart.py`.**Temporal tracing ownership is strict.** Register the SDK `TracingInterceptor` exactly once on the shared `Client.connect(...)`; the Python SDK automatically prepends compatible client interceptors to every worker. Never repeat `TracingInterceptor` in `Worker(..., interceptors=...)`. Worker lists contain the distinct `ObservabilityWorkerInterceptor` and plugin-specific worker interceptors only. Exact paired `StartActivity` / `RunActivity` / `CompleteWorkflow` console spans with identical Temporal identities indicate duplicate instrumentation, not duplicate execution; use Temporal Event History and the `node.<type>.execute` application span to verify actual attempts. See the tracing and debugging sections in [TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md) and [errors.md](./docs-internal/errors.md).