Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
feat(control): configurable recovery policies - crash pause, controller rebuild, failure circuit breaker
trohitg5dae0cec839a
diff --git a/.env.template b/.env.templateindex b02863e8..9f911eae 100644--- a/.env.template+++ b/.env.template@@ -121,6 +121,24 @@ TEMPORAL_GRACEFUL_SHUTDOWN_SECONDS=30# (backend-owned) servers only; remote clusters are never touched.TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS=15+# --- Workflow-control recovery policies ---+# Posture for deployments still ``running`` after an UNCLEAN shutdown+# (kill / crash, detected via a dirty-bit marker the graceful lifespan+# teardown clears). ``pause`` recovers them paused so the user+# consciously resumes; ``resume`` restores them running. A clean+# ``company stop`` + start always restores deployments as they were.+WORKFLOW_CONTROL_CRASH_RECOVERY=pause+# Convergence for a live generation whose controller execution vanished+# (terminated in the Temporal UI, killed, retention-deleted). ``pause``+# keeps it user-resumable — Resume rebuilds the controller from the+# durable row + graph snapshot; ``fail`` preserves the legacy behaviour+# where only Reset (which archives conversation state) recovers.+WORKFLOW_CONTROL_MISSING_CONTROLLER=pause+# Circuit breaker: when a controlled run fails with an error, pause the+# deployment (fix + Resume) instead of letting the trigger keep firing+# into the same error indefinitely. ``false`` disables the auto-pause.+WORKFLOW_CONTROL_PAUSE_ON_FAILURE=true+# --- Startup resilience (server-ready-before-workers) ---# After the gRPC port binds, the client polls the WorkflowService gRPC# health check until it reports SERVING before starting the worker /diff --git a/CLAUDE.md b/CLAUDE.mdindex ca1518b7..ac6451d5 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -742,7 +742,7 @@ Node groups (palette categories): agent, model, skill, tool, trigger, workflow,Workflows execute via Temporal for durability and horizontal scaling, gated by `TEMPORAL_ENABLED`. Three dispatch paths exist — legacy single `execute_node_activity` (fallback), per-type `node.{type}.v{version}` activities (F4.A, `TEMPORAL_PER_TYPE_DISPATCH=true`), and Agent-as-child-workflow (F4.B, `TEMPORAL_AGENT_WORKFLOW_ENABLED=true`) — with `rlm_agent`/`claude_code_agent` always bypassing AgentWorkflow. **Per-queue activity routing is production-default (Wave 16)**: a `TemporalWorkerPool` runs one activity-only worker per plugin-declared `cls.task_queue` (ai-heavy / browser / code-exec / rest-api / messaging / ...) with per-queue concurrency + rate limits + resource-based tuning for ai-heavy/browser; rollback via `TEMPORAL_WORKER_POOL_ENABLED=false`. Resilience knobs (Waves 17-18): `DEPLOYMENT_MODE` (local halves concurrency; sizes the sticky cache), one-shot retry on `agent.execute_llm_step.v1` (LLM calls are not idempotent), observability interceptors (`activity_retry` WARN on re-dispatch), 30s periodic heartbeat during long activity bodies, cron `catchup_window=24h`, poller autoscaling. Execution routing falls back Temporal → sequential; the Temporal dev server + embedded worker are managed in-process under `server/services/temporal/` (pooch-downloaded CLI, single `temporal server start-dev` process, gRPC `TEMPORAL_FRONTEND_GRPC_PORT` + Web UI `TEMPORAL_UI_PORT`, SQLite at `~/.opencompany/temporal.db`). Full architecture, dispatch matrix, per-node + agent-loop lifecycle, all 16 F4.B `agent.*.v1` activities, heartbeat semantics, delegation input contract, the worker tuning recipe, and all `.env` tunables live in [docs-internal/TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md).-**Months-long durability contract (July 2026).** Running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. Load-bearing invariants: (1) `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` stays `false` (debug-only sweep; even enabled, any control row in the shared `WORKFLOW_CONTROL_ACTIVE_STATES` — including `resetting` — vetoes it). (2) New child workflow starts carry **no** `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so lifetime caps silently terminated paused runs; liveness comes from activity heartbeats (30s beat / 2min timeout), never from workflow lifetime caps. Removing/adding start options or retry policies on recorded commands requires a `workflow.patched(...)` guard (see the `*-unbounded-*` / `*-history-bounded-can-*` markers). (3) Every long-lived workflow loop must continue-as-new under history pressure (`is_continue_as_new_suggested()` + a soft cap) — Temporal hard-terminates around ~51,200 history events; `WorkflowControlWorkflow` carries triggers/queued events/seen-ids/control state across rollovers, so controller handles are addressed **by workflow id only, never run_id-pinned**. (4) `dispatch.emit` skips controllers whose `ControlEventTypes` Search Attribute doesn't match the event (absent attribute = legacy match-all). (5) The Temporal lifecycle is owned by [services/temporal/lifecycle.py](./server/services/temporal/lifecycle.py) (`main.py` only schedules it): dev-server supervision + resident watchdog (`TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS`), worker crash-restarts that REBUILD the single-use SDK `Worker` per attempt (manager + every pool queue worker), and the boot-time `reconcile_active_controls_on_boot` pass (converges crash-stranded `starting` rows, re-arms running/paused generations from the persisted graph snapshot). Locked by `tests/temporal/test_durability_hardening.py` + `test_worker_restart.py`.+**Months-long durability contract (July 2026).** Running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. Load-bearing invariants: (1) `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` stays `false` (debug-only sweep; even enabled, any control row in the shared `WORKFLOW_CONTROL_ACTIVE_STATES` — including `resetting` — vetoes it). (2) New child workflow starts carry **no** `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so lifetime caps silently terminated paused runs; liveness comes from activity heartbeats (30s beat / 2min timeout), never from workflow lifetime caps. Removing/adding start options or retry policies on recorded commands requires a `workflow.patched(...)` guard (see the `*-unbounded-*` / `*-history-bounded-can-*` markers). (3) Every long-lived workflow loop must continue-as-new under history pressure (`is_continue_as_new_suggested()` + a soft cap) — Temporal hard-terminates around ~51,200 history events; `WorkflowControlWorkflow` carries triggers/queued events/seen-ids/control state across rollovers, so controller handles are addressed **by workflow id only, never run_id-pinned**. (4) `dispatch.emit` skips controllers whose `ControlEventTypes` Search Attribute doesn't match the event (absent attribute = legacy match-all). (5) The Temporal lifecycle is owned by [services/temporal/lifecycle.py](./server/services/temporal/lifecycle.py) (`main.py` only schedules it): dev-server supervision + resident watchdog (`TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS`), worker crash-restarts that REBUILD the single-use SDK `Worker` per attempt (manager + every pool queue worker), and the boot-time `reconcile_active_controls_on_boot` pass (converges crash-stranded `starting` rows, re-arms running/paused generations from the persisted graph snapshot). (6) **Recovery policies** (env-driven, defaults in `.env.template`; full semantics in [temporal-workflow-control.md → Recovery policies](./docs-internal/temporal-workflow-control.md)): `WORKFLOW_CONTROL_CRASH_RECOVERY=pause` — after a kill/crash (dirty-bit marker cleared only by graceful teardown) boot recovers running generations as **paused** so the user consciously resumes; `WORKFLOW_CONTROL_MISSING_CONTROLLER=pause` — a killed/vanished controller converges the generation to paused and **Resume rebuilds the controller** (`WorkflowIdConflictPolicy.USE_EXISTING`, re-arm from snapshot) instead of forcing a state-archiving Reset; `WORKFLOW_CONTROL_PAUSE_ON_FAILURE=true` — a failed trigger-spawned run pauses its deployment via the `workflow_control.pause_on_failure.v1` activity (circuit breaker; manual canvas runs never qualify). Temporal's native Pause/Unpause (server 1.28+) is deliberately NOT used — no Python SDK surface, and it halts workflow-task dispatch so a natively-paused controller couldn't process the resume Update. Locked by `tests/temporal/test_durability_hardening.py` + `test_worker_restart.py`.**Temporal tracing ownership is strict.** Register the SDK `TracingInterceptor` exactly once on the shared `Client.connect(...)`; the Python SDK automatically prepends compatible client interceptors to every worker. Never repeat `TracingInterceptor` in `Worker(..., interceptors=...)`. Worker lists contain the distinct `ObservabilityWorkerInterceptor` and plugin-specific worker interceptors only. Exact paired `StartActivity` / `RunActivity` / `CompleteWorkflow` console spans with identical Temporal identities indicate duplicate instrumentation, not duplicate execution; use Temporal Event History and the `node.<type>.execute` application span to verify actual attempts. See the tracing and debugging sections in [TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md) and [errors.md](./docs-internal/errors.md).diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.mdindex 910332ff..ec1eb506 100644--- a/docs-internal/TEMPORAL_ARCHITECTURE.md+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md@@ -618,6 +618,11 @@ The Temporal binary + persistence are managed in-process by the plugin-folder pa| `temporal_graceful_shutdown_seconds` | `TEMPORAL_GRACEFUL_SHUTDOWN_SECONDS` | `30` | `CTRL_BREAK_EVENT` (Windows) / `SIGTERM` (POSIX) → tree-kill grace window. Shared with the embedded worker shutdown. || `temporal_terminate_running_on_startup` | `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` | `false` | Debug-only startup sweep (see the durability contract above). Keep false so running and paused deployments survive restarts. || `temporal_health_monitor_interval_seconds` | `TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS` | `15` | Resident dev-server watchdog probe cadence (lifecycle module; loopback deployments only). |+| `workflow_control_crash_recovery` | `WORKFLOW_CONTROL_CRASH_RECOVERY` | `pause` | After an UNCLEAN shutdown (kill/crash, dirty-bit marker), boot pauses generations still `running` so the user consciously resumes; `resume` restores them running. Clean restarts always restore as-is. |+| `workflow_control_missing_controller` | `WORKFLOW_CONTROL_MISSING_CONTROLLER` | `pause` | A live generation whose controller vanished converges to `paused` (Resume rebuilds the controller); `fail` preserves the legacy Reset-only behaviour. |+| `workflow_control_pause_on_failure` | `WORKFLOW_CONTROL_PAUSE_ON_FAILURE` | `true` | Circuit breaker: a failed trigger-spawned run pauses its deployment (fix + Resume) instead of firing into the same error indefinitely. Evaluated activity-side; never touches recorded commands. |++Full recovery-policy semantics: [temporal-workflow-control.md → Recovery policies](./temporal-workflow-control.md#recovery-policies).The legacy `TEMPORAL_SERVER_READY_TIMEOUT_SECONDS` knob (CLI-supervised-era readiness wait) was removed — it had no consumer since the backend-owned cutover.diff --git a/docs-internal/temporal-workflow-control.md b/docs-internal/temporal-workflow-control.mdindex 75e5d83c..63af6b75 100644--- a/docs-internal/temporal-workflow-control.md+++ b/docs-internal/temporal-workflow-control.md@@ -167,6 +167,45 @@ the control plane is hardened against Temporal's per-run event-history ceilingticking through a pause, so the caps silently terminated paused work).Replay-patched; see TEMPORAL_ARCHITECTURE.md for the patch inventory.+## Recovery policies++Three env-driven policies (canonical defaults + semantics in+`.env.template`; consumed by `services/deployment/handlers.py`) govern how a+generation behaves around kills, crashes, and failures. All three reuse the+cooperative control-plane pause — Temporal's native Pause/Unpause (server+1.28+) is deliberately not used: it is an operational control with no Python+SDK client methods, and it halts workflow-task dispatch entirely, so a+natively-paused controller could not process the `set_control_state` Update+that Resume relies on.++- **`WORKFLOW_CONTROL_CRASH_RECOVERY`** (`pause` | `resume`, default+ `pause`): after an UNCLEAN shutdown — kill or crash, detected via a+ dirty-bit cache marker the graceful lifespan teardown clears (registered+ through the generic shutdown-hook registry) — the boot reconcile pauses+ every generation still `running` so the user consciously resumes it. A+ clean `company stop` + start always restores deployments as they were.+- **`WORKFLOW_CONTROL_MISSING_CONTROLLER`** (`pause` | `fail`, default+ `pause`): a live generation whose controller execution vanished+ (terminated in the Temporal UI, killed, retention-deleted) converges to+ `paused` instead of `failed`. Resume then **rebuilds the controller**:+ same generation-scoped workflow id started with the documented+ `WorkflowIdConflictPolicy.USE_EXISTING` semantics (race-safe — adopts a+ live controller or starts a fresh run after termination), persists the+ new run id, tears down stale local trigger state, and re-arms from the+ persisted graph snapshot before re-applying the running Update.+ `starting` rows always fail instead (nothing durable runs yet; Reset ++ Start rebuilds cleanly). `fail` preserves the legacy Reset-only+ behaviour.+- **`WORKFLOW_CONTROL_PAUSE_ON_FAILURE`** (default `true`): circuit+ breaker — when a trigger-spawned run fails with an error, MachinaWorkflow+ schedules `workflow_control.pause_on_failure.v1` (patch+ `machina-pause-on-failure-v1`) and the deployment pauses so the user+ fixes the cause and Resumes, instead of the trigger firing into the same+ error indefinitely. Only deployment-spawned runs qualify (they carry a+ `_pre_executed` firing trigger; a failed manual canvas run never pauses a+ live deployment), and the knob is evaluated on the activity side so+ flipping it never touches recorded workflow commands.+## Delegated-task tracesEach task attempt stores the actual parent, detached runner, and child Temporaldiff --git a/server/core/config.py b/server/core/config.pyindex 2c877cd8..732396db 100644--- a/server/core/config.py+++ b/server/core/config.py@@ -250,6 +250,32 @@ class Settings(BaseSettings):gt=0,)+ # Workflow-control recovery policies (canonical defaults + full+ # semantics in .env.template; Python defaults keep pre-feature .env+ # files booting). All three are consumed by+ # services/deployment/handlers.py — see the normalizers there.+ # "pause" | "resume": posture for generations still running after an+ # UNCLEAN shutdown (kill / crash). Clean restarts always restore as-is.+ workflow_control_crash_recovery: str = Field(+ default="pause",+ env="WORKFLOW_CONTROL_CRASH_RECOVERY",+ )+ # "pause" | "fail": convergence for a live generation whose controller+ # execution vanished (terminated in the Temporal UI, killed, deleted).+ # "pause" keeps it user-resumable — Resume rebuilds the controller from+ # the durable row; "fail" preserves the legacy Reset-only behaviour.+ workflow_control_missing_controller: str = Field(+ default="pause",+ env="WORKFLOW_CONTROL_MISSING_CONTROLLER",+ )+ # Circuit breaker: a controlled run failing with an error pauses the+ # deployment (user fixes + resumes) instead of the trigger firing into+ # the same error indefinitely.+ workflow_control_pause_on_failure: bool = Field(+ default=True,+ env="WORKFLOW_CONTROL_PAUSE_ON_FAILURE",+ )+# API Keys (all optional, injected at runtime)google_maps_api_key: Optional[str] = Field(default=None, env="GOOGLE_MAPS_API_KEY")openai_api_key: Optional[str] = Field(default=None, env="OPENAI_API_KEY")diff --git a/server/services/deployment/handlers.py b/server/services/deployment/handlers.pyindex eecf8f36..24b2b2b5 100644--- a/server/services/deployment/handlers.py+++ b/server/services/deployment/handlers.py@@ -420,16 +420,32 @@ def _control_service():return WorkflowControlService(container.database())-async def _start_controller(control) -> Optional[str]:- """Start the durable controller, or use local mode when Temporal is disabled."""+async def _start_controller(control, *, use_existing: bool = False) -> Optional[str]:+ """Start the durable controller, or use local mode when Temporal is disabled.++ ``use_existing=True`` is the rebuild path (Resume after the controller+ was killed): per the documented WorkflowIdConflictPolicy semantics,+ ``USE_EXISTING`` adopts a still-running controller (returning its run+ id) instead of erroring, while the default reuse policy permits a+ fresh run when the previous execution was Terminated/Failed — so the+ rebuild is race-safe against mis-detection.+ """from core.container import container- from temporalio.common import SearchAttributeKey, SearchAttributePair, TypedSearchAttributes+ from temporalio.common import (+ SearchAttributeKey,+ SearchAttributePair,+ TypedSearchAttributes,+ WorkflowIDConflictPolicy,+ )wrapper = container.temporal_client()if wrapper is None or wrapper.client is None:if container.settings().temporal_enabled:raise RuntimeError("temporal_control_unavailable")return None+ start_kwargs: Dict[str, Any] = {}+ if use_existing:+ start_kwargs["id_conflict_policy"] = WorkflowIDConflictPolicy.USE_EXISTINGhandle = await wrapper.client.start_workflow("WorkflowControlWorkflow",args=[{@@ -447,6 +463,7 @@ async def _start_controller(control) -> Optional[str]:SearchAttributeKey.for_keyword("EventWorkflowId"), control.workflow_id,)]),+ **start_kwargs,)return getattr(handle, "result_run_id", None) or getattr(handle, "first_execution_run_id", None)@@ -552,19 +569,58 @@ async def _query_controller_state(control) -> Optional[Dict[str, Any]]:_MISSING_CONTROLLER_FAILS = frozenset({"starting", "running", "pausing", "paused", "resuming"})+# The subset of live states that converge to ``paused`` (user-resumable)+# under WORKFLOW_CONTROL_MISSING_CONTROLLER=pause. ``starting`` always+# fails instead — nothing durable is running yet, and Reset + Start+# rebuilds a fresh generation cleanly.+_MISSING_CONTROLLER_PAUSES = frozenset({"running", "pausing", "paused", "resuming"})async def _fail_missing_controller(service: WorkflowControlService, control):- """Converge a generation whose controller has vanished to ``failed``.+ """Converge a generation whose controller execution has vanished.Without this the row keeps whatever live status it had, and because``begin_generation`` refuses to open a new generation unless the latest- one is ``reset``, the workflow could never be started again. ``failed`` is- both honest and recoverable -- Reset accepts it, and Reset is what returns- the workflow to ``ready``.+ one is ``reset``, the workflow could never be started again.++ Under the default ``WORKFLOW_CONTROL_MISSING_CONTROLLER=pause``, live+ states converge to ``paused``: a killed workflow (terminated in the+ Temporal UI, crashed server, retention-deleted execution) stays+ user-recoverable — Resume rebuilds the controller from the durable+ row + graph snapshot — whereas ``failed`` forces a Reset that+ archives conversation state. ``fail`` preserves the legacy+ Reset-only behaviour; ``starting`` rows always fail (nothing durable+ is running yet)."""if control.status not in _MISSING_CONTROLLER_FAILS:return control+ if (+ control.status in _MISSING_CONTROLLER_PAUSES+ and _missing_controller_policy() == "pause"+ ):+ if control.status == "paused":+ # Already user-resumable; Resume performs the rebuild.+ return control+ logger.warning(+ "Workflow controller execution is gone; pausing the generation so "+ "the user can resume it (Resume rebuilds the controller)",+ workflow_id=control.workflow_id,+ controller_workflow_id=control.controller_workflow_id,+ status=control.status,+ generation=control.generation,+ )+ try:+ recovered = await service.transition(+ control,+ expected_revision=control.revision,+ from_statuses={control.status},+ status="paused",+ )+ except ValueError:+ # Lost the CAS to a concurrent writer; their transition wins.+ return control+ await _broadcast_control(recovered, extra={"recovery": "controller_missing"})+ return recoveredlogger.warning("Workflow controller execution is gone; failing the generation so it ""can be reset",@@ -983,14 +1039,117 @@ async def _reconcile_control(service: WorkflowControlService, control):# ---------------------------------------------------------------------------+# Dirty-bit marker for crash detection. Boot stamps "running"; the graceful+# lifespan teardown stamps "clean" (registered as a shutdown hook at the+# bottom of this module). A boot that finds "running" already stamped knows+# the previous process was killed or crashed mid-flight.+_SHUTDOWN_STATE_CACHE_KEY = "workflow_control:shutdown_state"+++def _crash_recovery_policy() -> str:+ """Normalized WORKFLOW_CONTROL_CRASH_RECOVERY: ``pause`` | ``resume``."""+ from core.container import container++ value = str(+ getattr(container.settings(), "workflow_control_crash_recovery", "pause") or "pause"+ ).strip().lower()+ if value not in {"pause", "resume"}:+ logger.warning(f"Unknown WORKFLOW_CONTROL_CRASH_RECOVERY={value!r}; using 'pause'")+ return "pause"+ return value+++def _missing_controller_policy() -> str:+ """Normalized WORKFLOW_CONTROL_MISSING_CONTROLLER: ``pause`` | ``fail``."""+ from core.container import container++ value = str(+ getattr(container.settings(), "workflow_control_missing_controller", "pause") or "pause"+ ).strip().lower()+ if value not in {"pause", "fail"}:+ logger.warning(f"Unknown WORKFLOW_CONTROL_MISSING_CONTROLLER={value!r}; using 'pause'")+ return "pause"+ return value+++async def _consume_shutdown_state() -> bool:+ """True when the previous backend process exited cleanly.++ A missing marker (first boot / pre-feature database) counts as clean+ so a fresh install never boots into a recovery pause.+ """+ from core.container import container++ database = container.database()+ entry = await database.get_cache_entry(_SHUTDOWN_STATE_CACHE_KEY)+ previous = getattr(entry, "value", None) if entry is not None else None+ await database.set_cache_entry(_SHUTDOWN_STATE_CACHE_KEY, "running")+ return previous is None or previous == "clean"+++async def mark_clean_shutdown() -> None:+ """Stamp the clean-shutdown marker (lifespan teardown hook)."""+ from core.container import container++ try:+ await container.database().set_cache_entry(_SHUTDOWN_STATE_CACHE_KEY, "clean")+ except Exception as exc: # noqa: BLE001 — best-effort during teardown+ logger.warning(f"Failed to record clean shutdown marker: {exc}")+++async def _pause_for_recovery(service: WorkflowControlService, control, *, reason: str):+ """Durably pause a recovered generation so the user resumes it.++ After a kill or crash the deployment must come back ``paused`` rather+ than silently continuing — the user decides when it is safe to+ resume. Reuses the full pause ceremony (controller update, cron+ schedule pause, trigger flags, local admission) via+ :func:`handle_pause_workflow` so recovery lands in exactly the same+ posture a user-initiated pause produces. Temporal's native+ Pause/Unpause (server 1.28+) is deliberately NOT used here: it is an+ operational control without Python SDK client methods, and it halts+ workflow-task dispatch entirely — a natively-paused controller could+ not process the ``set_control_state`` Update that Resume relies on.+ """+ logger.warning(+ "Pausing recovered generation after unclean shutdown",+ workflow_id=control.workflow_id,+ generation=control.generation,+ reason=reason,+ )+ try:+ result = await handle_pause_workflow(+ {+ "workflow_id": control.workflow_id,+ "expected_revision": control.revision,+ "idempotency_key": f"recovery:{control.id}:{control.revision}",+ },+ None,+ )+ if not (isinstance(result, dict) and result.get("success")):+ logger.warning(+ "Recovery pause did not complete",+ workflow_id=control.workflow_id,+ error=(result or {}).get("error") if isinstance(result, dict) else None,+ )+ except Exception as exc: # noqa: BLE001 — lazy reconcile retries later+ logger.warning(+ "Recovery pause failed",+ workflow_id=control.workflow_id,+ error=str(exc),+ )+ latest = await service.database.get_latest_workflow_control(control.workflow_id)+ return latest if latest is not None else control++async def reconcile_active_controls_on_boot() -> int:"""Converge every active durable control after a backend restart.- Three responsibilities per active row:+ Responsibilities per active row:1. Run the same lazy :func:`_reconcile_control` used by status reads- (finishes interrupted pause/resume transitions, fails rows whose- controller vanished).+ (finishes interrupted pause/resume transitions, converges rows+ whose controller vanished).2. Converge ``starting`` rows — the one state the lazy path nevertouches while its controller exists, so a crash mid-start used towedge the workflow behind a permanent ``workflow_start_pending``.@@ -1002,6 +1161,11 @@ async def reconcile_active_controls_on_boot() -> int:state (runtime counts, admission, pause fan-in) and in-processcollectors for non-canary trigger types die with the process andpreviously stayed dead while the row reported ``running`` forever.+ 4. After an UNCLEAN shutdown (kill / crash — dirty-bit marker), pause+ every generation still ``running`` so the user consciously+ resumes it (``WORKFLOW_CONTROL_CRASH_RECOVERY=pause``, the+ default; ``resume`` restores them running). A clean+ ``company stop`` + start always restores deployments as they were.Returns the number of rows processed; per-row failures are logged andskipped so one broken generation cannot block the rest.@@ -1010,6 +1174,12 @@ async def reconcile_active_controls_on_boot() -> int:database = container.database()service = WorkflowControlService(database)+ try:+ clean_shutdown = await _consume_shutdown_state()+ except Exception as exc: # noqa: BLE001 — marker is best-effort+ logger.warning(f"Shutdown-state marker read failed; assuming clean: {exc}")+ clean_shutdown = True+ recovery_pause = not clean_shutdown and _crash_recovery_policy() == "pause"controls = await database.list_active_workflow_controls()processed = 0for control in controls:@@ -1021,6 +1191,10 @@ async def reconcile_active_controls_on_boot() -> int:)if control.status in {"running", "paused"}:await _rearm_generation(control)+ if recovery_pause and control.status == "running":+ control = await _pause_for_recovery(+ service, control, reason="unclean_shutdown"+ )processed += 1except Exception as exc: # noqa: BLE001 — per-row isolationlogger.warning(@@ -1186,6 +1360,57 @@ async def _restore_control_after_failed_update(latest = await service.database.get_latest_workflow_control(control.workflow_id)restored = latest if latest is not None else controlawait _broadcast_control(restored)+++async def _rebuild_missing_controller(service: WorkflowControlService, control):+ """Start a fresh controller for a live generation whose previous one died.++ Resume path for a killed workflow: the generation converged to+ ``paused`` when its controller vanished (terminated in the Temporal+ UI, crashed server, retention-deleted execution); Resume calls this+ to rebuild before applying the running state. Same generation-scoped+ controller workflow id, fresh run chain (``USE_EXISTING`` conflict+ policy makes it race-safe against a mis-detected live controller),+ then the local re-arm re-registers canary triggers with the fresh+ controller from the persisted graph snapshot.+ """+ from core.container import container++ logger.warning(+ "Rebuilding missing workflow controller",+ workflow_id=control.workflow_id,+ controller_workflow_id=control.controller_workflow_id,+ generation=control.generation,+ )+ run_id = await _start_controller(control, use_existing=True)+ if run_id:+ control = await service.transition(+ control,+ expected_revision=control.revision,+ from_statuses={control.status},+ status=control.status,+ values={"controller_run_id": run_id},+ )+ await service.database.update_workflow_run_data_scope(+ control.data_scope_id or control.execution_id,+ temporal_run_id=run_id,+ )+ # Local trigger state may still be armed against the dead controller —+ # tear it down so the re-arm registers with the fresh one. The+ # "not deployed" error from a cold process is expected and benign.+ workflow_service = container.workflow_service()+ if workflow_service.is_workflow_deployed(control.workflow_id):+ try:+ await handle_cancel_deployment({"workflow_id": control.workflow_id}, None)+ except Exception as exc: # noqa: BLE001 — re-arm still proceeds+ logger.warning(+ "Local cancel before controller rebuild re-arm failed",+ workflow_id=control.workflow_id,+ error=str(exc),+ )+ await _rearm_generation(control)+ await _broadcast_control(control, extra={"recovery": "controller_rebuilt"})+ return controlreturn restored@@ -1431,15 +1656,17 @@ async def handle_resume_workflow(data: Dict[str, Any], websocket: WebSocket) ->control, expected_revision=_expected_revision(data, control), from_statuses={"paused"}, status="resuming")await _broadcast_control(control)- try:- controller_status = await _update_controller_state(- control,+ resume_update_key = data.get("idempotency_key") or uuid.uuid4().hex++ async def _apply_resume_update(target):+ return await _update_controller_state(+ target,"running",- update_id=(- f"{control.id}:{control.revision}:"- f"{data.get('idempotency_key') or uuid.uuid4().hex}:running"- ),+ update_id=f"{target.id}:{target.revision}:{resume_update_key}:running",)++ try:+ controller_status = await _apply_resume_update(control)except (TemporalControlUnavailable, TemporalControlAckMismatch):await _restore_control_after_failed_update(service,@@ -1448,6 +1675,30 @@ async def handle_resume_workflow(data: Dict[str, Any], websocket: WebSocket) ->stable_state="paused",)raise+ except Exception as exc:+ if not _temporal_target_already_gone(exc):+ raise+ # The controller execution is gone (killed in the Temporal UI /+ # crashed / retention-deleted). Under the missing-controller pause+ # policy the row converged to paused precisely so this Resume can+ # rebuild the controller from the durable row + graph snapshot,+ # then re-apply the running state against the fresh execution.+ try:+ control = await _rebuild_missing_controller(service, control)+ controller_status = await _apply_resume_update(control)+ except Exception as rebuild_exc: # noqa: BLE001 — restore + surface+ logger.warning(+ "Controller rebuild during resume failed",+ workflow_id=workflow_id,+ error=str(rebuild_exc),+ )+ await _restore_control_after_failed_update(+ service,+ control,+ transitional_state="resuming",+ stable_state="paused",+ )+ raiseresumed_schedules = await _set_cron_pause(workflow_id,paused=False,@@ -1597,6 +1848,67 @@ async def handle_reset_workflow(data: Dict[str, Any], websocket: WebSocket) -> D}+async def pause_generation_on_failure(*, workflow_id: str, reason: str) -> Dict[str, Any]:+ """Circuit breaker: pause a controlled deployment after a failed run.++ Called by the ``workflow_control.pause_on_failure.v1`` activity that+ MachinaWorkflow schedules on its failure path. Without it a trigger+ keeps firing new runs into the same error indefinitely; with it the+ deployment lands ``paused`` so the user fixes the cause and Resumes.++ Gated on ``WORKFLOW_CONTROL_PAUSE_ON_FAILURE`` (default true) and+ applies only to a generation currently ``running`` — direct manual+ runs, legacy uncontrolled deployments, and already-paused+ generations are untouched. The knob is evaluated HERE (activity+ side) rather than inside workflow code so flipping it never touches+ recorded workflow commands.+ """+ from core.container import container++ if not workflow_id:+ return {"paused": False, "reason": "no_workflow_id"}+ if not bool(getattr(container.settings(), "workflow_control_pause_on_failure", True)):+ return {"paused": False, "reason": "disabled"}+ service = _control_service()+ control = await service.database.get_latest_workflow_control(workflow_id)+ if control is None or control.status != "running":+ return {+ "paused": False,+ "reason": "not_running",+ "status": getattr(control, "status", None),+ }+ logger.warning(+ "Pausing deployment after failed run (circuit breaker)",+ workflow_id=workflow_id,+ generation=control.generation,+ failure=reason,+ )+ try:+ result = await handle_pause_workflow(+ {+ "workflow_id": workflow_id,+ "expected_revision": control.revision,+ "idempotency_key": f"pause-on-failure:{control.id}:{control.revision}",+ },+ None,+ )+ except Exception as exc: # noqa: BLE001 — activity reports, never raises+ logger.warning(+ "Pause-on-failure did not complete",+ workflow_id=workflow_id,+ error=str(exc),+ )+ return {"paused": False, "error": str(exc)}+ paused = bool(isinstance(result, dict) and result.get("success"))+ if not paused:+ logger.warning(+ "Pause-on-failure did not complete",+ workflow_id=workflow_id,+ error=(result or {}).get("error") if isinstance(result, dict) else None,+ )+ return {"paused": paused, "failure": reason}++WS_HANDLERS: Dict[str, Any] = {"deploy_workflow": handle_deploy_workflow,"cancel_deployment": handle_cancel_deployment,@@ -1611,6 +1923,14 @@ WS_HANDLERS: Dict[str, Any] = {}+# Clean-shutdown marker for crash detection (see _consume_shutdown_state).+# Registered through the generic lifespan shutdown-hook registry so main.py+# never learns about the control plane's internals.+from services.plugin.shutdown_hooks import register_shutdown_hook # noqa: E402++register_shutdown_hook("workflow_control_clean_shutdown", mark_clean_shutdown)++__all__ = ["WS_HANDLERS","handle_cancel_deployment",@@ -1618,4 +1938,7 @@ __all__ = ["handle_get_deployment_status","handle_get_workflow_lock","handle_update_deployment_settings",+ "mark_clean_shutdown",+ "pause_generation_on_failure",+ "reconcile_active_controls_on_boot",]diff --git a/server/services/temporal/activities.py b/server/services/temporal/activities.pyindex 0724edd2..5005d948 100644--- a/server/services/temporal/activities.py+++ b/server/services/temporal/activities.py@@ -524,3 +524,32 @@ async def broadcast_trigger_status_activity(payload: Dict[str, Any]) -> None:)except Exception as exc: # noqa: BLE001 — non-fatalactivity.logger.warning(f"broadcast_trigger_status_activity failed for " f"node={payload.get('node_id')!r}: {exc}")++[email protected](name="workflow_control.pause_on_failure.v1")+async def pause_workflow_on_failure_activity(payload: Dict[str, Any]) -> Dict[str, Any]:+ """Circuit breaker: pause a controlled deployment after a failed run.++ Scheduled by MachinaWorkflow's failure path (patch-gated). All policy+ lives in ``services.deployment.handlers.pause_generation_on_failure``+ — including the WORKFLOW_CONTROL_PAUSE_ON_FAILURE knob, evaluated on+ the activity side so flipping it never touches recorded workflow+ commands. Never raises; the result dict lands in workflow history+ for ops inspection.++ Payload shape:+ {+ "workflow_id": str, # OpenCompany deployment workflow_id+ "reason": str, # first node error of the failed run+ }+ """+ from services.deployment.handlers import pause_generation_on_failure++ try:+ return await pause_generation_on_failure(+ workflow_id=str(payload.get("workflow_id") or ""),+ reason=str(payload.get("reason") or "run_failed"),+ )+ except Exception as exc: # noqa: BLE001 — non-fatal+ activity.logger.warning(f"pause_workflow_on_failure_activity failed for " f"workflow={payload.get('workflow_id')!r}: {exc}")+ return {"paused": False, "error": str(exc)}diff --git a/server/services/temporal/worker.py b/server/services/temporal/worker.pyindex 07b42c60..0f6780c4 100644--- a/server/services/temporal/worker.py+++ b/server/services/temporal/worker.py@@ -199,6 +199,7 @@ class TemporalWorkerManager:from services.temporal.activities import (broadcast_trigger_status_activity,load_persisted_workflow_graph_activity,+ pause_workflow_on_failure_activity,store_node_output_activity,)@@ -252,6 +253,7 @@ class TemporalWorkerManager:self._activities.execute_node_activity,broadcast_trigger_status_activity,load_persisted_workflow_graph_activity,+ pause_workflow_on_failure_activity,store_node_output_activity,*per_type,*agent_activities,@@ -704,6 +706,7 @@ async def run_standalone_worker(from services.temporal.activities import (broadcast_trigger_status_activity,load_persisted_workflow_graph_activity,+ pause_workflow_on_failure_activity,store_node_output_activity,)@@ -769,6 +772,7 @@ async def create_worker(from services.temporal.activities import (broadcast_trigger_status_activity,load_persisted_workflow_graph_activity,+ pause_workflow_on_failure_activity,store_node_output_activity,)@@ -785,6 +789,7 @@ async def create_worker(activities.execute_node_activity,broadcast_trigger_status_activity,load_persisted_workflow_graph_activity,+ pause_workflow_on_failure_activity,store_node_output_activity,],max_concurrent_activities=100,diff --git a/server/services/temporal/workflow.py b/server/services/temporal/workflow.pyindex 874adeb8..3566768e 100644--- a/server/services/temporal/workflow.py+++ b/server/services/temporal/workflow.py@@ -16,7 +16,7 @@ from typing import Any, Dict, List, Optional, Setfrom temporalio import workflow-from ._retry_policies import DEFAULT_ACTIVITY_RETRY+from ._retry_policies import DEFAULT_ACTIVITY_RETRY, QUICK_ACTIVITY_RETRYfrom services.workflow_naming import node_label_slug# Config handles - nodes connecting via these are config nodes (not executed)@@ -105,6 +105,14 @@ COOPERATIVE_PAUSE_SCHEDULING_PATCH = (# 30s), not by lifetime caps. Patch keeps replay compatibility for# histories recorded with the old caps.UNBOUNDED_LIFETIMES_PATCH = "machina-unbounded-lifetimes-v1"+# Circuit breaker: a failed trigger-spawned run schedules the+# workflow_control.pause_on_failure.v1 activity so the deployment pauses+# (user fixes + resumes) instead of the trigger firing into the same+# error indefinitely. The WORKFLOW_CONTROL_PAUSE_ON_FAILURE knob is+# evaluated on the activity side, so flipping config never touches+# recorded workflow commands; this patch only gates the activity+# scheduling itself for replay compatibility with older histories.+PAUSE_ON_FAILURE_PATCH = "machina-pause-on-failure-v1"# start_to_close for node activities on the patched path. Generous by# design: a single node step (agent turn batch, browser op, long shell# command) may run for hours; worker-death detection is the heartbeat@@ -163,6 +171,38 @@ class MachinaWorkflow:if self._control_paused:await workflow.wait_condition(lambda: not self._control_paused)+ async def _pause_deployment_on_failure(+ self,+ workflow_data: Dict[str, Any],+ nodes: List[Dict[str, Any]],+ errors: List[Dict],+ ) -> None:+ """Circuit breaker: a failed trigger-spawned run pauses its deployment.++ Only deployment-spawned runs qualify — they always carry a+ ``_pre_executed`` firing trigger, which one-off manual canvas runs+ never do, so a failed test run cannot pause a live deployment.+ Legacy uncontrolled deployments no-op inside the activity (no+ running control row). Non-fatal by design: the run's own failure+ result is the primary outcome either way.+ """+ workflow_id = workflow_data.get("workflow_id")+ spawned_by_trigger = any(node.get("_pre_executed") for node in nodes)+ if not workflow_id or not spawned_by_trigger:+ return+ try:+ await workflow.execute_activity(+ "workflow_control.pause_on_failure.v1",+ {+ "workflow_id": workflow_id,+ "reason": str((errors[0] or {}).get("error", "run_failed"))[:500],+ },+ start_to_close_timeout=timedelta(seconds=30),+ retry_policy=QUICK_ACTIVITY_RETRY,+ )+ except Exception as exc: # noqa: BLE001 — cosmetic relative to the run result+ workflow.logger.warning(f"pause-on-failure activity failed (non-fatal): {exc}")+@workflow.signalasync def on_event(self, event_payload: Dict[str, Any]) -> None:"""Wave 12 A7: receive an event from ``services.events.dispatch.emit``.@@ -275,6 +315,7 @@ class MachinaWorkflow:COOPERATIVE_PAUSE_SCHEDULING_PATCH)use_unbounded_lifetimes = workflow.patched(UNBOUNDED_LIFETIMES_PATCH)+ use_pause_on_failure = workflow.patched(PAUSE_ON_FAILURE_PATCH)workflow.logger.info(f"Starting workflow orchestration: {len(nodes)} nodes, {len(edges)} edges")@@ -519,6 +560,9 @@ class MachinaWorkflow:# Build final resultsuccess = len(errors) == 0 and len(completed) == len(node_map)+ if errors and use_pause_on_failure:+ await self._pause_deployment_on_failure(workflow_data, nodes, errors)+workflow.logger.info(f"Workflow complete: success={success}, " f"executed={len(execution_trace)}/{len(node_map)}")return {diff --git a/server/tests/services/test_workflow_control_handlers.py b/server/tests/services/test_workflow_control_handlers.pyindex 479ddb1c..f1e0fcdf 100644--- a/server/tests/services/test_workflow_control_handlers.py+++ b/server/tests/services/test_workflow_control_handlers.py@@ -1187,12 +1187,70 @@ async def test_terminal_generation_is_never_probed(monkeypatch, terminal_status)@pytest.mark.asyncio-async def test_live_generation_with_missing_controller_is_failed(monkeypatch):- """Otherwise it stays non-'reset' forever and blocks every future Start."""+async def test_live_generation_with_missing_controller_pauses_by_default(monkeypatch):+ """WORKFLOW_CONTROL_MISSING_CONTROLLER=pause (default): a killed+ controller leaves the generation user-resumable — Resume rebuilds it —+ instead of forcing a Reset that archives conversation state."""+ paused = _control("paused", 7)+ transition = AsyncMock(return_value=paused)+ fail = AsyncMock()+ broadcast = AsyncMock(return_value={})++ monkeypatch.setattr(handlers, "_missing_controller_policy", lambda: "pause")+ monkeypatch.setattr(+ handlers,+ "_query_controller_state",+ AsyncMock(side_effect=handlers.ControllerExecutionMissing("gone")),+ )+ monkeypatch.setattr(handlers, "_broadcast_control", broadcast)++ control = _control("pausing", 6)+ result, controller_status = await handlers._reconcile_control(+ _reconcile_service(fail=fail, transition=transition), control+ )++ assert result is paused+ assert controller_status is None+ assert transition.await_args.kwargs["status"] == "paused"+ fail.assert_not_awaited()+ broadcast.assert_awaited_once()+++async def test_already_paused_generation_with_missing_controller_is_left_alone(+ monkeypatch,+):+ """Paused is already the recoverable posture; Resume performs the rebuild."""+ transition = AsyncMock()+ fail = AsyncMock()+ monkeypatch.setattr(handlers, "_missing_controller_policy", lambda: "pause")+ monkeypatch.setattr(+ handlers,+ "_query_controller_state",+ AsyncMock(side_effect=handlers.ControllerExecutionMissing("gone")),+ )++ control = _control("paused", 6)+ result, _ = await handlers._reconcile_control(+ _reconcile_service(fail=fail, transition=transition), control+ )++ assert result is control+ transition.assert_not_awaited()+ fail.assert_not_awaited()+++async def test_live_generation_with_missing_controller_fails_under_legacy_policy(+ monkeypatch,+):+ """WORKFLOW_CONTROL_MISSING_CONTROLLER=fail preserves the Reset-only+ behaviour; otherwise the row stays non-'reset' forever and blocks Start."""failed = _control("failed", 7, terminal_reason="controller_execution_missing")fail = AsyncMock(return_value=failed)broadcast = AsyncMock(return_value={})+ monkeypatch.setattr(handlers, "_missing_controller_policy", lambda: "fail")monkeypatch.setattr(handlers,"_query_controller_state",@@ -1211,6 +1269,30 @@ async def test_live_generation_with_missing_controller_is_failed(monkeypatch):broadcast.assert_awaited_once()+async def test_starting_with_missing_controller_still_fails_under_pause_policy(+ monkeypatch,+):+ """Nothing durable runs yet mid-start; Reset + Start rebuilds cleanly."""+ failed = _control("failed", 7, terminal_reason="controller_execution_missing")+ fail = AsyncMock(return_value=failed)+ monkeypatch.setattr(handlers, "_missing_controller_policy", lambda: "pause")+ monkeypatch.setattr(+ handlers,+ "_query_controller_state",+ AsyncMock(side_effect=handlers.ControllerExecutionMissing("gone")),+ )+ monkeypatch.setattr(handlers, "_broadcast_control", AsyncMock(return_value={}))++ control = _control("starting", 6)+ result, _ = await handlers._reconcile_control(+ _reconcile_service(fail=fail), control+ )++ assert result is failed+ assert fail.await_args.args[1] == "controller_execution_missing"++@pytest.mark.asyncioasync def test_resetting_is_not_auto_failed(monkeypatch):"""Reset owns its own retry; auto-failing here could race a live reset."""diff --git a/server/tests/temporal/test_durability_hardening.py b/server/tests/temporal/test_durability_hardening.pyindex d5351361..3b2ccba5 100644--- a/server/tests/temporal/test_durability_hardening.py+++ b/server/tests/temporal/test_durability_hardening.py@@ -56,6 +56,7 @@ def test_durability_changes_are_replay_patch_guarded():expected = {machina_module.UNBOUNDED_LIFETIMES_PATCH: inspect.getsource(MachinaWorkflow.run),+ machina_module.PAUSE_ON_FAILURE_PATCH: inspect.getsource(MachinaWorkflow.run),agent_module.UNBOUNDED_LIFETIMES_PATCH: inspect.getsource(AgentWorkflow.run),trigger_module.UNBOUNDED_CHILD_RUNS_PATCH: inspect.getsource(TriggerListenerWorkflow._spawn_child_run@@ -80,6 +81,7 @@ def test_durability_changes_are_replay_patch_guarded():assert set(expected) == {"machina-unbounded-lifetimes-v1",+ "machina-pause-on-failure-v1","agent-unbounded-lifetimes-v1","trigger-unbounded-child-runs-v1","polling-unbounded-child-runs-v1",@@ -732,6 +734,7 @@ async def test_boot_reconcile_processes_each_active_row_with_isolation(monkeypatlist_active_workflow_controls=AsyncMock(return_value=[broken, healthy]),)monkeypatch.setattr(container, "database", MagicMock(return_value=database))+ monkeypatch.setattr(handlers, "_consume_shutdown_state", AsyncMock(return_value=True))async def reconcile(_service, control):if control.workflow_id == "wf-broken":@@ -747,3 +750,364 @@ async def test_boot_reconcile_processes_each_active_row_with_isolation(monkeypat# The broken row was isolated; the healthy one was re-armed.assert processed == 1rearm.assert_awaited_once_with(healthy)+++# ---------------------------------------------------------------------------+# Crash / kill recovery: unclean-shutdown detection + recovery pause.+# ---------------------------------------------------------------------------+++async def test_shutdown_marker_protocol(monkeypatch):+ """Dirty bit: boot stamps "running"; clean teardown stamps "clean";+ a missing marker (first boot) counts as clean."""+ from core.container import container+ from services.deployment import handlers++ store: dict = {}++ async def get_entry(key):+ value = store.get(key)+ return SimpleNamespace(value=value) if value is not None else None++ async def set_entry(key, value, *args, **kwargs):+ store[key] = value++ database = SimpleNamespace(get_cache_entry=get_entry, set_cache_entry=set_entry)+ monkeypatch.setattr(container, "database", MagicMock(return_value=database))++ # First boot: no marker -> clean; stamps the dirty bit.+ assert await handlers._consume_shutdown_state() is True+ assert store[handlers._SHUTDOWN_STATE_CACHE_KEY] == "running"+ # Crash (no clean stamp) -> next boot detects unclean.+ assert await handlers._consume_shutdown_state() is False+ # Graceful teardown stamps clean -> next boot is clean again.+ await handlers.mark_clean_shutdown()+ assert store[handlers._SHUTDOWN_STATE_CACHE_KEY] == "clean"+ assert await handlers._consume_shutdown_state() is True+++def test_clean_shutdown_hook_is_registered():+ from services.plugin.shutdown_hooks import registered_labels++ assert "workflow_control_clean_shutdown" in registered_labels()+++ ("clean", "policy", "expect_pause"),+ [+ (False, "pause", True), # crash + default policy -> recovery pause+ (False, "resume", False), # crash + resume policy -> keep running+ (True, "pause", False), # clean restart -> always restore as-is+ ],+)+async def test_boot_recovery_pauses_running_rows_after_unclean_shutdown(+ monkeypatch, clean, policy, expect_pause+):+ from core.container import container+ from services.deployment import handlers++ running = _boot_control("running")+ paused = _boot_control("paused")+ database = SimpleNamespace(+ list_active_workflow_controls=AsyncMock(return_value=[running, paused]),+ )+ monkeypatch.setattr(container, "database", MagicMock(return_value=database))+ monkeypatch.setattr(+ handlers, "_consume_shutdown_state", AsyncMock(return_value=clean)+ )+ monkeypatch.setattr(handlers, "_crash_recovery_policy", lambda: policy)+ monkeypatch.setattr(+ handlers,+ "_reconcile_control",+ AsyncMock(side_effect=lambda _s, c: (c, {"state": c.status})),+ )+ monkeypatch.setattr(handlers, "_rearm_generation", AsyncMock())+ recovery_pause = AsyncMock(side_effect=lambda _s, c, reason: c)+ monkeypatch.setattr(handlers, "_pause_for_recovery", recovery_pause)++ processed = await handlers.reconcile_active_controls_on_boot()++ assert processed == 2+ if expect_pause:+ # Only the RUNNING row gets the recovery pause; already-paused+ # generations stay untouched.+ recovery_pause.assert_awaited_once()+ assert recovery_pause.await_args.args[1] is running+ else:+ recovery_pause.assert_not_awaited()+++async def test_pause_for_recovery_reuses_the_full_pause_ceremony(monkeypatch):+ from services.deployment import handlers++ control = _boot_control("running")+ paused = _boot_control("paused")+ pause_handler = AsyncMock(return_value={"success": True, "state": "paused"})+ monkeypatch.setattr(handlers, "handle_pause_workflow", pause_handler)+ service = SimpleNamespace(+ database=SimpleNamespace(+ get_latest_workflow_control=AsyncMock(return_value=paused)+ )+ )++ result = await handlers._pause_for_recovery(+ service, control, reason="unclean_shutdown"+ )++ assert result is paused+ request = pause_handler.await_args.args[0]+ assert request["workflow_id"] == control.workflow_id+ assert request["expected_revision"] == control.revision+++# ---------------------------------------------------------------------------+# Killed controller: Resume rebuilds it from the durable row.+# ---------------------------------------------------------------------------+++async def test_rebuild_missing_controller_uses_existing_conflict_policy(monkeypatch):+ from core.container import container+ from services.deployment import handlers++ control = _boot_control("resuming")+ updated = _boot_control("resuming")+ updated.revision = 4+ start_controller = AsyncMock(return_value="fresh-run-1")+ monkeypatch.setattr(handlers, "_start_controller", start_controller)+ rearm = AsyncMock()+ monkeypatch.setattr(handlers, "_rearm_generation", rearm)+ monkeypatch.setattr(handlers, "_broadcast_control", AsyncMock(return_value={}))+ local_cancel = AsyncMock(return_value={"success": True})+ monkeypatch.setattr(handlers, "handle_cancel_deployment", local_cancel)+ workflow_service = MagicMock()+ workflow_service.is_workflow_deployed.return_value = True+ monkeypatch.setattr(+ container, "workflow_service", MagicMock(return_value=workflow_service)+ )+ service = SimpleNamespace(+ transition=AsyncMock(return_value=updated),+ database=SimpleNamespace(+ update_workflow_run_data_scope=AsyncMock(return_value=True)+ ),+ )++ result = await handlers._rebuild_missing_controller(service, control)++ assert result is updated+ # Race-safe start: adopt a live controller or restart a terminated one.+ assert start_controller.await_args.kwargs["use_existing"] is True+ # Fresh run id persisted on the row (status unchanged).+ assert service.transition.await_args.kwargs["values"] == {+ "controller_run_id": "fresh-run-1"+ }+ # Stale local trigger state torn down, then re-armed against the+ # fresh controller.+ local_cancel.assert_awaited_once()+ rearm.assert_awaited_once_with(updated)+++async def test_resume_rebuilds_a_gone_controller_then_reapplies_running(monkeypatch):+ from core.container import container+ from services.deployment import handlers++ paused = _boot_control("paused")+ resuming = _boot_control("resuming")+ resuming.revision = 4+ running = _boot_control("running")+ running.revision = 5++ class _Gone(Exception):+ pass++ service = SimpleNamespace(+ database=SimpleNamespace(+ get_latest_workflow_control=AsyncMock(return_value=paused)+ ),+ transition=AsyncMock(side_effect=[resuming, running]),+ )+ monkeypatch.setattr(handlers, "_control_service", lambda: service)+ monkeypatch.setattr(+ handlers, "_reconcile_control", AsyncMock(return_value=(paused, None))+ )+ monkeypatch.setattr(handlers, "_broadcast_control", AsyncMock(return_value={}))+ monkeypatch.setattr(+ handlers, "_control_payload", AsyncMock(return_value={"state": "running"})+ )+ update_state = AsyncMock(side_effect=[_Gone("not found"), {"state": "running"}])+ monkeypatch.setattr(handlers, "_update_controller_state", update_state)+ monkeypatch.setattr(handlers, "_temporal_target_already_gone", lambda exc: isinstance(exc, _Gone))+ rebuild = AsyncMock(return_value=resuming)+ monkeypatch.setattr(handlers, "_rebuild_missing_controller", rebuild)+ monkeypatch.setattr(handlers, "_set_cron_pause", AsyncMock(return_value=0))+ monkeypatch.setattr(+ handlers, "_signal_generation_workflows", AsyncMock(return_value=0)+ )+ workflow_service = MagicMock()+ workflow_service.resume_deployment = AsyncMock(return_value=0)+ workflow_service.update_trigger_pause_status = AsyncMock(return_value=0)+ monkeypatch.setattr(+ container, "workflow_service", MagicMock(return_value=workflow_service)+ )++ result = await handlers.handle_resume_workflow(+ {"workflow_id": "wf", "expected_revision": paused.revision}, None+ )++ assert result["success"] is True+ rebuild.assert_awaited_once()+ # The running update was re-applied against the rebuilt controller.+ assert update_state.await_count == 2+++# ---------------------------------------------------------------------------+# Circuit breaker: failed runs pause the deployment.+# ---------------------------------------------------------------------------+++async def test_pause_on_failure_is_config_gated_and_requires_running(monkeypatch):+ from core.container import container+ from services.deployment import handlers++ pause_handler = AsyncMock(return_value={"success": True})+ monkeypatch.setattr(handlers, "handle_pause_workflow", pause_handler)++ # Disabled by config -> untouched.+ monkeypatch.setattr(+ container,+ "settings",+ MagicMock(+ return_value=SimpleNamespace(workflow_control_pause_on_failure=False)+ ),+ )+ result = await handlers.pause_generation_on_failure(+ workflow_id="wf", reason="boom"+ )+ assert result == {"paused": False, "reason": "disabled"}++ # Enabled but the generation is already paused -> untouched.+ monkeypatch.setattr(+ container,+ "settings",+ MagicMock(+ return_value=SimpleNamespace(workflow_control_pause_on_failure=True)+ ),+ )+ service = SimpleNamespace(+ database=SimpleNamespace(+ get_latest_workflow_control=AsyncMock(return_value=_boot_control("paused"))+ )+ )+ monkeypatch.setattr(handlers, "_control_service", lambda: service)+ result = await handlers.pause_generation_on_failure(+ workflow_id="wf", reason="boom"+ )+ assert result["paused"] is False+ pause_handler.assert_not_awaited()+++async def test_pause_on_failure_pauses_a_running_generation(monkeypatch):+ from core.container import container+ from services.deployment import handlers++ running = _boot_control("running")+ monkeypatch.setattr(+ container,+ "settings",+ MagicMock(+ return_value=SimpleNamespace(workflow_control_pause_on_failure=True)+ ),+ )+ service = SimpleNamespace(+ database=SimpleNamespace(+ get_latest_workflow_control=AsyncMock(return_value=running)+ )+ )+ monkeypatch.setattr(handlers, "_control_service", lambda: service)+ pause_handler = AsyncMock(return_value={"success": True, "state": "paused"})+ monkeypatch.setattr(handlers, "handle_pause_workflow", pause_handler)++ result = await handlers.pause_generation_on_failure(+ workflow_id="wf", reason="node exploded"+ )++ assert result == {"paused": True, "failure": "node exploded"}+ request = pause_handler.await_args.args[0]+ assert request["workflow_id"] == "wf"+ assert request["expected_revision"] == running.revision+++async def test_machina_failure_schedules_the_circuit_breaker_activity(monkeypatch):+ from services.temporal import workflow as machina_module+ from services.temporal.workflow import MachinaWorkflow++ temporal_workflow = machina_module.workflow+ monkeypatch.setattr(temporal_workflow, "logger", MagicMock())+ captured: dict = {}++ async def capture_activity(name, payload, **kwargs):+ captured["name"] = name+ captured["payload"] = payload++ monkeypatch.setattr(temporal_workflow, "execute_activity", capture_activity)++ instance = MachinaWorkflow()+ nodes = [{"id": "t-1", "type": "webhookTrigger", "_pre_executed": True}]+ errors = [{"node_id": "n-2", "error": "credential expired"}]++ # Trigger-spawned run -> schedules the breaker.+ await instance._pause_deployment_on_failure({"workflow_id": "wf"}, nodes, errors)+ assert captured["name"] == "workflow_control.pause_on_failure.v1"+ assert captured["payload"] == {+ "workflow_id": "wf",+ "reason": "credential expired",+ }++ # Manual run (no pre-executed trigger) -> never pauses a deployment.+ captured.clear()+ await instance._pause_deployment_on_failure(+ {"workflow_id": "wf"}, [{"id": "n-1", "type": "aiAgent"}], errors+ )+ assert captured == {}++ # No workflow_id -> nothing to pause.+ await instance._pause_deployment_on_failure({}, nodes, errors)+ assert captured == {}+++def test_recovery_policies_normalize_unknown_values(monkeypatch):+ from core.container import container+ from services.deployment import handlers++ monkeypatch.setattr(+ container,+ "settings",+ MagicMock(+ return_value=SimpleNamespace(+ workflow_control_crash_recovery="bogus",+ workflow_control_missing_controller="also-bogus",+ )+ ),+ )+ assert handlers._crash_recovery_policy() == "pause"+ assert handlers._missing_controller_policy() == "pause"++ monkeypatch.setattr(+ container,+ "settings",+ MagicMock(+ return_value=SimpleNamespace(+ workflow_control_crash_recovery="RESUME",+ workflow_control_missing_controller="Fail",+ )+ ),+ )+ assert handlers._crash_recovery_policy() == "resume"+ assert handlers._missing_controller_policy() == "fail"