Skip to content

Self-improving AI Employees for everyone

feat(control): configurable recovery policies - crash pause, controller rebuild, failure circuit breaker

trohitg5dae0cec839a
diff --git a/.env.template b/.env.template
index b02863e8..9f911eae 100644
--- a/.env.template
+++ b/.env.template
@@ -121,6 +121,24 @@ TEMPORAL_GRACEFUL_SHUTDOWN_SECONDS=30
# (backend-owned) servers only; remote clusters are never touched.
TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS=15
+# --- Workflow-control recovery policies ---
+# Posture for deployments still ``running`` after an UNCLEAN shutdown
+# (kill / crash, detected via a dirty-bit marker the graceful lifespan
+# teardown clears). ``pause`` recovers them paused so the user
+# consciously resumes; ``resume`` restores them running. A clean
+# ``company stop`` + start always restores deployments as they were.
+WORKFLOW_CONTROL_CRASH_RECOVERY=pause
+# Convergence for a live generation whose controller execution vanished
+# (terminated in the Temporal UI, killed, retention-deleted). ``pause``
+# keeps it user-resumable — Resume rebuilds the controller from the
+# durable row + graph snapshot; ``fail`` preserves the legacy behaviour
+# where only Reset (which archives conversation state) recovers.
+WORKFLOW_CONTROL_MISSING_CONTROLLER=pause
+# Circuit breaker: when a controlled run fails with an error, pause the
+# deployment (fix + Resume) instead of letting the trigger keep firing
+# into the same error indefinitely. ``false`` disables the auto-pause.
+WORKFLOW_CONTROL_PAUSE_ON_FAILURE=true
+
# --- Startup resilience (server-ready-before-workers) ---
# After the gRPC port binds, the client polls the WorkflowService gRPC
# health check until it reports SERVING before starting the worker /
diff --git a/CLAUDE.md b/CLAUDE.md
index ca1518b7..ac6451d5 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -742,7 +742,7 @@ Node groups (palette categories): agent, model, skill, tool, trigger, workflow,
Workflows execute via Temporal for durability and horizontal scaling, gated by `TEMPORAL_ENABLED`. Three dispatch paths exist — legacy single `execute_node_activity` (fallback), per-type `node.{type}.v{version}` activities (F4.A, `TEMPORAL_PER_TYPE_DISPATCH=true`), and Agent-as-child-workflow (F4.B, `TEMPORAL_AGENT_WORKFLOW_ENABLED=true`) — with `rlm_agent`/`claude_code_agent` always bypassing AgentWorkflow. **Per-queue activity routing is production-default (Wave 16)**: a `TemporalWorkerPool` runs one activity-only worker per plugin-declared `cls.task_queue` (ai-heavy / browser / code-exec / rest-api / messaging / ...) with per-queue concurrency + rate limits + resource-based tuning for ai-heavy/browser; rollback via `TEMPORAL_WORKER_POOL_ENABLED=false`. Resilience knobs (Waves 17-18): `DEPLOYMENT_MODE` (local halves concurrency; sizes the sticky cache), one-shot retry on `agent.execute_llm_step.v1` (LLM calls are not idempotent), observability interceptors (`activity_retry` WARN on re-dispatch), 30s periodic heartbeat during long activity bodies, cron `catchup_window=24h`, poller autoscaling. Execution routing falls back Temporal → sequential; the Temporal dev server + embedded worker are managed in-process under `server/services/temporal/` (pooch-downloaded CLI, single `temporal server start-dev` process, gRPC `TEMPORAL_FRONTEND_GRPC_PORT` + Web UI `TEMPORAL_UI_PORT`, SQLite at `~/.opencompany/temporal.db`). Full architecture, dispatch matrix, per-node + agent-loop lifecycle, all 16 F4.B `agent.*.v1` activities, heartbeat semantics, delegation input contract, the worker tuning recipe, and all `.env` tunables live in [docs-internal/TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md).
-**Months-long durability contract (July 2026).** Running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. Load-bearing invariants: (1) `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` stays `false` (debug-only sweep; even enabled, any control row in the shared `WORKFLOW_CONTROL_ACTIVE_STATES` — including `resetting` — vetoes it). (2) New child workflow starts carry **no** `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so lifetime caps silently terminated paused runs; liveness comes from activity heartbeats (30s beat / 2min timeout), never from workflow lifetime caps. Removing/adding start options or retry policies on recorded commands requires a `workflow.patched(...)` guard (see the `*-unbounded-*` / `*-history-bounded-can-*` markers). (3) Every long-lived workflow loop must continue-as-new under history pressure (`is_continue_as_new_suggested()` + a soft cap) — Temporal hard-terminates around ~51,200 history events; `WorkflowControlWorkflow` carries triggers/queued events/seen-ids/control state across rollovers, so controller handles are addressed **by workflow id only, never run_id-pinned**. (4) `dispatch.emit` skips controllers whose `ControlEventTypes` Search Attribute doesn't match the event (absent attribute = legacy match-all). (5) The Temporal lifecycle is owned by [services/temporal/lifecycle.py](./server/services/temporal/lifecycle.py) (`main.py` only schedules it): dev-server supervision + resident watchdog (`TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS`), worker crash-restarts that REBUILD the single-use SDK `Worker` per attempt (manager + every pool queue worker), and the boot-time `reconcile_active_controls_on_boot` pass (converges crash-stranded `starting` rows, re-arms running/paused generations from the persisted graph snapshot). Locked by `tests/temporal/test_durability_hardening.py` + `test_worker_restart.py`.
+**Months-long durability contract (July 2026).** Running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. Load-bearing invariants: (1) `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` stays `false` (debug-only sweep; even enabled, any control row in the shared `WORKFLOW_CONTROL_ACTIVE_STATES` — including `resetting` — vetoes it). (2) New child workflow starts carry **no** `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so lifetime caps silently terminated paused runs; liveness comes from activity heartbeats (30s beat / 2min timeout), never from workflow lifetime caps. Removing/adding start options or retry policies on recorded commands requires a `workflow.patched(...)` guard (see the `*-unbounded-*` / `*-history-bounded-can-*` markers). (3) Every long-lived workflow loop must continue-as-new under history pressure (`is_continue_as_new_suggested()` + a soft cap) — Temporal hard-terminates around ~51,200 history events; `WorkflowControlWorkflow` carries triggers/queued events/seen-ids/control state across rollovers, so controller handles are addressed **by workflow id only, never run_id-pinned**. (4) `dispatch.emit` skips controllers whose `ControlEventTypes` Search Attribute doesn't match the event (absent attribute = legacy match-all). (5) The Temporal lifecycle is owned by [services/temporal/lifecycle.py](./server/services/temporal/lifecycle.py) (`main.py` only schedules it): dev-server supervision + resident watchdog (`TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS`), worker crash-restarts that REBUILD the single-use SDK `Worker` per attempt (manager + every pool queue worker), and the boot-time `reconcile_active_controls_on_boot` pass (converges crash-stranded `starting` rows, re-arms running/paused generations from the persisted graph snapshot). (6) **Recovery policies** (env-driven, defaults in `.env.template`; full semantics in [temporal-workflow-control.md → Recovery policies](./docs-internal/temporal-workflow-control.md)): `WORKFLOW_CONTROL_CRASH_RECOVERY=pause` — after a kill/crash (dirty-bit marker cleared only by graceful teardown) boot recovers running generations as **paused** so the user consciously resumes; `WORKFLOW_CONTROL_MISSING_CONTROLLER=pause` — a killed/vanished controller converges the generation to paused and **Resume rebuilds the controller** (`WorkflowIdConflictPolicy.USE_EXISTING`, re-arm from snapshot) instead of forcing a state-archiving Reset; `WORKFLOW_CONTROL_PAUSE_ON_FAILURE=true` — a failed trigger-spawned run pauses its deployment via the `workflow_control.pause_on_failure.v1` activity (circuit breaker; manual canvas runs never qualify). Temporal's native Pause/Unpause (server 1.28+) is deliberately NOT used — no Python SDK surface, and it halts workflow-task dispatch so a natively-paused controller couldn't process the resume Update. Locked by `tests/temporal/test_durability_hardening.py` + `test_worker_restart.py`.
**Temporal tracing ownership is strict.** Register the SDK `TracingInterceptor` exactly once on the shared `Client.connect(...)`; the Python SDK automatically prepends compatible client interceptors to every worker. Never repeat `TracingInterceptor` in `Worker(..., interceptors=...)`. Worker lists contain the distinct `ObservabilityWorkerInterceptor` and plugin-specific worker interceptors only. Exact paired `StartActivity` / `RunActivity` / `CompleteWorkflow` console spans with identical Temporal identities indicate duplicate instrumentation, not duplicate execution; use Temporal Event History and the `node.<type>.execute` application span to verify actual attempts. See the tracing and debugging sections in [TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md) and [errors.md](./docs-internal/errors.md).
diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.md
index 910332ff..ec1eb506 100644
--- a/docs-internal/TEMPORAL_ARCHITECTURE.md
+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md
@@ -618,6 +618,11 @@ The Temporal binary + persistence are managed in-process by the plugin-folder pa
| `temporal_graceful_shutdown_seconds` | `TEMPORAL_GRACEFUL_SHUTDOWN_SECONDS` | `30` | `CTRL_BREAK_EVENT` (Windows) / `SIGTERM` (POSIX) → tree-kill grace window. Shared with the embedded worker shutdown. |
| `temporal_terminate_running_on_startup` | `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` | `false` | Debug-only startup sweep (see the durability contract above). Keep false so running and paused deployments survive restarts. |
| `temporal_health_monitor_interval_seconds` | `TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS` | `15` | Resident dev-server watchdog probe cadence (lifecycle module; loopback deployments only). |
+| `workflow_control_crash_recovery` | `WORKFLOW_CONTROL_CRASH_RECOVERY` | `pause` | After an UNCLEAN shutdown (kill/crash, dirty-bit marker), boot pauses generations still `running` so the user consciously resumes; `resume` restores them running. Clean restarts always restore as-is. |
+| `workflow_control_missing_controller` | `WORKFLOW_CONTROL_MISSING_CONTROLLER` | `pause` | A live generation whose controller vanished converges to `paused` (Resume rebuilds the controller); `fail` preserves the legacy Reset-only behaviour. |
+| `workflow_control_pause_on_failure` | `WORKFLOW_CONTROL_PAUSE_ON_FAILURE` | `true` | Circuit breaker: a failed trigger-spawned run pauses its deployment (fix + Resume) instead of firing into the same error indefinitely. Evaluated activity-side; never touches recorded commands. |
+
+Full recovery-policy semantics: [temporal-workflow-control.md → Recovery policies](./temporal-workflow-control.md#recovery-policies).
The legacy `TEMPORAL_SERVER_READY_TIMEOUT_SECONDS` knob (CLI-supervised-era readiness wait) was removed — it had no consumer since the backend-owned cutover.
diff --git a/docs-internal/temporal-workflow-control.md b/docs-internal/temporal-workflow-control.md
index 75e5d83c..63af6b75 100644
--- a/docs-internal/temporal-workflow-control.md
+++ b/docs-internal/temporal-workflow-control.md
@@ -167,6 +167,45 @@ the control plane is hardened against Temporal's per-run event-history ceiling
ticking through a pause, so the caps silently terminated paused work).
Replay-patched; see TEMPORAL_ARCHITECTURE.md for the patch inventory.
+## Recovery policies
+
+Three env-driven policies (canonical defaults + semantics in
+`.env.template`; consumed by `services/deployment/handlers.py`) govern how a
+generation behaves around kills, crashes, and failures. All three reuse the
+cooperative control-plane pause — Temporal's native Pause/Unpause (server
+1.28+) is deliberately not used: it is an operational control with no Python
+SDK client methods, and it halts workflow-task dispatch entirely, so a
+natively-paused controller could not process the `set_control_state` Update
+that Resume relies on.
+
+- **`WORKFLOW_CONTROL_CRASH_RECOVERY`** (`pause` | `resume`, default
+ `pause`): after an UNCLEAN shutdown — kill or crash, detected via a
+ dirty-bit cache marker the graceful lifespan teardown clears (registered
+ through the generic shutdown-hook registry) — the boot reconcile pauses
+ every generation still `running` so the user consciously resumes it. A
+ clean `company stop` + start always restores deployments as they were.
+- **`WORKFLOW_CONTROL_MISSING_CONTROLLER`** (`pause` | `fail`, default
+ `pause`): a live generation whose controller execution vanished
+ (terminated in the Temporal UI, killed, retention-deleted) converges to
+ `paused` instead of `failed`. Resume then **rebuilds the controller**:
+ same generation-scoped workflow id started with the documented
+ `WorkflowIdConflictPolicy.USE_EXISTING` semantics (race-safe — adopts a
+ live controller or starts a fresh run after termination), persists the
+ new run id, tears down stale local trigger state, and re-arms from the
+ persisted graph snapshot before re-applying the running Update.
+ `starting` rows always fail instead (nothing durable runs yet; Reset +
+ Start rebuilds cleanly). `fail` preserves the legacy Reset-only
+ behaviour.
+- **`WORKFLOW_CONTROL_PAUSE_ON_FAILURE`** (default `true`): circuit
+ breaker — when a trigger-spawned run fails with an error, MachinaWorkflow
+ schedules `workflow_control.pause_on_failure.v1` (patch
+ `machina-pause-on-failure-v1`) and the deployment pauses so the user
+ fixes the cause and Resumes, instead of the trigger firing into the same
+ error indefinitely. Only deployment-spawned runs qualify (they carry a
+ `_pre_executed` firing trigger; a failed manual canvas run never pauses a
+ live deployment), and the knob is evaluated on the activity side so
+ flipping it never touches recorded workflow commands.
+
## Delegated-task traces
Each task attempt stores the actual parent, detached runner, and child Temporal
diff --git a/server/core/config.py b/server/core/config.py
index 2c877cd8..732396db 100644
--- a/server/core/config.py
+++ b/server/core/config.py
@@ -250,6 +250,32 @@ class Settings(BaseSettings):
gt=0,
)
+ # Workflow-control recovery policies (canonical defaults + full
+ # semantics in .env.template; Python defaults keep pre-feature .env
+ # files booting). All three are consumed by
+ # services/deployment/handlers.py — see the normalizers there.
+ # "pause" | "resume": posture for generations still running after an
+ # UNCLEAN shutdown (kill / crash). Clean restarts always restore as-is.
+ workflow_control_crash_recovery: str = Field(
+ default="pause",
+ env="WORKFLOW_CONTROL_CRASH_RECOVERY",
+ )
+ # "pause" | "fail": convergence for a live generation whose controller
+ # execution vanished (terminated in the Temporal UI, killed, deleted).
+ # "pause" keeps it user-resumable — Resume rebuilds the controller from
+ # the durable row; "fail" preserves the legacy Reset-only behaviour.
+ workflow_control_missing_controller: str = Field(
+ default="pause",
+ env="WORKFLOW_CONTROL_MISSING_CONTROLLER",
+ )
+ # Circuit breaker: a controlled run failing with an error pauses the
+ # deployment (user fixes + resumes) instead of the trigger firing into
+ # the same error indefinitely.
+ workflow_control_pause_on_failure: bool = Field(
+ default=True,
+ env="WORKFLOW_CONTROL_PAUSE_ON_FAILURE",
+ )
+
# API Keys (all optional, injected at runtime)
google_maps_api_key: Optional[str] = Field(default=None, env="GOOGLE_MAPS_API_KEY")
openai_api_key: Optional[str] = Field(default=None, env="OPENAI_API_KEY")
diff --git a/server/services/deployment/handlers.py b/server/services/deployment/handlers.py
index eecf8f36..24b2b2b5 100644
--- a/server/services/deployment/handlers.py
+++ b/server/services/deployment/handlers.py
@@ -420,16 +420,32 @@ def _control_service():
return WorkflowControlService(container.database())
-async def _start_controller(control) -> Optional[str]:
- """Start the durable controller, or use local mode when Temporal is disabled."""
+async def _start_controller(control, *, use_existing: bool = False) -> Optional[str]:
+ """Start the durable controller, or use local mode when Temporal is disabled.
+
+ ``use_existing=True`` is the rebuild path (Resume after the controller
+ was killed): per the documented WorkflowIdConflictPolicy semantics,
+ ``USE_EXISTING`` adopts a still-running controller (returning its run
+ id) instead of erroring, while the default reuse policy permits a
+ fresh run when the previous execution was Terminated/Failed — so the
+ rebuild is race-safe against mis-detection.
+ """
from core.container import container
- from temporalio.common import SearchAttributeKey, SearchAttributePair, TypedSearchAttributes
+ from temporalio.common import (
+ SearchAttributeKey,
+ SearchAttributePair,
+ TypedSearchAttributes,
+ WorkflowIDConflictPolicy,
+ )
wrapper = container.temporal_client()
if wrapper is None or wrapper.client is None:
if container.settings().temporal_enabled:
raise RuntimeError("temporal_control_unavailable")
return None
+ start_kwargs: Dict[str, Any] = {}
+ if use_existing:
+ start_kwargs["id_conflict_policy"] = WorkflowIDConflictPolicy.USE_EXISTING
handle = await wrapper.client.start_workflow(
"WorkflowControlWorkflow",
args=[{
@@ -447,6 +463,7 @@ async def _start_controller(control) -> Optional[str]:
SearchAttributeKey.for_keyword("EventWorkflowId"), control.workflow_id,
)
]),
+ **start_kwargs,
)
return getattr(handle, "result_run_id", None) or getattr(handle, "first_execution_run_id", None)
@@ -552,19 +569,58 @@ async def _query_controller_state(control) -> Optional[Dict[str, Any]]:
_MISSING_CONTROLLER_FAILS = frozenset(
{"starting", "running", "pausing", "paused", "resuming"}
)
+# The subset of live states that converge to ``paused`` (user-resumable)
+# under WORKFLOW_CONTROL_MISSING_CONTROLLER=pause. ``starting`` always
+# fails instead — nothing durable is running yet, and Reset + Start
+# rebuilds a fresh generation cleanly.
+_MISSING_CONTROLLER_PAUSES = frozenset({"running", "pausing", "paused", "resuming"})
async def _fail_missing_controller(service: WorkflowControlService, control):
- """Converge a generation whose controller has vanished to ``failed``.
+ """Converge a generation whose controller execution has vanished.
Without this the row keeps whatever live status it had, and because
``begin_generation`` refuses to open a new generation unless the latest
- one is ``reset``, the workflow could never be started again. ``failed`` is
- both honest and recoverable -- Reset accepts it, and Reset is what returns
- the workflow to ``ready``.
+ one is ``reset``, the workflow could never be started again.
+
+ Under the default ``WORKFLOW_CONTROL_MISSING_CONTROLLER=pause``, live
+ states converge to ``paused``: a killed workflow (terminated in the
+ Temporal UI, crashed server, retention-deleted execution) stays
+ user-recoverable — Resume rebuilds the controller from the durable
+ row + graph snapshot — whereas ``failed`` forces a Reset that
+ archives conversation state. ``fail`` preserves the legacy
+ Reset-only behaviour; ``starting`` rows always fail (nothing durable
+ is running yet).
"""
if control.status not in _MISSING_CONTROLLER_FAILS:
return control
+ if (
+ control.status in _MISSING_CONTROLLER_PAUSES
+ and _missing_controller_policy() == "pause"
+ ):
+ if control.status == "paused":
+ # Already user-resumable; Resume performs the rebuild.
+ return control
+ logger.warning(
+ "Workflow controller execution is gone; pausing the generation so "
+ "the user can resume it (Resume rebuilds the controller)",
+ workflow_id=control.workflow_id,
+ controller_workflow_id=control.controller_workflow_id,
+ status=control.status,
+ generation=control.generation,
+ )
+ try:
+ recovered = await service.transition(
+ control,
+ expected_revision=control.revision,
+ from_statuses={control.status},
+ status="paused",
+ )
+ except ValueError:
+ # Lost the CAS to a concurrent writer; their transition wins.
+ return control
+ await _broadcast_control(recovered, extra={"recovery": "controller_missing"})
+ return recovered
logger.warning(
"Workflow controller execution is gone; failing the generation so it "
"can be reset",
@@ -983,14 +1039,117 @@ async def _reconcile_control(service: WorkflowControlService, control):
# ---------------------------------------------------------------------------
+# Dirty-bit marker for crash detection. Boot stamps "running"; the graceful
+# lifespan teardown stamps "clean" (registered as a shutdown hook at the
+# bottom of this module). A boot that finds "running" already stamped knows
+# the previous process was killed or crashed mid-flight.
+_SHUTDOWN_STATE_CACHE_KEY = "workflow_control:shutdown_state"
+
+
+def _crash_recovery_policy() -> str:
+ """Normalized WORKFLOW_CONTROL_CRASH_RECOVERY: ``pause`` | ``resume``."""
+ from core.container import container
+
+ value = str(
+ getattr(container.settings(), "workflow_control_crash_recovery", "pause") or "pause"
+ ).strip().lower()
+ if value not in {"pause", "resume"}:
+ logger.warning(f"Unknown WORKFLOW_CONTROL_CRASH_RECOVERY={value!r}; using 'pause'")
+ return "pause"
+ return value
+
+
+def _missing_controller_policy() -> str:
+ """Normalized WORKFLOW_CONTROL_MISSING_CONTROLLER: ``pause`` | ``fail``."""
+ from core.container import container
+
+ value = str(
+ getattr(container.settings(), "workflow_control_missing_controller", "pause") or "pause"
+ ).strip().lower()
+ if value not in {"pause", "fail"}:
+ logger.warning(f"Unknown WORKFLOW_CONTROL_MISSING_CONTROLLER={value!r}; using 'pause'")
+ return "pause"
+ return value
+
+
+async def _consume_shutdown_state() -> bool:
+ """True when the previous backend process exited cleanly.
+
+ A missing marker (first boot / pre-feature database) counts as clean
+ so a fresh install never boots into a recovery pause.
+ """
+ from core.container import container
+
+ database = container.database()
+ entry = await database.get_cache_entry(_SHUTDOWN_STATE_CACHE_KEY)
+ previous = getattr(entry, "value", None) if entry is not None else None
+ await database.set_cache_entry(_SHUTDOWN_STATE_CACHE_KEY, "running")
+ return previous is None or previous == "clean"
+
+
+async def mark_clean_shutdown() -> None:
+ """Stamp the clean-shutdown marker (lifespan teardown hook)."""
+ from core.container import container
+
+ try:
+ await container.database().set_cache_entry(_SHUTDOWN_STATE_CACHE_KEY, "clean")
+ except Exception as exc: # noqa: BLE001 — best-effort during teardown
+ logger.warning(f"Failed to record clean shutdown marker: {exc}")
+
+
+async def _pause_for_recovery(service: WorkflowControlService, control, *, reason: str):
+ """Durably pause a recovered generation so the user resumes it.
+
+ After a kill or crash the deployment must come back ``paused`` rather
+ than silently continuing — the user decides when it is safe to
+ resume. Reuses the full pause ceremony (controller update, cron
+ schedule pause, trigger flags, local admission) via
+ :func:`handle_pause_workflow` so recovery lands in exactly the same
+ posture a user-initiated pause produces. Temporal's native
+ Pause/Unpause (server 1.28+) is deliberately NOT used here: it is an
+ operational control without Python SDK client methods, and it halts
+ workflow-task dispatch entirely — a natively-paused controller could
+ not process the ``set_control_state`` Update that Resume relies on.
+ """
+ logger.warning(
+ "Pausing recovered generation after unclean shutdown",
+ workflow_id=control.workflow_id,
+ generation=control.generation,
+ reason=reason,
+ )
+ try:
+ result = await handle_pause_workflow(
+ {
+ "workflow_id": control.workflow_id,
+ "expected_revision": control.revision,
+ "idempotency_key": f"recovery:{control.id}:{control.revision}",
+ },
+ None,
+ )
+ if not (isinstance(result, dict) and result.get("success")):
+ logger.warning(
+ "Recovery pause did not complete",
+ workflow_id=control.workflow_id,
+ error=(result or {}).get("error") if isinstance(result, dict) else None,
+ )
+ except Exception as exc: # noqa: BLE001 — lazy reconcile retries later
+ logger.warning(
+ "Recovery pause failed",
+ workflow_id=control.workflow_id,
+ error=str(exc),
+ )
+ latest = await service.database.get_latest_workflow_control(control.workflow_id)
+ return latest if latest is not None else control
+
+
async def reconcile_active_controls_on_boot() -> int:
"""Converge every active durable control after a backend restart.
- Three responsibilities per active row:
+ Responsibilities per active row:
1. Run the same lazy :func:`_reconcile_control` used by status reads
- (finishes interrupted pause/resume transitions, fails rows whose
- controller vanished).
+ (finishes interrupted pause/resume transitions, converges rows
+ whose controller vanished).
2. Converge ``starting`` rows — the one state the lazy path never
touches while its controller exists, so a crash mid-start used to
wedge the workflow behind a permanent ``workflow_start_pending``.
@@ -1002,6 +1161,11 @@ async def reconcile_active_controls_on_boot() -> int:
state (runtime counts, admission, pause fan-in) and in-process
collectors for non-canary trigger types die with the process and
previously stayed dead while the row reported ``running`` forever.
+ 4. After an UNCLEAN shutdown (kill / crash — dirty-bit marker), pause
+ every generation still ``running`` so the user consciously
+ resumes it (``WORKFLOW_CONTROL_CRASH_RECOVERY=pause``, the
+ default; ``resume`` restores them running). A clean
+ ``company stop`` + start always restores deployments as they were.
Returns the number of rows processed; per-row failures are logged and
skipped so one broken generation cannot block the rest.
@@ -1010,6 +1174,12 @@ async def reconcile_active_controls_on_boot() -> int:
database = container.database()
service = WorkflowControlService(database)
+ try:
+ clean_shutdown = await _consume_shutdown_state()
+ except Exception as exc: # noqa: BLE001 — marker is best-effort
+ logger.warning(f"Shutdown-state marker read failed; assuming clean: {exc}")
+ clean_shutdown = True
+ recovery_pause = not clean_shutdown and _crash_recovery_policy() == "pause"
controls = await database.list_active_workflow_controls()
processed = 0
for control in controls:
@@ -1021,6 +1191,10 @@ async def reconcile_active_controls_on_boot() -> int:
)
if control.status in {"running", "paused"}:
await _rearm_generation(control)
+ if recovery_pause and control.status == "running":
+ control = await _pause_for_recovery(
+ service, control, reason="unclean_shutdown"
+ )
processed += 1
except Exception as exc: # noqa: BLE001 — per-row isolation
logger.warning(
@@ -1186,6 +1360,57 @@ async def _restore_control_after_failed_update(
latest = await service.database.get_latest_workflow_control(control.workflow_id)
restored = latest if latest is not None else control
await _broadcast_control(restored)
+
+
+async def _rebuild_missing_controller(service: WorkflowControlService, control):
+ """Start a fresh controller for a live generation whose previous one died.
+
+ Resume path for a killed workflow: the generation converged to
+ ``paused`` when its controller vanished (terminated in the Temporal
+ UI, crashed server, retention-deleted execution); Resume calls this
+ to rebuild before applying the running state. Same generation-scoped
+ controller workflow id, fresh run chain (``USE_EXISTING`` conflict
+ policy makes it race-safe against a mis-detected live controller),
+ then the local re-arm re-registers canary triggers with the fresh
+ controller from the persisted graph snapshot.
+ """
+ from core.container import container
+
+ logger.warning(
+ "Rebuilding missing workflow controller",
+ workflow_id=control.workflow_id,
+ controller_workflow_id=control.controller_workflow_id,
+ generation=control.generation,
+ )
+ run_id = await _start_controller(control, use_existing=True)
+ if run_id:
+ control = await service.transition(
+ control,
+ expected_revision=control.revision,
+ from_statuses={control.status},
+ status=control.status,
+ values={"controller_run_id": run_id},
+ )
+ await service.database.update_workflow_run_data_scope(
+ control.data_scope_id or control.execution_id,
+ temporal_run_id=run_id,
+ )
+ # Local trigger state may still be armed against the dead controller —
+ # tear it down so the re-arm registers with the fresh one. The
+ # "not deployed" error from a cold process is expected and benign.
+ workflow_service = container.workflow_service()
+ if workflow_service.is_workflow_deployed(control.workflow_id):
+ try:
+ await handle_cancel_deployment({"workflow_id": control.workflow_id}, None)
+ except Exception as exc: # noqa: BLE001 — re-arm still proceeds
+ logger.warning(
+ "Local cancel before controller rebuild re-arm failed",
+ workflow_id=control.workflow_id,
+ error=str(exc),
+ )
+ await _rearm_generation(control)
+ await _broadcast_control(control, extra={"recovery": "controller_rebuilt"})
+ return control
return restored
@@ -1431,15 +1656,17 @@ async def handle_resume_workflow(data: Dict[str, Any], websocket: WebSocket) ->
control, expected_revision=_expected_revision(data, control), from_statuses={"paused"}, status="resuming"
)
await _broadcast_control(control)
- try:
- controller_status = await _update_controller_state(
- control,
+ resume_update_key = data.get("idempotency_key") or uuid.uuid4().hex
+
+ async def _apply_resume_update(target):
+ return await _update_controller_state(
+ target,
"running",
- update_id=(
- f"{control.id}:{control.revision}:"
- f"{data.get('idempotency_key') or uuid.uuid4().hex}:running"
- ),
+ update_id=f"{target.id}:{target.revision}:{resume_update_key}:running",
)
+
+ try:
+ controller_status = await _apply_resume_update(control)
except (TemporalControlUnavailable, TemporalControlAckMismatch):
await _restore_control_after_failed_update(
service,
@@ -1448,6 +1675,30 @@ async def handle_resume_workflow(data: Dict[str, Any], websocket: WebSocket) ->
stable_state="paused",
)
raise
+ except Exception as exc:
+ if not _temporal_target_already_gone(exc):
+ raise
+ # The controller execution is gone (killed in the Temporal UI /
+ # crashed / retention-deleted). Under the missing-controller pause
+ # policy the row converged to paused precisely so this Resume can
+ # rebuild the controller from the durable row + graph snapshot,
+ # then re-apply the running state against the fresh execution.
+ try:
+ control = await _rebuild_missing_controller(service, control)
+ controller_status = await _apply_resume_update(control)
+ except Exception as rebuild_exc: # noqa: BLE001 — restore + surface
+ logger.warning(
+ "Controller rebuild during resume failed",
+ workflow_id=workflow_id,
+ error=str(rebuild_exc),
+ )
+ await _restore_control_after_failed_update(
+ service,
+ control,
+ transitional_state="resuming",
+ stable_state="paused",
+ )
+ raise
resumed_schedules = await _set_cron_pause(
workflow_id,
paused=False,
@@ -1597,6 +1848,67 @@ async def handle_reset_workflow(data: Dict[str, Any], websocket: WebSocket) -> D
}
+async def pause_generation_on_failure(*, workflow_id: str, reason: str) -> Dict[str, Any]:
+ """Circuit breaker: pause a controlled deployment after a failed run.
+
+ Called by the ``workflow_control.pause_on_failure.v1`` activity that
+ MachinaWorkflow schedules on its failure path. Without it a trigger
+ keeps firing new runs into the same error indefinitely; with it the
+ deployment lands ``paused`` so the user fixes the cause and Resumes.
+
+ Gated on ``WORKFLOW_CONTROL_PAUSE_ON_FAILURE`` (default true) and
+ applies only to a generation currently ``running`` — direct manual
+ runs, legacy uncontrolled deployments, and already-paused
+ generations are untouched. The knob is evaluated HERE (activity
+ side) rather than inside workflow code so flipping it never touches
+ recorded workflow commands.
+ """
+ from core.container import container
+
+ if not workflow_id:
+ return {"paused": False, "reason": "no_workflow_id"}
+ if not bool(getattr(container.settings(), "workflow_control_pause_on_failure", True)):
+ return {"paused": False, "reason": "disabled"}
+ service = _control_service()
+ control = await service.database.get_latest_workflow_control(workflow_id)
+ if control is None or control.status != "running":
+ return {
+ "paused": False,
+ "reason": "not_running",
+ "status": getattr(control, "status", None),
+ }
+ logger.warning(
+ "Pausing deployment after failed run (circuit breaker)",
+ workflow_id=workflow_id,
+ generation=control.generation,
+ failure=reason,
+ )
+ try:
+ result = await handle_pause_workflow(
+ {
+ "workflow_id": workflow_id,
+ "expected_revision": control.revision,
+ "idempotency_key": f"pause-on-failure:{control.id}:{control.revision}",
+ },
+ None,
+ )
+ except Exception as exc: # noqa: BLE001 — activity reports, never raises
+ logger.warning(
+ "Pause-on-failure did not complete",
+ workflow_id=workflow_id,
+ error=str(exc),
+ )
+ return {"paused": False, "error": str(exc)}
+ paused = bool(isinstance(result, dict) and result.get("success"))
+ if not paused:
+ logger.warning(
+ "Pause-on-failure did not complete",
+ workflow_id=workflow_id,
+ error=(result or {}).get("error") if isinstance(result, dict) else None,
+ )
+ return {"paused": paused, "failure": reason}
+
+
WS_HANDLERS: Dict[str, Any] = {
"deploy_workflow": handle_deploy_workflow,
"cancel_deployment": handle_cancel_deployment,
@@ -1611,6 +1923,14 @@ WS_HANDLERS: Dict[str, Any] = {
}
+# Clean-shutdown marker for crash detection (see _consume_shutdown_state).
+# Registered through the generic lifespan shutdown-hook registry so main.py
+# never learns about the control plane's internals.
+from services.plugin.shutdown_hooks import register_shutdown_hook # noqa: E402
+
+register_shutdown_hook("workflow_control_clean_shutdown", mark_clean_shutdown)
+
+
__all__ = [
"WS_HANDLERS",
"handle_cancel_deployment",
@@ -1618,4 +1938,7 @@ __all__ = [
"handle_get_deployment_status",
"handle_get_workflow_lock",
"handle_update_deployment_settings",
+ "mark_clean_shutdown",
+ "pause_generation_on_failure",
+ "reconcile_active_controls_on_boot",
]
diff --git a/server/services/temporal/activities.py b/server/services/temporal/activities.py
index 0724edd2..5005d948 100644
--- a/server/services/temporal/activities.py
+++ b/server/services/temporal/activities.py
@@ -524,3 +524,32 @@ async def broadcast_trigger_status_activity(payload: Dict[str, Any]) -> None:
)
except Exception as exc: # noqa: BLE001 — non-fatal
activity.logger.warning(f"broadcast_trigger_status_activity failed for " f"node={payload.get('node_id')!r}: {exc}")
+
+
[email protected](name="workflow_control.pause_on_failure.v1")
+async def pause_workflow_on_failure_activity(payload: Dict[str, Any]) -> Dict[str, Any]:
+ """Circuit breaker: pause a controlled deployment after a failed run.
+
+ Scheduled by MachinaWorkflow's failure path (patch-gated). All policy
+ lives in ``services.deployment.handlers.pause_generation_on_failure``
+ — including the WORKFLOW_CONTROL_PAUSE_ON_FAILURE knob, evaluated on
+ the activity side so flipping it never touches recorded workflow
+ commands. Never raises; the result dict lands in workflow history
+ for ops inspection.
+
+ Payload shape:
+ {
+ "workflow_id": str, # OpenCompany deployment workflow_id
+ "reason": str, # first node error of the failed run
+ }
+ """
+ from services.deployment.handlers import pause_generation_on_failure
+
+ try:
+ return await pause_generation_on_failure(
+ workflow_id=str(payload.get("workflow_id") or ""),
+ reason=str(payload.get("reason") or "run_failed"),
+ )
+ except Exception as exc: # noqa: BLE001 — non-fatal
+ activity.logger.warning(f"pause_workflow_on_failure_activity failed for " f"workflow={payload.get('workflow_id')!r}: {exc}")
+ return {"paused": False, "error": str(exc)}
diff --git a/server/services/temporal/worker.py b/server/services/temporal/worker.py
index 07b42c60..0f6780c4 100644
--- a/server/services/temporal/worker.py
+++ b/server/services/temporal/worker.py
@@ -199,6 +199,7 @@ class TemporalWorkerManager:
from services.temporal.activities import (
broadcast_trigger_status_activity,
load_persisted_workflow_graph_activity,
+ pause_workflow_on_failure_activity,
store_node_output_activity,
)
@@ -252,6 +253,7 @@ class TemporalWorkerManager:
self._activities.execute_node_activity,
broadcast_trigger_status_activity,
load_persisted_workflow_graph_activity,
+ pause_workflow_on_failure_activity,
store_node_output_activity,
*per_type,
*agent_activities,
@@ -704,6 +706,7 @@ async def run_standalone_worker(
from services.temporal.activities import (
broadcast_trigger_status_activity,
load_persisted_workflow_graph_activity,
+ pause_workflow_on_failure_activity,
store_node_output_activity,
)
@@ -769,6 +772,7 @@ async def create_worker(
from services.temporal.activities import (
broadcast_trigger_status_activity,
load_persisted_workflow_graph_activity,
+ pause_workflow_on_failure_activity,
store_node_output_activity,
)
@@ -785,6 +789,7 @@ async def create_worker(
activities.execute_node_activity,
broadcast_trigger_status_activity,
load_persisted_workflow_graph_activity,
+ pause_workflow_on_failure_activity,
store_node_output_activity,
],
max_concurrent_activities=100,
diff --git a/server/services/temporal/workflow.py b/server/services/temporal/workflow.py
index 874adeb8..3566768e 100644
--- a/server/services/temporal/workflow.py
+++ b/server/services/temporal/workflow.py
@@ -16,7 +16,7 @@ from typing import Any, Dict, List, Optional, Set
from temporalio import workflow
-from ._retry_policies import DEFAULT_ACTIVITY_RETRY
+from ._retry_policies import DEFAULT_ACTIVITY_RETRY, QUICK_ACTIVITY_RETRY
from services.workflow_naming import node_label_slug
# Config handles - nodes connecting via these are config nodes (not executed)
@@ -105,6 +105,14 @@ COOPERATIVE_PAUSE_SCHEDULING_PATCH = (
# 30s), not by lifetime caps. Patch keeps replay compatibility for
# histories recorded with the old caps.
UNBOUNDED_LIFETIMES_PATCH = "machina-unbounded-lifetimes-v1"
+# Circuit breaker: a failed trigger-spawned run schedules the
+# workflow_control.pause_on_failure.v1 activity so the deployment pauses
+# (user fixes + resumes) instead of the trigger firing into the same
+# error indefinitely. The WORKFLOW_CONTROL_PAUSE_ON_FAILURE knob is
+# evaluated on the activity side, so flipping config never touches
+# recorded workflow commands; this patch only gates the activity
+# scheduling itself for replay compatibility with older histories.
+PAUSE_ON_FAILURE_PATCH = "machina-pause-on-failure-v1"
# start_to_close for node activities on the patched path. Generous by
# design: a single node step (agent turn batch, browser op, long shell
# command) may run for hours; worker-death detection is the heartbeat
@@ -163,6 +171,38 @@ class MachinaWorkflow:
if self._control_paused:
await workflow.wait_condition(lambda: not self._control_paused)
+ async def _pause_deployment_on_failure(
+ self,
+ workflow_data: Dict[str, Any],
+ nodes: List[Dict[str, Any]],
+ errors: List[Dict],
+ ) -> None:
+ """Circuit breaker: a failed trigger-spawned run pauses its deployment.
+
+ Only deployment-spawned runs qualify — they always carry a
+ ``_pre_executed`` firing trigger, which one-off manual canvas runs
+ never do, so a failed test run cannot pause a live deployment.
+ Legacy uncontrolled deployments no-op inside the activity (no
+ running control row). Non-fatal by design: the run's own failure
+ result is the primary outcome either way.
+ """
+ workflow_id = workflow_data.get("workflow_id")
+ spawned_by_trigger = any(node.get("_pre_executed") for node in nodes)
+ if not workflow_id or not spawned_by_trigger:
+ return
+ try:
+ await workflow.execute_activity(
+ "workflow_control.pause_on_failure.v1",
+ {
+ "workflow_id": workflow_id,
+ "reason": str((errors[0] or {}).get("error", "run_failed"))[:500],
+ },
+ start_to_close_timeout=timedelta(seconds=30),
+ retry_policy=QUICK_ACTIVITY_RETRY,
+ )
+ except Exception as exc: # noqa: BLE001 — cosmetic relative to the run result
+ workflow.logger.warning(f"pause-on-failure activity failed (non-fatal): {exc}")
+
@workflow.signal
async def on_event(self, event_payload: Dict[str, Any]) -> None:
"""Wave 12 A7: receive an event from ``services.events.dispatch.emit``.
@@ -275,6 +315,7 @@ class MachinaWorkflow:
COOPERATIVE_PAUSE_SCHEDULING_PATCH
)
use_unbounded_lifetimes = workflow.patched(UNBOUNDED_LIFETIMES_PATCH)
+ use_pause_on_failure = workflow.patched(PAUSE_ON_FAILURE_PATCH)
workflow.logger.info(f"Starting workflow orchestration: {len(nodes)} nodes, {len(edges)} edges")
@@ -519,6 +560,9 @@ class MachinaWorkflow:
# Build final result
success = len(errors) == 0 and len(completed) == len(node_map)
+ if errors and use_pause_on_failure:
+ await self._pause_deployment_on_failure(workflow_data, nodes, errors)
+
workflow.logger.info(f"Workflow complete: success={success}, " f"executed={len(execution_trace)}/{len(node_map)}")
return {
diff --git a/server/tests/services/test_workflow_control_handlers.py b/server/tests/services/test_workflow_control_handlers.py
index 479ddb1c..f1e0fcdf 100644
--- a/server/tests/services/test_workflow_control_handlers.py
+++ b/server/tests/services/test_workflow_control_handlers.py
@@ -1187,12 +1187,70 @@ async def test_terminal_generation_is_never_probed(monkeypatch, terminal_status)
@pytest.mark.asyncio
-async def test_live_generation_with_missing_controller_is_failed(monkeypatch):
- """Otherwise it stays non-'reset' forever and blocks every future Start."""
+async def test_live_generation_with_missing_controller_pauses_by_default(monkeypatch):
+ """WORKFLOW_CONTROL_MISSING_CONTROLLER=pause (default): a killed
+ controller leaves the generation user-resumable — Resume rebuilds it —
+ instead of forcing a Reset that archives conversation state."""
+ paused = _control("paused", 7)
+ transition = AsyncMock(return_value=paused)
+ fail = AsyncMock()
+ broadcast = AsyncMock(return_value={})
+
+ monkeypatch.setattr(handlers, "_missing_controller_policy", lambda: "pause")
+ monkeypatch.setattr(
+ handlers,
+ "_query_controller_state",
+ AsyncMock(side_effect=handlers.ControllerExecutionMissing("gone")),
+ )
+ monkeypatch.setattr(handlers, "_broadcast_control", broadcast)
+
+ control = _control("pausing", 6)
+ result, controller_status = await handlers._reconcile_control(
+ _reconcile_service(fail=fail, transition=transition), control
+ )
+
+ assert result is paused
+ assert controller_status is None
+ assert transition.await_args.kwargs["status"] == "paused"
+ fail.assert_not_awaited()
+ broadcast.assert_awaited_once()
+
+
+async def test_already_paused_generation_with_missing_controller_is_left_alone(
+ monkeypatch,
+):
+ """Paused is already the recoverable posture; Resume performs the rebuild."""
+ transition = AsyncMock()
+ fail = AsyncMock()
+ monkeypatch.setattr(handlers, "_missing_controller_policy", lambda: "pause")
+ monkeypatch.setattr(
+ handlers,
+ "_query_controller_state",
+ AsyncMock(side_effect=handlers.ControllerExecutionMissing("gone")),
+ )
+
+ control = _control("paused", 6)
+ result, _ = await handlers._reconcile_control(
+ _reconcile_service(fail=fail, transition=transition), control
+ )
+
+ assert result is control
+ transition.assert_not_awaited()
+ fail.assert_not_awaited()
+
+
+async def test_live_generation_with_missing_controller_fails_under_legacy_policy(
+ monkeypatch,
+):
+ """WORKFLOW_CONTROL_MISSING_CONTROLLER=fail preserves the Reset-only
+ behaviour; otherwise the row stays non-'reset' forever and blocks Start."""
failed = _control("failed", 7, terminal_reason="controller_execution_missing")
fail = AsyncMock(return_value=failed)
broadcast = AsyncMock(return_value={})
+ monkeypatch.setattr(handlers, "_missing_controller_policy", lambda: "fail")
monkeypatch.setattr(
handlers,
"_query_controller_state",
@@ -1211,6 +1269,30 @@ async def test_live_generation_with_missing_controller_is_failed(monkeypatch):
broadcast.assert_awaited_once()
+async def test_starting_with_missing_controller_still_fails_under_pause_policy(
+ monkeypatch,
+):
+ """Nothing durable runs yet mid-start; Reset + Start rebuilds cleanly."""
+ failed = _control("failed", 7, terminal_reason="controller_execution_missing")
+ fail = AsyncMock(return_value=failed)
+ monkeypatch.setattr(handlers, "_missing_controller_policy", lambda: "pause")
+ monkeypatch.setattr(
+ handlers,
+ "_query_controller_state",
+ AsyncMock(side_effect=handlers.ControllerExecutionMissing("gone")),
+ )
+ monkeypatch.setattr(handlers, "_broadcast_control", AsyncMock(return_value={}))
+
+ control = _control("starting", 6)
+ result, _ = await handlers._reconcile_control(
+ _reconcile_service(fail=fail), control
+ )
+
+ assert result is failed
+ assert fail.await_args.args[1] == "controller_execution_missing"
+
+
@pytest.mark.asyncio
async def test_resetting_is_not_auto_failed(monkeypatch):
"""Reset owns its own retry; auto-failing here could race a live reset."""
diff --git a/server/tests/temporal/test_durability_hardening.py b/server/tests/temporal/test_durability_hardening.py
index d5351361..3b2ccba5 100644
--- a/server/tests/temporal/test_durability_hardening.py
+++ b/server/tests/temporal/test_durability_hardening.py
@@ -56,6 +56,7 @@ def test_durability_changes_are_replay_patch_guarded():
expected = {
machina_module.UNBOUNDED_LIFETIMES_PATCH: inspect.getsource(MachinaWorkflow.run),
+ machina_module.PAUSE_ON_FAILURE_PATCH: inspect.getsource(MachinaWorkflow.run),
agent_module.UNBOUNDED_LIFETIMES_PATCH: inspect.getsource(AgentWorkflow.run),
trigger_module.UNBOUNDED_CHILD_RUNS_PATCH: inspect.getsource(
TriggerListenerWorkflow._spawn_child_run
@@ -80,6 +81,7 @@ def test_durability_changes_are_replay_patch_guarded():
assert set(expected) == {
"machina-unbounded-lifetimes-v1",
+ "machina-pause-on-failure-v1",
"agent-unbounded-lifetimes-v1",
"trigger-unbounded-child-runs-v1",
"polling-unbounded-child-runs-v1",
@@ -732,6 +734,7 @@ async def test_boot_reconcile_processes_each_active_row_with_isolation(monkeypat
list_active_workflow_controls=AsyncMock(return_value=[broken, healthy]),
)
monkeypatch.setattr(container, "database", MagicMock(return_value=database))
+ monkeypatch.setattr(handlers, "_consume_shutdown_state", AsyncMock(return_value=True))
async def reconcile(_service, control):
if control.workflow_id == "wf-broken":
@@ -747,3 +750,364 @@ async def test_boot_reconcile_processes_each_active_row_with_isolation(monkeypat
# The broken row was isolated; the healthy one was re-armed.
assert processed == 1
rearm.assert_awaited_once_with(healthy)
+
+
+# ---------------------------------------------------------------------------
+# Crash / kill recovery: unclean-shutdown detection + recovery pause.
+# ---------------------------------------------------------------------------
+
+
+async def test_shutdown_marker_protocol(monkeypatch):
+ """Dirty bit: boot stamps "running"; clean teardown stamps "clean";
+ a missing marker (first boot) counts as clean."""
+ from core.container import container
+ from services.deployment import handlers
+
+ store: dict = {}
+
+ async def get_entry(key):
+ value = store.get(key)
+ return SimpleNamespace(value=value) if value is not None else None
+
+ async def set_entry(key, value, *args, **kwargs):
+ store[key] = value
+
+ database = SimpleNamespace(get_cache_entry=get_entry, set_cache_entry=set_entry)
+ monkeypatch.setattr(container, "database", MagicMock(return_value=database))
+
+ # First boot: no marker -> clean; stamps the dirty bit.
+ assert await handlers._consume_shutdown_state() is True
+ assert store[handlers._SHUTDOWN_STATE_CACHE_KEY] == "running"
+ # Crash (no clean stamp) -> next boot detects unclean.
+ assert await handlers._consume_shutdown_state() is False
+ # Graceful teardown stamps clean -> next boot is clean again.
+ await handlers.mark_clean_shutdown()
+ assert store[handlers._SHUTDOWN_STATE_CACHE_KEY] == "clean"
+ assert await handlers._consume_shutdown_state() is True
+
+
+def test_clean_shutdown_hook_is_registered():
+ from services.plugin.shutdown_hooks import registered_labels
+
+ assert "workflow_control_clean_shutdown" in registered_labels()
+
+
+ ("clean", "policy", "expect_pause"),
+ [
+ (False, "pause", True), # crash + default policy -> recovery pause
+ (False, "resume", False), # crash + resume policy -> keep running
+ (True, "pause", False), # clean restart -> always restore as-is
+ ],
+)
+async def test_boot_recovery_pauses_running_rows_after_unclean_shutdown(
+ monkeypatch, clean, policy, expect_pause
+):
+ from core.container import container
+ from services.deployment import handlers
+
+ running = _boot_control("running")
+ paused = _boot_control("paused")
+ database = SimpleNamespace(
+ list_active_workflow_controls=AsyncMock(return_value=[running, paused]),
+ )
+ monkeypatch.setattr(container, "database", MagicMock(return_value=database))
+ monkeypatch.setattr(
+ handlers, "_consume_shutdown_state", AsyncMock(return_value=clean)
+ )
+ monkeypatch.setattr(handlers, "_crash_recovery_policy", lambda: policy)
+ monkeypatch.setattr(
+ handlers,
+ "_reconcile_control",
+ AsyncMock(side_effect=lambda _s, c: (c, {"state": c.status})),
+ )
+ monkeypatch.setattr(handlers, "_rearm_generation", AsyncMock())
+ recovery_pause = AsyncMock(side_effect=lambda _s, c, reason: c)
+ monkeypatch.setattr(handlers, "_pause_for_recovery", recovery_pause)
+
+ processed = await handlers.reconcile_active_controls_on_boot()
+
+ assert processed == 2
+ if expect_pause:
+ # Only the RUNNING row gets the recovery pause; already-paused
+ # generations stay untouched.
+ recovery_pause.assert_awaited_once()
+ assert recovery_pause.await_args.args[1] is running
+ else:
+ recovery_pause.assert_not_awaited()
+
+
+async def test_pause_for_recovery_reuses_the_full_pause_ceremony(monkeypatch):
+ from services.deployment import handlers
+
+ control = _boot_control("running")
+ paused = _boot_control("paused")
+ pause_handler = AsyncMock(return_value={"success": True, "state": "paused"})
+ monkeypatch.setattr(handlers, "handle_pause_workflow", pause_handler)
+ service = SimpleNamespace(
+ database=SimpleNamespace(
+ get_latest_workflow_control=AsyncMock(return_value=paused)
+ )
+ )
+
+ result = await handlers._pause_for_recovery(
+ service, control, reason="unclean_shutdown"
+ )
+
+ assert result is paused
+ request = pause_handler.await_args.args[0]
+ assert request["workflow_id"] == control.workflow_id
+ assert request["expected_revision"] == control.revision
+
+
+# ---------------------------------------------------------------------------
+# Killed controller: Resume rebuilds it from the durable row.
+# ---------------------------------------------------------------------------
+
+
+async def test_rebuild_missing_controller_uses_existing_conflict_policy(monkeypatch):
+ from core.container import container
+ from services.deployment import handlers
+
+ control = _boot_control("resuming")
+ updated = _boot_control("resuming")
+ updated.revision = 4
+ start_controller = AsyncMock(return_value="fresh-run-1")
+ monkeypatch.setattr(handlers, "_start_controller", start_controller)
+ rearm = AsyncMock()
+ monkeypatch.setattr(handlers, "_rearm_generation", rearm)
+ monkeypatch.setattr(handlers, "_broadcast_control", AsyncMock(return_value={}))
+ local_cancel = AsyncMock(return_value={"success": True})
+ monkeypatch.setattr(handlers, "handle_cancel_deployment", local_cancel)
+ workflow_service = MagicMock()
+ workflow_service.is_workflow_deployed.return_value = True
+ monkeypatch.setattr(
+ container, "workflow_service", MagicMock(return_value=workflow_service)
+ )
+ service = SimpleNamespace(
+ transition=AsyncMock(return_value=updated),
+ database=SimpleNamespace(
+ update_workflow_run_data_scope=AsyncMock(return_value=True)
+ ),
+ )
+
+ result = await handlers._rebuild_missing_controller(service, control)
+
+ assert result is updated
+ # Race-safe start: adopt a live controller or restart a terminated one.
+ assert start_controller.await_args.kwargs["use_existing"] is True
+ # Fresh run id persisted on the row (status unchanged).
+ assert service.transition.await_args.kwargs["values"] == {
+ "controller_run_id": "fresh-run-1"
+ }
+ # Stale local trigger state torn down, then re-armed against the
+ # fresh controller.
+ local_cancel.assert_awaited_once()
+ rearm.assert_awaited_once_with(updated)
+
+
+async def test_resume_rebuilds_a_gone_controller_then_reapplies_running(monkeypatch):
+ from core.container import container
+ from services.deployment import handlers
+
+ paused = _boot_control("paused")
+ resuming = _boot_control("resuming")
+ resuming.revision = 4
+ running = _boot_control("running")
+ running.revision = 5
+
+ class _Gone(Exception):
+ pass
+
+ service = SimpleNamespace(
+ database=SimpleNamespace(
+ get_latest_workflow_control=AsyncMock(return_value=paused)
+ ),
+ transition=AsyncMock(side_effect=[resuming, running]),
+ )
+ monkeypatch.setattr(handlers, "_control_service", lambda: service)
+ monkeypatch.setattr(
+ handlers, "_reconcile_control", AsyncMock(return_value=(paused, None))
+ )
+ monkeypatch.setattr(handlers, "_broadcast_control", AsyncMock(return_value={}))
+ monkeypatch.setattr(
+ handlers, "_control_payload", AsyncMock(return_value={"state": "running"})
+ )
+ update_state = AsyncMock(side_effect=[_Gone("not found"), {"state": "running"}])
+ monkeypatch.setattr(handlers, "_update_controller_state", update_state)
+ monkeypatch.setattr(handlers, "_temporal_target_already_gone", lambda exc: isinstance(exc, _Gone))
+ rebuild = AsyncMock(return_value=resuming)
+ monkeypatch.setattr(handlers, "_rebuild_missing_controller", rebuild)
+ monkeypatch.setattr(handlers, "_set_cron_pause", AsyncMock(return_value=0))
+ monkeypatch.setattr(
+ handlers, "_signal_generation_workflows", AsyncMock(return_value=0)
+ )
+ workflow_service = MagicMock()
+ workflow_service.resume_deployment = AsyncMock(return_value=0)
+ workflow_service.update_trigger_pause_status = AsyncMock(return_value=0)
+ monkeypatch.setattr(
+ container, "workflow_service", MagicMock(return_value=workflow_service)
+ )
+
+ result = await handlers.handle_resume_workflow(
+ {"workflow_id": "wf", "expected_revision": paused.revision}, None
+ )
+
+ assert result["success"] is True
+ rebuild.assert_awaited_once()
+ # The running update was re-applied against the rebuilt controller.
+ assert update_state.await_count == 2
+
+
+# ---------------------------------------------------------------------------
+# Circuit breaker: failed runs pause the deployment.
+# ---------------------------------------------------------------------------
+
+
+async def test_pause_on_failure_is_config_gated_and_requires_running(monkeypatch):
+ from core.container import container
+ from services.deployment import handlers
+
+ pause_handler = AsyncMock(return_value={"success": True})
+ monkeypatch.setattr(handlers, "handle_pause_workflow", pause_handler)
+
+ # Disabled by config -> untouched.
+ monkeypatch.setattr(
+ container,
+ "settings",
+ MagicMock(
+ return_value=SimpleNamespace(workflow_control_pause_on_failure=False)
+ ),
+ )
+ result = await handlers.pause_generation_on_failure(
+ workflow_id="wf", reason="boom"
+ )
+ assert result == {"paused": False, "reason": "disabled"}
+
+ # Enabled but the generation is already paused -> untouched.
+ monkeypatch.setattr(
+ container,
+ "settings",
+ MagicMock(
+ return_value=SimpleNamespace(workflow_control_pause_on_failure=True)
+ ),
+ )
+ service = SimpleNamespace(
+ database=SimpleNamespace(
+ get_latest_workflow_control=AsyncMock(return_value=_boot_control("paused"))
+ )
+ )
+ monkeypatch.setattr(handlers, "_control_service", lambda: service)
+ result = await handlers.pause_generation_on_failure(
+ workflow_id="wf", reason="boom"
+ )
+ assert result["paused"] is False
+ pause_handler.assert_not_awaited()
+
+
+async def test_pause_on_failure_pauses_a_running_generation(monkeypatch):
+ from core.container import container
+ from services.deployment import handlers
+
+ running = _boot_control("running")
+ monkeypatch.setattr(
+ container,
+ "settings",
+ MagicMock(
+ return_value=SimpleNamespace(workflow_control_pause_on_failure=True)
+ ),
+ )
+ service = SimpleNamespace(
+ database=SimpleNamespace(
+ get_latest_workflow_control=AsyncMock(return_value=running)
+ )
+ )
+ monkeypatch.setattr(handlers, "_control_service", lambda: service)
+ pause_handler = AsyncMock(return_value={"success": True, "state": "paused"})
+ monkeypatch.setattr(handlers, "handle_pause_workflow", pause_handler)
+
+ result = await handlers.pause_generation_on_failure(
+ workflow_id="wf", reason="node exploded"
+ )
+
+ assert result == {"paused": True, "failure": "node exploded"}
+ request = pause_handler.await_args.args[0]
+ assert request["workflow_id"] == "wf"
+ assert request["expected_revision"] == running.revision
+
+
+async def test_machina_failure_schedules_the_circuit_breaker_activity(monkeypatch):
+ from services.temporal import workflow as machina_module
+ from services.temporal.workflow import MachinaWorkflow
+
+ temporal_workflow = machina_module.workflow
+ monkeypatch.setattr(temporal_workflow, "logger", MagicMock())
+ captured: dict = {}
+
+ async def capture_activity(name, payload, **kwargs):
+ captured["name"] = name
+ captured["payload"] = payload
+
+ monkeypatch.setattr(temporal_workflow, "execute_activity", capture_activity)
+
+ instance = MachinaWorkflow()
+ nodes = [{"id": "t-1", "type": "webhookTrigger", "_pre_executed": True}]
+ errors = [{"node_id": "n-2", "error": "credential expired"}]
+
+ # Trigger-spawned run -> schedules the breaker.
+ await instance._pause_deployment_on_failure({"workflow_id": "wf"}, nodes, errors)
+ assert captured["name"] == "workflow_control.pause_on_failure.v1"
+ assert captured["payload"] == {
+ "workflow_id": "wf",
+ "reason": "credential expired",
+ }
+
+ # Manual run (no pre-executed trigger) -> never pauses a deployment.
+ captured.clear()
+ await instance._pause_deployment_on_failure(
+ {"workflow_id": "wf"}, [{"id": "n-1", "type": "aiAgent"}], errors
+ )
+ assert captured == {}
+
+ # No workflow_id -> nothing to pause.
+ await instance._pause_deployment_on_failure({}, nodes, errors)
+ assert captured == {}
+
+
+def test_recovery_policies_normalize_unknown_values(monkeypatch):
+ from core.container import container
+ from services.deployment import handlers
+
+ monkeypatch.setattr(
+ container,
+ "settings",
+ MagicMock(
+ return_value=SimpleNamespace(
+ workflow_control_crash_recovery="bogus",
+ workflow_control_missing_controller="also-bogus",
+ )
+ ),
+ )
+ assert handlers._crash_recovery_policy() == "pause"
+ assert handlers._missing_controller_policy() == "pause"
+
+ monkeypatch.setattr(
+ container,
+ "settings",
+ MagicMock(
+ return_value=SimpleNamespace(
+ workflow_control_crash_recovery="RESUME",
+ workflow_control_missing_controller="Fail",
+ )
+ ),
+ )
+ assert handlers._crash_recovery_policy() == "resume"
+ assert handlers._missing_controller_policy() == "fail"