Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
fix(control): circuit breaker trips on failure streaks, not single node hiccups
trohitgfd1921b4d9f4
diff --git a/.env.template b/.env.templateindex 9f911eae..26eac11a 100644--- a/.env.template+++ b/.env.template@@ -134,10 +134,16 @@ WORKFLOW_CONTROL_CRASH_RECOVERY=pause# durable row + graph snapshot; ``fail`` preserves the legacy behaviour# where only Reset (which archives conversation state) recovers.WORKFLOW_CONTROL_MISSING_CONTROLLER=pause-# Circuit breaker: when a controlled run fails with an error, pause the+# Circuit breaker: when controlled runs keep failing, pause the# deployment (fix + Resume) instead of letting the trigger keep firing# into the same error indefinitely. ``false`` disables the auto-pause.WORKFLOW_CONTROL_PAUSE_ON_FAILURE=true+# The breaker trips only after this many failed runs inside the rolling+# window below — one node hiccup on one firing (missing config,+# transient API error) never pauses a deployment. Resume resets the+# count; ``1`` restores pause-on-the-first-failure.+WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD=3+WORKFLOW_CONTROL_PAUSE_ON_FAILURE_WINDOW_SECONDS=600# --- Startup resilience (server-ready-before-workers) ---# After the gRPC port binds, the client polls the WorkflowService gRPCdiff --git a/CLAUDE.md b/CLAUDE.mdindex dee6ddaa..c1ee5851 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -742,7 +742,7 @@ Node groups (palette categories): agent, model, skill, tool, trigger, workflow,Workflows execute via Temporal for durability and horizontal scaling, gated by `TEMPORAL_ENABLED`. Three dispatch paths exist — legacy single `execute_node_activity` (fallback), per-type `node.{type}.v{version}` activities (F4.A, `TEMPORAL_PER_TYPE_DISPATCH=true`), and Agent-as-child-workflow (F4.B, `TEMPORAL_AGENT_WORKFLOW_ENABLED=true`) — with `rlm_agent`/`claude_code_agent` always bypassing AgentWorkflow. **Per-queue activity routing is production-default (Wave 16)**: a `TemporalWorkerPool` runs one activity-only worker per plugin-declared `cls.task_queue` (ai-heavy / browser / code-exec / rest-api / messaging / ...) with per-queue concurrency + rate limits + resource-based tuning for ai-heavy/browser; rollback via `TEMPORAL_WORKER_POOL_ENABLED=false`. Resilience knobs (Waves 17-18): `DEPLOYMENT_MODE` (local halves concurrency; sizes the sticky cache), one-shot retry on `agent.execute_llm_step.v1` (LLM calls are not idempotent), observability interceptors (`activity_retry` WARN on re-dispatch), 30s periodic heartbeat during long activity bodies, cron `catchup_window=24h`, poller autoscaling. Execution routing falls back Temporal → sequential; the Temporal dev server + embedded worker are managed in-process under `server/services/temporal/` (pooch-downloaded CLI, single `temporal server start-dev` process, gRPC `TEMPORAL_FRONTEND_GRPC_PORT` + Web UI `TEMPORAL_UI_PORT`, SQLite at `~/.opencompany/temporal.db`). Full architecture, dispatch matrix, per-node + agent-loop lifecycle, all 16 F4.B `agent.*.v1` activities, heartbeat semantics, delegation input contract, the worker tuning recipe, and all `.env` tunables live in [docs-internal/TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md).-**Months-long durability contract (July 2026).** Running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. Load-bearing invariants: (1) `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` stays `false` (debug-only sweep; even enabled, any control row in the shared `WORKFLOW_CONTROL_ACTIVE_STATES` — including `resetting` — vetoes it). (2) New child workflow starts carry **no** `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so lifetime caps silently terminated paused runs; liveness comes from activity heartbeats (30s beat / 2min timeout), never from workflow lifetime caps. Removing/adding start options or retry policies on recorded commands requires a `workflow.patched(...)` guard (see the `*-unbounded-*` / `*-history-bounded-can-*` markers). (3) Every long-lived workflow loop must continue-as-new under history pressure (`is_continue_as_new_suggested()` + a soft cap) — Temporal hard-terminates around ~51,200 history events; `WorkflowControlWorkflow` carries triggers/queued events/seen-ids/control state across rollovers, so controller handles are addressed **by workflow id only, never run_id-pinned**. (4) `dispatch.emit` skips controllers whose `ControlEventTypes` Search Attribute doesn't match the event (absent attribute = legacy match-all). (5) The Temporal lifecycle is owned by [services/temporal/lifecycle.py](./server/services/temporal/lifecycle.py) (`main.py` only schedules it): dev-server supervision + resident watchdog (`TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS`), worker crash-restarts that REBUILD the single-use SDK `Worker` per attempt (manager + every pool queue worker), and the boot-time `reconcile_active_controls_on_boot` pass (converges crash-stranded `starting` rows, re-arms running/paused generations from the persisted graph snapshot). (6) **Recovery policies** (env-driven, defaults in `.env.template`; full semantics in [temporal-workflow-control.md → Recovery policies](./docs-internal/temporal-workflow-control.md)): `WORKFLOW_CONTROL_CRASH_RECOVERY=pause` — after a kill/crash (dirty-bit marker cleared only by graceful teardown) boot recovers running generations as **paused** so the user consciously resumes; `WORKFLOW_CONTROL_MISSING_CONTROLLER=pause` — a killed/vanished controller converges the generation to paused and **Resume rebuilds the controller** (`WorkflowIdConflictPolicy.USE_EXISTING`, re-arm from snapshot) instead of forcing a state-archiving Reset; `WORKFLOW_CONTROL_PAUSE_ON_FAILURE=true` — a failed trigger-spawned run pauses its deployment via the `workflow_control.pause_on_failure.v1` activity (circuit breaker; manual canvas runs never qualify). Temporal's native Pause/Unpause (server 1.28+) is deliberately NOT used — no Python SDK surface, and it halts workflow-task dispatch so a natively-paused controller couldn't process the resume Update. Locked by `tests/temporal/test_durability_hardening.py` + `test_worker_restart.py`.+**Months-long durability contract (July 2026).** Running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. Load-bearing invariants: (1) `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` stays `false` (debug-only sweep; even enabled, any control row in the shared `WORKFLOW_CONTROL_ACTIVE_STATES` — including `resetting` — vetoes it). (2) New child workflow starts carry **no** `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so lifetime caps silently terminated paused runs; liveness comes from activity heartbeats (30s beat / 2min timeout), never from workflow lifetime caps. Removing/adding start options or retry policies on recorded commands requires a `workflow.patched(...)` guard (see the `*-unbounded-*` / `*-history-bounded-can-*` markers). (3) Every long-lived workflow loop must continue-as-new under history pressure (`is_continue_as_new_suggested()` + a soft cap) — Temporal hard-terminates around ~51,200 history events; `WorkflowControlWorkflow` carries triggers/queued events/seen-ids/control state across rollovers, so controller handles are addressed **by workflow id only, never run_id-pinned**. (4) `dispatch.emit` skips controllers whose `ControlEventTypes` Search Attribute doesn't match the event (absent attribute = legacy match-all). (5) The Temporal lifecycle is owned by [services/temporal/lifecycle.py](./server/services/temporal/lifecycle.py) (`main.py` only schedules it): dev-server supervision + resident watchdog (`TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS`), worker crash-restarts that REBUILD the single-use SDK `Worker` per attempt (manager + every pool queue worker), and the boot-time `reconcile_active_controls_on_boot` pass (converges crash-stranded `starting` rows, re-arms running/paused generations from the persisted graph snapshot). (6) **Recovery policies** (env-driven, defaults in `.env.template`; full semantics in [temporal-workflow-control.md → Recovery policies](./docs-internal/temporal-workflow-control.md)): `WORKFLOW_CONTROL_CRASH_RECOVERY=pause` — after a kill/crash (dirty-bit marker cleared only by graceful teardown) boot recovers running generations as **paused** so the user consciously resumes; `WORKFLOW_CONTROL_MISSING_CONTROLLER=pause` — a killed/vanished controller converges the generation to paused and **Resume rebuilds the controller** (`WorkflowIdConflictPolicy.USE_EXISTING`, re-arm from snapshot) instead of forcing a state-archiving Reset; `WORKFLOW_CONTROL_PAUSE_ON_FAILURE=true` — repeatedly-failing trigger-spawned runs pause their deployment via the `workflow_control.pause_on_failure.v1` activity (circuit breaker; trips only after `WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD` (default 3) failures inside a rolling `_WINDOW_SECONDS` (default 600) so one node hiccup never pauses anything; Resume resets the streak; manual canvas runs never qualify). Temporal's native Pause/Unpause (server 1.28+) is deliberately NOT used — no Python SDK surface, and it halts workflow-task dispatch so a natively-paused controller couldn't process the resume Update. Locked by `tests/temporal/test_durability_hardening.py` + `test_worker_restart.py`.**Temporal tracing ownership is strict.** Register the SDK `TracingInterceptor` exactly once on the shared `Client.connect(...)`; the Python SDK automatically prepends compatible client interceptors to every worker. Never repeat `TracingInterceptor` in `Worker(..., interceptors=...)`. Worker lists contain the distinct `ObservabilityWorkerInterceptor` and plugin-specific worker interceptors only. Exact paired `StartActivity` / `RunActivity` / `CompleteWorkflow` console spans with identical Temporal identities indicate duplicate instrumentation, not duplicate execution; use Temporal Event History and the `node.<type>.execute` application span to verify actual attempts. See the tracing and debugging sections in [TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md) and [errors.md](./docs-internal/errors.md).diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.mdindex ec1eb506..db03a09f 100644--- a/docs-internal/TEMPORAL_ARCHITECTURE.md+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md@@ -620,7 +620,9 @@ The Temporal binary + persistence are managed in-process by the plugin-folder pa| `temporal_health_monitor_interval_seconds` | `TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS` | `15` | Resident dev-server watchdog probe cadence (lifecycle module; loopback deployments only). || `workflow_control_crash_recovery` | `WORKFLOW_CONTROL_CRASH_RECOVERY` | `pause` | After an UNCLEAN shutdown (kill/crash, dirty-bit marker), boot pauses generations still `running` so the user consciously resumes; `resume` restores them running. Clean restarts always restore as-is. || `workflow_control_missing_controller` | `WORKFLOW_CONTROL_MISSING_CONTROLLER` | `pause` | A live generation whose controller vanished converges to `paused` (Resume rebuilds the controller); `fail` preserves the legacy Reset-only behaviour. |-| `workflow_control_pause_on_failure` | `WORKFLOW_CONTROL_PAUSE_ON_FAILURE` | `true` | Circuit breaker: a failed trigger-spawned run pauses its deployment (fix + Resume) instead of firing into the same error indefinitely. Evaluated activity-side; never touches recorded commands. |+| `workflow_control_pause_on_failure` | `WORKFLOW_CONTROL_PAUSE_ON_FAILURE` | `true` | Circuit breaker: repeatedly-failing trigger-spawned runs pause their deployment (fix + Resume) instead of firing into the same error indefinitely. Evaluated activity-side; never touches recorded commands. |+| `workflow_control_pause_on_failure_threshold` | `WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD` | `3` | Failed runs inside the rolling window required to trip the breaker — one node hiccup never pauses a deployment. `1` = pause on the first failure. Resume resets the streak. |+| `workflow_control_pause_on_failure_window_seconds` | `WORKFLOW_CONTROL_PAUSE_ON_FAILURE_WINDOW_SECONDS` | `600` | Rolling window for the failure streak; older failures age out (streak state lives in the cache table with a matching TTL). |Full recovery-policy semantics: [temporal-workflow-control.md → Recovery policies](./temporal-workflow-control.md#recovery-policies).diff --git a/docs-internal/temporal-workflow-control.md b/docs-internal/temporal-workflow-control.mdindex 63af6b75..403e68b0 100644--- a/docs-internal/temporal-workflow-control.md+++ b/docs-internal/temporal-workflow-control.md@@ -197,14 +197,22 @@ that Resume relies on.Start rebuilds cleanly). `fail` preserves the legacy Reset-onlybehaviour.- **`WORKFLOW_CONTROL_PAUSE_ON_FAILURE`** (default `true`): circuit- breaker — when a trigger-spawned run fails with an error, MachinaWorkflow+ breaker — when trigger-spawned runs keep failing, MachinaWorkflowschedules `workflow_control.pause_on_failure.v1` (patch`machina-pause-on-failure-v1`) and the deployment pauses so the userfixes the cause and Resumes, instead of the trigger firing into the same- error indefinitely. Only deployment-spawned runs qualify (they carry a- `_pre_executed` firing trigger; a failed manual canvas run never pauses a- live deployment), and the knob is evaluated on the activity side so- flipping it never touches recorded workflow commands.+ error indefinitely. The breaker trips only after+ **`WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD`** (default `3`) failed+ runs inside the rolling+ **`WORKFLOW_CONTROL_PAUSE_ON_FAILURE_WINDOW_SECONDS`** (default `600`)+ — one node hiccup on one firing (missing config, transient API error)+ never pauses a deployment; `1` restores pause-on-the-first-failure. The+ streak lives in the durable cache table keyed by the generation (Reset /+ Start begins fresh), Resume resets it, and tripping clears it. Only+ deployment-spawned runs qualify (they carry a `_pre_executed` firing+ trigger; a failed manual canvas run never pauses a live deployment), and+ every knob is evaluated on the activity side so flipping config never+ touches recorded workflow commands.## Delegated-task tracesdiff --git a/server/core/config.py b/server/core/config.pyindex 732396db..90cc3e56 100644--- a/server/core/config.py+++ b/server/core/config.py@@ -275,6 +275,20 @@ class Settings(BaseSettings):default=True,env="WORKFLOW_CONTROL_PAUSE_ON_FAILURE",)+ # The breaker trips only after this many failed runs inside the rolling+ # window below — one node hiccup on one firing (missing config,+ # transient API error) never pauses a deployment. 1 = pause on the+ # first failure.+ workflow_control_pause_on_failure_threshold: int = Field(+ default=3,+ env="WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD",+ ge=1,+ )+ workflow_control_pause_on_failure_window_seconds: float = Field(+ default=600.0,+ env="WORKFLOW_CONTROL_PAUSE_ON_FAILURE_WINDOW_SECONDS",+ gt=0,+ )# API Keys (all optional, injected at runtime)google_maps_api_key: Optional[str] = Field(default=None, env="GOOGLE_MAPS_API_KEY")diff --git a/server/services/deployment/handlers.py b/server/services/deployment/handlers.pyindex 24b2b2b5..6f495a05 100644--- a/server/services/deployment/handlers.py+++ b/server/services/deployment/handlers.py@@ -21,6 +21,7 @@ same shape — only the import path changes.from __future__ import annotationsimport asyncio+import jsonimport timeimport uuidfrom datetime import datetime, timezone@@ -1077,12 +1078,12 @@ async def _consume_shutdown_state() -> bool:A missing marker (first boot / pre-feature database) counts as cleanso a fresh install never boots into a recovery pause.+ ``Database.get_cache_entry`` returns the stored string directly."""from core.container import containerdatabase = container.database()- entry = await database.get_cache_entry(_SHUTDOWN_STATE_CACHE_KEY)- previous = getattr(entry, "value", None) if entry is not None else None+ previous = await database.get_cache_entry(_SHUTDOWN_STATE_CACHE_KEY)await database.set_cache_entry(_SHUTDOWN_STATE_CACHE_KEY, "running")return previous is None or previous == "clean"@@ -1712,6 +1713,9 @@ async def handle_resume_workflow(data: Dict[str, Any], websocket: WebSocket) ->queued = await container.workflow_service().resume_deployment(workflow_id)resumed_triggers = await container.workflow_service().update_trigger_pause_status(workflow_id, paused=False)control = await service.transition(control, expected_revision=control.revision, from_statuses={"resuming"}, status="running")+ # Operator intervention resets the circuit-breaker streak — the next+ # failure after a resume starts a fresh count, not a near-tripped one.+ await _clear_failure_streak(service.database, control)payload = await _broadcast_control(control, controller_status=controller_status, extra={"resumed_queued_events": queued,"signalled_executions": signalled,@@ -1848,20 +1852,93 @@ async def handle_reset_workflow(data: Dict[str, Any], websocket: WebSocket) -> D}+def _pause_on_failure_threshold() -> int:+ """Failures within the rolling window required to trip the breaker.++ Default 3 — a single node hiccup on one firing (missing config,+ transient API error) must never pause a live deployment; only a+ sustained failure streak does. 1 restores pause-on-first-failure.+ """+ from core.container import container++ try:+ value = int(getattr(container.settings(), "workflow_control_pause_on_failure_threshold", 3))+ except (TypeError, ValueError):+ value = 3+ return max(1, value)+++def _pause_on_failure_window_seconds() -> float:+ """Rolling window for the failure streak; older failures age out."""+ from core.container import container++ try:+ value = float(+ getattr(container.settings(), "workflow_control_pause_on_failure_window_seconds", 600.0)+ )+ except (TypeError, ValueError):+ value = 600.0+ return max(1.0, value)+++def _failure_streak_key(control) -> str:+ # Keyed by the control row id, which embeds workflow + generation —+ # a Reset/Start naturally begins with a fresh streak.+ return f"workflow_control:failure_streak:{control.id}"+++async def _bump_failure_streak(database, control, *, window_seconds: float) -> int:+ """Increment the rolling-window failure counter and return it.++ State lives in the durable cache table (value is a JSON blob; the+ TTL doubles as the window so stale streaks age out even if the+ timestamp check is bypassed). Concurrent failed runs can race the+ read-modify-write and miscount by one — acceptable slack for a+ breaker threshold.+ """+ key = _failure_streak_key(control)+ now = time.time()+ count = 0+ raw = await database.get_cache_entry(key)+ if raw:+ try:+ state = json.loads(raw)+ if now - float(state.get("last_at") or 0.0) <= window_seconds:+ count = int(state.get("count") or 0)+ except (TypeError, ValueError):+ count = 0+ count += 1+ await database.set_cache_entry(+ key,+ json.dumps({"count": count, "last_at": now}),+ int(window_seconds),+ )+ return count+++async def _clear_failure_streak(database, control) -> None:+ try:+ await database.delete_cache_entry(_failure_streak_key(control))+ except Exception as exc: # noqa: BLE001 — the TTL ages it out anyway+ logger.debug(f"Failed to clear failure streak: {exc}")++async def pause_generation_on_failure(*, workflow_id: str, reason: str) -> Dict[str, Any]:- """Circuit breaker: pause a controlled deployment after a failed run.+ """Circuit breaker: pause a controlled deployment after failed runs.Called by the ``workflow_control.pause_on_failure.v1`` activity thatMachinaWorkflow schedules on its failure path. Without it a triggerkeeps firing new runs into the same error indefinitely; with it thedeployment lands ``paused`` so the user fixes the cause and Resumes.- Gated on ``WORKFLOW_CONTROL_PAUSE_ON_FAILURE`` (default true) and- applies only to a generation currently ``running`` — direct manual- runs, legacy uncontrolled deployments, and already-paused- generations are untouched. The knob is evaluated HERE (activity- side) rather than inside workflow code so flipping it never touches- recorded workflow commands.+ Trips only after ``WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD``+ failed runs inside the rolling window — small things (one node+ failing on one firing) never pause the deployment. Gated on+ ``WORKFLOW_CONTROL_PAUSE_ON_FAILURE`` (default true) and applies+ only to a generation currently ``running`` — direct manual runs,+ legacy uncontrolled deployments, and already-paused generations are+ untouched. All knobs are evaluated HERE (activity side) rather than+ inside workflow code so config flips never touch recorded commands."""from core.container import container@@ -1877,6 +1954,28 @@ async def pause_generation_on_failure(*, workflow_id: str, reason: str) -> Dict["reason": "not_running","status": getattr(control, "status", None),}+ threshold = _pause_on_failure_threshold()+ if threshold > 1:+ streak = await _bump_failure_streak(+ service.database,+ control,+ window_seconds=_pause_on_failure_window_seconds(),+ )+ if streak < threshold:+ logger.info(+ "Run failure below circuit-breaker threshold; not pausing",+ workflow_id=workflow_id,+ streak=streak,+ threshold=threshold,+ failure=reason,+ )+ return {+ "paused": False,+ "reason": "below_threshold",+ "streak": streak,+ "threshold": threshold,+ }+ await _clear_failure_streak(service.database, control)logger.warning("Pausing deployment after failed run (circuit breaker)",workflow_id=workflow_id,diff --git a/server/tests/temporal/test_durability_hardening.py b/server/tests/temporal/test_durability_hardening.pyindex 3b2ccba5..52c02e69 100644--- a/server/tests/temporal/test_durability_hardening.py+++ b/server/tests/temporal/test_durability_hardening.py@@ -23,6 +23,8 @@ from __future__ import annotationsimport asyncioimport inspect+import json+import timefrom datetime import timedeltafrom pathlib import Pathfrom types import SimpleNamespace@@ -760,15 +762,20 @@ async def test_boot_reconcile_processes_each_active_row_with_isolation(monkeypat@pytest.mark.asyncioasync def test_shutdown_marker_protocol(monkeypatch):"""Dirty bit: boot stamps "running"; clean teardown stamps "clean";- a missing marker (first boot) counts as clean."""+ a missing marker (first boot) counts as clean.++ The fake mirrors the real contract: ``Database.get_cache_entry``+ returns the stored STRING directly, not an entry object — an+ entry-shaped fake previously masked a bug where crash detection+ could never fire against the real database.+ """from core.container import containerfrom services.deployment import handlersstore: dict = {}async def get_entry(key):- value = store.get(key)- return SimpleNamespace(value=value) if value is not None else None+ return store.get(key)async def set_entry(key, value, *args, **kwargs):store[key] = value@@ -947,6 +954,8 @@ async def test_resume_rebuilds_a_gone_controller_then_reapplies_running(monkeypamonkeypatch.setattr(handlers, "_signal_generation_workflows", AsyncMock(return_value=0))+ clear_streak = AsyncMock()+ monkeypatch.setattr(handlers, "_clear_failure_streak", clear_streak)workflow_service = MagicMock()workflow_service.resume_deployment = AsyncMock(return_value=0)workflow_service.update_trigger_pause_status = AsyncMock(return_value=0)@@ -962,6 +971,8 @@ async def test_resume_rebuilds_a_gone_controller_then_reapplies_running(monkeyparebuild.assert_awaited_once()# The running update was re-applied against the rebuilt controller.assert update_state.await_count == 2+ # Operator intervention resets the circuit-breaker streak.+ clear_streak.assert_awaited_once()# ---------------------------------------------------------------------------@@ -1011,24 +1022,50 @@ async def test_pause_on_failure_is_config_gated_and_requires_running(monkeypatchpause_handler.assert_not_awaited()-async def test_pause_on_failure_pauses_a_running_generation(monkeypatch):+def _streak_database(running, store: dict):+ """Dict-backed cache fake honouring the real string-value contract."""++ async def get_entry(key):+ return store.get(key)++ async def set_entry(key, value, *args, **kwargs):+ store[key] = value++ async def delete_entry(key):+ store.pop(key, None)+ return True++ return SimpleNamespace(+ get_latest_workflow_control=AsyncMock(return_value=running),+ get_cache_entry=get_entry,+ set_cache_entry=set_entry,+ delete_cache_entry=delete_entry,+ )+++def _breaker_settings(monkeypatch, *, threshold: int, window: float = 600.0):from core.container import container- from services.deployment import handlers- running = _boot_control("running")monkeypatch.setattr(container,"settings",MagicMock(- return_value=SimpleNamespace(workflow_control_pause_on_failure=True)+ return_value=SimpleNamespace(+ workflow_control_pause_on_failure=True,+ workflow_control_pause_on_failure_threshold=threshold,+ workflow_control_pause_on_failure_window_seconds=window,+ )),)- service = SimpleNamespace(- database=SimpleNamespace(- get_latest_workflow_control=AsyncMock(return_value=running)- )- )+++async def test_pause_on_failure_pauses_immediately_at_threshold_one(monkeypatch):+ from services.deployment import handlers++ running = _boot_control("running")+ _breaker_settings(monkeypatch, threshold=1)+ service = SimpleNamespace(database=_streak_database(running, {}))monkeypatch.setattr(handlers, "_control_service", lambda: service)pause_handler = AsyncMock(return_value={"success": True, "state": "paused"})monkeypatch.setattr(handlers, "handle_pause_workflow", pause_handler)@@ -1043,6 +1080,73 @@ async def test_pause_on_failure_pauses_a_running_generation(monkeypatch):assert request["expected_revision"] == running.revision+async def test_single_node_failure_does_not_trip_the_breaker(monkeypatch):+ """The reported regression: one telegramSend NodeUserError on one+ firing paused the whole deployment. Below the streak threshold the+ breaker only counts — the next firing proceeds."""+ from services.deployment import handlers++ running = _boot_control("running")+ store: dict = {}+ _breaker_settings(monkeypatch, threshold=3)+ service = SimpleNamespace(database=_streak_database(running, store))+ monkeypatch.setattr(handlers, "_control_service", lambda: service)+ pause_handler = AsyncMock()+ monkeypatch.setattr(handlers, "handle_pause_workflow", pause_handler)++ first = await handlers.pause_generation_on_failure(+ workflow_id="wf", reason="Bot owner not detected"+ )+ second = await handlers.pause_generation_on_failure(+ workflow_id="wf", reason="Bot owner not detected"+ )++ assert first == {+ "paused": False,+ "reason": "below_threshold",+ "streak": 1,+ "threshold": 3,+ }+ assert second["streak"] == 2+ pause_handler.assert_not_awaited()++ # The third failure inside the window trips the breaker and clears+ # the streak so a post-fix failure starts a fresh count.+ pause_handler.return_value = {"success": True, "state": "paused"}+ third = await handlers.pause_generation_on_failure(+ workflow_id="wf", reason="Bot owner not detected"+ )+ assert third["paused"] is True+ pause_handler.assert_awaited_once()+ assert handlers._failure_streak_key(running) not in store+++async def test_failure_streak_ages_out_beyond_the_window(monkeypatch):+ from services.deployment import handlers++ running = _boot_control("running")+ store = {+ handlers._failure_streak_key(running): json.dumps(+ {"count": 2, "last_at": time.time() - 10_000}+ )+ }+ _breaker_settings(monkeypatch, threshold=3, window=600.0)+ service = SimpleNamespace(database=_streak_database(running, store))+ monkeypatch.setattr(handlers, "_control_service", lambda: service)+ pause_handler = AsyncMock()+ monkeypatch.setattr(handlers, "handle_pause_workflow", pause_handler)++ result = await handlers.pause_generation_on_failure(+ workflow_id="wf", reason="sporadic blip"+ )++ # The stale streak restarted at 1 instead of tripping at 3.+ assert result["streak"] == 1+ pause_handler.assert_not_awaited()++@pytest.mark.asyncioasync def test_machina_failure_schedules_the_circuit_breaker_activity(monkeypatch):from services.temporal import workflow as machina_module