Skip to content

Self-improving AI Employees for everyone

fix(control): circuit breaker trips on failure streaks, not single node hiccups

trohitgfd1921b4d9f4
diff --git a/.env.template b/.env.template
index 9f911eae..26eac11a 100644
--- a/.env.template
+++ b/.env.template
@@ -134,10 +134,16 @@ WORKFLOW_CONTROL_CRASH_RECOVERY=pause
# durable row + graph snapshot; ``fail`` preserves the legacy behaviour
# where only Reset (which archives conversation state) recovers.
WORKFLOW_CONTROL_MISSING_CONTROLLER=pause
-# Circuit breaker: when a controlled run fails with an error, pause the
+# Circuit breaker: when controlled runs keep failing, pause the
# deployment (fix + Resume) instead of letting the trigger keep firing
# into the same error indefinitely. ``false`` disables the auto-pause.
WORKFLOW_CONTROL_PAUSE_ON_FAILURE=true
+# The breaker trips only after this many failed runs inside the rolling
+# window below — one node hiccup on one firing (missing config,
+# transient API error) never pauses a deployment. Resume resets the
+# count; ``1`` restores pause-on-the-first-failure.
+WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD=3
+WORKFLOW_CONTROL_PAUSE_ON_FAILURE_WINDOW_SECONDS=600
# --- Startup resilience (server-ready-before-workers) ---
# After the gRPC port binds, the client polls the WorkflowService gRPC
diff --git a/CLAUDE.md b/CLAUDE.md
index dee6ddaa..c1ee5851 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -742,7 +742,7 @@ Node groups (palette categories): agent, model, skill, tool, trigger, workflow,
Workflows execute via Temporal for durability and horizontal scaling, gated by `TEMPORAL_ENABLED`. Three dispatch paths exist — legacy single `execute_node_activity` (fallback), per-type `node.{type}.v{version}` activities (F4.A, `TEMPORAL_PER_TYPE_DISPATCH=true`), and Agent-as-child-workflow (F4.B, `TEMPORAL_AGENT_WORKFLOW_ENABLED=true`) — with `rlm_agent`/`claude_code_agent` always bypassing AgentWorkflow. **Per-queue activity routing is production-default (Wave 16)**: a `TemporalWorkerPool` runs one activity-only worker per plugin-declared `cls.task_queue` (ai-heavy / browser / code-exec / rest-api / messaging / ...) with per-queue concurrency + rate limits + resource-based tuning for ai-heavy/browser; rollback via `TEMPORAL_WORKER_POOL_ENABLED=false`. Resilience knobs (Waves 17-18): `DEPLOYMENT_MODE` (local halves concurrency; sizes the sticky cache), one-shot retry on `agent.execute_llm_step.v1` (LLM calls are not idempotent), observability interceptors (`activity_retry` WARN on re-dispatch), 30s periodic heartbeat during long activity bodies, cron `catchup_window=24h`, poller autoscaling. Execution routing falls back Temporal → sequential; the Temporal dev server + embedded worker are managed in-process under `server/services/temporal/` (pooch-downloaded CLI, single `temporal server start-dev` process, gRPC `TEMPORAL_FRONTEND_GRPC_PORT` + Web UI `TEMPORAL_UI_PORT`, SQLite at `~/.opencompany/temporal.db`). Full architecture, dispatch matrix, per-node + agent-loop lifecycle, all 16 F4.B `agent.*.v1` activities, heartbeat semantics, delegation input contract, the worker tuning recipe, and all `.env` tunables live in [docs-internal/TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md).
-**Months-long durability contract (July 2026).** Running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. Load-bearing invariants: (1) `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` stays `false` (debug-only sweep; even enabled, any control row in the shared `WORKFLOW_CONTROL_ACTIVE_STATES` — including `resetting` — vetoes it). (2) New child workflow starts carry **no** `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so lifetime caps silently terminated paused runs; liveness comes from activity heartbeats (30s beat / 2min timeout), never from workflow lifetime caps. Removing/adding start options or retry policies on recorded commands requires a `workflow.patched(...)` guard (see the `*-unbounded-*` / `*-history-bounded-can-*` markers). (3) Every long-lived workflow loop must continue-as-new under history pressure (`is_continue_as_new_suggested()` + a soft cap) — Temporal hard-terminates around ~51,200 history events; `WorkflowControlWorkflow` carries triggers/queued events/seen-ids/control state across rollovers, so controller handles are addressed **by workflow id only, never run_id-pinned**. (4) `dispatch.emit` skips controllers whose `ControlEventTypes` Search Attribute doesn't match the event (absent attribute = legacy match-all). (5) The Temporal lifecycle is owned by [services/temporal/lifecycle.py](./server/services/temporal/lifecycle.py) (`main.py` only schedules it): dev-server supervision + resident watchdog (`TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS`), worker crash-restarts that REBUILD the single-use SDK `Worker` per attempt (manager + every pool queue worker), and the boot-time `reconcile_active_controls_on_boot` pass (converges crash-stranded `starting` rows, re-arms running/paused generations from the persisted graph snapshot). (6) **Recovery policies** (env-driven, defaults in `.env.template`; full semantics in [temporal-workflow-control.md → Recovery policies](./docs-internal/temporal-workflow-control.md)): `WORKFLOW_CONTROL_CRASH_RECOVERY=pause` — after a kill/crash (dirty-bit marker cleared only by graceful teardown) boot recovers running generations as **paused** so the user consciously resumes; `WORKFLOW_CONTROL_MISSING_CONTROLLER=pause` — a killed/vanished controller converges the generation to paused and **Resume rebuilds the controller** (`WorkflowIdConflictPolicy.USE_EXISTING`, re-arm from snapshot) instead of forcing a state-archiving Reset; `WORKFLOW_CONTROL_PAUSE_ON_FAILURE=true` — a failed trigger-spawned run pauses its deployment via the `workflow_control.pause_on_failure.v1` activity (circuit breaker; manual canvas runs never qualify). Temporal's native Pause/Unpause (server 1.28+) is deliberately NOT used — no Python SDK surface, and it halts workflow-task dispatch so a natively-paused controller couldn't process the resume Update. Locked by `tests/temporal/test_durability_hardening.py` + `test_worker_restart.py`.
+**Months-long durability contract (July 2026).** Running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. Load-bearing invariants: (1) `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` stays `false` (debug-only sweep; even enabled, any control row in the shared `WORKFLOW_CONTROL_ACTIVE_STATES` — including `resetting` — vetoes it). (2) New child workflow starts carry **no** `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so lifetime caps silently terminated paused runs; liveness comes from activity heartbeats (30s beat / 2min timeout), never from workflow lifetime caps. Removing/adding start options or retry policies on recorded commands requires a `workflow.patched(...)` guard (see the `*-unbounded-*` / `*-history-bounded-can-*` markers). (3) Every long-lived workflow loop must continue-as-new under history pressure (`is_continue_as_new_suggested()` + a soft cap) — Temporal hard-terminates around ~51,200 history events; `WorkflowControlWorkflow` carries triggers/queued events/seen-ids/control state across rollovers, so controller handles are addressed **by workflow id only, never run_id-pinned**. (4) `dispatch.emit` skips controllers whose `ControlEventTypes` Search Attribute doesn't match the event (absent attribute = legacy match-all). (5) The Temporal lifecycle is owned by [services/temporal/lifecycle.py](./server/services/temporal/lifecycle.py) (`main.py` only schedules it): dev-server supervision + resident watchdog (`TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS`), worker crash-restarts that REBUILD the single-use SDK `Worker` per attempt (manager + every pool queue worker), and the boot-time `reconcile_active_controls_on_boot` pass (converges crash-stranded `starting` rows, re-arms running/paused generations from the persisted graph snapshot). (6) **Recovery policies** (env-driven, defaults in `.env.template`; full semantics in [temporal-workflow-control.md → Recovery policies](./docs-internal/temporal-workflow-control.md)): `WORKFLOW_CONTROL_CRASH_RECOVERY=pause` — after a kill/crash (dirty-bit marker cleared only by graceful teardown) boot recovers running generations as **paused** so the user consciously resumes; `WORKFLOW_CONTROL_MISSING_CONTROLLER=pause` — a killed/vanished controller converges the generation to paused and **Resume rebuilds the controller** (`WorkflowIdConflictPolicy.USE_EXISTING`, re-arm from snapshot) instead of forcing a state-archiving Reset; `WORKFLOW_CONTROL_PAUSE_ON_FAILURE=true` — repeatedly-failing trigger-spawned runs pause their deployment via the `workflow_control.pause_on_failure.v1` activity (circuit breaker; trips only after `WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD` (default 3) failures inside a rolling `_WINDOW_SECONDS` (default 600) so one node hiccup never pauses anything; Resume resets the streak; manual canvas runs never qualify). Temporal's native Pause/Unpause (server 1.28+) is deliberately NOT used — no Python SDK surface, and it halts workflow-task dispatch so a natively-paused controller couldn't process the resume Update. Locked by `tests/temporal/test_durability_hardening.py` + `test_worker_restart.py`.
**Temporal tracing ownership is strict.** Register the SDK `TracingInterceptor` exactly once on the shared `Client.connect(...)`; the Python SDK automatically prepends compatible client interceptors to every worker. Never repeat `TracingInterceptor` in `Worker(..., interceptors=...)`. Worker lists contain the distinct `ObservabilityWorkerInterceptor` and plugin-specific worker interceptors only. Exact paired `StartActivity` / `RunActivity` / `CompleteWorkflow` console spans with identical Temporal identities indicate duplicate instrumentation, not duplicate execution; use Temporal Event History and the `node.<type>.execute` application span to verify actual attempts. See the tracing and debugging sections in [TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md) and [errors.md](./docs-internal/errors.md).
diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.md
index ec1eb506..db03a09f 100644
--- a/docs-internal/TEMPORAL_ARCHITECTURE.md
+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md
@@ -620,7 +620,9 @@ The Temporal binary + persistence are managed in-process by the plugin-folder pa
| `temporal_health_monitor_interval_seconds` | `TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS` | `15` | Resident dev-server watchdog probe cadence (lifecycle module; loopback deployments only). |
| `workflow_control_crash_recovery` | `WORKFLOW_CONTROL_CRASH_RECOVERY` | `pause` | After an UNCLEAN shutdown (kill/crash, dirty-bit marker), boot pauses generations still `running` so the user consciously resumes; `resume` restores them running. Clean restarts always restore as-is. |
| `workflow_control_missing_controller` | `WORKFLOW_CONTROL_MISSING_CONTROLLER` | `pause` | A live generation whose controller vanished converges to `paused` (Resume rebuilds the controller); `fail` preserves the legacy Reset-only behaviour. |
-| `workflow_control_pause_on_failure` | `WORKFLOW_CONTROL_PAUSE_ON_FAILURE` | `true` | Circuit breaker: a failed trigger-spawned run pauses its deployment (fix + Resume) instead of firing into the same error indefinitely. Evaluated activity-side; never touches recorded commands. |
+| `workflow_control_pause_on_failure` | `WORKFLOW_CONTROL_PAUSE_ON_FAILURE` | `true` | Circuit breaker: repeatedly-failing trigger-spawned runs pause their deployment (fix + Resume) instead of firing into the same error indefinitely. Evaluated activity-side; never touches recorded commands. |
+| `workflow_control_pause_on_failure_threshold` | `WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD` | `3` | Failed runs inside the rolling window required to trip the breaker — one node hiccup never pauses a deployment. `1` = pause on the first failure. Resume resets the streak. |
+| `workflow_control_pause_on_failure_window_seconds` | `WORKFLOW_CONTROL_PAUSE_ON_FAILURE_WINDOW_SECONDS` | `600` | Rolling window for the failure streak; older failures age out (streak state lives in the cache table with a matching TTL). |
Full recovery-policy semantics: [temporal-workflow-control.md → Recovery policies](./temporal-workflow-control.md#recovery-policies).
diff --git a/docs-internal/temporal-workflow-control.md b/docs-internal/temporal-workflow-control.md
index 63af6b75..403e68b0 100644
--- a/docs-internal/temporal-workflow-control.md
+++ b/docs-internal/temporal-workflow-control.md
@@ -197,14 +197,22 @@ that Resume relies on.
Start rebuilds cleanly). `fail` preserves the legacy Reset-only
behaviour.
- **`WORKFLOW_CONTROL_PAUSE_ON_FAILURE`** (default `true`): circuit
- breaker — when a trigger-spawned run fails with an error, MachinaWorkflow
+ breaker — when trigger-spawned runs keep failing, MachinaWorkflow
schedules `workflow_control.pause_on_failure.v1` (patch
`machina-pause-on-failure-v1`) and the deployment pauses so the user
fixes the cause and Resumes, instead of the trigger firing into the same
- error indefinitely. Only deployment-spawned runs qualify (they carry a
- `_pre_executed` firing trigger; a failed manual canvas run never pauses a
- live deployment), and the knob is evaluated on the activity side so
- flipping it never touches recorded workflow commands.
+ error indefinitely. The breaker trips only after
+ **`WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD`** (default `3`) failed
+ runs inside the rolling
+ **`WORKFLOW_CONTROL_PAUSE_ON_FAILURE_WINDOW_SECONDS`** (default `600`)
+ — one node hiccup on one firing (missing config, transient API error)
+ never pauses a deployment; `1` restores pause-on-the-first-failure. The
+ streak lives in the durable cache table keyed by the generation (Reset /
+ Start begins fresh), Resume resets it, and tripping clears it. Only
+ deployment-spawned runs qualify (they carry a `_pre_executed` firing
+ trigger; a failed manual canvas run never pauses a live deployment), and
+ every knob is evaluated on the activity side so flipping config never
+ touches recorded workflow commands.
## Delegated-task traces
diff --git a/server/core/config.py b/server/core/config.py
index 732396db..90cc3e56 100644
--- a/server/core/config.py
+++ b/server/core/config.py
@@ -275,6 +275,20 @@ class Settings(BaseSettings):
default=True,
env="WORKFLOW_CONTROL_PAUSE_ON_FAILURE",
)
+ # The breaker trips only after this many failed runs inside the rolling
+ # window below — one node hiccup on one firing (missing config,
+ # transient API error) never pauses a deployment. 1 = pause on the
+ # first failure.
+ workflow_control_pause_on_failure_threshold: int = Field(
+ default=3,
+ env="WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD",
+ ge=1,
+ )
+ workflow_control_pause_on_failure_window_seconds: float = Field(
+ default=600.0,
+ env="WORKFLOW_CONTROL_PAUSE_ON_FAILURE_WINDOW_SECONDS",
+ gt=0,
+ )
# API Keys (all optional, injected at runtime)
google_maps_api_key: Optional[str] = Field(default=None, env="GOOGLE_MAPS_API_KEY")
diff --git a/server/services/deployment/handlers.py b/server/services/deployment/handlers.py
index 24b2b2b5..6f495a05 100644
--- a/server/services/deployment/handlers.py
+++ b/server/services/deployment/handlers.py
@@ -21,6 +21,7 @@ same shape — only the import path changes.
from __future__ import annotations
import asyncio
+import json
import time
import uuid
from datetime import datetime, timezone
@@ -1077,12 +1078,12 @@ async def _consume_shutdown_state() -> bool:
A missing marker (first boot / pre-feature database) counts as clean
so a fresh install never boots into a recovery pause.
+ ``Database.get_cache_entry`` returns the stored string directly.
"""
from core.container import container
database = container.database()
- entry = await database.get_cache_entry(_SHUTDOWN_STATE_CACHE_KEY)
- previous = getattr(entry, "value", None) if entry is not None else None
+ previous = await database.get_cache_entry(_SHUTDOWN_STATE_CACHE_KEY)
await database.set_cache_entry(_SHUTDOWN_STATE_CACHE_KEY, "running")
return previous is None or previous == "clean"
@@ -1712,6 +1713,9 @@ async def handle_resume_workflow(data: Dict[str, Any], websocket: WebSocket) ->
queued = await container.workflow_service().resume_deployment(workflow_id)
resumed_triggers = await container.workflow_service().update_trigger_pause_status(workflow_id, paused=False)
control = await service.transition(control, expected_revision=control.revision, from_statuses={"resuming"}, status="running")
+ # Operator intervention resets the circuit-breaker streak — the next
+ # failure after a resume starts a fresh count, not a near-tripped one.
+ await _clear_failure_streak(service.database, control)
payload = await _broadcast_control(control, controller_status=controller_status, extra={
"resumed_queued_events": queued,
"signalled_executions": signalled,
@@ -1848,20 +1852,93 @@ async def handle_reset_workflow(data: Dict[str, Any], websocket: WebSocket) -> D
}
+def _pause_on_failure_threshold() -> int:
+ """Failures within the rolling window required to trip the breaker.
+
+ Default 3 — a single node hiccup on one firing (missing config,
+ transient API error) must never pause a live deployment; only a
+ sustained failure streak does. 1 restores pause-on-first-failure.
+ """
+ from core.container import container
+
+ try:
+ value = int(getattr(container.settings(), "workflow_control_pause_on_failure_threshold", 3))
+ except (TypeError, ValueError):
+ value = 3
+ return max(1, value)
+
+
+def _pause_on_failure_window_seconds() -> float:
+ """Rolling window for the failure streak; older failures age out."""
+ from core.container import container
+
+ try:
+ value = float(
+ getattr(container.settings(), "workflow_control_pause_on_failure_window_seconds", 600.0)
+ )
+ except (TypeError, ValueError):
+ value = 600.0
+ return max(1.0, value)
+
+
+def _failure_streak_key(control) -> str:
+ # Keyed by the control row id, which embeds workflow + generation —
+ # a Reset/Start naturally begins with a fresh streak.
+ return f"workflow_control:failure_streak:{control.id}"
+
+
+async def _bump_failure_streak(database, control, *, window_seconds: float) -> int:
+ """Increment the rolling-window failure counter and return it.
+
+ State lives in the durable cache table (value is a JSON blob; the
+ TTL doubles as the window so stale streaks age out even if the
+ timestamp check is bypassed). Concurrent failed runs can race the
+ read-modify-write and miscount by one — acceptable slack for a
+ breaker threshold.
+ """
+ key = _failure_streak_key(control)
+ now = time.time()
+ count = 0
+ raw = await database.get_cache_entry(key)
+ if raw:
+ try:
+ state = json.loads(raw)
+ if now - float(state.get("last_at") or 0.0) <= window_seconds:
+ count = int(state.get("count") or 0)
+ except (TypeError, ValueError):
+ count = 0
+ count += 1
+ await database.set_cache_entry(
+ key,
+ json.dumps({"count": count, "last_at": now}),
+ int(window_seconds),
+ )
+ return count
+
+
+async def _clear_failure_streak(database, control) -> None:
+ try:
+ await database.delete_cache_entry(_failure_streak_key(control))
+ except Exception as exc: # noqa: BLE001 — the TTL ages it out anyway
+ logger.debug(f"Failed to clear failure streak: {exc}")
+
+
async def pause_generation_on_failure(*, workflow_id: str, reason: str) -> Dict[str, Any]:
- """Circuit breaker: pause a controlled deployment after a failed run.
+ """Circuit breaker: pause a controlled deployment after failed runs.
Called by the ``workflow_control.pause_on_failure.v1`` activity that
MachinaWorkflow schedules on its failure path. Without it a trigger
keeps firing new runs into the same error indefinitely; with it the
deployment lands ``paused`` so the user fixes the cause and Resumes.
- Gated on ``WORKFLOW_CONTROL_PAUSE_ON_FAILURE`` (default true) and
- applies only to a generation currently ``running`` — direct manual
- runs, legacy uncontrolled deployments, and already-paused
- generations are untouched. The knob is evaluated HERE (activity
- side) rather than inside workflow code so flipping it never touches
- recorded workflow commands.
+ Trips only after ``WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD``
+ failed runs inside the rolling window — small things (one node
+ failing on one firing) never pause the deployment. Gated on
+ ``WORKFLOW_CONTROL_PAUSE_ON_FAILURE`` (default true) and applies
+ only to a generation currently ``running`` — direct manual runs,
+ legacy uncontrolled deployments, and already-paused generations are
+ untouched. All knobs are evaluated HERE (activity side) rather than
+ inside workflow code so config flips never touch recorded commands.
"""
from core.container import container
@@ -1877,6 +1954,28 @@ async def pause_generation_on_failure(*, workflow_id: str, reason: str) -> Dict[
"reason": "not_running",
"status": getattr(control, "status", None),
}
+ threshold = _pause_on_failure_threshold()
+ if threshold > 1:
+ streak = await _bump_failure_streak(
+ service.database,
+ control,
+ window_seconds=_pause_on_failure_window_seconds(),
+ )
+ if streak < threshold:
+ logger.info(
+ "Run failure below circuit-breaker threshold; not pausing",
+ workflow_id=workflow_id,
+ streak=streak,
+ threshold=threshold,
+ failure=reason,
+ )
+ return {
+ "paused": False,
+ "reason": "below_threshold",
+ "streak": streak,
+ "threshold": threshold,
+ }
+ await _clear_failure_streak(service.database, control)
logger.warning(
"Pausing deployment after failed run (circuit breaker)",
workflow_id=workflow_id,
diff --git a/server/tests/temporal/test_durability_hardening.py b/server/tests/temporal/test_durability_hardening.py
index 3b2ccba5..52c02e69 100644
--- a/server/tests/temporal/test_durability_hardening.py
+++ b/server/tests/temporal/test_durability_hardening.py
@@ -23,6 +23,8 @@ from __future__ import annotations
import asyncio
import inspect
+import json
+import time
from datetime import timedelta
from pathlib import Path
from types import SimpleNamespace
@@ -760,15 +762,20 @@ async def test_boot_reconcile_processes_each_active_row_with_isolation(monkeypat
@pytest.mark.asyncio
async def test_shutdown_marker_protocol(monkeypatch):
"""Dirty bit: boot stamps "running"; clean teardown stamps "clean";
- a missing marker (first boot) counts as clean."""
+ a missing marker (first boot) counts as clean.
+
+ The fake mirrors the real contract: ``Database.get_cache_entry``
+ returns the stored STRING directly, not an entry object — an
+ entry-shaped fake previously masked a bug where crash detection
+ could never fire against the real database.
+ """
from core.container import container
from services.deployment import handlers
store: dict = {}
async def get_entry(key):
- value = store.get(key)
- return SimpleNamespace(value=value) if value is not None else None
+ return store.get(key)
async def set_entry(key, value, *args, **kwargs):
store[key] = value
@@ -947,6 +954,8 @@ async def test_resume_rebuilds_a_gone_controller_then_reapplies_running(monkeypa
monkeypatch.setattr(
handlers, "_signal_generation_workflows", AsyncMock(return_value=0)
)
+ clear_streak = AsyncMock()
+ monkeypatch.setattr(handlers, "_clear_failure_streak", clear_streak)
workflow_service = MagicMock()
workflow_service.resume_deployment = AsyncMock(return_value=0)
workflow_service.update_trigger_pause_status = AsyncMock(return_value=0)
@@ -962,6 +971,8 @@ async def test_resume_rebuilds_a_gone_controller_then_reapplies_running(monkeypa
rebuild.assert_awaited_once()
# The running update was re-applied against the rebuilt controller.
assert update_state.await_count == 2
+ # Operator intervention resets the circuit-breaker streak.
+ clear_streak.assert_awaited_once()
# ---------------------------------------------------------------------------
@@ -1011,24 +1022,50 @@ async def test_pause_on_failure_is_config_gated_and_requires_running(monkeypatch
pause_handler.assert_not_awaited()
-async def test_pause_on_failure_pauses_a_running_generation(monkeypatch):
+def _streak_database(running, store: dict):
+ """Dict-backed cache fake honouring the real string-value contract."""
+
+ async def get_entry(key):
+ return store.get(key)
+
+ async def set_entry(key, value, *args, **kwargs):
+ store[key] = value
+
+ async def delete_entry(key):
+ store.pop(key, None)
+ return True
+
+ return SimpleNamespace(
+ get_latest_workflow_control=AsyncMock(return_value=running),
+ get_cache_entry=get_entry,
+ set_cache_entry=set_entry,
+ delete_cache_entry=delete_entry,
+ )
+
+
+def _breaker_settings(monkeypatch, *, threshold: int, window: float = 600.0):
from core.container import container
- from services.deployment import handlers
- running = _boot_control("running")
monkeypatch.setattr(
container,
"settings",
MagicMock(
- return_value=SimpleNamespace(workflow_control_pause_on_failure=True)
+ return_value=SimpleNamespace(
+ workflow_control_pause_on_failure=True,
+ workflow_control_pause_on_failure_threshold=threshold,
+ workflow_control_pause_on_failure_window_seconds=window,
+ )
),
)
- service = SimpleNamespace(
- database=SimpleNamespace(
- get_latest_workflow_control=AsyncMock(return_value=running)
- )
- )
+
+
+async def test_pause_on_failure_pauses_immediately_at_threshold_one(monkeypatch):
+ from services.deployment import handlers
+
+ running = _boot_control("running")
+ _breaker_settings(monkeypatch, threshold=1)
+ service = SimpleNamespace(database=_streak_database(running, {}))
monkeypatch.setattr(handlers, "_control_service", lambda: service)
pause_handler = AsyncMock(return_value={"success": True, "state": "paused"})
monkeypatch.setattr(handlers, "handle_pause_workflow", pause_handler)
@@ -1043,6 +1080,73 @@ async def test_pause_on_failure_pauses_a_running_generation(monkeypatch):
assert request["expected_revision"] == running.revision
+async def test_single_node_failure_does_not_trip_the_breaker(monkeypatch):
+ """The reported regression: one telegramSend NodeUserError on one
+ firing paused the whole deployment. Below the streak threshold the
+ breaker only counts — the next firing proceeds."""
+ from services.deployment import handlers
+
+ running = _boot_control("running")
+ store: dict = {}
+ _breaker_settings(monkeypatch, threshold=3)
+ service = SimpleNamespace(database=_streak_database(running, store))
+ monkeypatch.setattr(handlers, "_control_service", lambda: service)
+ pause_handler = AsyncMock()
+ monkeypatch.setattr(handlers, "handle_pause_workflow", pause_handler)
+
+ first = await handlers.pause_generation_on_failure(
+ workflow_id="wf", reason="Bot owner not detected"
+ )
+ second = await handlers.pause_generation_on_failure(
+ workflow_id="wf", reason="Bot owner not detected"
+ )
+
+ assert first == {
+ "paused": False,
+ "reason": "below_threshold",
+ "streak": 1,
+ "threshold": 3,
+ }
+ assert second["streak"] == 2
+ pause_handler.assert_not_awaited()
+
+ # The third failure inside the window trips the breaker and clears
+ # the streak so a post-fix failure starts a fresh count.
+ pause_handler.return_value = {"success": True, "state": "paused"}
+ third = await handlers.pause_generation_on_failure(
+ workflow_id="wf", reason="Bot owner not detected"
+ )
+ assert third["paused"] is True
+ pause_handler.assert_awaited_once()
+ assert handlers._failure_streak_key(running) not in store
+
+
+async def test_failure_streak_ages_out_beyond_the_window(monkeypatch):
+ from services.deployment import handlers
+
+ running = _boot_control("running")
+ store = {
+ handlers._failure_streak_key(running): json.dumps(
+ {"count": 2, "last_at": time.time() - 10_000}
+ )
+ }
+ _breaker_settings(monkeypatch, threshold=3, window=600.0)
+ service = SimpleNamespace(database=_streak_database(running, store))
+ monkeypatch.setattr(handlers, "_control_service", lambda: service)
+ pause_handler = AsyncMock()
+ monkeypatch.setattr(handlers, "handle_pause_workflow", pause_handler)
+
+ result = await handlers.pause_generation_on_failure(
+ workflow_id="wf", reason="sporadic blip"
+ )
+
+ # The stale streak restarted at 1 instead of tripping at 3.
+ assert result["streak"] == 1
+ pause_handler.assert_not_awaited()
+
+
@pytest.mark.asyncio
async def test_machina_failure_schedules_the_circuit_breaker_activity(monkeypatch):
from services.temporal import workflow as machina_module