Skip to content

Self-improving AI Employees for everyone

fix(temporal): controller queue paging retries without limit (1b.16)

trohitg3e0d91f5b62c
diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.md
index 553ec2e6..8691d6a5 100644
--- a/docs-internal/TEMPORAL_ARCHITECTURE.md
+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md
@@ -1086,7 +1086,7 @@ guarantee applies to version 1 generations with their controller chain intact.
- **dispatch.emit controller narrowing.** Controllers advertise their push event types via the `ControlEventTypes` keyword-list Search Attribute (upserted as triggers register); `dispatch.emit` skips controllers with no matching trigger instead of signalling every running controller with every platform event (each unmatched signal was ~4 immutable history events — one busy deployment burned every other controller's rollover budget). Controllers without the attribute (pre-upgrade histories) keep match-all behaviour.
- **Boot-time reconcile** (`reconcile_active_controls_on_boot`, called from the lifecycle module after workers start): runs the lazy `_reconcile_control` over every active control row, converges `starting` rows a crash left behind (controller alive with triggers registered → `running`; alive-but-empty for a graph that declares triggers → `failed` + controller closed; vanished → `failed`), and re-arms the process-local half of running/paused generations from the persisted graph snapshot — DeploymentManager state, in-process collectors for non-canary trigger types, cron pause posture. Idempotent by construction (controller `register_trigger` keyed by listener id, legacy starts use `USE_EXISTING`, cron creation preserves server-owned pause state).
-**Controller event accumulation.** `controller-durable-queue-v2` preserves FIFO across carried events and the existing durable spillway. New execution-control version 1 histories also record `controller-event-accumulator-v1`: pending IDs remain protected beyond the recent-ID window, rollover includes Signals received during spill Activities, and a duplicate deterministic child start cannot block subsequent events. Legacy and pre-marker histories retain their recorded paths. Trigger processing stays immediate; no inactivity batching is added. See [event accumulation during Stop and rollover](./temporal-workflow-control.md#event-accumulation-during-stop-and-rollover) for scope and delivery limits.
+**Controller event accumulation.** `controller-durable-queue-v2` preserves FIFO across carried events and the existing durable spillway. New execution-control version 1 histories also record `controller-event-accumulator-v1`: pending IDs remain protected beyond the recent-ID window, rollover includes Signals received during spill Activities, and a duplicate deterministic child start cannot block subsequent events. Legacy and pre-marker histories retain their recorded paths. Trigger processing stays immediate; no inactivity batching is added. The queue-paging activities (`controller.queue.read`, `controller.queue.spill`) retry without limit (`QUEUE_PAGING_RETRY`): both are idempotent, so a database outage cannot fail a controller that a version 1 generation could not rebuild, and paging that keeps failing shows as an attention item. See [event accumulation during Stop and rollover](./temporal-workflow-control.md#event-accumulation-during-stop-and-rollover) for scope and delivery limits.
**Controller Update traffic.** Fresh version 1 histories record `controller-messaging-v1` so successful Updates can request history-pressure rollover even without event Signals or pollers. The same gate prevents producer-only Safe Apply compatibility messages from reopening a stopped generation. Pure Update validators reject malformed control requests before acceptance; the main loop waits for handlers before rollover. See [workflow messaging review](./temporal-workflow-control.md#workflow-messaging-review).
diff --git a/server/services/temporal/_retry_policies.py b/server/services/temporal/_retry_policies.py
index 7760f321..98399cd2 100644
--- a/server/services/temporal/_retry_policies.py
+++ b/server/services/temporal/_retry_policies.py
@@ -151,6 +151,22 @@ REVIEWED_DELIVERY_RETRY: RetryPolicy = RetryPolicy(
)
+# Controller queue paging (``controller.queue.read`` and
+# ``controller.queue.spill``) is awaited by WorkflowControlWorkflow outside any
+# try. An exhausted policy fails the controller, and a version 1 generation
+# cannot rebuild one (``handlers._rebuild_missing_controller``), so a short
+# database outage would end the deployment. Both are idempotent (a read is
+# keyed by its mutation id, a spill by the event's hash), so they retry until
+# the database answers; one that keeps failing shows as an attention item
+# (the ``activity_retry`` interceptor, limit 10).
+QUEUE_PAGING_RETRY: RetryPolicy = RetryPolicy(
+ initial_interval=timedelta(seconds=1),
+ maximum_interval=timedelta(seconds=30),
+ maximum_attempts=0,
+ non_retryable_error_types=list(NON_RETRYABLE_ERROR_TYPES),
+)
+
+
# Keep the existing Browser task policies unchanged for Temporal replay.
# A busy claim fails immediately; cleanup must wait for its owner to return.
BROWSER_TASK_CLAIM_RETRY: RetryPolicy = RetryPolicy(maximum_attempts=1)
@@ -183,6 +199,7 @@ __all__ = [
"DELEGATION_CLEANUP_RETRY",
"PERMIT_WAIT_RETRY",
"REVIEWED_DELIVERY_RETRY",
+ "QUEUE_PAGING_RETRY",
"BROWSER_TASK_CLAIM_RETRY",
"BROWSER_TASK_CLEANUP_RETRY",
"frozen_retry_policy",
diff --git a/server/services/temporal/workflow_control_workflow.py b/server/services/temporal/workflow_control_workflow.py
index 3bebb2c1..428e925a 100644
--- a/server/services/temporal/workflow_control_workflow.py
+++ b/server/services/temporal/workflow_control_workflow.py
@@ -9,7 +9,7 @@ from temporalio.common import SearchAttributeKey, SearchAttributePair
from temporalio.exceptions import ApplicationError, WorkflowAlreadyStartedError
from services.temporal._exits import is_workflow_bug
-from services.temporal._retry_policies import DEFAULT_ACTIVITY_RETRY
+from services.temporal._retry_policies import DEFAULT_ACTIVITY_RETRY, QUEUE_PAGING_RETRY
from services.temporal.execution_control import ExecutionControl
@@ -410,7 +410,10 @@ class WorkflowControlWorkflow:
"controller_id": workflow.info().workflow_id,
"request_id": f"{workflow.info().run_id}:queue:{self._queue_reads}",
**({"review_listener_ids": [listener_id for listener_id, spec in self._triggers.items() if (spec.get("listener_args") or {}).get("node_type") == "taskTrigger"]} if read_reviews else {})},
- start_to_close_timeout=timedelta(seconds=60), retry_policy=DEFAULT_ACTIVITY_RETRY)
+ # Paging retries without limit: an exhausted policy would
+ # fail the controller, which a version 1 generation cannot
+ # rebuild.
+ start_to_close_timeout=timedelta(seconds=60), retry_policy=QUEUE_PAGING_RETRY)
incoming = [(str(item[0]), item[1]) for item in page["events"]]
if queue_v2:
if accumulator_v1:
@@ -637,7 +640,7 @@ class WorkflowControlWorkflow:
async def spill(batch, prepend=False):
base = {"controller_id": workflow.info().workflow_id, "prepend": prepend}
async def invoke(payload):
- await workflow.execute_activity("controller.queue.spill", payload, start_to_close_timeout=timedelta(seconds=60), retry_policy=DEFAULT_ACTIVITY_RETRY)
+ await workflow.execute_activity("controller.queue.spill", payload, start_to_close_timeout=timedelta(seconds=60), retry_policy=QUEUE_PAGING_RETRY)
if len(batch) == 1:
encoded = json.dumps(batch[0], separators=(",", ":"), ensure_ascii=True)
if len(encoded.encode()) > 512_000:
@@ -687,7 +690,7 @@ class WorkflowControlWorkflow:
await workflow.execute_activity("controller.queue.spill", {
"controller_id": workflow.info().workflow_id,
"events": [list(item) for item in self._events[_MAX_CARRIED_EVENTS:]]},
- start_to_close_timeout=timedelta(seconds=60), retry_policy=DEFAULT_ACTIVITY_RETRY)
+ start_to_close_timeout=timedelta(seconds=60), retry_policy=QUEUE_PAGING_RETRY)
self._overflow = True
carry = self._events[:_MAX_CARRIED_EVENTS]
if self._execution_control.enabled and not accumulator_v1:
diff --git a/server/tests/temporal/test_activity_interceptor.py b/server/tests/temporal/test_activity_interceptor.py
index eedf191b..27cba65b 100644
--- a/server/tests/temporal/test_activity_interceptor.py
+++ b/server/tests/temporal/test_activity_interceptor.py
@@ -75,6 +75,13 @@ async def test_no_item_up_to_the_limit_or_for_a_planned_wait(store, activity_typ
store.resolve.assert_not_awaited()
[email protected]("activity_type", ["controller.queue.read", "controller.queue.spill"])
+async def test_controller_paging_past_its_limit_is_counted_not_a_planned_wait(store, activity_type):
+ # 1b.16: paging retries without limit, so a stuck outage must show on Home.
+ await _attempt(activity_type, 11)
+ store.touch.assert_awaited_once_with("stuck_retry", "wf:act-1")
+
+
async def test_the_retry_log_is_quiet_between_its_warnings(store):
await _attempt("node.httpRequest.v1", 2)
_interceptors.logger.warning.assert_called_once()
diff --git a/server/tests/test_retry_policies.py b/server/tests/test_retry_policies.py
index 821dc6c1..37298e0b 100644
--- a/server/tests/test_retry_policies.py
+++ b/server/tests/test_retry_policies.py
@@ -114,6 +114,38 @@ class TestReviewedDeliveryRetry:
assert policies == ["REVIEWED_DELIVERY_RETRY"] * count
+class TestQueuePagingRetry:
+ """1b.16: controller queue paging retries without limit."""
+
+ def test_the_policy_is_unlimited_and_keeps_the_non_retryable_list(self):
+ from datetime import timedelta
+
+ from services.temporal._retry_policies import NON_RETRYABLE_ERROR_TYPES, QUEUE_PAGING_RETRY
+
+ assert QUEUE_PAGING_RETRY.maximum_attempts == 0
+ assert QUEUE_PAGING_RETRY.initial_interval == timedelta(seconds=1)
+ assert QUEUE_PAGING_RETRY.maximum_interval == timedelta(seconds=30)
+ assert list(QUEUE_PAGING_RETRY.non_retryable_error_types) == list(NON_RETRYABLE_ERROR_TYPES)
+
+ def test_every_paging_call_uses_it(self):
+ import ast
+ import pathlib
+
+ path = pathlib.Path(__file__).resolve().parent.parent / "services/temporal/workflow_control_workflow.py"
+ tree = ast.parse(path.read_text(encoding="utf-8"))
+ policies = [
+ keyword.value.id
+ for node in ast.walk(tree)
+ if isinstance(node, ast.Call) and getattr(node.func, "attr", None) == "execute_activity"
+ and node.args and isinstance(node.args[0], ast.Constant)
+ and node.args[0].value in {"controller.queue.read", "controller.queue.spill"}
+ for keyword in node.keywords
+ if keyword.arg == "retry_policy" and isinstance(keyword.value, ast.Name)
+ ]
+ # The read, the batch and chunk spill, and the legacy spill.
+ assert policies == ["QUEUE_PAGING_RETRY"] * 3
+
+
class TestNoDuplicatedInlineRetryPolicy:
"""Workflow files must import the shared constant — not re-type
``RetryPolicy(initial_interval=..., maximum_interval=..., maximum_attempts=N)``