Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
fix(temporal): controller queue paging retries without limit (1b.16)
trohitg3e0d91f5b62c
diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.mdindex 553ec2e6..8691d6a5 100644--- a/docs-internal/TEMPORAL_ARCHITECTURE.md+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md@@ -1086,7 +1086,7 @@ guarantee applies to version 1 generations with their controller chain intact.- **dispatch.emit controller narrowing.** Controllers advertise their push event types via the `ControlEventTypes` keyword-list Search Attribute (upserted as triggers register); `dispatch.emit` skips controllers with no matching trigger instead of signalling every running controller with every platform event (each unmatched signal was ~4 immutable history events — one busy deployment burned every other controller's rollover budget). Controllers without the attribute (pre-upgrade histories) keep match-all behaviour.- **Boot-time reconcile** (`reconcile_active_controls_on_boot`, called from the lifecycle module after workers start): runs the lazy `_reconcile_control` over every active control row, converges `starting` rows a crash left behind (controller alive with triggers registered → `running`; alive-but-empty for a graph that declares triggers → `failed` + controller closed; vanished → `failed`), and re-arms the process-local half of running/paused generations from the persisted graph snapshot — DeploymentManager state, in-process collectors for non-canary trigger types, cron pause posture. Idempotent by construction (controller `register_trigger` keyed by listener id, legacy starts use `USE_EXISTING`, cron creation preserves server-owned pause state).-**Controller event accumulation.** `controller-durable-queue-v2` preserves FIFO across carried events and the existing durable spillway. New execution-control version 1 histories also record `controller-event-accumulator-v1`: pending IDs remain protected beyond the recent-ID window, rollover includes Signals received during spill Activities, and a duplicate deterministic child start cannot block subsequent events. Legacy and pre-marker histories retain their recorded paths. Trigger processing stays immediate; no inactivity batching is added. See [event accumulation during Stop and rollover](./temporal-workflow-control.md#event-accumulation-during-stop-and-rollover) for scope and delivery limits.+**Controller event accumulation.** `controller-durable-queue-v2` preserves FIFO across carried events and the existing durable spillway. New execution-control version 1 histories also record `controller-event-accumulator-v1`: pending IDs remain protected beyond the recent-ID window, rollover includes Signals received during spill Activities, and a duplicate deterministic child start cannot block subsequent events. Legacy and pre-marker histories retain their recorded paths. Trigger processing stays immediate; no inactivity batching is added. The queue-paging activities (`controller.queue.read`, `controller.queue.spill`) retry without limit (`QUEUE_PAGING_RETRY`): both are idempotent, so a database outage cannot fail a controller that a version 1 generation could not rebuild, and paging that keeps failing shows as an attention item. See [event accumulation during Stop and rollover](./temporal-workflow-control.md#event-accumulation-during-stop-and-rollover) for scope and delivery limits.**Controller Update traffic.** Fresh version 1 histories record `controller-messaging-v1` so successful Updates can request history-pressure rollover even without event Signals or pollers. The same gate prevents producer-only Safe Apply compatibility messages from reopening a stopped generation. Pure Update validators reject malformed control requests before acceptance; the main loop waits for handlers before rollover. See [workflow messaging review](./temporal-workflow-control.md#workflow-messaging-review).diff --git a/server/services/temporal/_retry_policies.py b/server/services/temporal/_retry_policies.pyindex 7760f321..98399cd2 100644--- a/server/services/temporal/_retry_policies.py+++ b/server/services/temporal/_retry_policies.py@@ -151,6 +151,22 @@ REVIEWED_DELIVERY_RETRY: RetryPolicy = RetryPolicy()+# Controller queue paging (``controller.queue.read`` and+# ``controller.queue.spill``) is awaited by WorkflowControlWorkflow outside any+# try. An exhausted policy fails the controller, and a version 1 generation+# cannot rebuild one (``handlers._rebuild_missing_controller``), so a short+# database outage would end the deployment. Both are idempotent (a read is+# keyed by its mutation id, a spill by the event's hash), so they retry until+# the database answers; one that keeps failing shows as an attention item+# (the ``activity_retry`` interceptor, limit 10).+QUEUE_PAGING_RETRY: RetryPolicy = RetryPolicy(+ initial_interval=timedelta(seconds=1),+ maximum_interval=timedelta(seconds=30),+ maximum_attempts=0,+ non_retryable_error_types=list(NON_RETRYABLE_ERROR_TYPES),+)++# Keep the existing Browser task policies unchanged for Temporal replay.# A busy claim fails immediately; cleanup must wait for its owner to return.BROWSER_TASK_CLAIM_RETRY: RetryPolicy = RetryPolicy(maximum_attempts=1)@@ -183,6 +199,7 @@ __all__ = ["DELEGATION_CLEANUP_RETRY","PERMIT_WAIT_RETRY","REVIEWED_DELIVERY_RETRY",+ "QUEUE_PAGING_RETRY","BROWSER_TASK_CLAIM_RETRY","BROWSER_TASK_CLEANUP_RETRY","frozen_retry_policy",diff --git a/server/services/temporal/workflow_control_workflow.py b/server/services/temporal/workflow_control_workflow.pyindex 3bebb2c1..428e925a 100644--- a/server/services/temporal/workflow_control_workflow.py+++ b/server/services/temporal/workflow_control_workflow.py@@ -9,7 +9,7 @@ from temporalio.common import SearchAttributeKey, SearchAttributePairfrom temporalio.exceptions import ApplicationError, WorkflowAlreadyStartedErrorfrom services.temporal._exits import is_workflow_bug-from services.temporal._retry_policies import DEFAULT_ACTIVITY_RETRY+from services.temporal._retry_policies import DEFAULT_ACTIVITY_RETRY, QUEUE_PAGING_RETRYfrom services.temporal.execution_control import ExecutionControl@@ -410,7 +410,10 @@ class WorkflowControlWorkflow:"controller_id": workflow.info().workflow_id,"request_id": f"{workflow.info().run_id}:queue:{self._queue_reads}",**({"review_listener_ids": [listener_id for listener_id, spec in self._triggers.items() if (spec.get("listener_args") or {}).get("node_type") == "taskTrigger"]} if read_reviews else {})},- start_to_close_timeout=timedelta(seconds=60), retry_policy=DEFAULT_ACTIVITY_RETRY)+ # Paging retries without limit: an exhausted policy would+ # fail the controller, which a version 1 generation cannot+ # rebuild.+ start_to_close_timeout=timedelta(seconds=60), retry_policy=QUEUE_PAGING_RETRY)incoming = [(str(item[0]), item[1]) for item in page["events"]]if queue_v2:if accumulator_v1:@@ -637,7 +640,7 @@ class WorkflowControlWorkflow:async def spill(batch, prepend=False):base = {"controller_id": workflow.info().workflow_id, "prepend": prepend}async def invoke(payload):- await workflow.execute_activity("controller.queue.spill", payload, start_to_close_timeout=timedelta(seconds=60), retry_policy=DEFAULT_ACTIVITY_RETRY)+ await workflow.execute_activity("controller.queue.spill", payload, start_to_close_timeout=timedelta(seconds=60), retry_policy=QUEUE_PAGING_RETRY)if len(batch) == 1:encoded = json.dumps(batch[0], separators=(",", ":"), ensure_ascii=True)if len(encoded.encode()) > 512_000:@@ -687,7 +690,7 @@ class WorkflowControlWorkflow:await workflow.execute_activity("controller.queue.spill", {"controller_id": workflow.info().workflow_id,"events": [list(item) for item in self._events[_MAX_CARRIED_EVENTS:]]},- start_to_close_timeout=timedelta(seconds=60), retry_policy=DEFAULT_ACTIVITY_RETRY)+ start_to_close_timeout=timedelta(seconds=60), retry_policy=QUEUE_PAGING_RETRY)self._overflow = Truecarry = self._events[:_MAX_CARRIED_EVENTS]if self._execution_control.enabled and not accumulator_v1:diff --git a/server/tests/temporal/test_activity_interceptor.py b/server/tests/temporal/test_activity_interceptor.pyindex eedf191b..27cba65b 100644--- a/server/tests/temporal/test_activity_interceptor.py+++ b/server/tests/temporal/test_activity_interceptor.py@@ -75,6 +75,13 @@ async def test_no_item_up_to_the_limit_or_for_a_planned_wait(store, activity_typstore.resolve.assert_not_awaited()[email protected]("activity_type", ["controller.queue.read", "controller.queue.spill"])+async def test_controller_paging_past_its_limit_is_counted_not_a_planned_wait(store, activity_type):+ # 1b.16: paging retries without limit, so a stuck outage must show on Home.+ await _attempt(activity_type, 11)+ store.touch.assert_awaited_once_with("stuck_retry", "wf:act-1")++async def test_the_retry_log_is_quiet_between_its_warnings(store):await _attempt("node.httpRequest.v1", 2)_interceptors.logger.warning.assert_called_once()diff --git a/server/tests/test_retry_policies.py b/server/tests/test_retry_policies.pyindex 821dc6c1..37298e0b 100644--- a/server/tests/test_retry_policies.py+++ b/server/tests/test_retry_policies.py@@ -114,6 +114,38 @@ class TestReviewedDeliveryRetry:assert policies == ["REVIEWED_DELIVERY_RETRY"] * count+class TestQueuePagingRetry:+ """1b.16: controller queue paging retries without limit."""++ def test_the_policy_is_unlimited_and_keeps_the_non_retryable_list(self):+ from datetime import timedelta++ from services.temporal._retry_policies import NON_RETRYABLE_ERROR_TYPES, QUEUE_PAGING_RETRY++ assert QUEUE_PAGING_RETRY.maximum_attempts == 0+ assert QUEUE_PAGING_RETRY.initial_interval == timedelta(seconds=1)+ assert QUEUE_PAGING_RETRY.maximum_interval == timedelta(seconds=30)+ assert list(QUEUE_PAGING_RETRY.non_retryable_error_types) == list(NON_RETRYABLE_ERROR_TYPES)++ def test_every_paging_call_uses_it(self):+ import ast+ import pathlib++ path = pathlib.Path(__file__).resolve().parent.parent / "services/temporal/workflow_control_workflow.py"+ tree = ast.parse(path.read_text(encoding="utf-8"))+ policies = [+ keyword.value.id+ for node in ast.walk(tree)+ if isinstance(node, ast.Call) and getattr(node.func, "attr", None) == "execute_activity"+ and node.args and isinstance(node.args[0], ast.Constant)+ and node.args[0].value in {"controller.queue.read", "controller.queue.spill"}+ for keyword in node.keywords+ if keyword.arg == "retry_policy" and isinstance(keyword.value, ast.Name)+ ]+ # The read, the batch and chunk spill, and the legacy spill.+ assert policies == ["QUEUE_PAGING_RETRY"] * 3++class TestNoDuplicatedInlineRetryPolicy:"""Workflow files must import the shared constant — not re-type``RetryPolicy(initial_interval=..., maximum_interval=..., maximum_attempts=N)``