Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
fix(temporal): an agent's tool call uses its plugin's declared options on every path (1b.3)
trohitg0b0225f04714
diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.mdindex b1e2e753..1c11cabc 100644--- a/docs-internal/TEMPORAL_ARCHITECTURE.md+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md@@ -127,7 +127,7 @@ This invokes `run_standalone_worker()` from `services/temporal/worker.py`. It alOrdinary nodes and the external steps of agent child Workflows run as Activities with:- **Own context** - No shared mutable state between nodes- **Own retry policy** - Plugin declarations take precedence over ordinary defaults. Agent LLM steps use unlimited transient retries; a declared one-attempt tool keeps that policy.-- **Own timeout** - Ordinary graph dispatch uses a 24 h ceiling and 2-minute heartbeat; Workspace-task nodes and version 1 agent-tool dispatch consume plugin values. Different Activities can progress independently.+- **Own timeout** - Ordinary graph dispatch uses a 24 h ceiling and 2-minute heartbeat; Workspace-task nodes and agent-tool dispatch consume plugin values. Different Activities can progress independently.- **Own worker** - Can execute on any available worker in the cluster### 2. Workflow = Pure Orchestrator@@ -635,8 +635,13 @@ explicit refresh produce these bindings; continue-as-new carries the resolvedset. Scheduling uses those recorded declarations, with specialized queuerouting only when the frozen worker-pool flag is on. A plugin update duringStop or worker restart does not implicitly replace an already-bound policy.-Legacy histories keep their prior scheduling options, including the existing-Workspace-task policy patch.+A tool call outside such a generation (a manual Run, or a history from before+version 1) uses the plugin's declared timeout, heartbeat and retry too, read+from the registered class (`agent_workflow._declared_tool_options`; BaseNode's+for a type no plugin registers), with the default cancellation and queue. That+is an options change, so it needs no marker, and those histories keep their+commands; the Workspace-task policy patch still adds its cancellation and+queue.`collect_agent_activities()` registers the agent activities — read the live setfrom that function rather than trusting a count here, which drifts on every@@ -752,7 +757,7 @@ cleanup.- **Browser history.** `WorkspaceTaskRecord` is a projection, not the execution authority. Admission and lifecycle writes are idempotent and terminal states cannot regress. Recent tasks contain prompt, timestamps, bounded safe result/error and authorized relative artifact links, excluding tool transcripts and credentials. Pages default to 20 (maximum 100); terminal records expire after 35 days, while active records and history survive Reset. Workflow authorization protects historical records after node deletion. Retained terminal records also prevent resubmission after controller dedup or Temporal history expiration.- **Reset.** The controller's `reset` Update first fences new submissions by advancing the epoch. It then cancels every admitted child and waits for each to close, and runs the `workspace_tasks.reset_runtime` activity (`services/temporal/workspace_task_activities.py::reset_workspace_task_runtime`). That activity cancels invocations started before the controller existed and calls each Workspace-task node's `reset_execution_state`. The Update returns only after that cleanup. If cleanup fails, the fence stays in place, and retrying Reset resumes the same request. The workflow toolbar's Reset reaches the controller through `services/deployment/handlers.py::_reset_workspace_tasks`, even for a workflow that was never started; see [temporal-workflow-control.md](./temporal-workflow-control.md#workspace-tasks).- **Registration and rollover.** Both workflow classes are in `worker.py::_framework_workflows`, and every framework worker registers `reset_workspace_task_runtime`. The controller continues-as-new under history pressure, carrying its epoch and its bounded submission and reset records.-- **Inside graph and agent runs.** Two patch markers give Workspace-task nodes their own activity options. `workspace-node-cancellation-v1` (`MachinaWorkflow`) uses the node class's `start_to_close_timeout` and `heartbeat_timeout`, and makes cancellation wait until the activity has finished cleaning up. `workspace-task-activity-policy-v1` (tool calls in `AgentWorkflow`) does the same and also applies the class's `retry_policy` and, with the worker pool on, its `task_queue`.+- **Inside graph and agent runs.** Two patch markers give Workspace-task nodes their own activity options. `workspace-node-cancellation-v1` (`MachinaWorkflow`) uses the node class's `start_to_close_timeout` and `heartbeat_timeout`, and makes cancellation wait until the activity has finished cleaning up. `workspace-task-activity-policy-v1` (tool calls in `AgentWorkflow`, which already run with the class's declared timeout, heartbeat and `retry_policy`) makes cancellation wait the same way and, with the worker pool on, uses its `task_queue`.Browser tasks hold the profile claim during reasoning, tools and human help.Finalization waits for matching-token owner cleanup; stale cancellation cannotdiff --git a/docs-internal/node_creation.md b/docs-internal/node_creation.mdindex ca482a52..5e4e3d80 100644--- a/docs-internal/node_creation.md+++ b/docs-internal/node_creation.md@@ -97,7 +97,8 @@ Declare the operation's Temporal policy on the node class:| `retry_policy` | Attempts, intervals, and non-retryable error types for the operation. |Ordinary per-type agent-tool dispatch applies all four declarations for new-controlled generations. Graph node dispatch currently applies plugin retry+controlled generations, and every declaration but the queue for a tool call+outside one (a manual Run). Graph node dispatch currently applies plugin retryand queue declarations but uses generic 24-hour attempt / 2-minute heartbeatdefaults; the existing Workspace-task override uses that plugin's timeout andheartbeat values. Do not assume a graph node's timeout declaration is enforceddiff --git a/docs-internal/plugin_system.md b/docs-internal/plugin_system.mdindex 14a2af32..16f41bdf 100644--- a/docs-internal/plugin_system.md+++ b/docs-internal/plugin_system.md@@ -492,6 +492,8 @@ does not redirect an already-running workflow. New controlled generationsthe plugin's timeout, heartbeat, retry, and queue declarations during payloadpreparation. Ordinary per-type tools use these policies and carry them throughContinue-As-New. Existing histories retain their recorded policy decisions.+A tool call outside a controlled generation (a manual Run) applies the+declared timeout, heartbeat and retry too, but not the queue.The in-process executor does not enforce Temporal Activity policies.Graph Activity starts currently honor plugin retry and queue declarations,diff --git a/server/nodes/README.md b/server/nodes/README.mdindex dc1d95bc..3834d3a3 100644--- a/server/nodes/README.md+++ b/server/nodes/README.md@@ -129,7 +129,9 @@ tool bindings and these policies during agent preparation. Ordinary per-typeagent tools use that recorded policy, including attempt limits andnon-retryable error types. Continue-As-New carries the prepared bindings andpolicy rather than reloading a pending tool from edited configuration. Old-histories retain their previous dispatch decisions.+histories retain their previous dispatch decisions. A tool call outside a+controlled generation (a manual Run) uses the declared timeout, heartbeat+and retry policy as well, never Temporal's unlimited default retries.Graph node dispatch has a narrower policy contract: plugin retry and queuedeclarations apply, but its ordinary Activity starts retain generic 24-hourdiff --git a/server/services/temporal/agent_workflow.py b/server/services/temporal/agent_workflow.pyindex 1e9558f2..5f073fa8 100644--- a/server/services/temporal/agent_workflow.py+++ b/server/services/temporal/agent_workflow.py@@ -59,6 +59,7 @@ from temporalio.workflow import ActivityCancellationType, ChildWorkflowCancellatfrom services.agent_bindings import unique_node_bindingsfrom services.coding_agent_runtime import agent_workflow_name, uses_pifrom services.node_registry import get_node_class+from services.plugin.base import BaseNodefrom services.tool_output import bound_tool_output, tool_output_is_cappedfrom ._retry_policies import (@@ -91,14 +92,21 @@ from .execution_control import ExecutionControlLLM_STEP_TIMEOUT = timedelta(minutes=10)PERSIST_TURN_TIMEOUT = timedelta(seconds=30)COMPACT_MEMORY_TIMEOUT = timedelta(minutes=5)--# Tool activity defaults — plugin classes can override via-# ``cls.start_to_close_timeout`` (F4.A); these are floor values.-TOOL_STEP_TIMEOUT = timedelta(minutes=10)-TOOL_HEARTBEAT_TIMEOUT = timedelta(minutes=2)LLM_STEP_HEARTBEAT_TIMEOUT = timedelta(minutes=1)+def _declared_tool_options(node_type: str) -> Dict[str, Any]:+ """A tool activity's timeouts and retry policy as its plugin declares+ them (BaseNode's for a type no plugin registers). Options only: on+ replay Temporal compares no option (RFC-0006 §5 rule 6)."""+ cls = get_node_class(node_type) or BaseNode+ return {+ "start_to_close_timeout": cls.start_to_close_timeout,+ "heartbeat_timeout": cls.heartbeat_timeout,+ "retry_policy": cls.retry_policy.to_temporal(),+ }++async def _execute_plugin_tool_activity(activity_name: str,activity_id: str,@@ -109,13 +117,10 @@ async def _execute_plugin_tool_activity("""Preserve legacy commands; use frozen plugin policy for new generations.These plugins can perform irreversible device actions. Temporal's implicit- retry defaults must not override their explicitly declared attempt limit.- Routing comes from frozen workflow input, never live process settings.+ retry defaults must not override their explicitly declared attempt limit,+ on either path. Routing comes from frozen workflow input, never live+ process settings."""- options: Dict[str, Any] = {- "start_to_close_timeout": TOOL_STEP_TIMEOUT,- "heartbeat_timeout": TOOL_HEARTBEAT_TIMEOUT,- }policy = (tool_info or {}).get("activity_policy")if (context.get("execution_control_version") == 1 or context.get("native_workspace_version") == 1or context.get("browser_runtime_version") == 1 or context.get("browser_routing_version") == 1@@ -123,14 +128,10 @@ async def _execute_plugin_tool_activity(options = _frozen_tool_activity_options(policy, context)tool_payload = {**tool_payload, **_inherited_scope(context), "user_id": str(context.get("user_id") or "owner")}else:+ options = _declared_tool_options(tool_payload["node_type"])cls = get_node_class(tool_payload["node_type"])if getattr(cls, "workspace_task", False) and workflow.patched("workspace-task-activity-policy-v1"):- options.update(- start_to_close_timeout=cls.start_to_close_timeout,- heartbeat_timeout=cls.heartbeat_timeout,- retry_policy=cls.retry_policy.to_temporal(),- cancellation_type=ActivityCancellationType.WAIT_CANCELLATION_COMPLETED,- )+ options["cancellation_type"] = ActivityCancellationType.WAIT_CANCELLATION_COMPLETEDif context.get("temporal_worker_pool_enabled") is True:options["task_queue"] = cls.task_queuetool_payload = {**tool_payload, **_inherited_scope(context), "user_id": str(context.get("user_id") or "owner")}@@ -867,7 +868,7 @@ class AgentWorkflow:if policy:options = {**_frozen_tool_activity_options(policy, context), "retry_policy": SEND_ONCE}else:- options = {"start_to_close_timeout": TOOL_STEP_TIMEOUT, "heartbeat_timeout": TOOL_HEARTBEAT_TIMEOUT, "retry_policy": SEND_ONCE}+ options = {**_declared_tool_options(tool_payload["node_type"]), "retry_policy": SEND_ONCE}base = {**tool_payload, **_inherited_scope(context), "user_id": str(context.get("user_id") or "owner")}def as_allowed(claimed: Dict[str, Any]) -> Dict[str, Any]:@@ -2054,10 +2055,7 @@ class AgentWorkflow:await self._wait_until_resumed()await self._execution_control.begin_action()preflight_action_active = self._execution_control.enabled- preflight_options = {- "start_to_close_timeout": TOOL_STEP_TIMEOUT,- "heartbeat_timeout": TOOL_HEARTBEAT_TIMEOUT,- }+ preflight_options = _declared_tool_options("taskManager")if (self._execution_control.enabled or pi_runtime) and preflight_tool.get("activity_policy"):preflight_options = _frozen_tool_activity_options(preflight_tool["activity_policy"], context)task_manager_preflight_handles.append(diff --git a/server/tests/temporal/test_agent_llm_contract.py b/server/tests/temporal/test_agent_llm_contract.pyindex 2787de3c..1654ba75 100644--- a/server/tests/temporal/test_agent_llm_contract.py+++ b/server/tests/temporal/test_agent_llm_contract.py@@ -81,7 +81,7 @@ def patched_workflow(monkeypatch):monkeypatch.setattr(workflow_module,"get_node_class",- lambda _node_type: SimpleNamespace(needs_canvas=False),+ lambda _node_type: None,)return temporal_workflowdiff --git a/server/tests/temporal/test_agent_workflow_pressure.py b/server/tests/temporal/test_agent_workflow_pressure.pyindex 102702c6..65328de2 100644--- a/server/tests/temporal/test_agent_workflow_pressure.py+++ b/server/tests/temporal/test_agent_workflow_pressure.py@@ -169,7 +169,7 @@ def patched_workflow(monkeypatch):monkeypatch.setattr(workflow_module,"get_node_class",- lambda _node_type: SimpleNamespace(needs_canvas=False),+ lambda _node_type: None,)return temporal_workflowdiff --git a/server/tests/temporal/test_cooperative_pause.py b/server/tests/temporal/test_cooperative_pause.pyindex 8823595f..cbd4a990 100644--- a/server/tests/temporal/test_cooperative_pause.py+++ b/server/tests/temporal/test_cooperative_pause.py@@ -371,7 +371,7 @@ async def test_agent_rechecks_pause_before_tool_command_batch(monkeypatch):monkeypatch.setattr(agent_module,"get_node_class",- lambda _node_type: SimpleNamespace(needs_canvas=False),+ lambda _node_type: None,)instance = AgentWorkflow()@@ -452,7 +452,7 @@ async def test_agent_rechecks_pause_after_tool_phase_before_activity(monkeypatchmonkeypatch.setattr(agent_module,"get_node_class",- lambda _node_type: SimpleNamespace(needs_canvas=False),+ lambda _node_type: None,)instance = AgentWorkflow()@@ -1195,7 +1195,7 @@ async def test_agent_rechecks_pause_before_task_manager_child_start(monkeypatch)monkeypatch.setattr(agent_module,"get_node_class",- lambda _node_type: SimpleNamespace(needs_canvas=False),+ lambda _node_type: None,)instance = AgentWorkflow()diff --git a/server/tests/temporal/test_temporal_identity.py b/server/tests/temporal/test_temporal_identity.pyindex c515d7d4..b3d7d2a1 100644--- a/server/tests/temporal/test_temporal_identity.py+++ b/server/tests/temporal/test_temporal_identity.py@@ -195,7 +195,7 @@ class TestAgentCallIdentity:raise AssertionError(f"Unexpected activity {name}")monkeypatch.setattr(patched_workflow, "execute_activity", fake_execute_activity)- monkeypatch.setattr(agent_module, "get_node_class", lambda _node_type: SimpleNamespace(needs_canvas=False))+ monkeypatch.setattr(agent_module, "get_node_class", lambda _node_type: None)result = await AgentWorkflow().run({"node_id": "agent-1", "execution_id": "root-run-1"})@@ -279,7 +279,7 @@ class TestAgentCallIdentity:raise AssertionError(f"Unexpected activity {name}")monkeypatch.setattr(patched_workflow, "execute_activity", fake_execute_activity)- monkeypatch.setattr(agent_module, "get_node_class", lambda _node_type: SimpleNamespace(needs_canvas=False))+ monkeypatch.setattr(agent_module, "get_node_class", lambda _node_type: None)result = await AgentWorkflow().run({"node_id": "agent-1", "execution_id": "root-run-1"})diff --git a/server/tests/temporal/test_workspace_task_activity_policy.py b/server/tests/temporal/test_workspace_task_activity_policy.pyindex 46beb65e..1b52f8cd 100644--- a/server/tests/temporal/test_workspace_task_activity_policy.py+++ b/server/tests/temporal/test_workspace_task_activity_policy.py@@ -43,8 +43,21 @@ async def test_workspace_task_honors_plugin_policy_and_authenticated_principal(massert tool_payload["user_id"] == "untrusted" # caller input is not mutated-async def test_prepatch_workspace_history_emits_identical_legacy_command(monkeypatch):- monkeypatch.setattr(module, "get_node_class", lambda _: SimpleNamespace(workspace_task=True))+def declared(**attributes):+ """A plugin class as the registry holds one: it declares its options."""+ return SimpleNamespace(+ start_to_close_timeout=timedelta(minutes=4),+ heartbeat_timeout=timedelta(seconds=40),+ retry_policy=RetryPolicy(maximum_attempts=2),+ **attributes,+ )+++async def test_prepatch_workspace_history_keeps_the_legacy_command(monkeypatch):+ # Before the patch, no workspace policy: the default cancellation, the+ # default queue and the payload as it was.+ plugin = declared(workspace_task=True)+ monkeypatch.setattr(module, "get_node_class", lambda _: plugin)monkeypatch.setattr(module.workflow, "patched", Mock(return_value=False))execute = AsyncMock()monkeypatch.setattr(module.workflow, "execute_activity", execute)@@ -56,14 +69,18 @@ async def test_prepatch_workspace_history_emits_identical_legacy_command(monkeyp"node.embedded_task.v1",args=[payload],activity_id="old-id",- start_to_close_timeout=module.TOOL_STEP_TIMEOUT,- heartbeat_timeout=module.TOOL_HEARTBEAT_TIMEOUT,+ start_to_close_timeout=timedelta(minutes=4),+ heartbeat_timeout=timedelta(seconds=40),+ retry_policy=plugin.retry_policy.to_temporal(),)assert "user_id" not in payload-async def test_ordinary_tools_do_not_add_patch_markers_or_change_policy(monkeypatch):- monkeypatch.setattr(module, "get_node_class", lambda _: SimpleNamespace(workspace_task=False))+async def test_ordinary_tools_retry_as_declared_without_a_patch_marker(monkeypatch):+ # A manual run's tool call never retries without limit (1b.3). Options+ # only, so nothing in history changes.+ plugin = declared(workspace_task=False)+ monkeypatch.setattr(module, "get_node_class", lambda _: plugin)patched = Mock(side_effect=AssertionError("Ordinary tool must not change history"))monkeypatch.setattr(module.workflow, "patched", patched)execute = AsyncMock()@@ -75,6 +92,18 @@ async def test_ordinary_tools_do_not_add_patch_markers_or_change_policy(monkeypa"node.ordinary_tool.v1",args=[payload],activity_id="ordinary-id",- start_to_close_timeout=module.TOOL_STEP_TIMEOUT,- heartbeat_timeout=module.TOOL_HEARTBEAT_TIMEOUT,+ start_to_close_timeout=timedelta(minutes=4),+ heartbeat_timeout=timedelta(seconds=40),+ retry_policy=plugin.retry_policy.to_temporal(),)+++def test_a_type_no_plugin_registers_takes_the_base_node_options(monkeypatch):+ from services.plugin.base import BaseNode++ monkeypatch.setattr(module, "get_node_class", lambda _: None)+ assert module._declared_tool_options("_builtin_skill") == {+ "start_to_close_timeout": BaseNode.start_to_close_timeout,+ "heartbeat_timeout": BaseNode.heartbeat_timeout,+ "retry_policy": BaseNode.retry_policy.to_temporal(),+ }