Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
fix(temporal): a cancel reaches the activity; a cut call leaves its card (RFC-0006 1b.2)
trohitg8f1db2df474e
diff --git a/docs-internal/chat_protocol.md b/docs-internal/chat_protocol.mdindex a6f3c1cd..755f1078 100644--- a/docs-internal/chat_protocol.md+++ b/docs-internal/chat_protocol.md@@ -578,7 +578,10 @@ What an employee sends to someone waits for the owner's OK while its workflow asshows what the call ran with, without secrets. A graph node's answer carries `requires_user_action` and a `hint`,so its employee pauses at once, and its employee reads an `[update]` note; a graph node's send keeps what it sent,its templates resolved and its secrets left out, so Send again resends it and the node's settings put the secrets- back. Held calls are drafts, made once per call, and still retry.+ back. A call cut midway (the run cancelled it, its heartbeat lapsed, or the worker shut down) leaves the same card+ at once. A cancel stays a cancel. A worker shutdown retries a call safe to repeat; any other answers at once with+ the same envelope, since a later attempt could only refuse it. Held calls are drafts, made once per call, and still+ retry.- **The employee hears how it went**: an `[update]{...}[/update]` note (`approval:<id>`) at the start of its nextturn in the chat, for a send that went, failed, was discarded or expired, and for a graph node's call that mayhave happened.diff --git a/docs-internal/node-logic-flows/workflow_triggers/cronScheduler.md b/docs-internal/node-logic-flows/workflow_triggers/cronScheduler.mdindex 7fdc85ac..cea78761 100644--- a/docs-internal/node-logic-flows/workflow_triggers/cronScheduler.md+++ b/docs-internal/node-logic-flows/workflow_triggers/cronScheduler.md@@ -82,7 +82,7 @@ flowchart TDC --> D[_calculate_wait_seconds<br/>maps frequency to seconds]D --> E[get_status_broadcaster.update_node_status 'waiting']E --> F[await asyncio.sleep wait_seconds]- F -- CancelledError --> G[Raise RuntimeError: Scheduler cancelled<br/>BaseNode returns failed envelope]+ F -- CancelledError --> G[The cancel propagates<br/>a Temporal activity reports a cancel; in process NodeExecutor answers Cancelled]F -- ok --> I[Build result_data: timestamp / iteration=1 / frequency / schedule / waited_seconds]I --> J{frequency == 'once'?}J -- yes --> K[message: 'Triggered after waiting ...']@@ -105,9 +105,10 @@ flowchart TD- **Once vs recurring**: determines whether `next_run` is added to thepayload and which `message` template is used. Both branches still return`success=True`.-- **Explicit cancellation**: the operation raises `RuntimeError("Scheduler- cancelled")`; `BaseNode.execute` turns that failure into its standard- envelope. Cooperative generation Stop does not enter this branch.+- **Explicit cancellation**: the cancel propagates. In a Temporal activity it+ stays a cancel (a worker shutdown retries the wait, which is safe to+ repeat); in process `NodeExecutor` answers `Cancelled`. Cooperative+ generation Stop does not enter this branch.## Deployed Stop/Resume boundarydiff --git a/docs-internal/node-logic-flows/workflow_triggers/timer.md b/docs-internal/node-logic-flows/workflow_triggers/timer.mdindex d61b51b8..9402f91d 100644--- a/docs-internal/node-logic-flows/workflow_triggers/timer.md+++ b/docs-internal/node-logic-flows/workflow_triggers/timer.md@@ -66,7 +66,7 @@ flowchart TDS3 --> DS4 --> DD --> E[await asyncio.sleep wait_seconds]- E -- CancelledError --> F[Raise RuntimeError: Timer cancelled<br/>BaseNode returns failed envelope]+ E -- CancelledError --> F[The cancel propagates<br/>a Temporal activity reports a cancel; in process NodeExecutor answers Cancelled]E -- ok --> H[elapsed_ms = now - start]H --> I[Return success envelope<br/>with timestamp / elapsed_ms / duration / unit / message]```@@ -76,9 +76,10 @@ flowchart TD- **Unit mapping**: `seconds` / `minutes` / `hours` multiply duration by1 / 60 / 3600. Any other value falls through to the `_` match case and istreated as raw seconds.-- **Explicit cancellation**: `asyncio.CancelledError` becomes- `RuntimeError("Timer cancelled")`; the node execution wrapper surfaces the- failed envelope. Cooperative generation Stop does not cancel this sleep.+- **Explicit cancellation**: `asyncio.CancelledError` propagates. In a+ Temporal activity it stays a cancel (a worker shutdown retries the wait,+ which is safe to repeat); in process `NodeExecutor` answers `Cancelled`.+ Cooperative generation Stop does not cancel this sleep.## Stop/Resume boundary@@ -116,9 +117,8 @@ count; this user-requested delay is admitted node work. See- `duration` is Pydantic-validated `ge=1, le=86400`; for `hours` that is upto 86400 hours, so a large value with `unit=hours` can sleep effectivelyindefinitely.-- Cancellation (`asyncio.CancelledError`) is re-raised as `RuntimeError("Timer- cancelled")`, surfaced as a failed envelope. Downstream nodes that key off- `success` will not execute when the timer is cancelled.+- Cancellation (`asyncio.CancelledError`) propagates as a cancel, so+ downstream nodes do not execute when the timer is cancelled.## Relateddiff --git a/server/nodes/scheduler/cron_scheduler/__init__.py b/server/nodes/scheduler/cron_scheduler/__init__.pyindex c44c891a..722c4009 100644--- a/server/nodes/scheduler/cron_scheduler/__init__.py+++ b/server/nodes/scheduler/cron_scheduler/__init__.py@@ -309,10 +309,8 @@ class CronSchedulerNode(ActionNode):trigger_time=trigger_time.isoformat(),)- try:- await asyncio.sleep(wait_seconds)- except asyncio.CancelledError:- raise RuntimeError("Scheduler cancelled")+ # A cancel stays a cancel, so Temporal sees it as one.+ await asyncio.sleep(wait_seconds)triggered_at = datetime.now()output = CronSchedulerOutput(diff --git a/server/nodes/scheduler/timer/__init__.py b/server/nodes/scheduler/timer/__init__.pyindex 30aabd12..f25f190a 100644--- a/server/nodes/scheduler/timer/__init__.py+++ b/server/nodes/scheduler/timer/__init__.py@@ -78,10 +78,8 @@ class TimerNode(ActionNode):workflow_id=ctx.workflow_id,)- try:- await asyncio.sleep(wait_seconds)- except asyncio.CancelledError:- raise RuntimeError("Timer cancelled")+ # A cancel stays a cancel, so Temporal sees it as one.+ await asyncio.sleep(wait_seconds)elapsed_ms = int((time.time() - start_time) * 1000)return {diff --git a/server/services/node_executor.py b/server/services/node_executor.pyindex 3b5e1a7d..9f8a4ff1 100644--- a/server/services/node_executor.py+++ b/server/services/node_executor.py@@ -187,10 +187,15 @@ class NodeExecutor:return resultexcept asyncio.CancelledError:- # Workspace engines own subprocesses/device leases. Temporal must- # observe cancellation after their finally blocks finish, not an- # ordinary failure envelope that can be mistaken for completion.- if getattr(get_node_class(node_type), "workspace_task", False):+ # Inside an activity Temporal must observe the cancellation, not+ # an ordinary failure envelope that can be mistaken for+ # completion: the activity wrapper reports a call that may have+ # happened (BaseNode.as_activity). Workspace engines own+ # subprocesses/device leases, so they re-raise after their+ # finally blocks wherever they run.+ from temporalio import activity++ if activity.in_activity() or getattr(get_node_class(node_type), "workspace_task", False):raisereturn ExecutionResult(False, node_id, node_type, error="Cancelled", execution_id=execution_id, execution_time=time.time() - start_timediff --git a/server/services/plugin/base.py b/server/services/plugin/base.pyindex 05a9bb0b..64295524 100644--- a/server/services/plugin/base.py+++ b/server/services/plugin/base.py@@ -137,6 +137,12 @@ class NodeWaitInterrupted(Exception):re-attaches to whatever durable state the wait was keyed on. Returningnormally instead would record the attempt as completed and fail the run.+ The wrapper raises the same retryable error when a worker shutdown+ cancels a call that is safe to repeat (services/approvals/repeats.py);+ a call that is not answers at once that it may have happened. The next+ attempt of such a call is refused anyway, so a body that raises this+ must be listed as safe to repeat.+Only raise it from a body that is idempotent across attempts. On thein-process path there is no retry, and the run fails as it would for anyother error.@@ -1348,8 +1354,6 @@ class BaseNode:success, payload, error = cls.interpret_result(result)if success:await chat_steps.number_sources(context, cls, result)- if checked.record is not None:- await checked.record(result)if chat_step is not None and not (not success and result.get("error_type") == NODE_WAIT_INTERRUPTED):@@ -1366,6 +1370,10 @@ class BaseNode:type=NODE_WAIT_INTERRUPTED,non_retryable=False,)+ # Recorded once the attempt is over: an interrupted wait+ # (above) runs again and is recorded then.+ if checked.record is not None:+ await checked.record(result)if success:correlated_payload = ({**payload, "execution_id": execution_id}@@ -1397,6 +1405,42 @@ class BaseNode:activity.heartbeat(f"Node {node_id} completed")return result+ except asyncio.CancelledError:+ # Cancelled mid-call: the run stopped it, its heartbeat lapsed,+ # or the worker is shutting down. A call that is not safe to+ # repeat may have happened, so the owner gets a card at once.+ # On a worker shutdown a call safe to repeat runs again+ # (retryable); any other answers now, since a later attempt+ # could only refuse it. Anything else stays a cancel.+ details = activity.cancellation_details()+ shutdown = bool(details is not None and details.worker_shutdown and not details.cancel_requested)+ answer: Optional[Dict[str, Any]] = None+ if context.get("approval_execution") is None and not await approval_calls.repeatable_call(context, cls, node_data):+ answer = {+ "node_id": node_id,+ "node_type": cls.type,+ "execution_id": execution_id,+ "timestamp": datetime.now().isoformat(),+ **approval_calls.unknown_answer(context, cls, node_data),+ }+ await approval_calls.record_unknown(context, cls, node_data, answer, checked=checked)+ elif shutdown:+ activity.logger.info(f"Node {node_id} interrupted by a worker shutdown; failing the attempt retryably")+ raise ApplicationError("Wait interrupted", type=NODE_WAIT_INTERRUPTED, non_retryable=False)+ maybe = None+ if answer:+ maybe = "It may already have gone out" if answer["error_type"] == "SendOutcomeUnknown" else "It may already have happened"+ await chat_steps.end(chat_step, state="failed", detail=maybe)+ await broadcaster.update_node_status(+ node_id,+ "error",+ {"error": answer["error"] if answer else "Cancelled", "execution_id": execution_id,+ **{key: answer[key] for key in ("hint", "requires_user_action", "error_type") if answer and key in answer}},+ workflow_id=workflow_id,+ )+ if shutdown:+ return answer+ raiseexcept Exception as e:if isinstance(e, ApplicationError) and e.type == NODE_WAIT_INTERRUPTED:raisediff --git a/server/tests/services/approvals/test_held_activity.py b/server/tests/services/approvals/test_held_activity.pyindex 03a32104..ed7560e0 100644--- a/server/tests/services/approvals/test_held_activity.py+++ b/server/tests/services/approvals/test_held_activity.py@@ -8,6 +8,7 @@ secrets) or an ``effect`` card to dismiss."""from __future__ import annotations+import asyncioimport dataclassesfrom types import SimpleNamespacefrom unittest.mock import AsyncMock, MagicMock@@ -240,6 +241,55 @@ async def test_a_graph_card_sends_again_as_it_was(harness):assert node_context(agent, "t1")["tool_args"] == {"message": "Hi"}+async def cut(service, node_type, call, **details):+ """Run a call that waits until its activity is cancelled with ``details``."""+ from temporalio.activity import ActivityCancellationDetails++ started = asyncio.Event()++ async def execute_node(**_):+ started.set()+ await asyncio.sleep(3600)++ service.execute_node = execute_node+ env = ActivityEnvironment()+ task = asyncio.ensure_future(env.run(get_node_class(node_type).as_activity(), call))+ await asyncio.wait_for(started.wait(), 5)+ env.cancel(cancellation_details=ActivityCancellationDetails(**details))+ return task+++async def test_a_call_cut_by_a_worker_shutdown_answers_at_once(harness, service):+ # A later attempt could only refuse it, so the model hears it now.+ command = {"command": "git push origin main"}+ call = context(node_id="wf:shell:1", node_data=dict(command), tool_args=dict(command))+ result = await (await cut(service, "shell", call, worker_shutdown=True))+ assert result["success"] is False and result["error_type"] == "OutcomeUnknown"+ (row,) = await store.list_approvals(harness.database, status="failed")+ assert (row.kind, row.outcome) == ("effect", "unknown")+++async def test_a_graph_call_cut_by_a_cancel_leaves_a_card_and_stays_cancelled(harness, service):+ command = {"command": "git push origin main"}+ task = await cut(service, "shell", graph_context(node_id="wf:shell:1", node_data=dict(command)), cancel_requested=True)+ with pytest.raises(asyncio.CancelledError):+ await task+ (row,) = await store.list_approvals(harness.database, status="failed")+ assert (row.kind, row.tool_call_id, row.idempotency_key[:2]) == ("effect", "", "e:")+++async def test_an_approved_send_cut_midway_is_left_to_its_workflow(harness, service, monkeypatch):+ # ApprovedToolCallWorkflow records how its own send went.+ from services.approvals import tool_calls++ monkeypatch.setattr(tool_calls, "check", AsyncMock(return_value=tool_calls.RUN))+ approved = context(approval_execution={"approval_id": "a1", "claim_token": "t1"})+ task = await cut(service, "whatsappSend", approved, cancel_requested=True)+ with pytest.raises(asyncio.CancelledError):+ await task+ assert await store.list_approvals(harness.database) == []++async def test_the_legacy_activity_refuses_a_later_attempt_too(harness):from services.temporal.activities import NodeExecutionActivitiesdiff --git a/server/tests/services/test_node_wait_interrupted.py b/server/tests/services/test_node_wait_interrupted.pyindex 6e99dcee..9b6803d8 100644--- a/server/tests/services/test_node_wait_interrupted.py+++ b/server/tests/services/test_node_wait_interrupted.py@@ -6,10 +6,14 @@ shutting down used to *complete* as a failure. Temporal never retried it, therun failed, and the circuit breaker counted it. ``NodeWaitInterrupted`` isreported as its own ``error_type``, which the activity wrapper re-raises as aretryable ``ApplicationError`` so the next attempt resumes the wait.++A cancel that reaches the activity stays a cancel; one caused by a worker+shutdown retries a call that is safe to repeat."""from __future__ import annotations+import asynciofrom unittest.mock import AsyncMock, MagicMockimport pytest@@ -70,6 +74,60 @@ async def test_activity_fails_retryably_when_the_wait_is_interrupted(monkeypatchassert "error" not in statuses, "an interrupted wait is not an error the user should see"+async def _cut(monkeypatch, details):+ """A console call that waits until its activity is cancelled with+ ``details`` (console is safe to repeat)."""+ from core.container import container+ from temporalio.testing import ActivityEnvironment++ broadcaster = _install_activity_fakes(monkeypatch, {})+ started = asyncio.Event()++ async def execute_node(**_):+ started.set()+ await asyncio.sleep(3600)++ container.workflow_service().execute_node = execute_node+ env = ActivityEnvironment()+ task = asyncio.ensure_future(env.run(get_node_class("console").as_activity(), {"node_id": "console_1", "workflow_id": "wf-1", "node_data": {}}))+ await asyncio.wait_for(started.wait(), 5)+ env.cancel(cancellation_details=details)+ return task, broadcaster+++async def test_a_cancel_stays_a_cancel(monkeypatch):+ from temporalio.activity import ActivityCancellationDetails++ task, broadcaster = await _cut(monkeypatch, ActivityCancellationDetails(cancel_requested=True))+ with pytest.raises(asyncio.CancelledError):+ await task+ statuses = [call.args[1] for call in broadcaster.update_node_status.await_args_list]+ assert statuses[-1] == "error"+++async def test_a_worker_shutdown_retries_a_call_safe_to_repeat(monkeypatch):+ from temporalio.activity import ActivityCancellationDetails+ from temporalio.exceptions import ApplicationError++ task, broadcaster = await _cut(monkeypatch, ActivityCancellationDetails(worker_shutdown=True))+ with pytest.raises(ApplicationError) as info:+ await task+ assert info.value.type == NODE_WAIT_INTERRUPTED and info.value.non_retryable is False+ statuses = [call.args[1] for call in broadcaster.update_node_status.await_args_list]+ assert "error" not in statuses+++async def test_a_cancel_requested_while_the_worker_shuts_down_stays_a_cancel(monkeypatch):+ from temporalio.activity import ActivityCancellationDetails++ task, _ = await _cut(monkeypatch, ActivityCancellationDetails(cancel_requested=True, worker_shutdown=True))+ with pytest.raises(asyncio.CancelledError):+ await task++@pytest.mark.asyncioasync def test_ordinary_failures_still_complete_the_attempt(monkeypatch):from temporalio.testing import ActivityEnvironmentdiff --git a/server/tests/test_workspace_invocations.py b/server/tests/test_workspace_invocations.pyindex aa6da919..8fab28e8 100644--- a/server/tests/test_workspace_invocations.py+++ b/server/tests/test_workspace_invocations.py@@ -148,6 +148,21 @@ async def test_legacy_reset_cancels_only_matching_roots(monkeypatch):cancel.assert_awaited_once_with(matching)+async def test_node_executor_lets_an_activity_see_a_cancel(monkeypatch):+ # The activity wrapper reports a call that may have happened, and+ # Temporal must see a cancel, not a completed failure.+ from temporalio.testing import ActivityEnvironment++ from services import node_executor as module++ executor = module.NodeExecutor.__new__(module.NodeExecutor)+ executor._prepare_parameters = AsyncMock(return_value={})+ executor._dispatch = AsyncMock(side_effect=asyncio.CancelledError())+ monkeypatch.setattr(module, "get_node_class", lambda _: SimpleNamespace(workspace_task=False))+ with pytest.raises(asyncio.CancelledError):+ await ActivityEnvironment().run(executor.execute, "node", "ordinary", {}, {})++@pytest.mark.parametrize("workspace", [True, False])async def test_node_executor_preserves_workspace_cancellation(monkeypatch, workspace):from services import node_executor as module