Skip to content

Self-improving AI Employees for everyone

fix(temporal): a cancel reaches the activity; a cut call leaves its card (RFC-0006 1b.2)

trohitg8f1db2df474e
diff --git a/docs-internal/chat_protocol.md b/docs-internal/chat_protocol.md
index a6f3c1cd..755f1078 100644
--- a/docs-internal/chat_protocol.md
+++ b/docs-internal/chat_protocol.md
@@ -578,7 +578,10 @@ What an employee sends to someone waits for the owner's OK while its workflow as
shows what the call ran with, without secrets. A graph node's answer carries `requires_user_action` and a `hint`,
so its employee pauses at once, and its employee reads an `[update]` note; a graph node's send keeps what it sent,
its templates resolved and its secrets left out, so Send again resends it and the node's settings put the secrets
- back. Held calls are drafts, made once per call, and still retry.
+ back. A call cut midway (the run cancelled it, its heartbeat lapsed, or the worker shut down) leaves the same card
+ at once. A cancel stays a cancel. A worker shutdown retries a call safe to repeat; any other answers at once with
+ the same envelope, since a later attempt could only refuse it. Held calls are drafts, made once per call, and still
+ retry.
- **The employee hears how it went**: an `[update]{...}[/update]` note (`approval:<id>`) at the start of its next
turn in the chat, for a send that went, failed, was discarded or expired, and for a graph node's call that may
have happened.
diff --git a/docs-internal/node-logic-flows/workflow_triggers/cronScheduler.md b/docs-internal/node-logic-flows/workflow_triggers/cronScheduler.md
index 7fdc85ac..cea78761 100644
--- a/docs-internal/node-logic-flows/workflow_triggers/cronScheduler.md
+++ b/docs-internal/node-logic-flows/workflow_triggers/cronScheduler.md
@@ -82,7 +82,7 @@ flowchart TD
C --> D[_calculate_wait_seconds<br/>maps frequency to seconds]
D --> E[get_status_broadcaster.update_node_status 'waiting']
E --> F[await asyncio.sleep wait_seconds]
- F -- CancelledError --> G[Raise RuntimeError: Scheduler cancelled<br/>BaseNode returns failed envelope]
+ F -- CancelledError --> G[The cancel propagates<br/>a Temporal activity reports a cancel; in process NodeExecutor answers Cancelled]
F -- ok --> I[Build result_data: timestamp / iteration=1 / frequency / schedule / waited_seconds]
I --> J{frequency == 'once'?}
J -- yes --> K[message: 'Triggered after waiting ...']
@@ -105,9 +105,10 @@ flowchart TD
- **Once vs recurring**: determines whether `next_run` is added to the
payload and which `message` template is used. Both branches still return
`success=True`.
-- **Explicit cancellation**: the operation raises `RuntimeError("Scheduler
- cancelled")`; `BaseNode.execute` turns that failure into its standard
- envelope. Cooperative generation Stop does not enter this branch.
+- **Explicit cancellation**: the cancel propagates. In a Temporal activity it
+ stays a cancel (a worker shutdown retries the wait, which is safe to
+ repeat); in process `NodeExecutor` answers `Cancelled`. Cooperative
+ generation Stop does not enter this branch.
## Deployed Stop/Resume boundary
diff --git a/docs-internal/node-logic-flows/workflow_triggers/timer.md b/docs-internal/node-logic-flows/workflow_triggers/timer.md
index d61b51b8..9402f91d 100644
--- a/docs-internal/node-logic-flows/workflow_triggers/timer.md
+++ b/docs-internal/node-logic-flows/workflow_triggers/timer.md
@@ -66,7 +66,7 @@ flowchart TD
S3 --> D
S4 --> D
D --> E[await asyncio.sleep wait_seconds]
- E -- CancelledError --> F[Raise RuntimeError: Timer cancelled<br/>BaseNode returns failed envelope]
+ E -- CancelledError --> F[The cancel propagates<br/>a Temporal activity reports a cancel; in process NodeExecutor answers Cancelled]
E -- ok --> H[elapsed_ms = now - start]
H --> I[Return success envelope<br/>with timestamp / elapsed_ms / duration / unit / message]
```
@@ -76,9 +76,10 @@ flowchart TD
- **Unit mapping**: `seconds` / `minutes` / `hours` multiply duration by
1 / 60 / 3600. Any other value falls through to the `_` match case and is
treated as raw seconds.
-- **Explicit cancellation**: `asyncio.CancelledError` becomes
- `RuntimeError("Timer cancelled")`; the node execution wrapper surfaces the
- failed envelope. Cooperative generation Stop does not cancel this sleep.
+- **Explicit cancellation**: `asyncio.CancelledError` propagates. In a
+ Temporal activity it stays a cancel (a worker shutdown retries the wait,
+ which is safe to repeat); in process `NodeExecutor` answers `Cancelled`.
+ Cooperative generation Stop does not cancel this sleep.
## Stop/Resume boundary
@@ -116,9 +117,8 @@ count; this user-requested delay is admitted node work. See
- `duration` is Pydantic-validated `ge=1, le=86400`; for `hours` that is up
to 86400 hours, so a large value with `unit=hours` can sleep effectively
indefinitely.
-- Cancellation (`asyncio.CancelledError`) is re-raised as `RuntimeError("Timer
- cancelled")`, surfaced as a failed envelope. Downstream nodes that key off
- `success` will not execute when the timer is cancelled.
+- Cancellation (`asyncio.CancelledError`) propagates as a cancel, so
+ downstream nodes do not execute when the timer is cancelled.
## Related
diff --git a/server/nodes/scheduler/cron_scheduler/__init__.py b/server/nodes/scheduler/cron_scheduler/__init__.py
index c44c891a..722c4009 100644
--- a/server/nodes/scheduler/cron_scheduler/__init__.py
+++ b/server/nodes/scheduler/cron_scheduler/__init__.py
@@ -309,10 +309,8 @@ class CronSchedulerNode(ActionNode):
trigger_time=trigger_time.isoformat(),
)
- try:
- await asyncio.sleep(wait_seconds)
- except asyncio.CancelledError:
- raise RuntimeError("Scheduler cancelled")
+ # A cancel stays a cancel, so Temporal sees it as one.
+ await asyncio.sleep(wait_seconds)
triggered_at = datetime.now()
output = CronSchedulerOutput(
diff --git a/server/nodes/scheduler/timer/__init__.py b/server/nodes/scheduler/timer/__init__.py
index 30aabd12..f25f190a 100644
--- a/server/nodes/scheduler/timer/__init__.py
+++ b/server/nodes/scheduler/timer/__init__.py
@@ -78,10 +78,8 @@ class TimerNode(ActionNode):
workflow_id=ctx.workflow_id,
)
- try:
- await asyncio.sleep(wait_seconds)
- except asyncio.CancelledError:
- raise RuntimeError("Timer cancelled")
+ # A cancel stays a cancel, so Temporal sees it as one.
+ await asyncio.sleep(wait_seconds)
elapsed_ms = int((time.time() - start_time) * 1000)
return {
diff --git a/server/services/node_executor.py b/server/services/node_executor.py
index 3b5e1a7d..9f8a4ff1 100644
--- a/server/services/node_executor.py
+++ b/server/services/node_executor.py
@@ -187,10 +187,15 @@ class NodeExecutor:
return result
except asyncio.CancelledError:
- # Workspace engines own subprocesses/device leases. Temporal must
- # observe cancellation after their finally blocks finish, not an
- # ordinary failure envelope that can be mistaken for completion.
- if getattr(get_node_class(node_type), "workspace_task", False):
+ # Inside an activity Temporal must observe the cancellation, not
+ # an ordinary failure envelope that can be mistaken for
+ # completion: the activity wrapper reports a call that may have
+ # happened (BaseNode.as_activity). Workspace engines own
+ # subprocesses/device leases, so they re-raise after their
+ # finally blocks wherever they run.
+ from temporalio import activity
+
+ if activity.in_activity() or getattr(get_node_class(node_type), "workspace_task", False):
raise
return ExecutionResult(
False, node_id, node_type, error="Cancelled", execution_id=execution_id, execution_time=time.time() - start_time
diff --git a/server/services/plugin/base.py b/server/services/plugin/base.py
index 05a9bb0b..64295524 100644
--- a/server/services/plugin/base.py
+++ b/server/services/plugin/base.py
@@ -137,6 +137,12 @@ class NodeWaitInterrupted(Exception):
re-attaches to whatever durable state the wait was keyed on. Returning
normally instead would record the attempt as completed and fail the run.
+ The wrapper raises the same retryable error when a worker shutdown
+ cancels a call that is safe to repeat (services/approvals/repeats.py);
+ a call that is not answers at once that it may have happened. The next
+ attempt of such a call is refused anyway, so a body that raises this
+ must be listed as safe to repeat.
+
Only raise it from a body that is idempotent across attempts. On the
in-process path there is no retry, and the run fails as it would for any
other error.
@@ -1348,8 +1354,6 @@ class BaseNode:
success, payload, error = cls.interpret_result(result)
if success:
await chat_steps.number_sources(context, cls, result)
- if checked.record is not None:
- await checked.record(result)
if chat_step is not None and not (
not success and result.get("error_type") == NODE_WAIT_INTERRUPTED
):
@@ -1366,6 +1370,10 @@ class BaseNode:
type=NODE_WAIT_INTERRUPTED,
non_retryable=False,
)
+ # Recorded once the attempt is over: an interrupted wait
+ # (above) runs again and is recorded then.
+ if checked.record is not None:
+ await checked.record(result)
if success:
correlated_payload = (
{**payload, "execution_id": execution_id}
@@ -1397,6 +1405,42 @@ class BaseNode:
activity.heartbeat(f"Node {node_id} completed")
return result
+ except asyncio.CancelledError:
+ # Cancelled mid-call: the run stopped it, its heartbeat lapsed,
+ # or the worker is shutting down. A call that is not safe to
+ # repeat may have happened, so the owner gets a card at once.
+ # On a worker shutdown a call safe to repeat runs again
+ # (retryable); any other answers now, since a later attempt
+ # could only refuse it. Anything else stays a cancel.
+ details = activity.cancellation_details()
+ shutdown = bool(details is not None and details.worker_shutdown and not details.cancel_requested)
+ answer: Optional[Dict[str, Any]] = None
+ if context.get("approval_execution") is None and not await approval_calls.repeatable_call(context, cls, node_data):
+ answer = {
+ "node_id": node_id,
+ "node_type": cls.type,
+ "execution_id": execution_id,
+ "timestamp": datetime.now().isoformat(),
+ **approval_calls.unknown_answer(context, cls, node_data),
+ }
+ await approval_calls.record_unknown(context, cls, node_data, answer, checked=checked)
+ elif shutdown:
+ activity.logger.info(f"Node {node_id} interrupted by a worker shutdown; failing the attempt retryably")
+ raise ApplicationError("Wait interrupted", type=NODE_WAIT_INTERRUPTED, non_retryable=False)
+ maybe = None
+ if answer:
+ maybe = "It may already have gone out" if answer["error_type"] == "SendOutcomeUnknown" else "It may already have happened"
+ await chat_steps.end(chat_step, state="failed", detail=maybe)
+ await broadcaster.update_node_status(
+ node_id,
+ "error",
+ {"error": answer["error"] if answer else "Cancelled", "execution_id": execution_id,
+ **{key: answer[key] for key in ("hint", "requires_user_action", "error_type") if answer and key in answer}},
+ workflow_id=workflow_id,
+ )
+ if shutdown:
+ return answer
+ raise
except Exception as e:
if isinstance(e, ApplicationError) and e.type == NODE_WAIT_INTERRUPTED:
raise
diff --git a/server/tests/services/approvals/test_held_activity.py b/server/tests/services/approvals/test_held_activity.py
index 03a32104..ed7560e0 100644
--- a/server/tests/services/approvals/test_held_activity.py
+++ b/server/tests/services/approvals/test_held_activity.py
@@ -8,6 +8,7 @@ secrets) or an ``effect`` card to dismiss."""
from __future__ import annotations
+import asyncio
import dataclasses
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock
@@ -240,6 +241,55 @@ async def test_a_graph_card_sends_again_as_it_was(harness):
assert node_context(agent, "t1")["tool_args"] == {"message": "Hi"}
+async def cut(service, node_type, call, **details):
+ """Run a call that waits until its activity is cancelled with ``details``."""
+ from temporalio.activity import ActivityCancellationDetails
+
+ started = asyncio.Event()
+
+ async def execute_node(**_):
+ started.set()
+ await asyncio.sleep(3600)
+
+ service.execute_node = execute_node
+ env = ActivityEnvironment()
+ task = asyncio.ensure_future(env.run(get_node_class(node_type).as_activity(), call))
+ await asyncio.wait_for(started.wait(), 5)
+ env.cancel(cancellation_details=ActivityCancellationDetails(**details))
+ return task
+
+
+async def test_a_call_cut_by_a_worker_shutdown_answers_at_once(harness, service):
+ # A later attempt could only refuse it, so the model hears it now.
+ command = {"command": "git push origin main"}
+ call = context(node_id="wf:shell:1", node_data=dict(command), tool_args=dict(command))
+ result = await (await cut(service, "shell", call, worker_shutdown=True))
+ assert result["success"] is False and result["error_type"] == "OutcomeUnknown"
+ (row,) = await store.list_approvals(harness.database, status="failed")
+ assert (row.kind, row.outcome) == ("effect", "unknown")
+
+
+async def test_a_graph_call_cut_by_a_cancel_leaves_a_card_and_stays_cancelled(harness, service):
+ command = {"command": "git push origin main"}
+ task = await cut(service, "shell", graph_context(node_id="wf:shell:1", node_data=dict(command)), cancel_requested=True)
+ with pytest.raises(asyncio.CancelledError):
+ await task
+ (row,) = await store.list_approvals(harness.database, status="failed")
+ assert (row.kind, row.tool_call_id, row.idempotency_key[:2]) == ("effect", "", "e:")
+
+
+async def test_an_approved_send_cut_midway_is_left_to_its_workflow(harness, service, monkeypatch):
+ # ApprovedToolCallWorkflow records how its own send went.
+ from services.approvals import tool_calls
+
+ monkeypatch.setattr(tool_calls, "check", AsyncMock(return_value=tool_calls.RUN))
+ approved = context(approval_execution={"approval_id": "a1", "claim_token": "t1"})
+ task = await cut(service, "whatsappSend", approved, cancel_requested=True)
+ with pytest.raises(asyncio.CancelledError):
+ await task
+ assert await store.list_approvals(harness.database) == []
+
+
async def test_the_legacy_activity_refuses_a_later_attempt_too(harness):
from services.temporal.activities import NodeExecutionActivities
diff --git a/server/tests/services/test_node_wait_interrupted.py b/server/tests/services/test_node_wait_interrupted.py
index 6e99dcee..9b6803d8 100644
--- a/server/tests/services/test_node_wait_interrupted.py
+++ b/server/tests/services/test_node_wait_interrupted.py
@@ -6,10 +6,14 @@ shutting down used to *complete* as a failure. Temporal never retried it, the
run failed, and the circuit breaker counted it. ``NodeWaitInterrupted`` is
reported as its own ``error_type``, which the activity wrapper re-raises as a
retryable ``ApplicationError`` so the next attempt resumes the wait.
+
+A cancel that reaches the activity stays a cancel; one caused by a worker
+shutdown retries a call that is safe to repeat.
"""
from __future__ import annotations
+import asyncio
from unittest.mock import AsyncMock, MagicMock
import pytest
@@ -70,6 +74,60 @@ async def test_activity_fails_retryably_when_the_wait_is_interrupted(monkeypatch
assert "error" not in statuses, "an interrupted wait is not an error the user should see"
+async def _cut(monkeypatch, details):
+ """A console call that waits until its activity is cancelled with
+ ``details`` (console is safe to repeat)."""
+ from core.container import container
+ from temporalio.testing import ActivityEnvironment
+
+ broadcaster = _install_activity_fakes(monkeypatch, {})
+ started = asyncio.Event()
+
+ async def execute_node(**_):
+ started.set()
+ await asyncio.sleep(3600)
+
+ container.workflow_service().execute_node = execute_node
+ env = ActivityEnvironment()
+ task = asyncio.ensure_future(env.run(get_node_class("console").as_activity(), {"node_id": "console_1", "workflow_id": "wf-1", "node_data": {}}))
+ await asyncio.wait_for(started.wait(), 5)
+ env.cancel(cancellation_details=details)
+ return task, broadcaster
+
+
+async def test_a_cancel_stays_a_cancel(monkeypatch):
+ from temporalio.activity import ActivityCancellationDetails
+
+ task, broadcaster = await _cut(monkeypatch, ActivityCancellationDetails(cancel_requested=True))
+ with pytest.raises(asyncio.CancelledError):
+ await task
+ statuses = [call.args[1] for call in broadcaster.update_node_status.await_args_list]
+ assert statuses[-1] == "error"
+
+
+async def test_a_worker_shutdown_retries_a_call_safe_to_repeat(monkeypatch):
+ from temporalio.activity import ActivityCancellationDetails
+ from temporalio.exceptions import ApplicationError
+
+ task, broadcaster = await _cut(monkeypatch, ActivityCancellationDetails(worker_shutdown=True))
+ with pytest.raises(ApplicationError) as info:
+ await task
+ assert info.value.type == NODE_WAIT_INTERRUPTED and info.value.non_retryable is False
+ statuses = [call.args[1] for call in broadcaster.update_node_status.await_args_list]
+ assert "error" not in statuses
+
+
+async def test_a_cancel_requested_while_the_worker_shuts_down_stays_a_cancel(monkeypatch):
+ from temporalio.activity import ActivityCancellationDetails
+
+ task, _ = await _cut(monkeypatch, ActivityCancellationDetails(cancel_requested=True, worker_shutdown=True))
+ with pytest.raises(asyncio.CancelledError):
+ await task
+
+
@pytest.mark.asyncio
async def test_ordinary_failures_still_complete_the_attempt(monkeypatch):
from temporalio.testing import ActivityEnvironment
diff --git a/server/tests/test_workspace_invocations.py b/server/tests/test_workspace_invocations.py
index aa6da919..8fab28e8 100644
--- a/server/tests/test_workspace_invocations.py
+++ b/server/tests/test_workspace_invocations.py
@@ -148,6 +148,21 @@ async def test_legacy_reset_cancels_only_matching_roots(monkeypatch):
cancel.assert_awaited_once_with(matching)
+async def test_node_executor_lets_an_activity_see_a_cancel(monkeypatch):
+ # The activity wrapper reports a call that may have happened, and
+ # Temporal must see a cancel, not a completed failure.
+ from temporalio.testing import ActivityEnvironment
+
+ from services import node_executor as module
+
+ executor = module.NodeExecutor.__new__(module.NodeExecutor)
+ executor._prepare_parameters = AsyncMock(return_value={})
+ executor._dispatch = AsyncMock(side_effect=asyncio.CancelledError())
+ monkeypatch.setattr(module, "get_node_class", lambda _: SimpleNamespace(workspace_task=False))
+ with pytest.raises(asyncio.CancelledError):
+ await ActivityEnvironment().run(executor.execute, "node", "ordinary", {}, {})
+
+
@pytest.mark.parametrize("workspace", [True, False])
async def test_node_executor_preserves_workspace_cancellation(monkeypatch, workspace):
from services import node_executor as module