Skip to content

Self-improving AI Employees for everyone

fix(agent): scope task completion events to their own workflow

trohitgdaddf460f439
diff --git a/CLAUDE.md b/CLAUDE.md
index bf4ecffb..93f7697f 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -755,7 +755,7 @@ Workflows execute via Temporal for durability and horizontal scaling, gated by `
Execution routing falls back Temporal → sequential; the Temporal dev server + embedded worker are managed in-process under `server/services/temporal/` (pooch-downloaded CLI, single `temporal server start-dev` process, gRPC `TEMPORAL_FRONTEND_GRPC_PORT` + Web UI `TEMPORAL_UI_PORT`, SQLite at `~/.opencompany/temporal.db`). Full architecture, dispatch matrix, per-node + agent-loop lifecycle, the F4.B `agent.*` activities (unsuffixed since the AgentWorkflow fold; read the live set from `collect_agent_activities()` rather than a hardcoded count), heartbeat semantics, delegation input contract, the worker tuning recipe, and all `.env` tunables live in [docs-internal/TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md).
-**Months-long durability contract (July 2026).** Running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. Load-bearing invariants: (1) `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` stays `false` (debug-only sweep; even enabled, any control row in the shared `WORKFLOW_CONTROL_ACTIVE_STATES` — including `resetting` — vetoes it). (2) New child workflow starts carry **no** `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so lifetime caps silently terminated paused runs; liveness comes from activity heartbeats (30s beat / 2min timeout), never from workflow lifetime caps. Removing/adding start options or retry policies on recorded commands requires a `workflow.patched(...)` guard (list the live markers with `grep -rn "workflow.patched(" server/services/temporal`; [TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md) says what each one gates). (3) Every long-lived workflow loop must continue-as-new under history pressure (`is_continue_as_new_suggested()` + a soft cap) — Temporal hard-terminates around ~51,200 history events; `WorkflowControlWorkflow` carries triggers/queued events/seen-ids/control state across rollovers, so controller handles are addressed **by workflow id only, never run_id-pinned**. (4) `dispatch.emit` skips controllers whose `ControlEventTypes` Search Attribute doesn't match the event (absent attribute = legacy match-all). (5) The Temporal lifecycle is owned by [services/temporal/lifecycle.py](./server/services/temporal/lifecycle.py) (`main.py` only schedules it): dev-server supervision + resident watchdog (`TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS`), worker crash-restarts that REBUILD the single-use SDK `Worker` per attempt (manager + every pool queue worker), and the boot-time `reconcile_active_controls_on_boot` pass (converges crash-stranded `starting` rows, re-arms running/paused generations from the persisted graph snapshot). (6) **Recovery policies** (env-driven, defaults in `.env.template`; full semantics in [temporal-workflow-control.md → Recovery policies](./docs-internal/temporal-workflow-control.md)): `WORKFLOW_CONTROL_CRASH_RECOVERY=pause` — after a kill/crash (dirty-bit marker cleared only by graceful teardown) boot recovers running generations as **paused** so the user consciously resumes; `WORKFLOW_CONTROL_MISSING_CONTROLLER=pause` — a killed/vanished controller converges the generation to paused and **Resume rebuilds the controller** (`WorkflowIdConflictPolicy.USE_EXISTING`, re-arm from snapshot) instead of forcing a state-archiving Reset; `WORKFLOW_CONTROL_PAUSE_ON_FAILURE=true` — repeatedly-failing trigger-spawned runs pause their deployment via the `workflow_control.pause_on_failure` activity (circuit breaker; trips only after `WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD` (default 3) failures inside a rolling `_WINDOW_SECONDS` (default 600) so one node hiccup never pauses anything; Resume resets the streak; manual canvas runs never qualify). Temporal's native Pause/Unpause (server 1.28+) is deliberately NOT used — no Python SDK surface, and it halts workflow-task dispatch so a natively-paused controller couldn't process the resume Update. (7) **Workflow-scoped event delivery**: an envelope whose `workflow_id` is set by its producer's call site reaches only that workflow's consumers (`dispatch.emit` narrows both Visibility clauses by `EventWorkflowId`); chat is the first scoped producer — the decision lives in `routers/websocket.py`, the narrowing in core dispatch, and plugin `_events.py` factories only plumb the field (see [event_framework.md](./docs-internal/event_framework.md)). (8) **Canvas editability is the server-owned `can_edit` capability** emitted by `serialize_control` (paused = editable; starting/running/transitional = locked); the FE renders it via `lib/canvasLock.ts` + Dashboard's `guardCanvasEdit` and never re-derives the rule from state strings (see [temporal-workflow-control.md → Canvas editability](./docs-internal/temporal-workflow-control.md)). Locked by `tests/temporal/test_durability_hardening.py` + `test_worker_restart.py`.
+**Months-long durability contract (July 2026).** Running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. Load-bearing invariants: (1) `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` stays `false` (debug-only sweep; even enabled, any control row in the shared `WORKFLOW_CONTROL_ACTIVE_STATES` — including `resetting` — vetoes it). (2) New child workflow starts carry **no** `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so lifetime caps silently terminated paused runs; liveness comes from activity heartbeats (30s beat / 2min timeout), never from workflow lifetime caps. Removing/adding start options or retry policies on recorded commands requires a `workflow.patched(...)` guard (list the live markers with `grep -rn "workflow.patched(" server/services/temporal`; [TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md) says what each one gates). (3) Every long-lived workflow loop must continue-as-new under history pressure (`is_continue_as_new_suggested()` + a soft cap) — Temporal hard-terminates around ~51,200 history events; `WorkflowControlWorkflow` carries triggers/queued events/seen-ids/control state across rollovers, so controller handles are addressed **by workflow id only, never run_id-pinned**. (4) `dispatch.emit` skips controllers whose `ControlEventTypes` Search Attribute doesn't match the event (absent attribute = legacy match-all). (5) The Temporal lifecycle is owned by [services/temporal/lifecycle.py](./server/services/temporal/lifecycle.py) (`main.py` only schedules it): dev-server supervision + resident watchdog (`TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS`), worker crash-restarts that REBUILD the single-use SDK `Worker` per attempt (manager + every pool queue worker), and the boot-time `reconcile_active_controls_on_boot` pass (converges crash-stranded `starting` rows, re-arms running/paused generations from the persisted graph snapshot). (6) **Recovery policies** (env-driven, defaults in `.env.template`; full semantics in [temporal-workflow-control.md → Recovery policies](./docs-internal/temporal-workflow-control.md)): `WORKFLOW_CONTROL_CRASH_RECOVERY=pause` — after a kill/crash (dirty-bit marker cleared only by graceful teardown) boot recovers running generations as **paused** so the user consciously resumes; `WORKFLOW_CONTROL_MISSING_CONTROLLER=pause` — a killed/vanished controller converges the generation to paused and **Resume rebuilds the controller** (`WorkflowIdConflictPolicy.USE_EXISTING`, re-arm from snapshot) instead of forcing a state-archiving Reset; `WORKFLOW_CONTROL_PAUSE_ON_FAILURE=true` — repeatedly-failing trigger-spawned runs pause their deployment via the `workflow_control.pause_on_failure` activity (circuit breaker; trips only after `WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD` (default 3) failures inside a rolling `_WINDOW_SECONDS` (default 600) so one node hiccup never pauses anything; Resume resets the streak; manual canvas runs never qualify). Temporal's native Pause/Unpause (server 1.28+) is deliberately NOT used — no Python SDK surface, and it halts workflow-task dispatch so a natively-paused controller couldn't process the resume Update. (7) **Workflow-scoped event delivery**: an envelope whose `workflow_id` is set by its producer's call site reaches only that workflow's consumers (`dispatch.emit` narrows both Visibility clauses by `EventWorkflowId`); chat is the first scoped producer and delegated-task completion (`nodes/agent/_events.py`) the second, whose broadcast frame carries only the task's identity — the decision lives in `routers/websocket.py`, the narrowing in core dispatch, and plugin `_events.py` factories only plumb the field (see [event_framework.md](./docs-internal/event_framework.md)). (8) **Canvas editability is the server-owned `can_edit` capability** emitted by `serialize_control` (paused = editable; starting/running/transitional = locked); the FE renders it via `lib/canvasLock.ts` + Dashboard's `guardCanvasEdit` and never re-derives the rule from state strings (see [temporal-workflow-control.md → Canvas editability](./docs-internal/temporal-workflow-control.md)). Locked by `tests/temporal/test_durability_hardening.py` + `test_worker_restart.py`.
**Temporal tracing ownership is strict.** Register the SDK `TracingInterceptor` exactly once on the shared `Client.connect(...)`; the Python SDK automatically prepends compatible client interceptors to every worker. Never repeat `TracingInterceptor` in `Worker(..., interceptors=...)`. Worker lists contain the distinct `ObservabilityWorkerInterceptor` and plugin-specific worker interceptors only. Exact paired `StartActivity` / `RunActivity` / `CompleteWorkflow` console spans with identical Temporal identities indicate duplicate instrumentation, not duplicate execution; use Temporal Event History and the `node.<type>.execute` application span to verify actual attempts. See the tracing and debugging sections in [TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md) and [errors.md](./docs-internal/errors.md).
diff --git a/docs-internal/event_framework.md b/docs-internal/event_framework.md
index 5974b71b..983de4d6 100644
--- a/docs-internal/event_framework.md
+++ b/docs-internal/event_framework.md
@@ -119,6 +119,15 @@ never ride `emit` or the broadcast at all: `services/chat/hub.py` delivers
them only to sockets subscribed to the session
([chat_protocol.md](./chat_protocol.md)).
+Second producer: delegated-task completion (`nodes/agent/_events.py`,
+`com.opencompany.agent.task.completed`). Without the scope, one workflow's
+finished task fired every deployed workflow's `taskTrigger` (RFC-0004 H11).
+The envelope is emitted with `broadcast=False`, because its data holds the
+child agent's answer or error. The `task_completed` WebSocket frame is a
+second envelope of the same type and id whose data is only `task_id`,
+`status`, `team_id` and `workflow_id`, which is all the Task Manager panel
+reads to refresh.
+
Worker is embedded in the FastAPI process (the `main.py` lifespan schedules
`run_temporal_lifecycle` from `services/temporal/lifecycle.py` as one
`asyncio.create_task()` named `temporal-init`; that module owns the connect loop and the
diff --git a/server/nodes/agent/_events.py b/server/nodes/agent/_events.py
index ec0bc149..1b4f51fa 100644
--- a/server/nodes/agent/_events.py
+++ b/server/nodes/agent/_events.py
@@ -10,13 +10,15 @@ wrappers below live with the agent plugin folder so the 3 callsites in
dispatcher) import from ``nodes.agent._events`` rather than building
the wire frame inline.
-Delivery: a single ``dispatch.emit`` call. ``dispatch.emit`` Signals
-running :class:`TriggerListenerWorkflow` consumers via Temporal
-Visibility AND broadcasts the same envelope to in-process WS clients on
-the ``task_completed`` wire-routing key — one canonical path. Pre-Wave-13
-this also called ``broadcaster.send_custom_event`` for the legacy
-collector path; that's dead since taskTrigger is canary-registered and
-the deployment manager skips ``setup_event_trigger`` for canary types.
+Delivery: ``dispatch.emit`` Signals the running consumers of the task's
+own workflow (the envelope's ``workflow_id`` scopes the Visibility
+query), so a completion never fires another workflow's taskTrigger.
+The WebSocket frame on the ``task_completed`` key carries the task's
+identity only: the broadcast reaches every connected socket, and the
+Task Manager panel needs just the team id to refresh. Pre-Wave-13 this
+also called ``broadcaster.send_custom_event`` for the legacy collector
+path; that's dead since taskTrigger is canary-registered and the
+deployment manager skips ``setup_event_trigger`` for canary types.
"""
from __future__ import annotations
@@ -93,18 +95,16 @@ async def _broadcast_task_event(
event_id: Optional[str] = None,
lifecycle_data: Optional[dict[str, Any]] = None,
) -> None:
- """Single delivery path: build the envelope, route through
- :func:`services.events.dispatch.emit`.
-
- ``emit`` does two things in one call: (a) Visibility query for
- running :class:`TriggerListenerWorkflow` consumers tagged with
- ``EventType='com.opencompany.agent.task.completed'`` and signals each;
- (b) direct in-process WS broadcast on the ``task_completed`` wire
- key so any FE consumers receive the envelope. Payload shape in
- ``data`` is exactly what the taskTrigger filter reads.
+ """Build the envelope and route it through
+ :func:`services.events.dispatch.emit` to the consumers of
+ ``workflow_id`` only, then broadcast the task's identity.
+
+ The envelope's ``data`` is exactly what the taskTrigger filter reads.
+ It holds the child agent's answer or error, so it is not broadcast.
"""
from services.events.dispatch import emit
from services.events.envelope import WorkflowEvent
+ from services.status_broadcaster import get_status_broadcaster
payload: dict[str, Any] = {
"task_id": task_id,
@@ -131,7 +131,22 @@ async def _broadcast_task_event(
# delivered immediately before a worker failure/retry.
if event_id:
envelope.id = event_id
- await emit(envelope, wire_routing_key=_WIRE_ROUTING_KEY)
+ envelope.workflow_id = workflow_id
+ await emit(envelope, wire_routing_key=_WIRE_ROUTING_KEY, broadcast=False)
+
+ identity = WorkflowEvent.task_completed(
+ task_id=task_id,
+ status="completed" if status == "completed" else "error",
+ agent=agent_name,
+ data={
+ "task_id": task_id,
+ "status": status,
+ "team_id": (lifecycle_data or {}).get("team_id"),
+ "workflow_id": workflow_id,
+ },
+ )
+ identity.id = envelope.id
+ await get_status_broadcaster().broadcast({"type": _WIRE_ROUTING_KEY, "data": identity.model_dump(mode="json")})
__all__ = [
diff --git a/server/tests/test_task_trigger_canary_producer.py b/server/tests/test_task_trigger_canary_producer.py
index 6d68dede..15d8f50e 100644
--- a/server/tests/test_task_trigger_canary_producer.py
+++ b/server/tests/test_task_trigger_canary_producer.py
@@ -54,8 +54,8 @@ class TestTaskTriggerProducerCanaryEmit:
assert _EVENTS_EMIT_PATTERN.search(src), (
"_broadcast_task_event must call "
"services.events.dispatch.emit(envelope, ...) — the canary "
- "CloudEvents path Signals running TriggerListenerWorkflow "
- "consumers AND broadcasts to FE on the task_completed wire key."
+ "CloudEvents path Signals the running consumers of the task's "
+ "own workflow."
)
assert not _SEND_CUSTOM_EVENT_PATTERN.search(src), (
"_broadcast_task_event must NOT call "
@@ -164,6 +164,49 @@ class TestTaskTriggerProducerCanaryEmit:
assert envelope.data["root_execution_id"] == "root-1"
assert envelope.data["trace_id"] == "call-1"
+ @pytest.mark.asyncio
+ async def test_delivery_is_scoped_and_the_broadcast_carries_identity_only(self, monkeypatch):
+ """A completion reaches only its own workflow's taskTrigger (RFC-0004
+ H11). The WebSocket broadcast reaches every socket, so it carries the
+ task's identity and never the child agent's answer or error."""
+ from nodes.agent import _events
+ from services import status_broadcaster
+ from services.events import dispatch as dispatch_mod
+
+ emit_calls: List[Any] = []
+ frames: List[Any] = []
+
+ async def fake_emit(event, **kwargs):
+ emit_calls.append({"event": event, **kwargs})
+ return event
+
+ class _Broadcaster:
+ async def broadcast(self, frame):
+ frames.append(frame)
+
+ monkeypatch.setattr(dispatch_mod, "emit", fake_emit)
+ monkeypatch.setattr(status_broadcaster, "get_status_broadcaster", lambda: _Broadcaster())
+
+ await _events.broadcast_agent_task_completed(
+ task_id="task-4",
+ agent_name="writer",
+ agent_node_id="agent-4",
+ parent_node_id="lead-1",
+ workflow_id="wf-4",
+ result="the customer's private answer",
+ event_id="task-4:completed",
+ lifecycle_data={"team_id": "team-4", "trace_id": "call-4"},
+ )
+
+ assert emit_calls[0]["event"].workflow_id == "wf-4"
+ assert emit_calls[0]["broadcast"] is False
+ assert len(frames) == 1
+ assert frames[0]["type"] == "task_completed"
+ envelope = frames[0]["data"]
+ assert envelope["id"] == "task-4:completed"
+ assert envelope["type"] == "com.opencompany.agent.task.completed"
+ assert envelope["data"] == {"task_id": "task-4", "status": "completed", "team_id": "team-4", "workflow_id": "wf-4"}
+
def test_temporal_completion_emits_after_authoritative_state_read(self):
"""Temporal parity: retries/requeues are decided before taskTrigger."""
from services.temporal import agent_activities