Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
fix(agent): scope task completion events to their own workflow
trohitgdaddf460f439
diff --git a/CLAUDE.md b/CLAUDE.mdindex bf4ecffb..93f7697f 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -755,7 +755,7 @@ Workflows execute via Temporal for durability and horizontal scaling, gated by `Execution routing falls back Temporal → sequential; the Temporal dev server + embedded worker are managed in-process under `server/services/temporal/` (pooch-downloaded CLI, single `temporal server start-dev` process, gRPC `TEMPORAL_FRONTEND_GRPC_PORT` + Web UI `TEMPORAL_UI_PORT`, SQLite at `~/.opencompany/temporal.db`). Full architecture, dispatch matrix, per-node + agent-loop lifecycle, the F4.B `agent.*` activities (unsuffixed since the AgentWorkflow fold; read the live set from `collect_agent_activities()` rather than a hardcoded count), heartbeat semantics, delegation input contract, the worker tuning recipe, and all `.env` tunables live in [docs-internal/TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md).-**Months-long durability contract (July 2026).** Running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. Load-bearing invariants: (1) `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` stays `false` (debug-only sweep; even enabled, any control row in the shared `WORKFLOW_CONTROL_ACTIVE_STATES` — including `resetting` — vetoes it). (2) New child workflow starts carry **no** `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so lifetime caps silently terminated paused runs; liveness comes from activity heartbeats (30s beat / 2min timeout), never from workflow lifetime caps. Removing/adding start options or retry policies on recorded commands requires a `workflow.patched(...)` guard (list the live markers with `grep -rn "workflow.patched(" server/services/temporal`; [TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md) says what each one gates). (3) Every long-lived workflow loop must continue-as-new under history pressure (`is_continue_as_new_suggested()` + a soft cap) — Temporal hard-terminates around ~51,200 history events; `WorkflowControlWorkflow` carries triggers/queued events/seen-ids/control state across rollovers, so controller handles are addressed **by workflow id only, never run_id-pinned**. (4) `dispatch.emit` skips controllers whose `ControlEventTypes` Search Attribute doesn't match the event (absent attribute = legacy match-all). (5) The Temporal lifecycle is owned by [services/temporal/lifecycle.py](./server/services/temporal/lifecycle.py) (`main.py` only schedules it): dev-server supervision + resident watchdog (`TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS`), worker crash-restarts that REBUILD the single-use SDK `Worker` per attempt (manager + every pool queue worker), and the boot-time `reconcile_active_controls_on_boot` pass (converges crash-stranded `starting` rows, re-arms running/paused generations from the persisted graph snapshot). (6) **Recovery policies** (env-driven, defaults in `.env.template`; full semantics in [temporal-workflow-control.md → Recovery policies](./docs-internal/temporal-workflow-control.md)): `WORKFLOW_CONTROL_CRASH_RECOVERY=pause` — after a kill/crash (dirty-bit marker cleared only by graceful teardown) boot recovers running generations as **paused** so the user consciously resumes; `WORKFLOW_CONTROL_MISSING_CONTROLLER=pause` — a killed/vanished controller converges the generation to paused and **Resume rebuilds the controller** (`WorkflowIdConflictPolicy.USE_EXISTING`, re-arm from snapshot) instead of forcing a state-archiving Reset; `WORKFLOW_CONTROL_PAUSE_ON_FAILURE=true` — repeatedly-failing trigger-spawned runs pause their deployment via the `workflow_control.pause_on_failure` activity (circuit breaker; trips only after `WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD` (default 3) failures inside a rolling `_WINDOW_SECONDS` (default 600) so one node hiccup never pauses anything; Resume resets the streak; manual canvas runs never qualify). Temporal's native Pause/Unpause (server 1.28+) is deliberately NOT used — no Python SDK surface, and it halts workflow-task dispatch so a natively-paused controller couldn't process the resume Update. (7) **Workflow-scoped event delivery**: an envelope whose `workflow_id` is set by its producer's call site reaches only that workflow's consumers (`dispatch.emit` narrows both Visibility clauses by `EventWorkflowId`); chat is the first scoped producer — the decision lives in `routers/websocket.py`, the narrowing in core dispatch, and plugin `_events.py` factories only plumb the field (see [event_framework.md](./docs-internal/event_framework.md)). (8) **Canvas editability is the server-owned `can_edit` capability** emitted by `serialize_control` (paused = editable; starting/running/transitional = locked); the FE renders it via `lib/canvasLock.ts` + Dashboard's `guardCanvasEdit` and never re-derives the rule from state strings (see [temporal-workflow-control.md → Canvas editability](./docs-internal/temporal-workflow-control.md)). Locked by `tests/temporal/test_durability_hardening.py` + `test_worker_restart.py`.+**Months-long durability contract (July 2026).** Running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. Load-bearing invariants: (1) `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` stays `false` (debug-only sweep; even enabled, any control row in the shared `WORKFLOW_CONTROL_ACTIVE_STATES` — including `resetting` — vetoes it). (2) New child workflow starts carry **no** `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so lifetime caps silently terminated paused runs; liveness comes from activity heartbeats (30s beat / 2min timeout), never from workflow lifetime caps. Removing/adding start options or retry policies on recorded commands requires a `workflow.patched(...)` guard (list the live markers with `grep -rn "workflow.patched(" server/services/temporal`; [TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md) says what each one gates). (3) Every long-lived workflow loop must continue-as-new under history pressure (`is_continue_as_new_suggested()` + a soft cap) — Temporal hard-terminates around ~51,200 history events; `WorkflowControlWorkflow` carries triggers/queued events/seen-ids/control state across rollovers, so controller handles are addressed **by workflow id only, never run_id-pinned**. (4) `dispatch.emit` skips controllers whose `ControlEventTypes` Search Attribute doesn't match the event (absent attribute = legacy match-all). (5) The Temporal lifecycle is owned by [services/temporal/lifecycle.py](./server/services/temporal/lifecycle.py) (`main.py` only schedules it): dev-server supervision + resident watchdog (`TEMPORAL_HEALTH_MONITOR_INTERVAL_SECONDS`), worker crash-restarts that REBUILD the single-use SDK `Worker` per attempt (manager + every pool queue worker), and the boot-time `reconcile_active_controls_on_boot` pass (converges crash-stranded `starting` rows, re-arms running/paused generations from the persisted graph snapshot). (6) **Recovery policies** (env-driven, defaults in `.env.template`; full semantics in [temporal-workflow-control.md → Recovery policies](./docs-internal/temporal-workflow-control.md)): `WORKFLOW_CONTROL_CRASH_RECOVERY=pause` — after a kill/crash (dirty-bit marker cleared only by graceful teardown) boot recovers running generations as **paused** so the user consciously resumes; `WORKFLOW_CONTROL_MISSING_CONTROLLER=pause` — a killed/vanished controller converges the generation to paused and **Resume rebuilds the controller** (`WorkflowIdConflictPolicy.USE_EXISTING`, re-arm from snapshot) instead of forcing a state-archiving Reset; `WORKFLOW_CONTROL_PAUSE_ON_FAILURE=true` — repeatedly-failing trigger-spawned runs pause their deployment via the `workflow_control.pause_on_failure` activity (circuit breaker; trips only after `WORKFLOW_CONTROL_PAUSE_ON_FAILURE_THRESHOLD` (default 3) failures inside a rolling `_WINDOW_SECONDS` (default 600) so one node hiccup never pauses anything; Resume resets the streak; manual canvas runs never qualify). Temporal's native Pause/Unpause (server 1.28+) is deliberately NOT used — no Python SDK surface, and it halts workflow-task dispatch so a natively-paused controller couldn't process the resume Update. (7) **Workflow-scoped event delivery**: an envelope whose `workflow_id` is set by its producer's call site reaches only that workflow's consumers (`dispatch.emit` narrows both Visibility clauses by `EventWorkflowId`); chat is the first scoped producer and delegated-task completion (`nodes/agent/_events.py`) the second, whose broadcast frame carries only the task's identity — the decision lives in `routers/websocket.py`, the narrowing in core dispatch, and plugin `_events.py` factories only plumb the field (see [event_framework.md](./docs-internal/event_framework.md)). (8) **Canvas editability is the server-owned `can_edit` capability** emitted by `serialize_control` (paused = editable; starting/running/transitional = locked); the FE renders it via `lib/canvasLock.ts` + Dashboard's `guardCanvasEdit` and never re-derives the rule from state strings (see [temporal-workflow-control.md → Canvas editability](./docs-internal/temporal-workflow-control.md)). Locked by `tests/temporal/test_durability_hardening.py` + `test_worker_restart.py`.**Temporal tracing ownership is strict.** Register the SDK `TracingInterceptor` exactly once on the shared `Client.connect(...)`; the Python SDK automatically prepends compatible client interceptors to every worker. Never repeat `TracingInterceptor` in `Worker(..., interceptors=...)`. Worker lists contain the distinct `ObservabilityWorkerInterceptor` and plugin-specific worker interceptors only. Exact paired `StartActivity` / `RunActivity` / `CompleteWorkflow` console spans with identical Temporal identities indicate duplicate instrumentation, not duplicate execution; use Temporal Event History and the `node.<type>.execute` application span to verify actual attempts. See the tracing and debugging sections in [TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md) and [errors.md](./docs-internal/errors.md).diff --git a/docs-internal/event_framework.md b/docs-internal/event_framework.mdindex 5974b71b..983de4d6 100644--- a/docs-internal/event_framework.md+++ b/docs-internal/event_framework.md@@ -119,6 +119,15 @@ never ride `emit` or the broadcast at all: `services/chat/hub.py` deliversthem only to sockets subscribed to the session([chat_protocol.md](./chat_protocol.md)).+Second producer: delegated-task completion (`nodes/agent/_events.py`,+`com.opencompany.agent.task.completed`). Without the scope, one workflow's+finished task fired every deployed workflow's `taskTrigger` (RFC-0004 H11).+The envelope is emitted with `broadcast=False`, because its data holds the+child agent's answer or error. The `task_completed` WebSocket frame is a+second envelope of the same type and id whose data is only `task_id`,+`status`, `team_id` and `workflow_id`, which is all the Task Manager panel+reads to refresh.+Worker is embedded in the FastAPI process (the `main.py` lifespan schedules`run_temporal_lifecycle` from `services/temporal/lifecycle.py` as one`asyncio.create_task()` named `temporal-init`; that module owns the connect loop and thediff --git a/server/nodes/agent/_events.py b/server/nodes/agent/_events.pyindex ec0bc149..1b4f51fa 100644--- a/server/nodes/agent/_events.py+++ b/server/nodes/agent/_events.py@@ -10,13 +10,15 @@ wrappers below live with the agent plugin folder so the 3 callsites indispatcher) import from ``nodes.agent._events`` rather than buildingthe wire frame inline.-Delivery: a single ``dispatch.emit`` call. ``dispatch.emit`` Signals-running :class:`TriggerListenerWorkflow` consumers via Temporal-Visibility AND broadcasts the same envelope to in-process WS clients on-the ``task_completed`` wire-routing key — one canonical path. Pre-Wave-13-this also called ``broadcaster.send_custom_event`` for the legacy-collector path; that's dead since taskTrigger is canary-registered and-the deployment manager skips ``setup_event_trigger`` for canary types.+Delivery: ``dispatch.emit`` Signals the running consumers of the task's+own workflow (the envelope's ``workflow_id`` scopes the Visibility+query), so a completion never fires another workflow's taskTrigger.+The WebSocket frame on the ``task_completed`` key carries the task's+identity only: the broadcast reaches every connected socket, and the+Task Manager panel needs just the team id to refresh. Pre-Wave-13 this+also called ``broadcaster.send_custom_event`` for the legacy collector+path; that's dead since taskTrigger is canary-registered and the+deployment manager skips ``setup_event_trigger`` for canary types."""from __future__ import annotations@@ -93,18 +95,16 @@ async def _broadcast_task_event(event_id: Optional[str] = None,lifecycle_data: Optional[dict[str, Any]] = None,) -> None:- """Single delivery path: build the envelope, route through- :func:`services.events.dispatch.emit`.-- ``emit`` does two things in one call: (a) Visibility query for- running :class:`TriggerListenerWorkflow` consumers tagged with- ``EventType='com.opencompany.agent.task.completed'`` and signals each;- (b) direct in-process WS broadcast on the ``task_completed`` wire- key so any FE consumers receive the envelope. Payload shape in- ``data`` is exactly what the taskTrigger filter reads.+ """Build the envelope and route it through+ :func:`services.events.dispatch.emit` to the consumers of+ ``workflow_id`` only, then broadcast the task's identity.++ The envelope's ``data`` is exactly what the taskTrigger filter reads.+ It holds the child agent's answer or error, so it is not broadcast."""from services.events.dispatch import emitfrom services.events.envelope import WorkflowEvent+ from services.status_broadcaster import get_status_broadcasterpayload: dict[str, Any] = {"task_id": task_id,@@ -131,7 +131,22 @@ async def _broadcast_task_event(# delivered immediately before a worker failure/retry.if event_id:envelope.id = event_id- await emit(envelope, wire_routing_key=_WIRE_ROUTING_KEY)+ envelope.workflow_id = workflow_id+ await emit(envelope, wire_routing_key=_WIRE_ROUTING_KEY, broadcast=False)++ identity = WorkflowEvent.task_completed(+ task_id=task_id,+ status="completed" if status == "completed" else "error",+ agent=agent_name,+ data={+ "task_id": task_id,+ "status": status,+ "team_id": (lifecycle_data or {}).get("team_id"),+ "workflow_id": workflow_id,+ },+ )+ identity.id = envelope.id+ await get_status_broadcaster().broadcast({"type": _WIRE_ROUTING_KEY, "data": identity.model_dump(mode="json")})__all__ = [diff --git a/server/tests/test_task_trigger_canary_producer.py b/server/tests/test_task_trigger_canary_producer.pyindex 6d68dede..15d8f50e 100644--- a/server/tests/test_task_trigger_canary_producer.py+++ b/server/tests/test_task_trigger_canary_producer.py@@ -54,8 +54,8 @@ class TestTaskTriggerProducerCanaryEmit:assert _EVENTS_EMIT_PATTERN.search(src), ("_broadcast_task_event must call ""services.events.dispatch.emit(envelope, ...) — the canary "- "CloudEvents path Signals running TriggerListenerWorkflow "- "consumers AND broadcasts to FE on the task_completed wire key."+ "CloudEvents path Signals the running consumers of the task's "+ "own workflow.")assert not _SEND_CUSTOM_EVENT_PATTERN.search(src), ("_broadcast_task_event must NOT call "@@ -164,6 +164,49 @@ class TestTaskTriggerProducerCanaryEmit:assert envelope.data["root_execution_id"] == "root-1"assert envelope.data["trace_id"] == "call-1"+ @pytest.mark.asyncio+ async def test_delivery_is_scoped_and_the_broadcast_carries_identity_only(self, monkeypatch):+ """A completion reaches only its own workflow's taskTrigger (RFC-0004+ H11). The WebSocket broadcast reaches every socket, so it carries the+ task's identity and never the child agent's answer or error."""+ from nodes.agent import _events+ from services import status_broadcaster+ from services.events import dispatch as dispatch_mod++ emit_calls: List[Any] = []+ frames: List[Any] = []++ async def fake_emit(event, **kwargs):+ emit_calls.append({"event": event, **kwargs})+ return event++ class _Broadcaster:+ async def broadcast(self, frame):+ frames.append(frame)++ monkeypatch.setattr(dispatch_mod, "emit", fake_emit)+ monkeypatch.setattr(status_broadcaster, "get_status_broadcaster", lambda: _Broadcaster())++ await _events.broadcast_agent_task_completed(+ task_id="task-4",+ agent_name="writer",+ agent_node_id="agent-4",+ parent_node_id="lead-1",+ workflow_id="wf-4",+ result="the customer's private answer",+ event_id="task-4:completed",+ lifecycle_data={"team_id": "team-4", "trace_id": "call-4"},+ )++ assert emit_calls[0]["event"].workflow_id == "wf-4"+ assert emit_calls[0]["broadcast"] is False+ assert len(frames) == 1+ assert frames[0]["type"] == "task_completed"+ envelope = frames[0]["data"]+ assert envelope["id"] == "task-4:completed"+ assert envelope["type"] == "com.opencompany.agent.task.completed"+ assert envelope["data"] == {"task_id": "task-4", "status": "completed", "team_id": "team-4", "workflow_id": "wf-4"}+def test_temporal_completion_emits_after_authoritative_state_read(self):"""Temporal parity: retries/requeues are decided before taskTrigger."""from services.temporal import agent_activities