Skip to content

Self-improving AI Employees for everyone

docs(temporal): correct stale statements in Temporal and event docs

trohitg3ccb986f8750
diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.md
index a1195acd..5fe939d3 100644
--- a/docs-internal/TEMPORAL_ARCHITECTURE.md
+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md
@@ -16,7 +16,9 @@ Each workflow node executes as a **Temporal activity** with its own isolated con
| **Per-type activity** (`node.{type}.v{version}`) | `TEMPORAL_PER_TYPE_DISPATCH=true` (production default) | Each plugin gets its own `@activity.defn`. Per-plugin retry / timeout / heartbeat configs apply. With `TEMPORAL_WORKER_POOL_ENABLED=true` (default since Wave 16.4) the activity also carries `task_queue=cls.task_queue`, landing it on its specialised `TemporalWorkerPool` worker (browser / code-exec / ai-heavy / ...). Shipped in F4.A (commit `8261b05`); queue routing activated in Wave 16. |
| **Agent-as-child-workflow** (`AgentWorkflow`) | `TEMPORAL_AGENT_WORKFLOW_ENABLED=true` | AI Agents (aiAgent, chatAgent, 11 specialized agents, 2 team leads) run as Temporal child workflows. Each LLM turn = activity; each tool call = per-type activity. Mirrors Temporal's AI Cookbook canonical pattern. F4.B infrastructure shipped (commit `a4d009e`); per-agent migrations follow. |
-`rlm_agent`, `claude_code_agent` are intentionally excluded from AgentWorkflow — their externalised loops (RLM REPL / Claude CLI `--resume`) require single-process state continuity.
+`rlm_agent`, `claude_code_agent` and `vertex_managed_agent` are intentionally excluded from AgentWorkflow — their externalised loops (RLM REPL / Claude CLI `--resume` / Vertex Interactions API `previous_interaction_id` chaining) require single-process state continuity. The authoritative list is `AGENT_WORKFLOW_TYPES` in `services/temporal/workflow.py`; any agent type outside it runs as an ordinary per-type activity.
+
+**Routing snapshot (`_temporal_routing_v1`).** The three dispatch flags — `TEMPORAL_PER_TYPE_DISPATCH`, `TEMPORAL_AGENT_WORKFLOW_ENABLED` and `TEMPORAL_WORKER_POOL_ENABLED` (which picks the task queue) — are read once by whoever starts the run and frozen into the workflow input under `_temporal_routing_v1` (`services/temporal/executor.py::capture_temporal_routing_input`). `TemporalExecutor.execute_workflow` captures them per run; `DeploymentManager` captures them when it registers a push/poll listener or creates a cron Schedule, and the listener hands the same snapshot to every run it spawns. `MachinaWorkflow.run` dispatches only from that snapshot (`_frozen_routing_from_input`) and never re-reads `Settings`, so changing a flag affects runs started, and triggers registered, afterwards — never a run already in flight. An input without a snapshot (or with an unknown version) gets `_SAFE_FROZEN_ROUTING`: agent workflow and per-type dispatch on, worker pool off. Locked by `tests/temporal/test_frozen_routing.py`.
## Execution Routing & Running
@@ -178,15 +180,22 @@ while True:
completed.add(node_id)
continue
- # F4.A: resolve to per-type name + queue when the flag is on,
- # else fall back to the legacy single dispatcher.
- activity_name, activity_queue = self._resolve_activity(node_type)
+ # F4.A: resolve to per-type name + queue when the run's frozen
+ # routing snapshot has the flag on, else fall back to the legacy
+ # single dispatcher. (Simplified: the real loop calls
+ # _resolve_dispatch(..., routing_snapshot=frozen_routing), which
+ # also picks the AgentWorkflow child path.)
+ activity_name, activity_queue = self._resolve_activity(
+ node_type,
+ per_type_dispatch_enabled=frozen_routing["per_type_dispatch_enabled"],
+ worker_pool_enabled=frozen_routing["worker_pool_enabled"],
+ )
start_kwargs = dict(
args=[context],
activity_id=node_id,
# Heartbeat is the liveness mechanism; start_to_close only
- # bounds a single legitimate step (24h; pre-patch histories
- # replay against the old 10min cap).
+ # bounds a single legitimate step (24h; unconditional, no
+ # workflow.patched marker).
start_to_close_timeout=_NODE_ACTIVITY_START_TO_CLOSE,
heartbeat_timeout=timedelta(minutes=2),
retry_policy=activity_retry_policy, # plugin cls.retry_policy wins when declared
@@ -288,7 +297,7 @@ Add more workers = handle more concurrent nodes
### Specialized Worker Pools (Wave 16 — `TEMPORAL_WORKER_POOL_ENABLED`, default on since 16.4)
-`TemporalWorkerPool` is wired into the `main.py` lifespan (starts right after `TemporalWorkerManager`, stops before it). One activity-only Worker per plugin-declared queue polls its specialised queue, and `MachinaWorkflow._resolve_activity` returns `(activity_name, cls.task_queue)` so per-type activities land there. Setting `TEMPORAL_WORKER_POOL_ENABLED=false` stops the pool and routes every activity back to the single manager worker on `machina-tasks` — **the flag is the rollback channel** (locked default-on by `test_task_queue_coverage.py::TestWorkerPoolDefaultOn`).
+`TemporalWorkerPool` is started by the Temporal lifecycle task (`services/temporal/lifecycle.py::_start_execution_engine`, right after `TemporalWorkerManager`) and stopped before it by the `main.py` lifespan teardown. One activity-only Worker per plugin-declared queue polls its specialised queue, and `MachinaWorkflow._resolve_activity` returns `(activity_name, cls.task_queue)` when the run's frozen routing snapshot has `worker_pool_enabled` set, so per-type activities land there. Setting `TEMPORAL_WORKER_POOL_ENABLED=false` (then restarting) stops the pool, and runs started afterwards route every activity to the single manager worker on `machina-tasks` — **the flag is the rollback channel** (locked default-on by `test_task_queue_coverage.py::TestWorkerPoolDefaultOn`). It is not retroactive: `_resolve_activity` receives the flag from the run's `_temporal_routing_v1` snapshot rather than reading `Settings`, so a run (or a deployed trigger) that started with the pool on keeps scheduling onto the specialised queues, which have no poller while the pool is off.
```
Queue: rest-api Queue: ai-heavy Queue: code-exec
@@ -436,11 +445,11 @@ request starts.
`emit_phase(phase, status?)` is a thin helper that schedules `agent.broadcast_progress`. The activity emits `WorkflowEvent.agent_progress` (CloudEvents v1.0, `type="com.opencompany.agent.progress"`) for FE consumers; when `status` is supplied it also drives a raw-dict `update_node_status` for the canvas-glow color (executing / success / error). Same dual-channel pattern F4.A's `_node_activity` uses. When this workflow is itself a delegated child (`context["parent_node_id"]` set), every `emit_phase` call ALSO schedules a second broadcast against the parent's `node_id` with `phase="delegating"` — the parent's canvas badge then advances in real time while the child loops, instead of freezing at "executing" glow until the child completes.
-Each LLM step is one activity and each ordinary tool call is one per-type activity. Team-lead Task Manager assignments are different: persistence happens first, then the lead starts a deterministic detached `DelegatedTaskWorkflow` with `ParentClosePolicy.ABANDON` and receives `queued` immediately. The runner owns the root-wide permit, claim, child `AgentWorkflow`, terminal result/usage persistence, `taskTrigger`, and permit release, so the assigning lead can return without polling. Direct non-team delegation retains the child-workflow path. Non-agent tools and excluded types (`rlm_agent`, `claude_code_agent`) still go through `execute_activity`. Failures surface as durable task failures and trigger review rather than being lost when the lead invocation closes.
+Each LLM step is one activity and each ordinary tool call is one per-type activity. Team-lead Task Manager assignments are different: persistence happens first, then the lead starts a deterministic detached `DelegatedTaskWorkflow` with `ParentClosePolicy.ABANDON` and receives `queued` immediately. The runner owns the root-wide permit, claim, child `AgentWorkflow`, terminal result/usage persistence, `taskTrigger`, and permit release, so the assigning lead can return without polling. Direct non-team delegation retains the child-workflow path. Non-agent tools and excluded types (any agent type outside `AGENT_WORKFLOW_TYPES`: `rlm_agent`, `claude_code_agent`, `vertex_managed_agent`) still go through `execute_activity`. Failures surface as durable task failures and trigger review rather than being lost when the lead invocation closes.
-Durable event listeners retain their deployment graph as a fallback, but each firing resolves the latest persisted workflow graph through `load_persisted_workflow_graph_activity` before filtering downstream nodes. This makes tools added after deployment available to `taskTrigger` and other triggered agent runs. Edge traversal accepts both canonical `targetHandle` and legacy `target_handle`; tool choice remains entirely with the agent and no trigger-specific tool-use prompt is injected.
+Which graph a firing runs depends on the deployment kind. Controlled generations (the listener payload carries a `data_scope_id`) execute the graph snapshot carried in their trigger registration on every firing, with no per-firing graph lookup. Legacy uncontrolled deployments (no `data_scope_id`) keep the hot lookup: each push or poll firing in `TriggerListenerWorkflow._spawn_child_run` / `PollingTriggerWorkflow._spawn_child_run` resolves the latest persisted workflow graph through `load_persisted_workflow_graph_activity` before filtering downstream nodes, falling back to the deployment snapshot if the lookup fails; that is what makes tools added after deployment available to `taskTrigger` and other triggered agent runs on those deployments. Edge traversal accepts both canonical `targetHandle` and legacy `target_handle`; tool choice remains entirely with the agent and no trigger-specific tool-use prompt is injected.
-**Delegation input contract (input-vs-config separation).** The LLM's `{task, context}` args are per-invocation *input*, not node configuration, and travel as the child workflow input's `invocation` field. `prepare_agent_payload` applies it AFTER its config resolution (`{**node_data, **db_params}` — DB wins for config liveness): `task` → system_message, `context`-or-`task` → prompt — the same semantics as the legacy `handlers.tools._execute_delegated_agent`. Stored node parameters (including the empty default `prompt` the frontend persists on drop) therefore never override the delegated task. A call with both fields empty is rejected at the parent's call boundary (tool-error message to the LLM, no child spawn). Bypass agents dispatched as plain activities (`rlm_agent` / `claude_code_agent`) instead receive the remap directly in `node_data` — their per-type activity consumes `node_data` verbatim with no DB re-merge.
+**Delegation input contract (input-vs-config separation).** The LLM's `{task, context}` args are per-invocation *input*, not node configuration, and travel as the child workflow input's `invocation` field. `prepare_agent_payload` applies it AFTER its config resolution (`{**node_data, **db_params}` — DB wins for config liveness): `task` → system_message, `context`-or-`task` → prompt — the same semantics as the legacy `handlers.tools._execute_delegated_agent`. Stored node parameters (including the empty default `prompt` the frontend persists on drop) therefore never override the delegated task. A call with both fields empty is rejected at the parent's call boundary (tool-error message to the LLM, no child spawn). Bypass agents dispatched as plain activities (any type outside `AGENT_WORKFLOW_TYPES`: `rlm_agent` / `claude_code_agent` / `vertex_managed_agent`) instead receive the remap directly in `node_data` — their per-type activity consumes `node_data` verbatim with no DB re-merge.
**Canvas-aware tools** opt into receiving the parent workflow's `nodes`/`edges` by declaring `needs_canvas: ClassVar[bool] = True` on their `BaseNode` subclass. The F4.B tool dispatch reads this via `services.node_registry.get_node_class(node_type).needs_canvas` and forwards `context.get("nodes")` / `context.get("edges")` into `tool_payload`; default plugins keep the empty-canvas optimisation. Today only `agentBuilder` opts in (it walks edges to resolve its calling agent and mutates the canvas). Operations inside agentBuilder reload via `database.get_workflow(workflow_id)` so in-run duplicate detection sees mutations from earlier calls in the same workflow run — see [agent_builder section in CLAUDE.md](../CLAUDE.md).
@@ -500,7 +509,7 @@ the tool-result cap apply whether or not compaction is on (see
**Broadcasts inside the loop** wrap `WorkflowEvent` (CloudEvents v1.0) per RFC §6.4: `agent_progress` events (`com.opencompany.agent.progress`) and `node_parameters_updated` events (`com.opencompany.node.parameters.updated`) flow through the `StatusBroadcaster.broadcast_agent_progress` and `StatusBroadcaster.broadcast_node_parameters_updated` wrappers respectively. The latter is reused by the legacy `routers/websocket.py:handle_save_node_parameters` (user-source) and `services/cli_agent/service.py:_persist_memory` (cli-source) — all three emission sites share the same envelope, distinguished by `source_hint` (`"user"` / `"cli"` / `"agent"`).
-`rlm_agent`, `claude_code_agent` are NOT migrated — their internal session state (RLM REPL / Claude CLI `--resume` with stable `cwd`) requires single-process continuity and would break across activity boundaries.
+`rlm_agent`, `claude_code_agent` and `vertex_managed_agent` are NOT migrated (they are absent from `AGENT_WORKFLOW_TYPES`) — their internal session state (RLM REPL / Claude CLI `--resume` with stable `cwd` / Vertex Interactions API `previous_interaction_id` and environment chaining) requires single-process continuity and would break across activity boundaries.
References: [Temporal AI Cookbook](https://docs.temporal.io/ai-cookbook), [`temporal-community/temporal-ai-agent`](https://github.com/temporal-community/temporal-ai-agent), [`temporalio.contrib.openai_agents`](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/openai_agents).
@@ -694,7 +703,7 @@ The Temporal binary + persistence are managed in-process by the plugin-folder pa
Full recovery-policy semantics: [temporal-workflow-control.md → Recovery policies](./temporal-workflow-control.md#recovery-policies).
-The legacy `TEMPORAL_SERVER_READY_TIMEOUT_SECONDS` knob (CLI-supervised-era readiness wait) was removed — it had no consumer since the backend-owned cutover. The temporary agent-engine cutover variable that was read directly rather than through `Settings` is gone too: the engine selector is now recorded per execution by `agent.prepare_payload` (see above).
+The legacy `TEMPORAL_SERVER_READY_TIMEOUT_SECONDS` knob (CLI-supervised-era readiness wait) was removed — it had no consumer since the backend-owned cutover. The temporary agent-engine cutover variable that was read directly rather than through `Settings` is gone too, and nothing replaced it: there is one engine, and no engine selector is recorded per execution (see the `agent.prepare_payload` note above).
## Debugging
diff --git a/server/services/deployment/canary_registry.py b/server/services/deployment/canary_registry.py
index 143ff5f6..5abe2a03 100644
--- a/server/services/deployment/canary_registry.py
+++ b/server/services/deployment/canary_registry.py
@@ -23,7 +23,7 @@ Membership semantics (set-shaped):
register_canary_trigger_type("webhookTrigger", "com.opencompany.webhook.received")
register_canary_trigger_type("chatTrigger", "com.opencompany.chat.message.received")
is_canary_trigger_type("webhookTrigger") # True
- is_canary_trigger_type("whatsappReceive") # False (not yet opted in)
+ is_canary_trigger_type("twitterReceive") # False (not opted in)
cloudevent_type_for("webhookTrigger") # "com.opencompany.webhook.received"
Idempotent on re-import (multiple registrations of the same type with
diff --git a/server/services/events/dispatch.py b/server/services/events/dispatch.py
index 04356e79..26c1e145 100644
--- a/server/services/events/dispatch.py
+++ b/server/services/events/dispatch.py
@@ -11,16 +11,17 @@ routes it to:
2. **In-process FastAPI WebSocket clients** via direct
``status_broadcaster.broadcast()`` call. Worker + WS pool share
- memory + event loop, so no IPC hop is needed (audit confirmed —
- ``main.py:211-292``).
+ memory + event loop, so no IPC hop is needed (the ``main.py``
+ lifespan schedules ``services.temporal.lifecycle.run_temporal_lifecycle``
+ as an in-process task, and that task starts the workers).
No EventDispatchWorkflow, no Redis Streams, no DLQ table. Temporal's
Visibility + Signal API + Event History provide the durability primitives
this function depends on; everything else is a thin pass-through.
-Behind ``Settings.event_framework_enabled`` (default off in Phase A).
-When disabled, :func:`emit` is a no-op pass-through that returns the
-envelope unchanged.
+Behind ``Settings.event_framework_enabled`` (default on since Wave 12;
+``EVENT_FRAMEWORK_ENABLED=false`` is the rollback). When disabled,
+:func:`emit` is a no-op pass-through that returns the envelope unchanged.
"""
from __future__ import annotations
@@ -74,9 +75,9 @@ async def emit(
Returns:
The envelope unchanged — callers may chain.
- Behaviour when ``Settings.event_framework_enabled=False`` (Phase A
- default): pass-through no-op. Logged at DEBUG so opt-in dogfooding
- is observable without flipping the flag globally.
+ Behaviour when ``Settings.event_framework_enabled=False`` (the
+ rollback setting; the default is True): pass-through no-op. Logged at
+ DEBUG so a rolled-back process is observable.
"""
if not Settings().event_framework_enabled:
logger.debug(
@@ -250,8 +251,9 @@ async def _signal_one(client, workflow_id: str, event: WorkflowEvent) -> None:
async def _broadcast_in_process(event: WorkflowEvent, wire_routing_key: str) -> None:
"""Direct in-process WS fan-out via the status broadcaster.
- Same asyncio event loop as the FastAPI handlers — see ``main.py:
- 211-292`` (TemporalWorkerManager starts as ``asyncio.create_task``).
+ Same asyncio event loop as the FastAPI handlers — the ``main.py``
+ lifespan schedules ``run_temporal_lifecycle`` with
+ ``asyncio.create_task``, and it starts the TemporalWorkerManager.
Activity → broadcaster is a direct method call against in-memory
``Set[WebSocket]``; no IPC.
"""
diff --git a/server/services/temporal/polling_trigger_workflow.py b/server/services/temporal/polling_trigger_workflow.py
index 129c0d63..3839ed35 100644
--- a/server/services/temporal/polling_trigger_workflow.py
+++ b/server/services/temporal/polling_trigger_workflow.py
@@ -75,7 +75,8 @@ _HISTORY_SOFT_CAP = 10_000
# interval burns ~100K+ history events/day. Mirrors the plugin-side
# clamp (PollingTriggerNode.poll_interval_clamp) that the legacy
# asyncio path applies but workflow payloads historically did not.
-# Timer durations are recorded commands, so the clamp rides the patch.
+# Applied unconditionally in ``run`` (no ``workflow.patched`` marker), as
+# is the identical floor in WorkflowControlWorkflow's polling path.
_MIN_POLL_INTERVAL_S = 30
diff --git a/server/services/temporal/workflow.py b/server/services/temporal/workflow.py
index 1835f572..16d56707 100644
--- a/server/services/temporal/workflow.py
+++ b/server/services/temporal/workflow.py
@@ -404,10 +404,10 @@ class MachinaWorkflow:
# session-keyed nodes (browser) share one instance per run instead
# of minting a fresh uuid per call (node_executor.py fallback).
execution_id = workflow_data.get("execution_id") or workflow.info().workflow_id
- # Command attributes are part of Temporal Event History. Existing
- # histories must keep the label-derived child ids they recorded;
- # new histories use the root Temporal id + exact canvas node id so
- # two same-label agents can start in the same graph execution.
+ # Dispatch flags come from the routing snapshot the starter froze
+ # into the workflow input (``_temporal_routing_v1``), never from
+ # Settings, so an environment change cannot alter how a run that
+ # is already in flight dispatches.
frozen_routing = _frozen_routing_from_input(workflow_data)
workflow.logger.info(f"Starting workflow orchestration: {len(nodes)} nodes, {len(edges)} edges")
@@ -615,11 +615,11 @@ class MachinaWorkflow:
# blocking on child completion). The handle is a
# Task-like with ``.done()`` so it slots into the same
# FIRST_COMPLETED loop as activity handles.
- # Patch-gated for replay safety. The legacy label-derived
- # id is preserved for histories recorded before
- # ``machina-agent-child-id-v2``. New runs use the actual
- # root Temporal workflow id plus the exact canvas node id;
- # labels are mutable and need not be unique.
+ # Not patch-gated: every run, new or replayed, derives the
+ # id from the actual root Temporal workflow id plus the
+ # exact canvas node id (labels are mutable and need not
+ # be unique). There is no ``workflow.patched`` marker
+ # and no label-derived fallback for this id.
child_workflow_id = _agent_child_workflow_id(
workflow.info().workflow_id,
node_id,
@@ -656,8 +656,8 @@ class MachinaWorkflow:
activity_id=node_id,
# Heartbeat is the liveness mechanism (activities
# self-heartbeat every 30s); start_to_close only
- # bounds a single legitimate step. Pre-patch
- # histories replay against the old 10min cap.
+ # bounds a single legitimate step. Unconditional:
+ # no ``workflow.patched`` marker gates this value.
start_to_close_timeout=_NODE_ACTIVITY_START_TO_CLOSE,
heartbeat_timeout=timedelta(minutes=2),
retry_policy=activity_retry_policy,
@@ -749,9 +749,12 @@ class MachinaWorkflow:
— F4.A per-type activity OR legacy fallback, depending on
``temporal_per_type_dispatch``.
- New histories pass ``routing_snapshot`` from workflow input so no
- mutable Settings value can alter their command shape. ``None`` is the
- replay-only compatibility path for pre-patch histories.
+ ``MachinaWorkflow.run`` always passes ``routing_snapshot`` from the
+ workflow input (``_frozen_routing_from_input`` substitutes
+ ``_SAFE_FROZEN_ROUTING`` when the input carries none), so no mutable
+ Settings value can alter a run's command shape. ``None`` falls back
+ to reading Settings and is reached only by direct callers such as
+ unit tests.
"""
if routing_snapshot is None:
from core.config import Settings
@@ -801,26 +804,28 @@ class MachinaWorkflow:
) -> tuple[str, str | None]:
"""Resolve (activity_name, task_queue) for a node type.
- F4.A: when ``settings.temporal_per_type_dispatch`` is on AND the
- plugin class is registered, returns the per-type activity name
+ F4.A: when per-type dispatch is on AND the plugin class is
+ registered, returns the per-type activity name
``node.{type}.v{version}``.
- Wave 16.3: the returned queue is ``cls.task_queue`` when
- ``settings.temporal_worker_pool_enabled`` is on (each declared
- queue then has a dedicated ``TemporalWorkerPool`` worker polling
- it — wired in main.py right after the manager starts), or
- ``None`` otherwise so the activity stays on the workflow's
- default queue, which the single ``TemporalWorkerManager`` polls.
- The flag is the rollback channel: flipping it off routes every
- activity back to the manager worker without code changes.
+ Wave 16.3: the returned queue is ``cls.task_queue`` when the worker
+ pool is on (each declared queue then has a dedicated
+ ``TemporalWorkerPool`` worker polling it — started by
+ ``services/temporal/lifecycle.py::_start_execution_engine`` right
+ after the manager), or ``None`` otherwise so the activity stays on
+ the workflow's default queue, which the single
+ ``TemporalWorkerManager`` polls. The flag is the rollback channel,
+ but only for runs started after the flip: ``MachinaWorkflow.run``
+ passes both flags from the run's frozen routing snapshot, and only
+ an omitted argument falls back to reading Settings.
Falls back to ``("execute_node_activity", None)`` when:
- the flag is off (preserves pre-F4.A behavior exactly), OR
- the node type isn't registered as a BaseNode subclass
(covers legacy types still on the metadata-only path).
- Determinism: lookups go through frozen module-level dicts
- (``_NODE_CLASS_REGISTRY``, ``Settings``) — no I/O. Safe inside
+ Determinism: lookups go through the frozen module-level
+ ``_NODE_CLASS_REGISTRY`` and the caller-supplied flags — no I/O. Safe inside
``MachinaWorkflow.run`` per the workflow-definition contract.
Imports are inside the method to keep the workflow module's
top-level import set minimal and to avoid import-cycle drift.