Skip to content

Self-improving AI Employees for everyone

fix(Wave 13): canary event framework — EventType SA + canary-only emit + status lifecycle + OOM + templates + cancel sweep

trohitgbae0954cdd21
diff --git a/CLAUDE.md b/CLAUDE.md
index 32c63fce..fcd5e629 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -5233,4 +5233,5 @@ This function:
- **Telegram owner detection — three layers** (f746ddf): the bot owner is captured by THREE independent paths so the realistic setup flows all converge to a working state. (1) **Explicit field**: secondary `FieldDef` in the credentials modal lets the user paste their Telegram user ID directly (recommended; works without DM'ing the bot). (2) **Pre-poll peek** in `connect()` via `_capture_owner_from_pending_updates()` — calls `bot.get_updates(timeout=0)` BEFORE `start_polling(drop_pending_updates=True)` discards the queue, scans for any historical private DM, captures + persists atomically. (3) **Atomic write-through in `_on_message_received`** — persists FIRST, sets in-memory ONLY on success. Invariant: "in-memory has owner ⇒ DB has owner" so a process restart can re-capture cleanly. Failure logged at `ERROR` with `exc_info=True` (was `WARNING` previously, masking persist failures). The lazy fallback in `telegram_send.py` (read DB → `service.set_owner`) sits on top of all three.
- **Supervisor Job Object loud-failure (machina/tree.py + supervisor.py)** (8a64eb9): the Windows process-tree mechanism depends entirely on a Job Object with `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE` (no fallback — `start_new_session` is POSIX-only). `pywin32>=308` is a hard supervisor dependency on Windows; the declaration in `machina/pyproject.toml` carries an inline comment so future maintainers don't drop it. `_JobObject.__init__` splits failures into explicit `ImportError` (with reinstall command) and `Exception` (with `repr`) branches — both write to `stderr` instead of swallowing. `_JobObject.add()` verifies enrollment via `win32job.IsProcessInJob(handle, self._handle)` after assignment; on mismatch it queries `IsProcessInJob(handle, None)` so the warning explicitly names the wrapper-job hypothesis (npm / pnpm / conhost wrappers occasionally place us in their own non-nesting Job). `supervisor._spawn_once` checks `add_to_job` return value and emits a yellow `WARN: pid=N not enrolled in Job Object` per child if False. Without these guards, a stale pywin32 install or a wrapping Job Object silently leaks orphan Python processes on every Ctrl-C and they accumulate to the point of holding SQLite locks that block subsequent backend startup.
- **Credentials envelope-shape invariant** (dc94cde): the `useCredentialPanel` query stores `{values: CredentialFormValues, hadStored: boolean}` (an envelope, NOT a flat dict). The earlier `writeValues` called `setQueryData<CredentialFormValues>` with a spread `(prev) => ({...prev, [key]: value})`, which at runtime merged the typed character at the envelope level next to `values` and `hadStored` instead of inside `.values` — the input selector `panel.values[field.key]` re-rendered with the original (server) value on every keystroke and the input felt frozen. Fix: `writeValues` now preserves the envelope shape and updates only the inner `values` dict; `hadStored` is preserved verbatim (it reflects real server state, not local edits). When introducing other queries shaped as `{data, meta}` envelopes, follow the same pattern — never spread `prev` as if it were the inner payload.
+- **Wave 13 canary fixes** — six load-bearing corrections to the Wave 12 event framework. See [docs-internal/event_framework.md → Wave 13 fixes](./docs-internal/event_framework.md#wave-13-fixes) for the full breakdown. Key invariants to remember when working on canary triggers: (1) `register_canary_trigger_type(node_type, cloudevent_type)` requires the CloudEvents reverse-DNS string as second arg — must match the producer's `WorkflowEvent.type` exactly or the deployment manager's `EventType` SA won't match `dispatch.emit`'s Visibility query (silent firing failure). Diverging re-registration raises `ValueError` so plugin upgrades surface loudly. (2) Plugin `_events.py` for canary-registered triggers is canary-only — no `event_waiter.dispatch` / `send_custom_event` calls (those have zero consumers in canary-on mode). `dispatch.emit(envelope, wire_routing_key=...)` handles BOTH Temporal Signal fan-out AND in-process WS broadcast. (3) `TriggerListenerWorkflow` + `PollingTriggerWorkflow` call `broadcast_trigger_status_activity` before/after each child spawn for firing-pulse UX (matches legacy `triggers.py` collector/processor). (4) `PollingTriggerNode.as_poll_activity` returns `seen_ids: list(current)` — NOT `list(prior_seen | current)` (the latter grows unboundedly; Gmail at ~100/day hit ~36K entries in a year). The legacy `_build_poll_coroutine` does `seen = set(current)` at end of cycle for the same reason. Visibility-filtered providers (Gmail-unread) re-emit on re-surface, which is correct. (5) Canary trigger output persistence: `MachinaWorkflow.run`'s pre-executed loop schedules `store_node_output_activity` for every firing trigger so `ParameterResolver` can resolve `{{triggerNode.field}}` in downstream nodes (the legacy `_execute_from_trigger` did this via `_store_output(trigger_node_id, "output_0", ...)`; canary skipped it pre-fix). Skips non-firing siblings (`_trigger_output={"not_triggered": True}`). (6) `DeploymentManager.cancel` sweeps stuck node statuses via `_clear_stuck_node_statuses(workflow_id, include_waiting=True)` + emits terminal `update_workflow_status(executing=False, workflow_id=...)`. Without these the FE leaves downstream nodes glowing forever after deployment cancel and the toolbar Start/Stop indicator stays at `executing=True`. The delegation guard inside `_clear_stuck_node_statuses` still protects in-flight fire-and-forget child agents.
- never use emojis in prints
\ No newline at end of file
diff --git a/docs-internal/event_framework.md b/docs-internal/event_framework.md
index 75b2a473..9b790566 100644
--- a/docs-internal/event_framework.md
+++ b/docs-internal/event_framework.md
@@ -30,6 +30,7 @@ phase plan lives in `~/.claude/plans/properly-fix-the-tech-dreamy-tarjan.md`.
| B11 — FE `plugin_connection_status` envelope handler | ✅ commit `899771c` |
| D4 — Drop legacy `*_status` raw frames (status-only round) | ✅ commit `5ea4e90` (whatsapp/android/telegram status retired; FE consumes typed channel) |
| Canary flag default flip + invariant lock | ✅ this commit (default `True`; `EVENT_FRAMEWORK_ENABLED=false` is the rollback) |
+| Wave 13 — EventType SA mismatch fix + canary-only emit path + trigger status lifecycle + polling OOM + template resolution + cancel sweep | ✅ shipped (see "Wave 13 fixes" below) |
### Dropped / Deferred / Pending
@@ -201,6 +202,68 @@ This is why Wave 12 explicitly does NOT add a custom `event_dlq` SQLModel table.
Wave 12 D3 (pending) adds thin WS handlers that wrap these Visibility queries for the FE admin surface, so operators can inspect failed runs without leaving the MachinaOs UI.
+## Wave 13 fixes
+
+Six load-bearing corrections to the Wave 12 canary path. All locked by regression tests.
+
+### 1. `EventType` Search Attribute must equal the producer's `event.type`
+
+Pre-fix the deployment manager registered the **legacy snake_case** `event_type` from `TriggerConfig` (e.g. `"chat_message_received"`) as the `EventType` SA on every canary listener. But `dispatch.emit` Visibility-queries by `event.type` from the CloudEvents envelope (reverse-DNS, e.g. `"com.machinaos.chat.message.received"`). The strings never matched — the listener started OK, never reacted to incoming events.
+
+Fix:
+- `register_canary_trigger_type(node_type, cloudevent_type)` now requires the CloudEvents type as a second arg ([canary_registry.py](../server/services/deployment/canary_registry.py)). Re-registering with a diverging `cloudevent_type` raises `ValueError` so plugin upgrades that change the envelope shape surface loudly.
+- `cloudevent_type_for(node_type)` lookup, used by `DeploymentManager._start_canary_listener` for the `EventType` SA value (was `config.event_type`).
+- `WorkflowEvent.task_completed` factory unified to single type `com.machinaos.agent.task.completed` (was `.succeeded` / `.failed` split — broke single-SA listener matching).
+
+Locked by `TestCloudEventTypeMatchesSearchAttribute` in [`test_canary_registry.py`](../server/tests/test_canary_registry.py).
+
+### 2. Plugin `_events.py` is canary-only — no more dual-emit
+
+`event_waiter.dispatch` / `broadcaster.send_custom_event` calls inside canary-registered plugin `_events.py` files (chat / webhook / task / telegram / email) had zero consumers in canary-on mode — the deployment manager skips `setup_event_trigger` when `is_canary_trigger_type(...)` is True, so no legacy waiter ever registered. Removed.
+
+- `dispatch.emit(envelope, wire_routing_key=...)` is now the single delivery path. It Signals running `TriggerListenerWorkflow` consumers via Temporal Visibility AND broadcasts to FE on the same wire key.
+- `google/_events.py:dispatch_gmail_received` deleted (gmail polling uses `PollingTriggerWorkflow`, not `event_waiter`).
+- `whatsapp/_events.py` kept the legacy raw frame on `whatsapp_message_received` for received messages because the FE message-list handler reads `data.sender` (legacy shape) directly — drop blocked on FE migration (D4 follow-up). The duplicate typed-envelope sibling on the same wire key was dropped.
+- `twitter/_events.py` keeps both paths — twitter is the only deferred canary plugin (needs PollingTriggerNode subclass refactor first).
+
+### 3. Trigger node status pulse on every firing
+
+Pre-fix the canary listener broadcast `waiting` once at deploy and stayed there forever — when an event fired, FE saw downstream nodes light up but no visual signal on the trigger node itself. The legacy `services/deployment/triggers.py` collector/processor did `waiting → idle (Graph executing...) → waiting` per event; canary skipped this.
+
+Fix: new `broadcast_trigger_status_activity` ([activities.py](../server/services/temporal/activities.py)). `TriggerListenerWorkflow._spawn_child_run` calls it before + after each child spawn:
+- Before: `status="idle"` with `message="Graph executing..."` data
+- After: `status="waiting"` with the next-event message
+
+`PollingTriggerWorkflow._spawn_child_run` does the same. Matches legacy UX.
+
+### 4. Polling `seen` set OOM leak
+
+Both polling paths grew `seen: Set[str]` unboundedly:
+- `as_poll_activity` returned `seen_ids: list(prior_seen | current)` every cycle — Temporal payload paid for the union forever.
+- `_build_poll_coroutine` called `seen.add(msg_id)` per emit with no eviction.
+
+At Gmail's ~100 msgs/day / 60s poll, this hit ~36K entries / ~1.4MB just for IDs in a year. Fix: rebase `seen = current` at end of every cycle in both paths ([polling.py](../server/services/plugin/polling.py)). Items the provider no longer reports drop out. Bounded by the provider's natural window size.
+
+Semantic note: if a filtered item disappears then reappears (e.g. user marks an email unread again under `is:unread`), it re-emits. That's the correct semantic for visibility-filtered providers; the old unbounded set suppressed legitimate re-emits.
+
+### 5. Template resolution against trigger output (canary path)
+
+Pre-fix the canary path's pre-executed trigger output never reached the workflow output store. `MachinaWorkflow.run` set `outputs[node_id]` in workflow memory only — never persisted. `ParameterResolver._gather_connected_outputs` reads from the store, so `{{triggerNode.field}}` templates in downstream nodes resolved to empty.
+
+The legacy `DeploymentManager._execute_from_trigger` called `_store_output(trigger_node_id, "output_0", trigger_output)` explicitly. Canary skipped it.
+
+Fix: new `store_node_output_activity` ([activities.py](../server/services/temporal/activities.py)). `MachinaWorkflow.run`'s pre-executed loop schedules it for every firing trigger (skips non-firing siblings with `_trigger_output={"not_triggered": True}`). Writes to `output_main` + `output_top` + `output_0` so any downstream edge handle resolves.
+
+### 6. `DeploymentManager.cancel` full status sweep
+
+Pre-fix cancel reset only cron + listener trigger nodes to `idle`. Downstream agents/tools/actions that were mid-execute when cancel hit stayed in `executing` forever on FE. Toolbar Start/Stop indicator also stayed at `executing=True` because no terminal `executing=False` workflow_status broadcast was emitted.
+
+Fix ([manager.py](../server/services/deployment/manager.py) `DeploymentManager.cancel`):
+- `await broadcaster._clear_stuck_node_statuses(workflow_id, include_waiting=True)` — sweeps every node still broadcast as `executing`/`waiting`. The delegation guard inside `_clear_stuck_node_statuses` still protects in-flight fire-and-forget child agents.
+- `await broadcaster.update_workflow_status(executing=False, workflow_id=...)` — terminal toolbar state. Avoids relying on `workflow_run_ended`'s counter (can race against in-flight `workflow_run_started` callers).
+
+Locked by `TestCancelSweepsStuckNodeStatuses` in [`test_deployment_canary_listener.py`](../server/tests/test_deployment_canary_listener.py).
+
## References
- Plan: `~/.claude/plans/properly-fix-the-tech-dreamy-tarjan.md`
diff --git a/server/nodes/agent/_events.py b/server/nodes/agent/_events.py
index 87e2ad40..a7166436 100644
--- a/server/nodes/agent/_events.py
+++ b/server/nodes/agent/_events.py
@@ -1,34 +1,33 @@
-"""Wave 12 B8: CloudEvents broadcaster wrappers for agent task delegation.
+"""CloudEvents broadcaster wrappers for agent task delegation.
Plugin-folder-owned wrappers around the cross-cutting
-``WorkflowEvent.task_completed`` factory (cross-cutting factory itself
-stays in ``services/events/envelope.py`` per RFC §6.4 — it's
-parametrised by ``task_id`` + ``agent`` + ``status`` and uniformly
-applies to every delegated agent task, not single-plugin shape).
-
-What lives here vs the central factory:
- - Central ``WorkflowEvent.task_completed(task_id, status, agent, data)``
- factory STAYS in services/events/envelope.py (cross-cutting).
- - The broadcaster *wrappers* below live with the agent plugin folder
- so the 3 callsites in ``services/handlers/tools.py`` (the
- fire-and-forget delegation dispatcher) import from
- ``nodes.agent._events`` rather than building the wire frame
- inline.
-
-Wire-routing key ``"task_completed"`` is preserved — the
-``taskTrigger`` plugin filter still subscribes on this string. The
-inner envelope carries the spec-compliant ``com.machinaos.agent.task.{
-succeeded, failed}`` type.
+``WorkflowEvent.task_completed`` factory. The factory itself stays in
+``services/events/envelope.py`` per RFC §6.4 — it's parametrised by
+``task_id`` + ``agent`` + ``status`` and uniformly applies to every
+delegated agent task, not a single-plugin shape. The broadcaster
+wrappers below live with the agent plugin folder so the 3 callsites in
+``services/handlers/tools.py`` (the fire-and-forget delegation
+dispatcher) import from ``nodes.agent._events`` rather than building
+the wire frame inline.
+
+Delivery: a single ``dispatch.emit`` call. ``dispatch.emit`` Signals
+running :class:`TriggerListenerWorkflow` consumers via Temporal
+Visibility AND broadcasts the same envelope to in-process WS clients on
+the ``task_completed`` wire-routing key — one canonical path. Pre-Wave-13
+this also called ``broadcaster.send_custom_event`` for the legacy
+collector path; that's dead since taskTrigger is canary-registered and
+the deployment manager skips ``setup_event_trigger`` for canary types.
"""
from __future__ import annotations
-from typing import Any, Mapping, Optional
+from typing import Any, Optional
-# Legacy wire-routing key. The ``taskTrigger`` plugin filters by this
-# string today (see ``services/event_waiter.py:TRIGGER_REGISTRY``).
-_LEGACY_EVENT_TYPE = "task_completed"
+# Outer wire-routing key. FE consumers (if any) and the legacy
+# back-compat receipt path still switch on this string; the inner
+# envelope carries ``com.machinaos.agent.task.completed``.
+_WIRE_ROUTING_KEY = "task_completed"
async def broadcast_agent_task_completed(
@@ -40,12 +39,7 @@ async def broadcast_agent_task_completed(
workflow_id: Optional[str] = None,
result: Optional[str] = None,
) -> None:
- """Broadcast a delegated agent task completing successfully.
-
- Replaces inline ``broadcaster.send_custom_event('task_completed',
- {...})`` calls in ``services/handlers/tools.py``. Builds the
- payload exactly as the consumer (taskTrigger filter) expects.
- """
+ """Broadcast a delegated agent task completing successfully."""
await _broadcast_task_event(
task_id=task_id,
status="completed",
@@ -66,12 +60,7 @@ async def broadcast_agent_task_failed(
workflow_id: Optional[str] = None,
error: str,
) -> None:
- """Broadcast a delegated agent task failing.
-
- Replaces inline ``broadcaster.send_custom_event('task_completed',
- {status: 'error', ...})`` calls in
- ``services/handlers/tools.py``.
- """
+ """Broadcast a delegated agent task failing."""
await _broadcast_task_event(
task_id=task_id,
status="error",
@@ -94,24 +83,18 @@ async def _broadcast_task_event(
result: Optional[str] = None,
error: Optional[str] = None,
) -> None:
- """Internal helper. Two delivery paths (in priority order):
-
- 1. **Legacy WS broadcast + asyncio.Future event_waiter** via
- ``broadcaster.send_custom_event`` — the in-process collector/
- processor path; default while the canary flag is off.
- 2. **Temporal-durable listeners** via
- ``services.events.dispatch.emit`` (Wave 12 C1 rollout #2). The
- envelope reuses the cross-cutting :meth:`WorkflowEvent.task_completed`
- factory (parametrised by task_id + agent + status — RFC §6.4
- cross-cutting tier). ``emit`` is a no-op when the feature flag
- is off, so the legacy path keeps working unchanged.
-
- Payload shape preserved exactly so the taskTrigger filter and the
- legacy FE consumers don't see any wire change.
+ """Single delivery path: build the envelope, route through
+ :func:`services.events.dispatch.emit`.
+
+ ``emit`` does two things in one call: (a) Visibility query for
+ running :class:`TriggerListenerWorkflow` consumers tagged with
+ ``EventType='com.machinaos.agent.task.completed'`` and signals each;
+ (b) direct in-process WS broadcast on the ``task_completed`` wire
+ key so any FE consumers receive the envelope. Payload shape in
+ ``data`` is exactly what the taskTrigger filter reads.
"""
from services.events.dispatch import emit
from services.events.envelope import WorkflowEvent
- from services.status_broadcaster import get_status_broadcaster
payload: dict[str, Any] = {
"task_id": task_id,
@@ -126,19 +109,13 @@ async def _broadcast_task_event(
elif status == "error" and error is not None:
payload["error"] = error
- broadcaster = get_status_broadcaster()
- await broadcaster.send_custom_event(_LEGACY_EVENT_TYPE, payload)
-
- # Temporal-durable fan-out (Wave 12 C1 rollout #2). Cross-cutting
- # factory because task_completed is parametrised by (task_id,
- # agent, status) and uniformly applies across every agent type.
envelope = WorkflowEvent.task_completed(
task_id=task_id,
status="completed" if status == "completed" else "error",
agent=agent_name,
data=payload,
)
- await emit(envelope, wire_routing_key=_LEGACY_EVENT_TYPE)
+ await emit(envelope, wire_routing_key=_WIRE_ROUTING_KEY)
__all__ = [
diff --git a/server/nodes/email/__init__.py b/server/nodes/email/__init__.py
index 6808e157..0fa40dc2 100644
--- a/server/nodes/email/__init__.py
+++ b/server/nodes/email/__init__.py
@@ -15,4 +15,4 @@ from ._events import dispatch_email_received # noqa: F401 — re-export
from ._filters import build_filter as build_email_filter
register_filter_builder("emailReceive", build_email_filter)
-register_canary_trigger_type("emailReceive")
+register_canary_trigger_type("emailReceive", "com.machinaos.email.message.received")
diff --git a/server/nodes/email/_events.py b/server/nodes/email/_events.py
index a2864a47..3c0cf612 100644
--- a/server/nodes/email/_events.py
+++ b/server/nodes/email/_events.py
@@ -1,22 +1,14 @@
-"""Wave 12 B4 + canary opt-in: CloudEvents factories + dispatcher for email.
-
-Plugin-specific event emission — replaces:
- - ``event_waiter.dispatch("email_received", email_data)`` in
- ``email_receive/__init__.py``
+"""CloudEvents factory + dispatcher for email.
Per RFC plugin_authoring_rfc.md §6.4: plugin-specific factories live in
the plugin folder.
-Legacy ``event_type`` (``"email_received"``) is preserved on the
-dispatch path so the ``emailReceive`` trigger node's ``event_type``
-ClassVar still matches without a coordinated registry-side rename.
-
-Dual-dispatch (canary opt-in, post-2026-05-15):
- - Legacy ``event_waiter.dispatch`` for the in-process collector/processor
- (still production-live for non-canary triggers).
- - Typed ``services.events.dispatch.emit`` for Temporal-durable
- ``TriggerListenerWorkflow`` consumers (when canary registry +
- ``event_framework_enabled`` are both on — both default-true).
+Delivery: single ``dispatch.emit`` call routes events to running
+:class:`TriggerListenerWorkflow` consumers via Temporal Visibility AND
+broadcasts the envelope to FE on the ``email_received`` wire key.
+emailReceive is canary-registered (see ``nodes/email/__init__.py``) so
+the deployment manager skips ``setup_event_trigger`` and the legacy
+``event_waiter.dispatch`` path has zero consumers — removed.
"""
from __future__ import annotations
@@ -26,12 +18,10 @@ from typing import Any, Mapping
from services.events.envelope import WorkflowEvent
-# Legacy event_type the event_waiter dispatches by; trigger nodes
-# subscribe on this string (matches ``EmailReceiveNode.event_type``).
-_LEGACY_EVENT_TYPE = "email_received"
-
-
-# ---- Typed factory ---------------------------------------------------------
+# Outer wire-routing key. Matches ``EmailReceiveNode.event_type`` and
+# the FE WS channel; the inner envelope carries
+# ``com.machinaos.email.message.received``.
+_WIRE_ROUTING_KEY = "email_received"
def email_message_received(email_data: Mapping[str, Any]) -> WorkflowEvent:
@@ -48,42 +38,15 @@ def email_message_received(email_data: Mapping[str, Any]) -> WorkflowEvent:
)
-# ---- Dispatcher wrapper ----------------------------------------------------
-
-
-async def dispatch_email_received(email_data: Mapping[str, Any]) -> int:
- """Dispatch an incoming email to waiting ``emailReceive`` trigger
- nodes.
-
- Two delivery paths (mirrors :func:`telegram._events.dispatch_telegram_message_received`):
-
- 1. **Legacy event_waiter waiters** via :func:`event_waiter.dispatch`
- (in-process collector/processor; still default for trigger nodes
- outside the canary registry).
- 2. **Temporal-durable listeners** via
- :func:`services.events.dispatch.emit`. ``emit`` is a pass-through
- no-op when ``event_framework_enabled`` is off; otherwise it
- Visibility-queries running ``TriggerListenerWorkflow`` instances
- and signals each.
-
- Returns the count of legacy waiters resolved.
- """
- from services import event_waiter
+async def dispatch_email_received(email_data: Mapping[str, Any]) -> None:
+ """Dispatch an incoming email via the canary CloudEvents path."""
from services.events.dispatch import emit
- payload = dict(email_data)
- resolved = event_waiter.dispatch(_LEGACY_EVENT_TYPE, payload)
-
- # Temporal-durable fan-out (canary opt-in via
- # ``register_canary_trigger_type("emailReceive")`` in
- # ``nodes/email/__init__.py``).
await emit(
- email_message_received(payload),
- wire_routing_key=_LEGACY_EVENT_TYPE,
+ email_message_received(dict(email_data)),
+ wire_routing_key=_WIRE_ROUTING_KEY,
)
- return resolved
-
__all__ = [
"dispatch_email_received",
diff --git a/server/nodes/google/__init__.py b/server/nodes/google/__init__.py
index 32a95ddd..11155443 100644
--- a/server/nodes/google/__init__.py
+++ b/server/nodes/google/__init__.py
@@ -35,7 +35,6 @@ from services.ws_handler_registry import (
)
from . import _router
-from ._events import dispatch_gmail_received # noqa: F401 — re-export
from ._filters import build_gmail_filter
from ._handlers import WS_HANDLERS
from ._option_loaders import (
@@ -65,4 +64,4 @@ register_option_loader("googleTasklists", load_tasklists)
# Temporal activity is emitted by GmailReceiveNode.as_poll_activity()
# and registered via collect_polling_activities() in the worker. See
# services/deployment/canary_registry.py.
-register_canary_trigger_type("googleGmailReceive")
+register_canary_trigger_type("googleGmailReceive", "com.machinaos.gmail.message.received")
diff --git a/server/nodes/google/_events.py b/server/nodes/google/_events.py
index b04882db..73a575c1 100644
--- a/server/nodes/google/_events.py
+++ b/server/nodes/google/_events.py
@@ -1,20 +1,24 @@
-"""Wave 12 B5: CloudEvents factories + broadcaster wrappers for google.
+"""CloudEvents factory for gmail.
-Plugin-specific event emission — replaces:
- - ``event_waiter.dispatch("gmail_email_received", email_data)`` in
- ``gmail_receive/__init__.py``.
+Per RFC plugin_authoring_rfc.md §6.4: plugin-specific factories live in
+the plugin folder.
+
+No dispatcher wrapper: the canary delivery path for ``googleGmailReceive``
+is :class:`services.temporal.polling_trigger_workflow.PollingTriggerWorkflow`
+(not Signal-based). The polling activity (built by
+:meth:`services.plugin.polling.PollingTriggerNode.as_poll_activity`)
+returns the raw email dict back to the workflow which spawns the child
+MachinaWorkflow directly — no envelope-on-the-wire step is needed.
+
+The factory below is kept for parity with the other plugin ``_events.py``
+modules (and for any future ad-hoc emit usage); the legacy
+``dispatch_gmail_received`` shim was deleted in Wave 13 because it
+dispatched to ``event_waiter`` waiters that were never registered in
+canary-on mode (the polling workflow owns delivery).
OAuth completion broadcasts route through the cross-cutting
``StatusBroadcaster.broadcast_credential_event("credential.oauth.connected", ...)``
path (RFC §6.4 cross-cutting tier) — NOT a plugin-named broadcast.
-No migration needed here for that flow.
-
-Per RFC plugin_authoring_rfc.md §6.4: plugin-specific factories live
-in the plugin folder.
-
-Legacy ``event_type`` (``"gmail_email_received"``) is preserved so
-the ``gmailReceive`` trigger node's ``event_type`` ClassVar still
-matches without a coordinated registry-side rename.
"""
from __future__ import annotations
@@ -24,14 +28,6 @@ from typing import Any, Mapping
from services.events.envelope import WorkflowEvent
-# Legacy event_type the event_waiter dispatches by; trigger nodes
-# subscribe on this string (matches ``GmailReceiveNode.event_type``).
-_LEGACY_EVENT_TYPE = "gmail_email_received"
-
-
-# ---- Typed factory ---------------------------------------------------------
-
-
def gmail_message_received(email_data: Mapping[str, Any]) -> WorkflowEvent:
"""Incoming Gmail message envelope. ``subject`` is the Gmail
message_id (correlation key for the consumer-side ``markAsRead``
@@ -46,18 +42,6 @@ def gmail_message_received(email_data: Mapping[str, Any]) -> WorkflowEvent:
)
-# ---- Dispatcher wrapper ----------------------------------------------------
-
-
-def dispatch_gmail_received(email_data: Mapping[str, Any]) -> int:
- """Dispatch an incoming Gmail message to waiting ``gmailReceive``
- trigger nodes. Returns the count of resolved waiters."""
- from services import event_waiter
-
- return event_waiter.dispatch(_LEGACY_EVENT_TYPE, dict(email_data))
-
-
__all__ = [
- "dispatch_gmail_received",
"gmail_message_received",
]
diff --git a/server/nodes/google/gmail_receive/__init__.py b/server/nodes/google/gmail_receive/__init__.py
index 7a697b33..1356df31 100644
--- a/server/nodes/google/gmail_receive/__init__.py
+++ b/server/nodes/google/gmail_receive/__init__.py
@@ -169,10 +169,11 @@ class GmailReceiveNode(PollingTriggerNode):
"gmail", node_id, "receive", 1,
{"workflow_id": context.workflow_id, "session_id": context.session_id},
)
- # Wave 12 B5: route through plugin _events.py wrapper.
- from nodes.google import dispatch_gmail_received
-
- dispatch_gmail_received(email_data)
+ # Inline Run path: the node returns ``email_data`` as
+ # its output and the activity wrapper broadcasts
+ # ``success`` status to FE. No separate dispatch
+ # needed — deployment mode uses PollingTriggerWorkflow
+ # which owns event fan-out in-workflow.
logger.info(
f"[GmailReceive] New email: {email_data.get('subject', 'no subject')}",
)
diff --git a/server/nodes/scheduler/cron_scheduler/__init__.py b/server/nodes/scheduler/cron_scheduler/__init__.py
index fb430d54..42e380fa 100644
--- a/server/nodes/scheduler/cron_scheduler/__init__.py
+++ b/server/nodes/scheduler/cron_scheduler/__init__.py
@@ -42,7 +42,7 @@ logger = get_logger(__name__)
register_temporal_plugin(
SimplePlugin(name="cron-scheduler", workflows=[CronTriggerWorkflow])
)
-register_canary_trigger_type("cronScheduler")
+register_canary_trigger_type("cronScheduler", "com.machinaos.cron.tick")
def _calculate_wait_seconds(p: Dict[str, Any]) -> int:
diff --git a/server/nodes/telegram/__init__.py b/server/nodes/telegram/__init__.py
index 39405e00..50929ce1 100644
--- a/server/nodes/telegram/__init__.py
+++ b/server/nodes/telegram/__init__.py
@@ -65,7 +65,7 @@ register_output_schema("telegramSend", TelegramSendOutput)
# unconditionally; emit() is gated by Settings.event_framework_enabled
# so the legacy path stays default. See
# services/deployment/canary_registry.py.
-register_canary_trigger_type(TelegramReceiveNode.type)
+register_canary_trigger_type(TelegramReceiveNode.type, "com.machinaos.telegram.message.received")
__all__ = [
"TelegramCredential",
diff --git a/server/nodes/telegram/_events.py b/server/nodes/telegram/_events.py
index 35b2b81f..e539437c 100644
--- a/server/nodes/telegram/_events.py
+++ b/server/nodes/telegram/_events.py
@@ -144,39 +144,22 @@ async def broadcast_telegram_status(
})
-async def dispatch_telegram_message_received(event_data: Mapping[str, Any]) -> int:
- """Dispatch an incoming Telegram message to waiting
- ``telegramReceive`` trigger nodes.
-
- Two delivery paths (in priority order):
-
- 1. **Legacy event_waiter waiters** via :func:`event_waiter.dispatch`
- (in-process collector/processor; default while the canary flag
- is off).
- 2. **Temporal-durable listeners** via
- :func:`services.events.dispatch.emit` (Wave 12 C1 rollout #3).
- ``emit`` is a no-op pass-through when the feature flag is off,
- so the legacy path keeps working unchanged.
-
- Returns the count of legacy waiters resolved.
+async def dispatch_telegram_message_received(event_data: Mapping[str, Any]) -> None:
+ """Dispatch an incoming Telegram message via the canary CloudEvents path.
+
+ Single delivery: :func:`services.events.dispatch.emit` Signals running
+ :class:`TriggerListenerWorkflow` consumers via Temporal Visibility AND
+ broadcasts the envelope to FE on the ``telegram_message_received``
+ wire key. telegramReceive is canary-registered so no legacy
+ ``event_waiter`` waiter is ever registered for it.
"""
- from services import event_waiter
from services.events.dispatch import emit
- payload = dict(event_data)
- # Trigger nodes subscribe by event_type string; keep the legacy
- # string so the `telegramReceive.event_type` ClassVar still matches.
- resolved = event_waiter.dispatch(_MESSAGE_LEGACY_EVENT_TYPE, payload)
-
- # Temporal-durable fan-out (Wave 12 C1 rollout #3). No-op when
- # event_framework_enabled is off.
await emit(
- telegram_message_received(payload),
+ telegram_message_received(dict(event_data)),
wire_routing_key=_MESSAGE_LEGACY_EVENT_TYPE,
)
- return resolved
-
__all__ = [
"broadcast_telegram_status",
diff --git a/server/nodes/telegram/_service.py b/server/nodes/telegram/_service.py
index 0a9d12fd..1e0b8ff6 100644
--- a/server/nodes/telegram/_service.py
+++ b/server/nodes/telegram/_service.py
@@ -493,12 +493,11 @@ class TelegramService(ServiceSingleton):
f"chat_type={event_data.get('chat_type')}"
)
- # Wave 12 B3: route via plugin _events.py wrapper (single
- # source of truth for the wire format).
+ # Route via plugin _events.py wrapper — canary CloudEvents
+ # path (Visibility-query Signal + WS broadcast).
from . import dispatch_telegram_message_received
- resolved = await dispatch_telegram_message_received(event_data)
- logger.info(f"[Telegram] Dispatched to {resolved} waiter(s)")
+ await dispatch_telegram_message_received(event_data)
except Exception as e:
logger.error(f"[Telegram] Message handler error: {e}")
diff --git a/server/nodes/tool/agent_builder/_events.py b/server/nodes/tool/agent_builder/_events.py
index 5e6400cc..94cd3049 100644
--- a/server/nodes/tool/agent_builder/_events.py
+++ b/server/nodes/tool/agent_builder/_events.py
@@ -1,18 +1,19 @@
-"""Wave 12 B10: CloudEvents factory + broadcaster wrapper for agent_builder.
-
-Plugin-specific event emission — replaces the inline
-``broadcaster.send_custom_event("workflow_ops_apply", {...})`` call in
-``agent_builder/__init__.py:_broadcast``.
-
-The agent_builder canvas-mutation broadcast carries a flat shape
-(``{workflow_id, caller_node_id, operations}``) consumed by the
-frontend's ``useWorkflowOpsListener`` hook directly (NOT via the
-typed-envelope path). Per the existing X4-allowlist rationale, the
-WIRE format stays flat for FE back-compat; the typed envelope is
-constructed for audit/future use only.
-
-Per RFC plugin_authoring_rfc.md §6.4: plugin-specific factories live
-in the plugin folder.
+"""CloudEvents factory + broadcaster wrapper for agent_builder.
+
+Per RFC plugin_authoring_rfc.md §6.4: plugin-specific factories live in
+the plugin folder.
+
+This is NOT a trigger event — it's a canvas-mutation broadcast consumed
+by the frontend's ``useWorkflowOpsListener`` hook. The wire format is
+flat (``{workflow_id, caller_node_id, operations}``) for FE back-compat;
+the typed envelope is constructed for parity with the other plugin
+``_events.py`` modules but not consumed today.
+
+Delivery: direct WS broadcast (no ``dispatch.emit`` — no
+:class:`TriggerListenerWorkflow` consumer exists for canvas mutations,
+and no legacy ``event_waiter`` waiter listens on ``workflow_ops_apply``
+either). Pre-Wave-13 this routed through ``send_custom_event`` which
+also called ``event_waiter.dispatch_async`` — that dispatch was dead.
"""
from __future__ import annotations
@@ -22,11 +23,8 @@ from typing import Any, Dict, List, Optional
from services.events.envelope import WorkflowEvent
-# Legacy wire-routing key consumed by ``client/src/hooks/useWorkflowOpsListener``.
-_LEGACY_EVENT_TYPE = "workflow_ops_apply"
-
-
-# ---- Typed factory ---------------------------------------------------------
+# Flat wire-routing key consumed by ``client/src/hooks/useWorkflowOpsListener``.
+_WIRE_ROUTING_KEY = "workflow_ops_apply"
def workflow_ops_applied(
@@ -50,18 +48,17 @@ def workflow_ops_applied(
)
-# ---- Broadcaster wrapper ---------------------------------------------------
-
-
async def broadcast_workflow_ops(
*,
workflow_id: Optional[str],
caller_node_id: str,
operations: List[Dict[str, Any]],
) -> None:
- """Emit a workflow_ops_apply event. Routes through
- ``broadcaster.send_custom_event`` (raw flat payload for FE
- back-compat; envelope constructed for future audit use).
+ """Broadcast a canvas-mutation event for the FE to apply.
+
+ Flat payload (FE back-compat with ``useWorkflowOpsListener``); the
+ typed envelope is constructed for audit/future migration but not
+ on the wire today.
"""
if not operations:
return
@@ -73,14 +70,17 @@ async def broadcast_workflow_ops(
"caller_node_id": caller_node_id,
"operations": list(operations),
}
- # Typed envelope constructed for future audit; not on the wire today
- # because FE listener consumes the flat payload via send_custom_event.
+ # Envelope constructed for parity / future migration; not on the
+ # wire today because FE listener reads the flat shape.
_ = workflow_ops_applied(
workflow_id=workflow_id,
caller_node_id=caller_node_id,
operations=operations,
)
- await broadcaster.send_custom_event(_LEGACY_EVENT_TYPE, payload)
+ await broadcaster.broadcast({
+ "type": _WIRE_ROUTING_KEY,
+ "data": payload,
+ })
__all__ = [
diff --git a/server/nodes/trigger/chat_trigger/__init__.py b/server/nodes/trigger/chat_trigger/__init__.py
index 0d7ed808..9058e295 100644
--- a/server/nodes/trigger/chat_trigger/__init__.py
+++ b/server/nodes/trigger/chat_trigger/__init__.py
@@ -77,4 +77,4 @@ class ChatTriggerNode(TriggerNode):
# ./_events.py calls dispatch.emit unconditionally; the canary registry
# tells DeploymentManager to start a listener for this type. See
# services/deployment/canary_registry.py.
-register_canary_trigger_type(ChatTriggerNode.type)
+register_canary_trigger_type(ChatTriggerNode.type, "com.machinaos.chat.message.received")
diff --git a/server/nodes/trigger/chat_trigger/_events.py b/server/nodes/trigger/chat_trigger/_events.py
index ed5c17da..066dce3a 100644
--- a/server/nodes/trigger/chat_trigger/_events.py
+++ b/server/nodes/trigger/chat_trigger/_events.py
@@ -1,19 +1,14 @@
-"""Wave 12 B7: CloudEvents factory + dispatch wrapper for chat_trigger.
+"""CloudEvents factory + dispatch wrapper for chat_trigger.
-Plugin-specific event emission — replaces:
- - ``event_waiter.dispatch("chat_message_received", event_data)`` in
- ``routers/websocket.py:handle_send_chat_message``.
+Per RFC plugin_authoring_rfc.md §6.4: plugin-specific factories live in
+the plugin folder.
-The Chat console message-send handler in the WS router previously
-dispatched the event itself, hardcoding the plugin-specific event_type
-string in framework code. Phase B7 inverts this: the chat_trigger
-plugin owns its own dispatch primitive, the WS handler imports it.
-
-Per RFC plugin_authoring_rfc.md §6.4: plugin-specific factories live
-in the plugin folder.
-
-Legacy ``event_type`` (``"chat_message_received"``) is preserved so
-the ``ChatTriggerNode.event_type`` ClassVar still matches.
+Delivery: single ``dispatch.emit`` call routes events to running
+:class:`TriggerListenerWorkflow` consumers via Temporal Visibility AND
+broadcasts the envelope to FE on the ``chat_message_received`` wire key.
+chatTrigger is canary-registered (see ``nodes/trigger/chat_trigger/__init__.py``)
+so the deployment manager skips ``setup_event_trigger`` and the legacy
+``event_waiter.dispatch`` path has zero consumers — removed.
"""
from __future__ import annotations
@@ -23,12 +18,10 @@ from typing import Any, Mapping
from services.events.envelope import WorkflowEvent
-# Legacy event_type the event_waiter dispatches by; trigger nodes
-# subscribe on this string (matches ``ChatTriggerNode.event_type``).
-_LEGACY_EVENT_TYPE = "chat_message_received"
-
-
-# ---- Typed factory ---------------------------------------------------------
+# Outer wire-routing key. Matches ``ChatTriggerNode.event_type`` and the
+# FE WS channel; the inner envelope carries
+# ``com.machinaos.chat.message.received``.
+_WIRE_ROUTING_KEY = "chat_message_received"
def chat_message_received(event_data: Mapping[str, Any]) -> WorkflowEvent:
@@ -44,37 +37,14 @@ def chat_message_received(event_data: Mapping[str, Any]) -> WorkflowEvent:
)
-# ---- Dispatcher wrapper ----------------------------------------------------
-
-
-async def dispatch_chat_message_received(event_data: Mapping[str, Any]) -> int:
- """Dispatch an incoming chat message to waiting ``chatTrigger`` nodes.
-
- Two delivery paths (in priority order):
-
- 1. **Legacy event_waiter waiters** via :func:`event_waiter.dispatch`
- (in-process collector/processor; default while
- ``Settings.event_framework_enabled`` is off).
- 2. **Temporal-durable listeners** via :func:`services.events.dispatch.emit`
- (Wave 12 C1 canary). ``emit`` is a no-op pass-through when the
- feature flag is off, so the legacy path keeps working unchanged.
-
- Returns the count of legacy waiters resolved (Temporal listeners
- are signalled asynchronously through Visibility queries and don't
- show up in this count — that's by design, the chat panel UI only
- cares about the legacy resolved-count for its "delivered" badge).
- """
- from services import event_waiter
+async def dispatch_chat_message_received(event_data: Mapping[str, Any]) -> None:
+ """Dispatch an incoming chat message via the canary CloudEvents path."""
from services.events.dispatch import emit
- payload = dict(event_data)
- resolved = event_waiter.dispatch(_LEGACY_EVENT_TYPE, payload)
-
- # Temporal-durable fan-out (Wave 12 C1 canary). Always-call;
- # dispatch.emit is a no-op when the feature flag is off.
- await emit(chat_message_received(payload), wire_routing_key=_LEGACY_EVENT_TYPE)
-
- return resolved
+ await emit(
+ chat_message_received(dict(event_data)),
+ wire_routing_key=_WIRE_ROUTING_KEY,
+ )
__all__ = [
diff --git a/server/nodes/trigger/task_trigger/__init__.py b/server/nodes/trigger/task_trigger/__init__.py
index 579cd39e..4a40f5ca 100644
--- a/server/nodes/trigger/task_trigger/__init__.py
+++ b/server/nodes/trigger/task_trigger/__init__.py
@@ -85,4 +85,4 @@ class TaskTriggerNode(TriggerNode):
# consumer path. The producer side lives in nodes/agent/_events.py
# (broadcast_agent_task_completed / _failed dual-emit via dispatch.emit).
# See services/deployment/canary_registry.py for the contract.
-register_canary_trigger_type(TaskTriggerNode.type)
+register_canary_trigger_type(TaskTriggerNode.type, "com.machinaos.agent.task.completed")
diff --git a/server/nodes/trigger/webhook_trigger/__init__.py b/server/nodes/trigger/webhook_trigger/__init__.py
index 7c512622..7365a609 100644
--- a/server/nodes/trigger/webhook_trigger/__init__.py
+++ b/server/nodes/trigger/webhook_trigger/__init__.py
@@ -115,4 +115,4 @@ class WebhookTriggerNode(TriggerNode):
# Gated runtime by Settings.event_framework_enabled in DeploymentManager;
# the producer side (broadcast_webhook_received calling dispatch.emit) is
# unconditional. See services/deployment/canary_registry.py.
-register_canary_trigger_type(WebhookTriggerNode.type)
+register_canary_trigger_type(WebhookTriggerNode.type, "com.machinaos.webhook.received")
diff --git a/server/nodes/trigger/webhook_trigger/_events.py b/server/nodes/trigger/webhook_trigger/_events.py
index 7b16a531..e60192dd 100644
--- a/server/nodes/trigger/webhook_trigger/_events.py
+++ b/server/nodes/trigger/webhook_trigger/_events.py
@@ -1,20 +1,15 @@
-"""Wave 12 B9: CloudEvents factory + dispatch wrapper for webhook_trigger.
-
-Plugin-specific event emission — replaces:
- - ``broadcaster.send_custom_event("webhook_received", webhook_data)``
- in ``routers/webhook.py`` (unmapped-path fallback).
-
-The webhook router previously emitted the event itself, hardcoding the
-plugin-specific event_type string in framework code. Phase B9 inverts
-this: the webhook_trigger plugin owns its dispatch primitive.
-
-Per RFC plugin_authoring_rfc.md §6.4: plugin-specific factories live
-in the plugin folder.
-
-Wire-key behaviour preserved: still goes through
-``broadcaster.send_custom_event(...)`` which (a) does the legacy raw
-WS broadcast AND (b) dispatches the event into ``event_waiter`` so
-``webhookTrigger`` nodes match it.
+"""CloudEvents factory + broadcaster wrapper for webhook_trigger.
+
+Per RFC plugin_authoring_rfc.md §6.4: plugin-specific factories live in
+the plugin folder.
+
+Delivery: single ``dispatch.emit`` call routes events to running
+:class:`TriggerListenerWorkflow` consumers via Temporal Visibility AND
+broadcasts the envelope to FE on the ``webhook_received`` wire key.
+webhookTrigger is canary-registered (see ``nodes/trigger/webhook_trigger/__init__.py``)
+so the deployment manager skips ``setup_event_trigger`` and the legacy
+``broadcaster.send_custom_event`` path (which dispatched to
+``event_waiter``) has zero consumers — removed.
"""
from __future__ import annotations
@@ -24,13 +19,10 @@ from typing import Any, Mapping
from services.events.envelope import WorkflowEvent
-# Legacy wire-routing key. The frontend has no listener for it today
-# (webhook receivers don't have a status panel); the consumer is the
-# ``webhookTrigger`` node via the event_waiter dispatch.
-_LEGACY_EVENT_TYPE = "webhook_received"
-
-
-# ---- Typed factory ---------------------------------------------------------
+# Outer wire-routing key. Matches ``WebhookTriggerNode.event_type`` and
+# the FE WS channel; the inner envelope carries
+# ``com.machinaos.webhook.received``.
+_WIRE_ROUTING_KEY = "webhook_received"
def webhook_received(webhook_data: Mapping[str, Any]) -> WorkflowEvent:
@@ -46,42 +38,14 @@ def webhook_received(webhook_data: Mapping[str, Any]) -> WorkflowEvent:
)
-# ---- Broadcaster wrapper ---------------------------------------------------
-
-
async def broadcast_webhook_received(webhook_data: Mapping[str, Any]) -> None:
- """Broadcast an incoming webhook to ``webhookTrigger`` nodes.
-
- Three delivery paths (kept in this order for back-compat):
-
- 1. **Legacy WS broadcast + asyncio.Future waiters** via
- ``broadcaster.send_custom_event`` (the in-process collector /
- processor path; default while the canary flag is off).
- 2. **Temporal Signal fan-out** via ``services.events.dispatch.emit``
- (Wave 12 C1 canary). Gated by ``Settings.event_framework_enabled``;
- inside :func:`emit` the flag-off branch is a no-op pass-through
- so it's cheap to always invoke. When on, the dispatch helper
- runs a Visibility query and signals every running
- :class:`TriggerListenerWorkflow` whose ``EventType`` matches
- the envelope.
-
- Both paths are non-overlapping consumers — legacy reaches the
- in-process collector/processor task pair; the Temporal path
- reaches the durable listener workflow. Producer doesn't need to
- pick one.
- """
- from services.status_broadcaster import get_status_broadcaster
-
- broadcaster = get_status_broadcaster()
- envelope = webhook_received(webhook_data)
- await broadcaster.send_custom_event(_LEGACY_EVENT_TYPE, dict(webhook_data))
-
- # Temporal-durable fan-out (Wave 12 C1 canary). Always-call;
- # dispatch.emit is a no-op when the feature flag is off, so the
- # legacy path keeps working unchanged.
+ """Broadcast an incoming webhook via the canary CloudEvents path."""
from services.events.dispatch import emit
- await emit(envelope, wire_routing_key=_LEGACY_EVENT_TYPE)
+ await emit(
+ webhook_received(dict(webhook_data)),
+ wire_routing_key=_WIRE_ROUTING_KEY,
+ )
__all__ = [
diff --git a/server/nodes/whatsapp/__init__.py b/server/nodes/whatsapp/__init__.py
index 84a37324..c27ca2ae 100644
--- a/server/nodes/whatsapp/__init__.py
+++ b/server/nodes/whatsapp/__init__.py
@@ -74,7 +74,7 @@ register_option_loader("whatsappGroupMembers", load_group_members)
# fans out via services.events.dispatch.emit; emit() is gated by
# Settings.event_framework_enabled so the legacy path stays default.
# See services/deployment/canary_registry.py.
-register_canary_trigger_type("whatsappReceive")
+register_canary_trigger_type("whatsappReceive", "com.machinaos.whatsapp.message.received")
# Wave 12 C4 sub-piece A: opt this plugin into the social-provider
# registry so the social node dispatches by platform identifier
diff --git a/server/nodes/whatsapp/_events.py b/server/nodes/whatsapp/_events.py
index 5adc0347..1dd78d52 100644
--- a/server/nodes/whatsapp/_events.py
+++ b/server/nodes/whatsapp/_events.py
@@ -188,87 +188,80 @@ async def broadcast_whatsapp_message(
direction: Literal["sent", "received"],
params: Mapping[str, Any],
) -> None:
- """Emit a message event (sent or received). Wire-routing key
- preserves the legacy ``whatsapp_message_<direction>`` channel.
-
- For inbound (``received``) messages, also fans out via
- :func:`services.events.dispatch.emit` so Wave 12 C1 canary
- :class:`TriggerListenerWorkflow` consumers can pick up the
- envelope (no-op when ``Settings.event_framework_enabled`` is off,
- so the legacy WS path stays default).
+ """Emit a message event (sent or received).
+
+ Inbound (``received``) routes via :func:`services.events.dispatch.emit`
+ — the canary CloudEvents path Signals running
+ :class:`TriggerListenerWorkflow` consumers AND broadcasts the
+ envelope on the ``whatsapp_message_received`` wire key.
+
+ Outbound (``sent``) is observation-only: no trigger consumer
+ exists, so it bypasses ``dispatch.emit`` and goes out as a raw
+ legacy frame on ``whatsapp_message_sent`` for the FE message-list
+ UI.
"""
from services.status_broadcaster import get_status_broadcaster
broadcaster = get_status_broadcaster()
- wire_key = (
- _MESSAGE_SENT_WIRE_KEY if direction == "sent"
- else _MESSAGE_RECEIVED_WIRE_KEY
- )
payload = dict(params)
- # Legacy raw frame.
- await broadcaster.broadcast({
- "type": wire_key,
- "data": payload,
- })
-
- # Typed CloudEvents sibling — same outer wire key (the inner
- # envelope is what carries the typed contract).
- event = whatsapp_message_event(direction, payload)
- await broadcaster.broadcast({
- "type": wire_key,
- "data": event.model_dump(mode="json"),
- })
-
- # Wave 12 C1 rollout #4: Temporal-durable canary fan-out (received
- # only; outbound message events are pure observation, no trigger
- # node consumes them). emit() no-ops when the feature flag is off.
if direction == "received":
+ # Canary path — single ``emit`` call delivers to Temporal
+ # listeners + FE WS. The FE message-list handler at
+ # WebSocketContext.tsx still reads ``data.*`` for back-compat,
+ # so we also broadcast the legacy raw frame until that handler
+ # migrates to envelope-shape (Wave 12 D4 follow-up).
from services.events.dispatch import emit
- await emit(event, wire_routing_key=wire_key)
+ await broadcaster.broadcast({
+ "type": _MESSAGE_RECEIVED_WIRE_KEY,
+ "data": payload,
+ })
+ await emit(
+ whatsapp_message_event("received", payload),
+ wire_routing_key=_MESSAGE_RECEIVED_WIRE_KEY,
+ )
+ else:
+ # Outbound observation only — direct legacy raw broadcast.
+ await broadcaster.broadcast({
+ "type": _MESSAGE_SENT_WIRE_KEY,
+ "data": payload,
+ })
async def broadcast_whatsapp_newsletter(
verb: Literal["joined", "left", "muted", "live_updated"],
params: Mapping[str, Any],
) -> None:
- """Emit a newsletter-lifecycle event. ``verb`` selects the
- legacy wire key from :data:`_NEWSLETTER_WIRE_KEYS`."""
+ """Emit a newsletter-lifecycle event for FE observation.
+
+ Not a trigger event — no canary consumer. Direct raw broadcast on
+ the legacy wire key until the FE newsletter UI migrates to read
+ the envelope shape, at which point this becomes a ``dispatch.emit``
+ call like :func:`broadcast_whatsapp_message`.
+ """
from services.status_broadcaster import get_status_broadcaster
broadcaster = get_status_broadcaster()
- wire_key = _NEWSLETTER_WIRE_KEYS[verb]
-
await broadcaster.broadcast({
- "type": wire_key,
+ "type": _NEWSLETTER_WIRE_KEYS[verb],
"data": dict(params),
})
- event = whatsapp_newsletter_event(verb, params)
- await broadcaster.broadcast({
- "type": wire_key,
- "data": event.model_dump(mode="json"),
- })
-
async def broadcast_whatsapp_history_synced(params: Mapping[str, Any]) -> None:
- """Emit a history-sync-complete event."""
+ """Emit a history-sync-complete event for FE observation.
+
+ Not a trigger event — no canary consumer. Direct raw broadcast.
+ """
from services.status_broadcaster import get_status_broadcaster
broadcaster = get_status_broadcaster()
-
await broadcaster.broadcast({
"type": _HISTORY_SYNCED_WIRE_KEY,
"data": dict(params),
})
- event = whatsapp_history_synced(params)
- await broadcaster.broadcast({
- "type": _HISTORY_SYNCED_WIRE_KEY,
- "data": event.model_dump(mode="json"),
- })
-
__all__ = [
"broadcast_whatsapp_history_synced",
diff --git a/server/routers/websocket.py b/server/routers/websocket.py
index 462a6ec0..22a58310 100644
--- a/server/routers/websocket.py
+++ b/server/routers/websocket.py
@@ -1001,16 +1001,16 @@ async def handle_send_chat_message(data: Dict[str, Any], websocket: WebSocket) -
"session_id": session_id
}
- # Dispatch to chatTrigger waiters (legacy event_waiter + Temporal-durable
- # canary via dispatch.emit; the wrapper is async since Wave 12 C1).
- resolved = await dispatch_chat_message_received(event_data)
+ # Dispatch via canary CloudEvents path — Visibility-query Signal
+ # fan-out to running TriggerListenerWorkflow consumers + in-process
+ # WS broadcast on ``chat_message_received``.
+ await dispatch_chat_message_received(event_data)
- logger.info(f"[ChatMessage] Dispatched message to {resolved} chatTrigger waiter(s)")
+ logger.info(f"[ChatMessage] Dispatched canary event for session={session_id}")
return {
"success": True,
"message": "Chat message sent",
- "resolved_count": resolved,
"timestamp": timestamp
}
diff --git a/server/services/deployment/canary_registry.py b/server/services/deployment/canary_registry.py
index 3d023f27..e769c6c7 100644
--- a/server/services/deployment/canary_registry.py
+++ b/server/services/deployment/canary_registry.py
@@ -20,46 +20,86 @@ ensures events reach that listener.
Membership semantics (set-shaped):
- register_canary_trigger_type("webhookTrigger")
- register_canary_trigger_type("chatTrigger")
+ register_canary_trigger_type("webhookTrigger", "com.machinaos.webhook.received")
+ register_canary_trigger_type("chatTrigger", "com.machinaos.chat.message.received")
is_canary_trigger_type("webhookTrigger") # True
is_canary_trigger_type("whatsappReceive") # False (not yet opted in)
-
-Idempotent on re-import (multiple registrations of the same type are a
-no-op). No collision semantics because there's no "value" to conflict —
-membership is the contract.
+ cloudevent_type_for("webhookTrigger") # "com.machinaos.webhook.received"
+
+Idempotent on re-import (multiple registrations of the same type with
+the same cloudevent_type are a no-op; a conflicting cloudevent_type
+raises so plugin upgrades surface drift loudly).
+
+**Why cloudevent_type is required (Wave 12 bug fix, 2026-05-15).** The
+``TriggerListenerWorkflow`` is tagged at start with an ``EventType``
+Search Attribute. :func:`services.events.dispatch.emit` runs a
+Visibility query ``EventType='<value>' AND ExecutionStatus='Running'``
+to find listeners and signal them. The query uses ``event.type`` from
+the CloudEvents envelope (reverse-DNS, e.g.
+``com.machinaos.chat.message.received``). The Search Attribute MUST
+carry the same value or the query never matches and no signal is
+delivered — symptom was "TriggerListener started ok but never reacts
+to incoming events". Pre-fix the deployment manager registered the
+legacy snake_case ``event_type`` from ``TriggerConfig`` (e.g.
+``chat_message_received``), which silently broke every C1 canary.
"""
from __future__ import annotations
-from typing import FrozenSet, Iterator, Set
+from typing import Dict, FrozenSet, Iterator, Optional
-_REGISTERED: Set[str] = set()
+_REGISTERED: Dict[str, str] = {}
-def register_canary_trigger_type(node_type: str) -> None:
+def register_canary_trigger_type(node_type: str, cloudevent_type: str) -> None:
"""Opt this trigger node type into the Wave 12 C1 canary path.
- Called from a plugin's ``__init__.py``. Idempotent: re-importing the
- plugin module (e.g. during ``importlib.reload`` in the test suite)
- is a no-op.
+ Called from a plugin's ``__init__.py``. Idempotent on re-import when
+ the same ``cloudevent_type`` is supplied; a conflicting
+ ``cloudevent_type`` raises :class:`ValueError` so an accidentally
+ diverging plugin upgrade can't silently produce events that the
+ listener can't see.
Args:
node_type: The trigger node's ``BaseNode.type`` string. The
string MUST match what the workflow JSON carries for nodes
of this type — that's the key DeploymentManager looks up
when deciding which trigger goes to the canary listener.
+ cloudevent_type: The CloudEvents ``type`` field the plugin's
+ ``_events.py`` factory will populate on outgoing envelopes
+ (reverse-DNS, e.g. ``"com.machinaos.chat.message.received"``).
+ Used as the ``EventType`` Search Attribute value when the
+ listener workflow is started so
+ :func:`services.events.dispatch.emit`'s Visibility query
+ matches it.
Raises:
- TypeError: if ``node_type`` is not a non-empty string.
+ TypeError: if ``node_type`` or ``cloudevent_type`` is not a
+ non-empty string.
+ ValueError: if ``node_type`` was already registered with a
+ different ``cloudevent_type``.
"""
if not isinstance(node_type, str) or not node_type:
raise TypeError(
- f"register_canary_trigger_type expected a non-empty str, "
- f"got {type(node_type).__name__}: {node_type!r}"
+ f"register_canary_trigger_type expected a non-empty str for "
+ f"node_type, got {type(node_type).__name__}: {node_type!r}"
+ )
+ if not isinstance(cloudevent_type, str) or not cloudevent_type:
+ raise TypeError(
+ f"register_canary_trigger_type expected a non-empty str for "
+ f"cloudevent_type, got {type(cloudevent_type).__name__}: "
+ f"{cloudevent_type!r}"
+ )
+ existing = _REGISTERED.get(node_type)
+ if existing is not None and existing != cloudevent_type:
+ raise ValueError(
+ f"register_canary_trigger_type: node_type={node_type!r} already "
+ f"registered with cloudevent_type={existing!r}; refusing to "
+ f"overwrite with {cloudevent_type!r}. A diverging registration "
+ f"would silently break the Visibility-query → Signal fan-out."
)
- _REGISTERED.add(node_type)
+ _REGISTERED[node_type] = cloudevent_type
def is_canary_trigger_type(node_type: str) -> bool:
@@ -67,6 +107,15 @@ def is_canary_trigger_type(node_type: str) -> bool:
return node_type in _REGISTERED
+def cloudevent_type_for(node_type: str) -> Optional[str]:
+ """Return the CloudEvents ``type`` registered for ``node_type``.
+
+ ``None`` when the trigger isn't canary-registered — caller decides
+ whether to fall back to a legacy event_type or error.
+ """
+ return _REGISTERED.get(node_type)
+
+
def canary_trigger_types() -> FrozenSet[str]:
"""Return an immutable snapshot of registered canary trigger types.
@@ -84,5 +133,6 @@ def __iter__() -> Iterator[str]: # pragma: no cover — module dunders only
__all__ = [
"register_canary_trigger_type",
"is_canary_trigger_type",
+ "cloudevent_type_for",
"canary_trigger_types",
]
diff --git a/server/services/deployment/manager.py b/server/services/deployment/manager.py
index a81b9b52..d3f271d9 100644
--- a/server/services/deployment/manager.py
+++ b/server/services/deployment/manager.py
@@ -283,6 +283,31 @@ class DeploymentManager:
# (client.list_schedules vs client.list_workflows).
cron_schedule_count = await self._cancel_canary_cron_schedules(workflow_id)
+ # Sweep any downstream nodes still glowing on the canvas. The
+ # cron + listener resets above only cover the trigger nodes the
+ # manager owns directly; if a child run was mid-execution when
+ # cancellation hit, downstream agents / tools / actions may have
+ # been broadcast ``executing`` and would otherwise stay glowing
+ # forever. ``include_waiting=True`` because explicit user-cancel
+ # is the "every indicator goes quiet" signal — matches the
+ # behaviour of the ``handle_cancel_execution`` WS path. The
+ # delegation guard inside ``_clear_stuck_node_statuses`` still
+ # protects in-flight fire-and-forget child agents.
+ await self._broadcaster._clear_stuck_node_statuses(
+ workflow_id, include_waiting=True,
+ )
+
+ # And broadcast a final ``executing=False`` for the deployment
+ # so the toolbar Start/Stop indicator reflects the cancel. The
+ # legacy path's run-counter eviction (``workflow_run_ended``)
+ # doesn't fire here because deployment-level cancel can race
+ # ahead of in-flight ``workflow_run_started`` callers — emit
+ # the terminal state directly to avoid a stuck ``executing=True``
+ # on the FE toolbar.
+ await self._broadcaster.update_workflow_status(
+ executing=False, workflow_id=workflow_id,
+ )
+
# Clear cron iteration counters for this workflow's cron nodes
for node_id in cron_node_ids:
self._cron_iterations.pop(node_id, None)
@@ -675,9 +700,37 @@ class DeploymentManager:
if state is None:
raise RuntimeError(f"No deployment state for workflow {workflow_id}")
+ from services.deployment.canary_registry import cloudevent_type_for
+
config = event_waiter.get_trigger_config(node_type)
+ # Legacy snake_case event_type — only used in the listener_args
+ # payload + node-status display strings. NOT the Search Attribute
+ # value; that has to be the CloudEvents reverse-DNS string the
+ # producer puts on outgoing envelopes (see EventType SA pair
+ # below).
event_type = config.event_type if config else f"unknown_{node_type}"
+ # CloudEvents type the producer's _events.py factory emits on its
+ # outgoing envelope. Used as the EventType Search Attribute value
+ # so services.events.dispatch.emit's Visibility query
+ # ``ListWorkflows(query="EventType='<event.type>'")`` actually
+ # matches this listener.
+ #
+ # Pre-fix (2026-05-15) the EventType SA was set to ``event_type``
+ # (the legacy snake_case), but dispatch.emit queries with the
+ # producer's CloudEvents reverse-DNS string. The mismatch silently
+ # zeroed the signal fan-out — listener started OK, never reacted.
+ cloudevent_type = cloudevent_type_for(node_type)
+ if cloudevent_type is None:
+ logger.warning(
+ "Canary listener: no cloudevent_type registered for node_type; "
+ "falling back to legacy event_type for EventType SA. This will "
+ "silently fail dispatch.emit fan-out — register cloudevent_type "
+ "in the plugin's __init__.py.",
+ node_id=node_id, workflow_id=workflow_id, node_type=node_type,
+ )
+ cloudevent_type = event_type
+
# Pick workflow type by plugin class. EventTriggerKind picks up
# the "polling" classification for ops dashboards independent
# of the per-plugin kind (gmail / twitter / ...).
@@ -730,7 +783,10 @@ class DeploymentManager:
task_queue="machina-tasks",
id_conflict_policy=WorkflowIDConflictPolicy.USE_EXISTING,
search_attributes=TypedSearchAttributes([
- SearchAttributePair(event_type_key, event_type),
+ # MUST be the CloudEvents type the producer emits on
+ # outgoing envelopes — dispatch.emit's Visibility query
+ # substitutes ``event.type`` into the EventType filter.
+ SearchAttributePair(event_type_key, cloudevent_type),
SearchAttributePair(trigger_node_id_key, node_id),
SearchAttributePair(event_workflow_id_key, workflow_id),
SearchAttributePair(event_trigger_kind_key, trigger_kind),
diff --git a/server/services/events/envelope.py b/server/services/events/envelope.py
index 9b7cc749..661fc60e 100644
--- a/server/services/events/envelope.py
+++ b/server/services/events/envelope.py
@@ -324,11 +324,24 @@ class WorkflowEvent(BaseModel):
agent: str,
data: Optional[Mapping[str, Any]] = None,
) -> "WorkflowEvent":
- """Delegated child-agent completion. Type discriminates on
- succeeded vs failed; ``taskTrigger`` filters by both."""
+ """Delegated child-agent completion.
+
+ Both success and failure share one ``type``
+ (``com.machinaos.agent.task.completed``) so the
+ ``TriggerListenerWorkflow`` for ``taskTrigger`` can register a
+ single ``EventType`` Search Attribute and match every
+ completion event via :func:`services.events.dispatch.emit`'s
+ Visibility query. Status discrimination lives in ``data.status``
+ (``"completed"`` or ``"error"``) — :class:`TaskTriggerNode`'s
+ filter already keys off this field.
+
+ Pre-fix (commit ?) the type split into ``.succeeded`` /
+ ``.failed`` which left the listener's SA value matching only
+ one branch and silently lost the other half of events.
+ """
return cls(
source="machinaos://services/agent",
- type=f"{_TYPE_PREFIX}agent.task.{'succeeded' if status == 'completed' else 'failed'}",
+ type=f"{_TYPE_PREFIX}agent.task.completed",
subject=task_id,
data=dict(data) if data else {"task_id": task_id, "agent": agent, "status": status},
)
diff --git a/server/services/plugin/polling.py b/server/services/plugin/polling.py
index 2dcb00b2..bbfd70ab 100644
--- a/server/services/plugin/polling.py
+++ b/server/services/plugin/polling.py
@@ -260,7 +260,16 @@ class PollingTriggerNode(TriggerNode, abstract=True):
except Exception:
pass
- return {"events": events, "seen_ids": list(prior_seen | current)}
+ # OOM fix: bound ``seen_ids`` to what the provider currently
+ # reports as visible — old IDs that have dropped off the
+ # provider's window (archived / deleted / aged out) fall
+ # out of seen too. Pre-fix this was ``prior_seen | current``
+ # which grew forever (every poll appended new IDs; nothing
+ # ever got evicted). At Gmail's ~100 msgs/day cadence the
+ # set hit ~36K entries in a year — ~1.4MB just for IDs,
+ # serialised through Temporal payload on every cycle. Now
+ # bounded by the provider's natural window size.
+ return {"events": events, "seen_ids": list(current)}
return _poll_cycle_activity
@@ -313,6 +322,11 @@ class PollingTriggerNode(TriggerNode, abstract=True):
current = await self.fetch_ids(service, params)
new_ids = current - seen
if not new_ids:
+ # OOM fix: even when nothing new arrives we
+ # still rebase ``seen`` to ``current`` so old
+ # IDs that dropped off the provider's window
+ # don't linger forever.
+ seen = set(current)
continue
logger.debug(
"Polling trigger cycle",
@@ -321,7 +335,6 @@ class PollingTriggerNode(TriggerNode, abstract=True):
new=len(new_ids),
)
for msg_id in new_ids:
- seen.add(msg_id)
payload = await self.fetch_detail(
service, msg_id, params
)
@@ -333,6 +346,18 @@ class PollingTriggerNode(TriggerNode, abstract=True):
await self.post_emit(service, msg_id, params)
except Exception:
pass
+ # OOM fix: rebase ``seen`` to the provider's current
+ # window. Pre-fix this added every new ID to ``seen``
+ # forever (``seen.add(msg_id)`` inside the loop, no
+ # eviction). Long-running pollers (Gmail with the
+ # default 60s interval) accumulated tens of thousands
+ # of entries over weeks/months. Now bounded by the
+ # provider's natural window size — items that have
+ # been emitted are still in ``current`` until they
+ # age/archive out, at which point dropping them is
+ # correct (the provider will never re-surface them
+ # under the same filter).
+ seen = set(current)
except asyncio.CancelledError:
raise
except Exception as exc: # noqa: BLE001 -- per-cycle isolation
diff --git a/server/services/temporal/activities.py b/server/services/temporal/activities.py
index 73f404c6..9d595db9 100644
--- a/server/services/temporal/activities.py
+++ b/server/services/temporal/activities.py
@@ -372,3 +372,97 @@ async def emit_event_activity(event_payload: Dict[str, Any]) -> Dict[str, Any]:
"event_id": event.id,
"event_type": event.type,
}
+
+
+# ---------------------------------------------------------------------------
+# Trigger-node status broadcast activity
+#
+# Used by :class:`services.temporal.trigger_listener_workflow.TriggerListenerWorkflow`
+# and :class:`services.temporal.polling_trigger_workflow.PollingTriggerWorkflow`
+# to surface firing/waiting transitions to FE. Workflows can't broadcast
+# directly (sandboxed, deterministic) so they offload to this activity.
+#
+# Mirrors the legacy ``services/deployment/triggers.py`` collector/processor
+# transitions: ``"waiting" → "idle (processing)" → "waiting"`` per event.
+# Without this, the canary path would keep the trigger node visually
+# stuck on "waiting" with no firing pulse — a UX regression vs legacy.
+
+async def store_node_output_activity(payload: Dict[str, Any]) -> None:
+ """Persist a node's output to the workflow output store so
+ :class:`services.parameter_resolver.ParameterResolver` can resolve
+ ``{{nodeName.field}}`` templates in downstream nodes.
+
+ Used by :class:`services.temporal.workflow.MachinaWorkflow` for
+ pre-executed trigger nodes — they bypass ``NodeExecutor.execute``
+ (which is what normally persists outputs) because the workflow
+ received their output via the parent ``TriggerListenerWorkflow`` /
+ ``PollingTriggerWorkflow`` signal, not via the standard activity
+ dispatch. Without this persist step, ``ParameterResolver`` reads
+ ``None`` from the store and downstream templates like
+ ``{{chatTrigger.message}}`` silently resolve to empty.
+
+ Mirrors the per-handle write pattern in
+ :func:`services.temporal.agent_activities.store_agent_output`.
+
+ Payload shape::
+
+ {
+ "node_id": str,
+ "session_id": str,
+ "result": dict, # the trigger output payload
+ }
+ """
+ from core.container import container
+
+ workflow_service = container.workflow_service()
+ node_id = payload["node_id"]
+ session_id = payload.get("session_id", "default")
+ data = payload.get("result") or {}
+ # Mirror NodeExecutor — write to every standard output handle so
+ # the resolver finds the data regardless of which handle name the
+ # downstream edge targets.
+ for output_name in ("output_main", "output_top", "output_0"):
+ try:
+ await workflow_service.store_node_output(
+ session_id, node_id, output_name, data,
+ )
+ except Exception as exc: # noqa: BLE001 — non-fatal
+ activity.logger.warning(
+ f"store_node_output_activity failed for "
+ f"node={node_id!r} handle={output_name!r}: {exc}"
+ )
+
+
+async def broadcast_trigger_status_activity(payload: Dict[str, Any]) -> None:
+ """Broadcast a trigger node status update for FE animation.
+
+ Payload shape:
+ {
+ "node_id": str,
+ "status": "waiting" | "idle" | ...,
+ "data": dict, # optional, defaults to {}
+ "workflow_id": str | None,
+ }
+
+ Embedded-worker mode (default): direct in-process call to the
+ status_broadcaster singleton — same event loop as FastAPI handlers.
+ Non-fatal: broadcast failures are logged at WARNING and swallowed
+ so they don't fail the calling workflow.
+ """
+ from services.status_broadcaster import get_status_broadcaster
+
+ try:
+ broadcaster = get_status_broadcaster()
+ await broadcaster.update_node_status(
+ node_id=payload["node_id"],
+ status=payload.get("status", "waiting"),
+ data=payload.get("data", {}) or {},
+ workflow_id=payload.get("workflow_id"),
+ )
+ except Exception as exc: # noqa: BLE001 — non-fatal
+ activity.logger.warning(
+ f"broadcast_trigger_status_activity failed for "
+ f"node={payload.get('node_id')!r}: {exc}"
+ )
diff --git a/server/services/temporal/polling_trigger_workflow.py b/server/services/temporal/polling_trigger_workflow.py
index f4fbe5c8..9ca0b468 100644
--- a/server/services/temporal/polling_trigger_workflow.py
+++ b/server/services/temporal/polling_trigger_workflow.py
@@ -201,7 +201,11 @@ class PollingTriggerWorkflow:
nodes) stay single-source. Mirrors
:meth:`TriggerListenerWorkflow._spawn_child_run` exactly.
"""
- from services.temporal.trigger_listener_workflow import _build_run_graph
+ from services.temporal.trigger_listener_workflow import (
+ _broadcast_trigger_idle,
+ _broadcast_trigger_waiting,
+ _build_run_graph,
+ )
trigger_node_id = listener_data["trigger_node_id"]
nodes = listener_data["nodes"]
@@ -231,6 +235,13 @@ class PollingTriggerWorkflow:
event_id = event.get("id") or workflow.uuid4().hex
child_id = f"run-{workflow_id}-{event_id}"
+ await _broadcast_trigger_idle(
+ node_id=trigger_node_id,
+ workflow_id=workflow_id,
+ event_id=event_id,
+ event_type=listener_data.get("event_type", ""),
+ )
+
await workflow.start_child_workflow(
"MachinaWorkflow",
args=[{
@@ -252,5 +263,12 @@ class PollingTriggerWorkflow:
f"event.id={event.get('id')}"
)
+ await _broadcast_trigger_waiting(
+ node_id=trigger_node_id,
+ workflow_id=workflow_id,
+ event_type=listener_data.get("event_type", ""),
+ processed_count=self._processed_count + 1,
+ )
+
__all__ = ["PollingTriggerWorkflow"]
diff --git a/server/services/temporal/trigger_listener_workflow.py b/server/services/temporal/trigger_listener_workflow.py
index 8fced073..37ccd403 100644
--- a/server/services/temporal/trigger_listener_workflow.py
+++ b/server/services/temporal/trigger_listener_workflow.py
@@ -149,6 +149,12 @@ class TriggerListenerWorkflow:
the same deployment are marked ``_pre_executed=True`` with
``{not_triggered: True}`` so MachinaWorkflow doesn't block on
them (mirrors ``DeploymentManager._execute_from_trigger``).
+
+ Status broadcast lifecycle (matches legacy
+ ``deployment/triggers.py`` collector/processor):
+ - Before spawn → trigger node ``"idle"`` with "Graph executing..."
+ - After spawn returns → trigger node ``"waiting"`` (child runs
+ independently per ``parent_close_policy=ABANDON``).
"""
trigger_node_id = listener_data["trigger_node_id"]
nodes = listener_data["nodes"]
@@ -180,6 +186,13 @@ class TriggerListenerWorkflow:
event_id = event.get("id") or workflow.uuid4().hex
child_id = f"run-{workflow_id}-{event_id}"
+ await _broadcast_trigger_idle(
+ node_id=trigger_node_id,
+ workflow_id=workflow_id,
+ event_id=event_id,
+ event_type=event.get("type", ""),
+ )
+
await workflow.start_child_workflow(
"MachinaWorkflow",
args=[{
@@ -201,6 +214,78 @@ class TriggerListenerWorkflow:
f"event.id={event.get('id')} event.type={event.get('type')}"
)
+ await _broadcast_trigger_waiting(
+ node_id=trigger_node_id,
+ workflow_id=workflow_id,
+ event_type=listener_data.get("event_type", ""),
+ processed_count=self._processed_count + 1,
+ )
+
+
+# ---------------------------------------------------------------------------
+# Status-broadcast helpers — thin wrappers around the activity so the
+# spawn loop reads cleanly. Activity name string matches the @activity.defn
+# registration in services/temporal/activities.py.
+# ---------------------------------------------------------------------------
+
+
+_STATUS_ACTIVITY_NAME = "broadcast_trigger_status_activity"
+_STATUS_ACTIVITY_TIMEOUT = timedelta(seconds=5)
+
+
+async def _broadcast_trigger_idle(
+ *,
+ node_id: str,
+ workflow_id: Optional[str],
+ event_id: str,
+ event_type: str,
+) -> None:
+ """Broadcast trigger node ``"idle"`` status with a "Graph executing..."
+ message — matches the legacy collector/processor transition so FE
+ shows a firing pulse instead of a stuck "waiting" indicator."""
+ await workflow.execute_activity(
+ _STATUS_ACTIVITY_NAME,
+ {
+ "node_id": node_id,
+ "status": "idle",
+ "data": {
+ "message": "Graph executing...",
+ "is_processing": True,
+ "event_id": event_id,
+ "event_type": event_type,
+ },
+ "workflow_id": workflow_id,
+ },
+ start_to_close_timeout=_STATUS_ACTIVITY_TIMEOUT,
+ )
+
+
+async def _broadcast_trigger_waiting(
+ *,
+ node_id: str,
+ workflow_id: Optional[str],
+ event_type: str,
+ processed_count: int,
+) -> None:
+ """Broadcast trigger node back to ``"waiting"`` after the child run
+ has been spawned (child completes independently per
+ ``parent_close_policy=ABANDON``)."""
+ await workflow.execute_activity(
+ _STATUS_ACTIVITY_NAME,
+ {
+ "node_id": node_id,
+ "status": "waiting",
+ "data": {
+ "message": "Waiting for next event...",
+ "is_processing": False,
+ "event_type": event_type,
+ "processed_count": processed_count,
+ },
+ "workflow_id": workflow_id,
+ },
+ start_to_close_timeout=_STATUS_ACTIVITY_TIMEOUT,
+ )
+
# ---------------------------------------------------------------------------
# Pure graph builder — kept in this module on purpose: the only consumer is
diff --git a/server/services/temporal/worker.py b/server/services/temporal/worker.py
index 8e8fd784..77a41a10 100644
--- a/server/services/temporal/worker.py
+++ b/server/services/temporal/worker.py
@@ -140,6 +140,10 @@ class TemporalWorkerManager:
)
from services.temporal.agent_activities import collect_agent_activities
from services.temporal.agent_workflow import AgentWorkflow
+ from services.temporal.activities import (
+ broadcast_trigger_status_activity,
+ store_node_output_activity,
+ )
per_type = collect_plugin_activities() # no queue filter; all plugins
agent_activities = collect_agent_activities()
@@ -164,6 +168,8 @@ class TemporalWorkerManager:
],
activities=[
self._activities.execute_node_activity,
+ broadcast_trigger_status_activity,
+ store_node_output_activity,
*per_type,
*agent_activities,
*polling_activities,
@@ -183,7 +189,6 @@ class TemporalWorkerManager:
span.set_attribute("task_queue", self.task_queue)
span.set_attribute("pool_size", self.pool_size)
- print(f"[Temporal] Worker started (queue={self.task_queue}, pool={self.pool_size})", flush=True)
logger.info(
"Starting Temporal worker",
task_queue=self.task_queue,
@@ -385,6 +390,11 @@ async def run_standalone_worker(
session = await create_shared_session(pool_size)
activities = NodeExecutionActivities(session)
+ from services.temporal.activities import (
+ broadcast_trigger_status_activity,
+ store_node_output_activity,
+ )
+
try:
worker = Worker(
client,
@@ -394,7 +404,11 @@ async def run_standalone_worker(
TriggerListenerWorkflow,
PollingTriggerWorkflow,
],
- activities=[activities.execute_node_activity], # Pass bound method
+ activities=[
+ activities.execute_node_activity,
+ broadcast_trigger_status_activity,
+ store_node_output_activity,
+ ],
max_concurrent_activities=pool_size,
max_concurrent_workflow_tasks=10,
graceful_shutdown_timeout=_graceful_shutdown_timeout(),
@@ -428,6 +442,10 @@ async def create_worker(
session = await create_shared_session()
activities = NodeExecutionActivities(session)
+ from services.temporal.activities import (
+ broadcast_trigger_status_activity,
+ store_node_output_activity,
+ )
return Worker(
client,
@@ -437,7 +455,11 @@ async def create_worker(
TriggerListenerWorkflow,
PollingTriggerWorkflow,
],
- activities=[activities.execute_node_activity], # Pass bound method
+ activities=[
+ activities.execute_node_activity,
+ broadcast_trigger_status_activity,
+ store_node_output_activity,
+ ],
max_concurrent_activities=100,
max_concurrent_workflow_tasks=10,
graceful_shutdown_timeout=_GRACEFUL_SHUTDOWN_TIMEOUT,
diff --git a/server/services/temporal/workflow.py b/server/services/temporal/workflow.py
index 4446b282..07d00430 100644
--- a/server/services/temporal/workflow.py
+++ b/server/services/temporal/workflow.py
@@ -231,9 +231,10 @@ class MachinaWorkflow:
for node in exec_nodes:
if node.get("_pre_executed"):
node_id = node["id"]
+ trigger_output = node.get("_trigger_output", {})
outputs[node_id] = {
"success": True,
- "result": node.get("_trigger_output", {}),
+ "result": trigger_output,
"pre_executed": True,
}
completed.add(node_id)
@@ -241,6 +242,28 @@ class MachinaWorkflow:
pre_executed_count += 1
workflow.logger.info(f"Pre-executed trigger: {node_id}")
+ # Persist the trigger output to the workflow output
+ # store so ParameterResolver can resolve
+ # ``{{triggerNodeName.field}}`` in downstream nodes.
+ # Pre-executed nodes bypass NodeExecutor.execute (the
+ # normal write site), so without this activity the
+ # legacy path's ``_store_output(trigger_node_id, ...)``
+ # behaviour is missing on the canary path and templates
+ # against the trigger silently resolve to empty.
+ # Skip the persist for non-firing siblings — their
+ # ``_trigger_output`` is ``{not_triggered: True}``, no
+ # downstream template should resolve against them.
+ if not trigger_output.get("not_triggered"):
+ await workflow.execute_activity(
+ "store_node_output_activity",
+ {
+ "node_id": node_id,
+ "session_id": session_id,
+ "result": trigger_output,
+ },
+ start_to_close_timeout=timedelta(seconds=10),
+ )
+
workflow.logger.info(f"Pre-executed: {pre_executed_count}, To execute: {len(node_map) - pre_executed_count}")
# 5. Retry policy for node activities. Wave 12 D1: imported
diff --git a/server/tests/nodes/test_email.py b/server/tests/nodes/test_email.py
index bcd84458..740e49c7 100644
--- a/server/tests/nodes/test_email.py
+++ b/server/tests/nodes/test_email.py
@@ -229,8 +229,9 @@ class TestEmailReceive:
patch.object(EmailService, "fetch_detail", fetch_detail_mock), \
patch("asyncio.sleep", new=AsyncMock(return_value=None)), \
patch("services.status_broadcaster.get_status_broadcaster") as bcast, \
- patch("services.event_waiter.dispatch") as dispatch_mock:
+ patch("services.events.dispatch.emit") as emit_mock:
bcast.return_value.update_node_status = AsyncMock(return_value=None)
+ emit_mock.return_value = None
result = await harness.execute(
"emailReceive",
@@ -244,9 +245,13 @@ class TestEmailReceive:
assert payload["subject"] == "new!"
# Baseline + one diffing poll
assert poll_ids_mock.await_count == 2
- # Event dispatched exactly once
- dispatch_mock.assert_called_once()
- assert dispatch_mock.call_args.args[0] == "email_received"
+ # Event routed through the canary CloudEvents path (legacy
+ # event_waiter.dispatch was retired in Wave 13 — emailReceive
+ # is canary-registered and the legacy collector has no consumer).
+ emit_mock.assert_called_once()
+ envelope = emit_mock.call_args.args[0]
+ assert envelope.type == "com.machinaos.email.message.received"
+ assert emit_mock.call_args.kwargs["wire_routing_key"] == "email_received"
async def test_mark_as_read_adds_seen_flag(self, harness):
from nodes.email._service import EmailService
@@ -262,7 +267,7 @@ class TestEmailReceive:
patch.object(HimalayaService, "flag_message", flag_mock), \
patch("asyncio.sleep", new=AsyncMock(return_value=None)), \
patch("services.status_broadcaster.get_status_broadcaster") as bcast, \
- patch("services.event_waiter.dispatch"):
+ patch("services.events.dispatch.emit"):
bcast.return_value.update_node_status = AsyncMock(return_value=None)
result = await harness.execute(
@@ -300,7 +305,7 @@ class TestEmailReceive:
patch.object(EmailService, "poll_ids", poll_ids_mock), \
patch("asyncio.sleep", new=AsyncMock(return_value=None)), \
patch("services.status_broadcaster.get_status_broadcaster") as bcast, \
- patch("services.event_waiter.dispatch"):
+ patch("services.events.dispatch.emit"):
bcast.return_value.update_node_status = AsyncMock(return_value=None)
result = await harness.execute(
diff --git a/server/tests/test_canary_registry.py b/server/tests/test_canary_registry.py
index 25df0a5a..9b7a266c 100644
--- a/server/tests/test_canary_registry.py
+++ b/server/tests/test_canary_registry.py
@@ -39,55 +39,87 @@ if "machina" not in sys.modules:
@pytest.fixture
def fresh_registry(monkeypatch):
- """Reset the registry's backing set so each test runs in isolation."""
+ """Reset the registry's backing dict so each test runs in isolation."""
from services.deployment import canary_registry
- monkeypatch.setattr(canary_registry, "_REGISTERED", set())
+ monkeypatch.setattr(canary_registry, "_REGISTERED", {})
return canary_registry
+# Dummy CloudEvents types used in registry tests — the real values come
+# from each plugin's ``_events.py`` factory.
+_CE_WEBHOOK = "com.machinaos.webhook.received"
+_CE_CHAT = "com.machinaos.chat.message.received"
+_CE_TASK = "com.machinaos.agent.task.completed"
+
+
class TestRegistryContract:
- """Surface API: register, query, snapshot."""
+ """Surface API: register, query, snapshot, cloudevent_type lookup."""
def test_unregistered_type_is_not_canary(self, fresh_registry):
assert fresh_registry.is_canary_trigger_type("webhookTrigger") is False
+ assert fresh_registry.cloudevent_type_for("webhookTrigger") is None
def test_register_then_query_returns_true(self, fresh_registry):
- fresh_registry.register_canary_trigger_type("webhookTrigger")
+ fresh_registry.register_canary_trigger_type("webhookTrigger", _CE_WEBHOOK)
assert fresh_registry.is_canary_trigger_type("webhookTrigger") is True
+ assert fresh_registry.cloudevent_type_for("webhookTrigger") == _CE_WEBHOOK
- def test_register_is_idempotent(self, fresh_registry):
- fresh_registry.register_canary_trigger_type("chatTrigger")
- fresh_registry.register_canary_trigger_type("chatTrigger")
- fresh_registry.register_canary_trigger_type("chatTrigger")
+ def test_register_is_idempotent_with_same_cloudevent_type(self, fresh_registry):
+ fresh_registry.register_canary_trigger_type("chatTrigger", _CE_CHAT)
+ fresh_registry.register_canary_trigger_type("chatTrigger", _CE_CHAT)
+ fresh_registry.register_canary_trigger_type("chatTrigger", _CE_CHAT)
assert fresh_registry.canary_trigger_types() == frozenset({"chatTrigger"})
+ assert fresh_registry.cloudevent_type_for("chatTrigger") == _CE_CHAT
+
+ def test_register_rejects_conflicting_cloudevent_type(self, fresh_registry):
+ """Reload-tolerance is bounded: same type re-registered with a
+ diverging CloudEvents type is a loud :class:`ValueError`, not a
+ silent overwrite. Catches plugin upgrades that change the
+ producer envelope shape without coordinating with the listener."""
+ fresh_registry.register_canary_trigger_type("chatTrigger", _CE_CHAT)
+ with pytest.raises(ValueError, match="diverging"):
+ fresh_registry.register_canary_trigger_type(
+ "chatTrigger", "com.machinaos.chat.message.changed"
+ )
def test_multiple_types_coexist(self, fresh_registry):
- for t in ("webhookTrigger", "chatTrigger", "taskTrigger"):
- fresh_registry.register_canary_trigger_type(t)
+ for t, ce in (
+ ("webhookTrigger", _CE_WEBHOOK),
+ ("chatTrigger", _CE_CHAT),
+ ("taskTrigger", _CE_TASK),
+ ):
+ fresh_registry.register_canary_trigger_type(t, ce)
assert fresh_registry.canary_trigger_types() == frozenset({
"webhookTrigger", "chatTrigger", "taskTrigger",
})
+ assert fresh_registry.cloudevent_type_for("taskTrigger") == _CE_TASK
def test_snapshot_is_immutable(self, fresh_registry):
"""``canary_trigger_types()`` returns a frozenset — mutating it
must not leak back into the registry."""
- fresh_registry.register_canary_trigger_type("webhookTrigger")
+ fresh_registry.register_canary_trigger_type("webhookTrigger", _CE_WEBHOOK)
snap = fresh_registry.canary_trigger_types()
assert isinstance(snap, frozenset)
- # frozenset has no add() — TypeError if attempted.
+ # frozenset has no add() — AttributeError if attempted.
with pytest.raises(AttributeError):
snap.add("chatTrigger") # type: ignore[attr-defined]
- def test_rejects_non_string(self, fresh_registry):
+ def test_rejects_non_string_node_type(self, fresh_registry):
+ with pytest.raises(TypeError):
+ fresh_registry.register_canary_trigger_type(None, _CE_WEBHOOK) # type: ignore[arg-type]
with pytest.raises(TypeError):
- fresh_registry.register_canary_trigger_type(None) # type: ignore[arg-type]
+ fresh_registry.register_canary_trigger_type(123, _CE_WEBHOOK) # type: ignore[arg-type]
+
+ def test_rejects_empty_node_type(self, fresh_registry):
with pytest.raises(TypeError):
- fresh_registry.register_canary_trigger_type(123) # type: ignore[arg-type]
+ fresh_registry.register_canary_trigger_type("", _CE_WEBHOOK)
- def test_rejects_empty_string(self, fresh_registry):
+ def test_rejects_missing_cloudevent_type(self, fresh_registry):
+ with pytest.raises(TypeError):
+ fresh_registry.register_canary_trigger_type("webhookTrigger", None) # type: ignore[arg-type]
with pytest.raises(TypeError):
- fresh_registry.register_canary_trigger_type("")
+ fresh_registry.register_canary_trigger_type("webhookTrigger", "")
class TestNoFrameworkSideAllowlist:
@@ -154,12 +186,14 @@ class TestPluginSelfRegistration:
rather than masking the import failure.
"""
- @pytest.mark.parametrize("plugin_module,expected_type", [
- ("nodes.trigger.webhook_trigger", "webhookTrigger"),
- ("nodes.trigger.chat_trigger", "chatTrigger"),
- ("nodes.trigger.task_trigger", "taskTrigger"),
+ @pytest.mark.parametrize("plugin_module,expected_type,expected_cloudevent_type", [
+ ("nodes.trigger.webhook_trigger", "webhookTrigger", "com.machinaos.webhook.received"),
+ ("nodes.trigger.chat_trigger", "chatTrigger", "com.machinaos.chat.message.received"),
+ ("nodes.trigger.task_trigger", "taskTrigger", "com.machinaos.agent.task.completed"),
])
- def test_plugin_import_registers_its_type(self, plugin_module, expected_type):
+ def test_plugin_import_registers_its_type(
+ self, plugin_module, expected_type, expected_cloudevent_type
+ ):
from services.deployment import canary_registry
try:
@@ -169,7 +203,108 @@ class TestPluginSelfRegistration:
assert canary_registry.is_canary_trigger_type(expected_type), (
f"Importing {plugin_module} should call "
- f"register_canary_trigger_type({expected_type!r}). Check the "
+ f"register_canary_trigger_type({expected_type!r}, ...). Check the "
"module's __init__.py for the register_canary_trigger_type "
"call at module scope."
)
+ # The cloudevent_type MUST match what the producer's _events.py
+ # factory puts on outgoing envelopes — otherwise dispatch.emit's
+ # Visibility query never finds the listener.
+ assert canary_registry.cloudevent_type_for(expected_type) == expected_cloudevent_type, (
+ f"{plugin_module} registered {expected_type!r} with the wrong "
+ f"cloudevent_type. Expected {expected_cloudevent_type!r} to match "
+ f"the producer's WorkflowEvent.type. If the producer factory "
+ "changed, update both the factory AND the register_canary_trigger_type "
+ "call together — drift here silently breaks the canary signal "
+ "fan-out (TriggerListener starts ok, never reacts to events)."
+ )
+
+
+class TestCloudEventTypeMatchesSearchAttribute:
+ """Regression: every canary trigger's registered cloudevent_type MUST
+ equal the ``event.type`` field a producer-side factory call would
+ emit. The deployment manager uses ``cloudevent_type_for(node_type)``
+ as the ``EventType`` Search Attribute on the listener workflow; the
+ producer side uses ``event.type`` from ``WorkflowEvent`` factories
+ as the Visibility query value in ``services.events.dispatch.emit``.
+
+ These two strings MUST agree or the Signal fan-out silently zeroes
+ out — symptom is "TriggerListenerWorkflow started but never reacts
+ to incoming events". This invariant locks the contract.
+ """
+
+ def test_every_registered_type_has_reverse_dns_cloudevent_type(self):
+ """All registered cloudevent_types are reverse-DNS strings
+ (``com.machinaos.*``) — that's the CloudEvents §3.1.2 type
+ convention this codebase declares (see envelope.py:_TYPE_PREFIX).
+ Anything else means a plugin registered the legacy snake_case
+ event_waiter event_type by mistake."""
+ from services.deployment import canary_registry
+
+ for node_type in canary_registry.canary_trigger_types():
+ ce_type = canary_registry.cloudevent_type_for(node_type)
+ assert ce_type is not None, (
+ f"canary_trigger_types includes {node_type!r} but "
+ f"cloudevent_type_for() returned None — registry corrupted."
+ )
+ assert ce_type.startswith("com.machinaos."), (
+ f"{node_type!r} registered with cloudevent_type={ce_type!r}; "
+ f"expected a reverse-DNS string starting with 'com.machinaos.'. "
+ f"That's the format dispatch.emit's Visibility query "
+ f"substitutes from envelope.type — anything else fails to "
+ f"match the listener's EventType Search Attribute."
+ )
+
+ def test_dispatch_emit_query_uses_event_type_field(self):
+ """Locks the contract on dispatch.emit's Visibility query: it
+ MUST substitute ``event.type`` (CloudEvents reverse-DNS), not
+ any other field. If this regresses to ``event.subject`` /
+ legacy event_type / etc. the SA lookup breaks."""
+ from services.events import dispatch as dispatch_mod
+
+ src = inspect.getsource(dispatch_mod._signal_running_consumers)
+ # The query template MUST format the CloudEvents type into the
+ # EventType filter.
+ assert "EventType=" in dispatch_mod._RUNNING_CONSUMERS_QUERY, (
+ "dispatch.emit's _RUNNING_CONSUMERS_QUERY no longer filters by "
+ "EventType — listener-side SA registration relies on this key."
+ )
+ assert "event.type" in src or "event_type=event.type" in src, (
+ "dispatch.emit no longer substitutes event.type into the "
+ "Visibility query. The listener registered SA value must match "
+ "what's substituted here; using subject / id / data fields "
+ "breaks the contract."
+ )
+
+ def test_deployment_manager_uses_cloudevent_type_for_event_type_sa(self):
+ """Source-introspection regression on the deployment-side fix:
+ ``_start_canary_listener`` must construct the EventType
+ SearchAttributePair from ``cloudevent_type_for(node_type)``,
+ NOT from ``config.event_type`` (the legacy event_waiter string)."""
+ from services.deployment.manager import DeploymentManager
+
+ src = inspect.getsource(DeploymentManager._start_canary_listener)
+ assert "cloudevent_type_for" in src, (
+ "DeploymentManager._start_canary_listener no longer calls "
+ "cloudevent_type_for() to resolve the EventType Search Attribute. "
+ "Pre-fix it used event_waiter.TriggerConfig.event_type (legacy "
+ "snake_case) which never matched dispatch.emit's Visibility "
+ "query (CloudEvents reverse-DNS). Restore the registry lookup."
+ )
+ # The SearchAttributePair for EventType MUST use the cloudevent_type
+ # variable, not the legacy event_type one.
+ pair_pattern = re.search(
+ r"SearchAttributePair\s*\(\s*event_type_key\s*,\s*(\w+)",
+ src,
+ )
+ assert pair_pattern is not None, (
+ "Couldn't find the SearchAttributePair(event_type_key, ...) "
+ "construction. Check _start_canary_listener for the SA wiring."
+ )
+ sa_value_name = pair_pattern.group(1)
+ assert sa_value_name == "cloudevent_type", (
+ f"SearchAttributePair(event_type_key, {sa_value_name}) — expected "
+ f"'cloudevent_type'. The legacy 'event_type' variable holds the "
+ f"snake_case event_waiter string and will silently break the "
+ f"Signal fan-out (TriggerListener starts ok, never reacts)."
+ )
diff --git a/server/tests/test_chat_trigger_canary_producer.py b/server/tests/test_chat_trigger_canary_producer.py
index 1a3da1da..322ac109 100644
--- a/server/tests/test_chat_trigger_canary_producer.py
+++ b/server/tests/test_chat_trigger_canary_producer.py
@@ -1,16 +1,16 @@
-"""Wave 12 C1 rollout #1: chatTrigger producer dual-emit invariant.
+"""chatTrigger producer canary-emit invariant.
Locks the contract: ``nodes.trigger.chat_trigger._events.dispatch_chat_message_received``
-must route to BOTH the legacy in-process ``event_waiter.dispatch`` AND
-the Temporal-durable ``services.events.dispatch.emit`` path. Without
-the second call, a canary-enabled chatTrigger deployment would silently
-miss all incoming chat messages — the TriggerListenerWorkflow has no
-other event source.
+routes through the canary CloudEvents path
+(:func:`services.events.dispatch.emit`) ONLY. The legacy
+``event_waiter.dispatch`` path was removed in Wave 13 — chatTrigger is
+canary-registered, the deployment manager skips ``setup_event_trigger``,
+and the legacy collector has zero consumers in production.
Same regex-introspection invariant style as
-``tests/test_credential_broadcasts.py`` — source-level assertions
-catch the wire contract drifting without paying the cost of standing
-up Temporal in CI.
+``tests/test_credential_broadcasts.py`` — source-level assertions catch
+the wire contract drifting without paying the cost of standing up
+Temporal in CI.
"""
from __future__ import annotations
@@ -19,8 +19,8 @@ import inspect
import re
import sys
import types
-from typing import Any, Dict, List
-from unittest.mock import AsyncMock, MagicMock
+from typing import Any, List
+from unittest.mock import MagicMock
import pytest
@@ -38,64 +38,50 @@ _EVENT_WAITER_DISPATCH_PATTERN = re.compile(r"event_waiter\.dispatch\s*\(")
_EVENTS_EMIT_PATTERN = re.compile(r"\bemit\s*\(")
-class TestChatTriggerProducerDualEmit:
- """Producer wrapper emits BOTH legacy and Temporal envelopes."""
+class TestChatTriggerProducerCanaryEmit:
+ """Producer wrapper emits via the canary CloudEvents path only."""
def test_dispatcher_is_async(self):
from nodes.trigger.chat_trigger._events import dispatch_chat_message_received
- # async coroutine functions return a coroutine when called.
- # Both `asyncio.iscoroutinefunction` and `inspect.iscoroutinefunction`
- # detect this.
assert inspect.iscoroutinefunction(dispatch_chat_message_received), (
- "dispatch_chat_message_received must be async since Wave 12 C1 — "
- "it awaits services.events.dispatch.emit. Sync callers must "
- "switch to `await dispatch_chat_message_received(...)`."
+ "dispatch_chat_message_received must be async — it awaits "
+ "services.events.dispatch.emit."
)
- def test_dispatcher_routes_both_legacy_and_temporal(self):
+ def test_dispatcher_uses_canary_path_only(self):
from nodes.trigger.chat_trigger import _events
src = inspect.getsource(_events.dispatch_chat_message_received)
- # Legacy event_waiter.dispatch path stays for back-compat.
- assert _EVENT_WAITER_DISPATCH_PATTERN.search(src), (
- "dispatch_chat_message_received must still call "
- "event_waiter.dispatch(_LEGACY_EVENT_TYPE, ...) for the "
- "in-process waiter path. Without it, canary-flag-off "
- "deployments lose their existing chat dispatch."
- )
-
- # Temporal-durable path via dispatch.emit.
assert _EVENTS_EMIT_PATTERN.search(src), (
"dispatch_chat_message_received must call "
- "services.events.dispatch.emit(envelope, ...) for the "
- "Temporal-durable canary path. Without it, the "
- "TriggerListenerWorkflow receives nothing when the "
- "event_framework_enabled flag is on."
+ "services.events.dispatch.emit(envelope, ...) — the canary "
+ "CloudEvents path Signals running TriggerListenerWorkflow "
+ "consumers AND broadcasts to FE on the chat_message_received "
+ "wire key."
+ )
+ assert not _EVENT_WAITER_DISPATCH_PATTERN.search(src), (
+ "dispatch_chat_message_received must NOT call "
+ "event_waiter.dispatch — chatTrigger is canary-registered, "
+ "the legacy collector path has zero consumers, and that "
+ "call was removed in Wave 13. Reintroducing it would "
+ "double-dispatch through dead infrastructure."
)
@pytest.mark.asyncio
- async def test_runtime_dual_emit_calls_both_paths(self, monkeypatch):
- """Integration smoke: invoke the dispatcher and assert both
- downstream calls fire. event_framework_enabled=False so emit()
- no-ops on the Temporal side; the call still happens though."""
+ async def test_runtime_emits_canary_envelope(self, monkeypatch):
+ """Invoking the dispatcher calls dispatch.emit with the right
+ envelope. The legacy event_waiter is not touched."""
from nodes.trigger.chat_trigger import _events
- from services import event_waiter
from services.events import dispatch as dispatch_mod
- legacy_calls: List[Dict[str, Any]] = []
emit_calls: List[Any] = []
- def fake_dispatch(event_type, data):
- legacy_calls.append({"event_type": event_type, "data": data})
- return 0
-
async def fake_emit(event, **kwargs):
emit_calls.append({"event": event, **kwargs})
return event
- monkeypatch.setattr(event_waiter, "dispatch", fake_dispatch)
monkeypatch.setattr(dispatch_mod, "emit", fake_emit)
result = await _events.dispatch_chat_message_received({
@@ -104,12 +90,8 @@ class TestChatTriggerProducerDualEmit:
"timestamp": "2026-05-14T00:00:00",
})
- # Legacy returned the resolved count.
- assert result == 0
-
- assert len(legacy_calls) == 1
- assert legacy_calls[0]["event_type"] == "chat_message_received"
- assert legacy_calls[0]["data"]["message"] == "hello"
+ # No return value — canary-only emit doesn't carry a waiter count.
+ assert result is None
assert len(emit_calls) == 1
event = emit_calls[0]["event"]
diff --git a/server/tests/test_deployment_canary_listener.py b/server/tests/test_deployment_canary_listener.py
index 3014c7ca..9fec426e 100644
--- a/server/tests/test_deployment_canary_listener.py
+++ b/server/tests/test_deployment_canary_listener.py
@@ -55,7 +55,7 @@ def fresh_canary_registry(monkeypatch):
"""
from services.deployment import canary_registry
- monkeypatch.setattr(canary_registry, "_REGISTERED", set())
+ monkeypatch.setattr(canary_registry, "_REGISTERED", {})
return canary_registry
@@ -139,7 +139,9 @@ class TestCanaryScope:
async def test_disabled_when_flag_off(self, monkeypatch, fresh_canary_registry):
from services.deployment.manager import DeploymentManager
- fresh_canary_registry.register_canary_trigger_type("webhookTrigger")
+ fresh_canary_registry.register_canary_trigger_type(
+ "webhookTrigger", "com.machinaos.webhook.received"
+ )
class _Off:
event_framework_enabled = False
@@ -170,8 +172,12 @@ class TestCanaryScope:
async def test_enabled_when_registered_and_flag_on(self, monkeypatch, fresh_canary_registry):
from services.deployment.manager import DeploymentManager
- fresh_canary_registry.register_canary_trigger_type("webhookTrigger")
- fresh_canary_registry.register_canary_trigger_type("chatTrigger")
+ fresh_canary_registry.register_canary_trigger_type(
+ "webhookTrigger", "com.machinaos.webhook.received"
+ )
+ fresh_canary_registry.register_canary_trigger_type(
+ "chatTrigger", "com.machinaos.chat.message.received"
+ )
class _On:
event_framework_enabled = True
@@ -307,8 +313,13 @@ class TestStartCanaryListener:
# EventTriggerKind derived via _trigger_kind_for (strips "Trigger" /
# "Receive" suffix) — NOT hardcoded "webhook".
assert attrs_by_name["EventTriggerKind"] == "webhook"
- # EventType comes from event_waiter.TRIGGER_REGISTRY['webhookTrigger'].
- assert attrs_by_name["EventType"] == "webhook_received"
+ # EventType MUST be the CloudEvents reverse-DNS string the
+ # producer puts on outgoing envelopes (registered via
+ # canary_registry). dispatch.emit's Visibility query substitutes
+ # event.type into the EventType filter — if the SA carries the
+ # legacy snake_case event_waiter string instead, the query
+ # never matches and no signal reaches the listener.
+ assert attrs_by_name["EventType"] == "com.machinaos.webhook.received"
@pytest.mark.asyncio
async def test_chat_trigger_uses_chat_kind_in_search_attrs(self, monkeypatch):
@@ -345,7 +356,9 @@ class TestStartCanaryListener:
sa = recorded[0]["search_attributes"]
attrs_by_name = {pair.key.name: pair.value for pair in sa}
assert attrs_by_name["EventTriggerKind"] == "chat"
- assert attrs_by_name["EventType"] == "chat_message_received"
+ # CloudEvents reverse-DNS — see test_search_attributes_include_event_workflow_id
+ # for the full rationale.
+ assert attrs_by_name["EventType"] == "com.machinaos.chat.message.received"
@pytest.mark.asyncio
async def test_returns_none_when_temporal_not_connected(self, monkeypatch):
@@ -516,6 +529,139 @@ class TestCancelCanaryListeners:
# ---------------------------------------------------------------------------
+class TestCancelSweepsStuckNodeStatuses:
+ """Regression: ``DeploymentManager.cancel`` must broadcast a full
+ status cleanup so the FE doesn't leave downstream nodes glowing
+ forever after the deployment goes down.
+
+ Pre-fix the cancel only reset trigger nodes (cron + listener) to
+ "idle". Downstream agents/tools/actions that were mid-execute when
+ cancel hit stayed in "executing" status on every connected client
+ because no sweep was issued. Also the toolbar Start/Stop indicator
+ stuck at ``executing=True`` because no terminal
+ ``workflow_run_ended`` / ``update_workflow_status(executing=False)``
+ was emitted.
+ """
+
+ def test_cancel_source_calls_stuck_node_sweep_with_include_waiting(self):
+ """Source-introspection: ``cancel`` must call
+ ``_clear_stuck_node_statuses(..., include_waiting=True)`` so
+ every node currently broadcast as ``executing`` OR ``waiting``
+ for this deployment goes back to idle on the FE. Without
+ ``include_waiting=True`` non-firing trigger siblings (and any
+ other ``waiting`` indicators outside the cron/listener buckets
+ the manager owns directly) stay glowing."""
+ import inspect
+
+ from services.deployment.manager import DeploymentManager
+
+ src = inspect.getsource(DeploymentManager.cancel)
+ assert "_clear_stuck_node_statuses" in src, (
+ "DeploymentManager.cancel no longer sweeps stuck node "
+ "statuses. Downstream nodes mid-execute at cancel-time "
+ "will stay glowing on FE forever. Restore the "
+ "``_broadcaster._clear_stuck_node_statuses(workflow_id, "
+ "include_waiting=True)`` call after the trigger resets."
+ )
+ assert "include_waiting=True" in src, (
+ "DeploymentManager.cancel must pass include_waiting=True to "
+ "the stuck-node sweep — explicit user-cancel is the "
+ "'every indicator goes quiet' signal. Without it, sibling "
+ "trigger nodes (or other waiting indicators) outlive the "
+ "deployment."
+ )
+
+ def test_cancel_source_broadcasts_terminal_workflow_status(self):
+ """Source-introspection: ``cancel`` must emit a final
+ ``update_workflow_status(executing=False, workflow_id=...)`` so
+ the toolbar Start/Stop indicator reflects the cancel. The
+ run-counter eviction in ``workflow_run_ended`` can race against
+ in-flight child runs that already incremented the counter."""
+ import inspect
+
+ from services.deployment.manager import DeploymentManager
+
+ src = inspect.getsource(DeploymentManager.cancel)
+ assert "update_workflow_status" in src, (
+ "DeploymentManager.cancel must broadcast "
+ "``update_workflow_status(executing=False, workflow_id=...)`` "
+ "so the FE toolbar Start/Stop indicator goes quiet after a "
+ "deployment is cancelled."
+ )
+ assert "executing=False" in src, (
+ "DeploymentManager.cancel must pass executing=False to the "
+ "terminal workflow_status broadcast. Anything else leaves "
+ "the toolbar showing the deployment as still active."
+ )
+
+ @pytest.mark.asyncio
+ async def test_cancel_runtime_calls_sweep_and_terminal_broadcast(self):
+ """Runtime smoke: stub the broadcaster + state + trigger_manager
+ and assert cancel actually invokes the sweep + terminal status
+ broadcast with the right args."""
+ from services.deployment.manager import DeploymentManager
+ from services.deployment.state import DeploymentState
+
+ sweep_calls: List[Dict[str, Any]] = []
+ status_calls: List[Dict[str, Any]] = []
+
+ broadcaster = MagicMock()
+ broadcaster.update_node_status = AsyncMock()
+ broadcaster.update_workflow_status = AsyncMock(
+ side_effect=lambda **kw: status_calls.append(kw),
+ )
+ broadcaster._clear_stuck_node_statuses = AsyncMock(
+ side_effect=lambda workflow_id, include_waiting=False: sweep_calls.append({
+ "workflow_id": workflow_id,
+ "include_waiting": include_waiting,
+ }) or 0,
+ )
+
+ database = MagicMock()
+ mgr = DeploymentManager(
+ database=database,
+ execute_workflow_fn=AsyncMock(return_value={"success": True}),
+ store_output_fn=AsyncMock(),
+ broadcaster=broadcaster,
+ )
+ # Seed deployment state — the cancel path bails early if
+ # the workflow isn't in self._deployments.
+ mgr._deployments["wf-1"] = DeploymentState(
+ deployment_id="deploy_wf-1",
+ workflow_id="wf-1",
+ is_running=True,
+ nodes=[{"id": "n1", "type": "aiAgent"}],
+ edges=[],
+ session_id="sess",
+ )
+
+ # Patch _cancel_canary_listeners / _cancel_canary_cron_schedules
+ # to be no-ops so the test focuses on the sweep + terminal
+ # broadcast contract.
+ mgr._cancel_canary_listeners = AsyncMock(return_value=0)
+ mgr._cancel_canary_cron_schedules = AsyncMock(return_value=0)
+
+ result = await mgr.cancel("wf-1")
+ assert result["success"] is True
+
+ # Sweep ran once with include_waiting=True (every indicator
+ # goes quiet on explicit user cancel).
+ assert len(sweep_calls) == 1, (
+ f"Expected one stuck-node sweep on cancel, got "
+ f"{len(sweep_calls)}: {sweep_calls!r}"
+ )
+ assert sweep_calls[0] == {
+ "workflow_id": "wf-1",
+ "include_waiting": True,
+ }
+
+ # Terminal executing=False broadcast for the toolbar.
+ assert {"executing": False, "workflow_id": "wf-1"} in status_calls, (
+ f"Expected update_workflow_status(executing=False, "
+ f"workflow_id='wf-1') on cancel; got {status_calls!r}"
+ )
+
+
class TestNoInstanceStateForCanaryListeners:
"""Lock the architectural decision: DeploymentManager does NOT keep
a Python dict of canary listener handles or ids. Temporal's Visibility
diff --git a/server/tests/test_event_framework_phase_a.py b/server/tests/test_event_framework_phase_a.py
index 3149b252..eb980bc8 100644
--- a/server/tests/test_event_framework_phase_a.py
+++ b/server/tests/test_event_framework_phase_a.py
@@ -32,6 +32,7 @@ from __future__ import annotations
import sys
import types
from datetime import timedelta
+from typing import Any, Dict, List
from unittest.mock import MagicMock
import pytest
@@ -425,3 +426,105 @@ class TestA8EmitEventActivity:
assert result["delivered"] is False
assert "error" in result
+
+
+class TestTriggerOutputPersistenceForTemplateResolution:
+ """Wave 13: the canary path's pre-executed trigger nodes must
+ persist their output to the workflow output store, otherwise
+ :class:`services.parameter_resolver.ParameterResolver` can't see
+ them and downstream ``{{triggerNode.field}}`` templates silently
+ resolve to empty.
+
+ Pre-fix MachinaWorkflow's pre-executed handler set
+ ``outputs[node_id]`` in WORKFLOW MEMORY only — never persisted.
+ The legacy ``DeploymentManager._execute_from_trigger`` path called
+ ``_store_output(trigger_node_id, "output_0", trigger_output)``
+ before executing downstream. The fix adds a
+ ``store_node_output_activity`` call inside MachinaWorkflow.run's
+ pre-executed loop that mirrors that persist.
+ """
+
+ def test_activity_exists_and_is_async(self):
+ import inspect
+
+ from services.temporal.activities import store_node_output_activity
+
+ assert inspect.iscoroutinefunction(store_node_output_activity), (
+ "store_node_output_activity must be async — workflow.execute_activity "
+ "awaits it."
+ )
+
+ def test_machina_workflow_calls_persist_activity_for_pre_executed(self):
+ """Source-introspection regression: MachinaWorkflow.run's
+ pre-executed handler must call store_node_output_activity so
+ ParameterResolver can read the trigger output back."""
+ import inspect
+
+ from services.temporal.workflow import MachinaWorkflow
+
+ src = inspect.getsource(MachinaWorkflow.run)
+
+ assert "store_node_output_activity" in src, (
+ "MachinaWorkflow.run no longer schedules the "
+ "store_node_output_activity for pre-executed trigger nodes. "
+ "Without this persist, the legacy "
+ "DeploymentManager._execute_from_trigger contract "
+ "(``_store_output(trigger_node_id, 'output_0', ...)``) is "
+ "missing on the canary path and ``{{triggerNode.field}}`` "
+ "templates resolve to empty."
+ )
+ # Sanity: the persist guard skips non-firing siblings (their
+ # _trigger_output is ``{not_triggered: True}``).
+ assert "not_triggered" in src, (
+ "MachinaWorkflow.run must skip the persist for non-firing "
+ "trigger siblings (``_trigger_output={'not_triggered': True}``). "
+ "Otherwise sibling triggers in the same deployment write empty "
+ "outputs that template references could pick up by mistake."
+ )
+
+ @pytest.mark.asyncio
+ async def test_store_activity_writes_three_handles(self, monkeypatch):
+ """The activity must write to every standard output handle —
+ downstream edges target different handle names (``output_main``
+ on agent nodes, ``output_0`` on trigger nodes, ``output_top`` on
+ legacy code paths). Writing only one would miss the others."""
+ from services.temporal.activities import store_node_output_activity
+
+ calls: List[Dict[str, Any]] = []
+
+ class _StubWorkflowService:
+ async def store_node_output(self, session_id, node_id, output_name, data):
+ calls.append({
+ "session_id": session_id,
+ "node_id": node_id,
+ "output_name": output_name,
+ "data": data,
+ })
+
+ class _StubContainer:
+ def workflow_service(self):
+ return _StubWorkflowService()
+
+ from core import container as container_mod
+ monkeypatch.setattr(container_mod, "container", _StubContainer())
+
+ await store_node_output_activity({
+ "node_id": "trig-1",
+ "session_id": "sess-1",
+ "result": {"message": "hello", "session_id": "sess-1"},
+ })
+
+ # All three standard handles get the data — same write pattern
+ # NodeExecutor uses for non-pre-executed nodes.
+ output_names = {c["output_name"] for c in calls}
+ assert output_names == {"output_main", "output_top", "output_0"}, (
+ f"Expected writes to output_main + output_top + output_0; "
+ f"got {sorted(output_names)!r}. Missing handles cause "
+ f"downstream template resolution to fail when the edge "
+ f"targets a handle that wasn't written."
+ )
+ # Same node_id + session_id + data on every write.
+ assert {c["node_id"] for c in calls} == {"trig-1"}
+ assert {c["session_id"] for c in calls} == {"sess-1"}
+ for c in calls:
+ assert c["data"] == {"message": "hello", "session_id": "sess-1"}
diff --git a/server/tests/test_polling_trigger_workflow.py b/server/tests/test_polling_trigger_workflow.py
index 58438210..c8a65ce8 100644
--- a/server/tests/test_polling_trigger_workflow.py
+++ b/server/tests/test_polling_trigger_workflow.py
@@ -157,9 +157,53 @@ class TestAsPollActivity:
new_ids = {e["id"] for e in result["events"]}
assert new_ids == {"c", "d"}
- # seen_ids is the union of prior + current.
+ # OOM fix: seen_ids is bounded by ``current`` — items the
+ # provider no longer reports drop out. Here ``current`` still
+ # includes a/b/c/d so all four stay.
assert set(result["seen_ids"]) == {"a", "b", "c", "d"}
+ @pytest.mark.asyncio
+ async def test_seen_ids_bounded_by_current_window_no_unbounded_growth(self):
+ """OOM regression guard: when items drop off the provider's
+ ``current`` window (archived / aged out), they must drop out of
+ ``seen_ids`` too. Pre-fix this returned ``prior_seen | current``
+ which grew forever; a Gmail trigger polling every 60s with ~100
+ emails/day would accumulate ~36K IDs in a year (~1.4MB just for
+ the seen set, serialised through Temporal payload every cycle)."""
+ # Provider only reports {e, f} as currently visible — a/b/c/d
+ # have aged out (e.g. archived in Gmail, marked read+filter is
+ # ``is:unread``, paged off the latest-N window, etc.).
+ cls = _make_fake_polling_class("test.bound", current_ids={"e", "f"})
+ activity_fn = cls.as_poll_activity()
+ inner = self._underlying(activity_fn)
+
+ result = await inner({
+ "node_id": "n1",
+ "params": {},
+ # Carry-forward from an earlier cycle.
+ "seen_ids": ["a", "b", "c", "d", "e"],
+ "baseline_only": False,
+ })
+
+ # Only the genuinely new id (``f``) emits — ``e`` was already
+ # in prior_seen.
+ new_ids = {ev["id"] for ev in result["events"]}
+ assert new_ids == {"f"}, (
+ "Only items in ``current`` that weren't in ``prior_seen`` "
+ "should emit."
+ )
+ # And the returned seen_ids must NOT contain a/b/c/d — those
+ # are gone from the provider's window. If they linger, every
+ # subsequent cycle pays Temporal-payload cost for IDs that will
+ # never matter again, and the set grows without bound.
+ assert set(result["seen_ids"]) == {"e", "f"}, (
+ f"seen_ids must be bounded by the provider's current window "
+ f"(got {sorted(result['seen_ids'])!r}). Old IDs that have "
+ f"dropped off the provider must NOT be carried forward — "
+ f"pre-fix this returned ``list(prior_seen | current)`` and "
+ f"leaked unboundedly."
+ )
+
@pytest.mark.asyncio
async def test_detail_payload_id_fallback(self):
"""When fetch_detail returns a payload without 'id', the
@@ -216,6 +260,71 @@ class TestAsPollActivity:
# ---------------------------------------------------------------------------
+class TestPollCoroutineSeenSetBounded:
+ """OOM regression guard for the legacy poll coroutine path
+ (``_build_poll_coroutine``).
+
+ Pre-fix the coroutine maintained ``seen: Set[str]`` that grew on
+ every emit via ``seen.add(msg_id)`` with no eviction — long-running
+ pollers (Gmail with the default 60s interval) accumulated tens of
+ thousands of entries over weeks/months. The fix rebases ``seen``
+ to ``current`` at the end of every cycle so items that drop off
+ the provider's window drop out of ``seen`` too.
+ """
+
+ def test_coroutine_source_rebases_seen_to_current(self):
+ """Source-introspection lock: ``_build_poll_coroutine`` must
+ contain ``seen = set(current)`` (or equivalent rebase) and must
+ NOT contain ``seen.add(`` inside the new-id emit loop. Driving
+ the live coroutine through cycles is fragile because the
+ per-class ``poll_interval_clamp`` enforces a 10s minimum
+ sleep — and the bug is in single-line state mutation that
+ source introspection catches reliably."""
+ import inspect
+
+ from services.plugin import PollingTriggerNode
+
+ # Build a coroutine on a real subclass — we only need its
+ # source, not its execution.
+ class _T(PollingTriggerNode, abstract=True):
+ type = "introspect.bound"
+ display_name = "Introspect"
+
+ async def setup_service(self, params): return MagicMock()
+ async def fetch_ids(self, service, params): return set()
+ async def fetch_detail(self, service, msg_id, params): return {"id": msg_id}
+ async def post_emit(self, service, msg_id, params): pass
+
+ # _build_poll_coroutine returns a closure; introspect the
+ # MRO-resolved method body instead. Strip comments and
+ # docstrings so the regex only sees executable code.
+ raw = inspect.getsource(PollingTriggerNode._build_poll_coroutine)
+ code_only = "\n".join(
+ line.split("#", 1)[0] # drop trailing/inline comments
+ for line in raw.splitlines()
+ )
+
+ # The fix: rebase seen to the provider's current snapshot at
+ # the end of every cycle.
+ assert "seen = set(current)" in code_only, (
+ "OOM fix regressed: _build_poll_coroutine no longer rebases "
+ "``seen`` to ``current`` at end of cycle. Pre-fix it called "
+ "``seen.add(msg_id)`` per emit with no eviction; old IDs "
+ "accumulated forever (~36K entries in a year for a 60s "
+ "poller). Restore the ``seen = set(current)`` rebase after "
+ "the new-id emit loop."
+ )
+
+ # And the anti-pattern (per-emit unbounded add) is gone from
+ # executable code. Comments / docstrings that REFERENCE the
+ # historical bug are fine (and useful for future maintainers).
+ assert "seen.add(" not in code_only, (
+ "OOM fix regressed: _build_poll_coroutine reintroduced "
+ "``seen.add(msg_id)``. The rebase-to-current pattern makes "
+ "this redundant AND unbounded — drop it."
+ )
+
+
class TestPollingTriggerWorkflowBody:
"""The workflow's signal/sleep/spawn loop runs deterministically."""
@@ -242,6 +351,11 @@ class TestPollingTriggerWorkflowBody:
from temporalio import workflow as temporal_workflow
async def fake_execute_activity(name, payload, **kwargs):
+ # Trigger-status broadcasts (idle/waiting around each spawn)
+ # are side-effect-only; they're driven by the workflow body
+ # but don't consume the scripted poll-cycle responses.
+ if name == "broadcast_trigger_status_activity":
+ return None
return next(responses)
sleep_calls: List[Any] = []
@@ -312,6 +426,10 @@ class TestPollingTriggerWorkflowBody:
call_count = [0]
async def fake_execute_activity(name, payload, **kwargs):
+ # Trigger-status broadcasts are side-effect-only; don't
+ # count them against the poll-cycle script.
+ if name == "broadcast_trigger_status_activity":
+ return None
call_count[0] += 1
if call_count[0] == 2: # second call fails
raise RuntimeError("transient API timeout")
diff --git a/server/tests/test_task_trigger_canary_producer.py b/server/tests/test_task_trigger_canary_producer.py
index 85fec1e1..d655a98f 100644
--- a/server/tests/test_task_trigger_canary_producer.py
+++ b/server/tests/test_task_trigger_canary_producer.py
@@ -1,11 +1,12 @@
-"""Wave 12 C1 rollout #2: taskTrigger producer dual-emit invariant.
+"""taskTrigger producer canary-emit invariant.
-Locks the contract: ``nodes.agent._events._broadcast_task_event`` must
-route delegated-task completion to BOTH the legacy
-``broadcaster.send_custom_event`` (which dispatches via event_waiter)
-AND the Temporal-durable ``services.events.dispatch.emit`` path. Without
-the second call, canary-enabled taskTrigger consumers miss every
-child-agent completion.
+Locks the contract: ``nodes.agent._events._broadcast_task_event`` routes
+delegated-task completion through the canary CloudEvents path
+(:func:`services.events.dispatch.emit`) ONLY. The legacy
+``broadcaster.send_custom_event`` path (which internally dispatched to
+``event_waiter``) was removed in Wave 13 — taskTrigger is canary-registered,
+the deployment manager skips ``setup_event_trigger``, and the legacy
+collector has zero consumers.
Cross-cutting factory: this rollout uses ``WorkflowEvent.task_completed``
which stays central per RFC §6.4 (parametrised by task_id + agent +
@@ -19,8 +20,8 @@ import inspect
import re
import sys
import types
-from typing import Any, Dict, List
-from unittest.mock import AsyncMock, MagicMock
+from typing import Any, List
+from unittest.mock import MagicMock
import pytest
@@ -37,54 +38,44 @@ _SEND_CUSTOM_EVENT_PATTERN = re.compile(r"send_custom_event\s*\(")
_EVENTS_EMIT_PATTERN = re.compile(r"\bemit\s*\(")
-class TestTaskTriggerProducerDualEmit:
- """Producer wrapper emits BOTH legacy and Temporal envelopes."""
+class TestTaskTriggerProducerCanaryEmit:
+ """Producer wrapper emits via the canary CloudEvents path only."""
def test_broadcast_helper_is_async(self):
from nodes.agent._events import _broadcast_task_event
assert inspect.iscoroutinefunction(_broadcast_task_event)
- def test_helper_routes_both_legacy_and_temporal(self):
+ def test_helper_uses_canary_path_only(self):
from nodes.agent import _events
src = inspect.getsource(_events._broadcast_task_event)
- assert _SEND_CUSTOM_EVENT_PATTERN.search(src), (
- "_broadcast_task_event must still call "
- "broadcaster.send_custom_event(_LEGACY_EVENT_TYPE, payload) "
- "so canary-flag-off deployments keep their existing dispatch."
- )
assert _EVENTS_EMIT_PATTERN.search(src), (
"_broadcast_task_event must call "
- "services.events.dispatch.emit(envelope, ...) for the "
- "Temporal-durable canary path. Without it, taskTrigger "
- "TriggerListenerWorkflows receive nothing when the "
- "event_framework_enabled flag is on."
+ "services.events.dispatch.emit(envelope, ...) — the canary "
+ "CloudEvents path Signals running TriggerListenerWorkflow "
+ "consumers AND broadcasts to FE on the task_completed wire key."
+ )
+ assert not _SEND_CUSTOM_EVENT_PATTERN.search(src), (
+ "_broadcast_task_event must NOT call "
+ "broadcaster.send_custom_event — taskTrigger is canary-registered "
+ "and the legacy event_waiter dispatch path (which "
+ "send_custom_event drives internally) has zero consumers "
+ "(removed in Wave 13)."
)
@pytest.mark.asyncio
- async def test_completed_event_routes_both_paths(self, monkeypatch):
+ async def test_completed_event_emits_canary_envelope(self, monkeypatch):
from nodes.agent import _events
from services.events import dispatch as dispatch_mod
- custom_calls: List[Dict[str, Any]] = []
emit_calls: List[Any] = []
- broadcaster = MagicMock()
-
- async def fake_send_custom_event(event_type, payload):
- custom_calls.append({"event_type": event_type, "payload": payload})
-
- broadcaster.send_custom_event = fake_send_custom_event
-
async def fake_emit(event, **kwargs):
emit_calls.append({"event": event, **kwargs})
return event
- # Patch the get_status_broadcaster singleton.
- from services import status_broadcaster as sb
- monkeypatch.setattr(sb, "get_status_broadcaster", lambda: broadcaster)
monkeypatch.setattr(dispatch_mod, "emit", fake_emit)
await _events.broadcast_agent_task_completed(
@@ -96,40 +87,31 @@ class TestTaskTriggerProducerDualEmit:
result="done",
)
- # Legacy path: wire key = task_completed, payload carries
- # the existing fields the taskTrigger filter reads.
- assert len(custom_calls) == 1
- assert custom_calls[0]["event_type"] == "task_completed"
- legacy_payload = custom_calls[0]["payload"]
- assert legacy_payload["task_id"] == "task-1"
- assert legacy_payload["status"] == "completed"
- assert legacy_payload["result"] == "done"
-
- # Temporal path: envelope.type discriminates on succeeded vs
- # failed; subject is the task_id; wire_routing_key preserved.
+ # Single type (``.completed``) so the listener's EventType SA
+ # matches via dispatch.emit's Visibility query. Status
+ # discrimination lives in data.status.
assert len(emit_calls) == 1
envelope = emit_calls[0]["event"]
- assert envelope.type == "com.machinaos.agent.task.succeeded"
+ assert envelope.type == "com.machinaos.agent.task.completed"
assert envelope.subject == "task-1"
+ assert envelope.data["status"] == "completed"
+ assert envelope.data["result"] == "done"
assert emit_calls[0]["wire_routing_key"] == "task_completed"
@pytest.mark.asyncio
- async def test_failed_event_routes_both_paths_with_error_type(self, monkeypatch):
- """Type discriminates between succeeded/failed; subject still task_id."""
+ async def test_failed_event_shares_type_with_status_discriminator(self, monkeypatch):
+ """Success and failure share a single CloudEvents type so the
+ listener's EventType Search Attribute matches both branches;
+ ``status='error'`` lives in the payload."""
from nodes.agent import _events
from services.events import dispatch as dispatch_mod
emit_calls: List[Any] = []
- broadcaster = MagicMock()
- broadcaster.send_custom_event = AsyncMock()
-
async def fake_emit(event, **kwargs):
emit_calls.append(event)
return event
- from services import status_broadcaster as sb
- monkeypatch.setattr(sb, "get_status_broadcaster", lambda: broadcaster)
monkeypatch.setattr(dispatch_mod, "emit", fake_emit)
await _events.broadcast_agent_task_failed(
@@ -143,7 +125,7 @@ class TestTaskTriggerProducerDualEmit:
assert len(emit_calls) == 1
envelope = emit_calls[0]
- # Status='error' maps to the .failed type variant per the
- # central factory's discrimination.
- assert envelope.type == "com.machinaos.agent.task.failed"
+ assert envelope.type == "com.machinaos.agent.task.completed"
assert envelope.subject == "task-2"
+ assert envelope.data["status"] == "error"
+ assert envelope.data["error"] == "timeout"
diff --git a/server/tests/test_telegram_trigger_canary_producer.py b/server/tests/test_telegram_trigger_canary_producer.py
index 2d5bf133..b3317df9 100644
--- a/server/tests/test_telegram_trigger_canary_producer.py
+++ b/server/tests/test_telegram_trigger_canary_producer.py
@@ -1,12 +1,11 @@
-"""Wave 12 C1 rollout #3: telegramReceive producer dual-emit invariant.
+"""telegramReceive producer canary-emit invariant.
Locks the contract: ``nodes.telegram._events.dispatch_telegram_message_received``
-must route incoming Telegram messages to BOTH the legacy
-``event_waiter.dispatch`` and the Temporal-durable
-``services.events.dispatch.emit`` path. Without the second call, a
-canary-enabled telegramReceive deployment silently misses every
-incoming message — the TriggerListenerWorkflow has no other event
-source.
+routes incoming Telegram messages through the canary CloudEvents path
+(:func:`services.events.dispatch.emit`) ONLY. The legacy
+``event_waiter.dispatch`` path was removed in Wave 13 — telegramReceive
+is canary-registered, the deployment manager skips
+``setup_event_trigger``, and the legacy collector has zero consumers.
Same regex-introspection + runtime smoke pattern used for the
webhookTrigger / chatTrigger / taskTrigger producers in this series.
@@ -18,8 +17,8 @@ import inspect
import re
import sys
import types
-from typing import Any, Dict, List
-from unittest.mock import AsyncMock, MagicMock
+from typing import Any, List
+from unittest.mock import MagicMock
import pytest
@@ -37,51 +36,44 @@ _EVENT_WAITER_DISPATCH_PATTERN = re.compile(r"event_waiter\.dispatch\s*\(")
_EVENTS_EMIT_PATTERN = re.compile(r"\bemit\s*\(")
-class TestTelegramProducerDualEmit:
- """Producer wrapper emits BOTH legacy and Temporal envelopes."""
+class TestTelegramProducerCanaryEmit:
+ """Producer wrapper emits via the canary CloudEvents path only."""
def test_dispatcher_is_async(self):
from nodes.telegram._events import dispatch_telegram_message_received
assert inspect.iscoroutinefunction(dispatch_telegram_message_received)
- def test_dispatcher_routes_both_legacy_and_temporal(self):
+ def test_dispatcher_uses_canary_path_only(self):
from nodes.telegram import _events
src = inspect.getsource(_events.dispatch_telegram_message_received)
- assert _EVENT_WAITER_DISPATCH_PATTERN.search(src), (
- "dispatch_telegram_message_received must still call "
- "event_waiter.dispatch(_MESSAGE_LEGACY_EVENT_TYPE, ...) for "
- "the in-process waiter path. Without it, canary-flag-off "
- "deployments lose their existing telegram dispatch."
- )
assert _EVENTS_EMIT_PATTERN.search(src), (
"dispatch_telegram_message_received must call "
- "services.events.dispatch.emit(envelope, ...) for the "
- "Temporal-durable canary path. Without it, telegramReceive "
- "TriggerListenerWorkflows receive nothing when the "
- "event_framework_enabled flag is on."
+ "services.events.dispatch.emit(envelope, ...) — the canary "
+ "CloudEvents path Signals running TriggerListenerWorkflow "
+ "consumers AND broadcasts to FE on the "
+ "telegram_message_received wire key."
+ )
+ assert not _EVENT_WAITER_DISPATCH_PATTERN.search(src), (
+ "dispatch_telegram_message_received must NOT call "
+ "event_waiter.dispatch — telegramReceive is canary-registered "
+ "and the legacy collector path has zero consumers (removed "
+ "in Wave 13)."
)
@pytest.mark.asyncio
- async def test_runtime_dual_emit_calls_both_paths(self, monkeypatch):
+ async def test_runtime_emits_canary_envelope(self, monkeypatch):
from nodes.telegram import _events
- from services import event_waiter
from services.events import dispatch as dispatch_mod
- legacy_calls: List[Dict[str, Any]] = []
emit_calls: List[Any] = []
- def fake_dispatch(event_type, data):
- legacy_calls.append({"event_type": event_type, "data": data})
- return 0
-
async def fake_emit(event, **kwargs):
emit_calls.append({"event": event, **kwargs})
return event
- monkeypatch.setattr(event_waiter, "dispatch", fake_dispatch)
monkeypatch.setattr(dispatch_mod, "emit", fake_emit)
result = await _events.dispatch_telegram_message_received({
@@ -91,11 +83,7 @@ class TestTelegramProducerDualEmit:
"timestamp": "2026-05-14T00:00:00",
})
- assert result == 0
-
- assert len(legacy_calls) == 1
- assert legacy_calls[0]["event_type"] == "telegram_message_received"
- assert legacy_calls[0]["data"]["chat_id"] == 12345
+ assert result is None
assert len(emit_calls) == 1
envelope = emit_calls[0]["event"]
@@ -118,6 +106,6 @@ class TestTelegramPluginSelfRegistersCanary:
assert canary_registry.is_canary_trigger_type("telegramReceive"), (
"Importing nodes.telegram should call "
- "register_canary_trigger_type('telegramReceive') — see "
+ "register_canary_trigger_type('telegramReceive', ...) — see "
"the __init__.py bottom section."
)
diff --git a/server/tests/test_trigger_listener_workflow.py b/server/tests/test_trigger_listener_workflow.py
index 8095fbdb..03ba0a25 100644
--- a/server/tests/test_trigger_listener_workflow.py
+++ b/server/tests/test_trigger_listener_workflow.py
@@ -277,6 +277,10 @@ class TestSpawnChildRun:
recorded.append({"name": workflow_name, **kwargs})
return MagicMock()
+ async def fake_execute_activity(*args, **kwargs):
+ # Status-broadcast activities — no-op for the spawn test.
+ return None
+
from temporalio import workflow as temporal_workflow
monkeypatch.setattr(
@@ -284,6 +288,11 @@ class TestSpawnChildRun:
"start_child_workflow",
fake_start_child_workflow,
)
+ monkeypatch.setattr(
+ temporal_workflow,
+ "execute_activity",
+ fake_execute_activity,
+ )
wf = tlw.TriggerListenerWorkflow()
event = {
@@ -327,6 +336,9 @@ class TestSpawnChildRun:
async def boom(*args, **kwargs):
raise RuntimeError("temporal unreachable")
+ async def fake_execute_activity(*args, **kwargs):
+ return None
+
from temporalio import workflow as temporal_workflow
monkeypatch.setattr(
@@ -334,6 +346,11 @@ class TestSpawnChildRun:
"start_child_workflow",
boom,
)
+ monkeypatch.setattr(
+ temporal_workflow,
+ "execute_activity",
+ fake_execute_activity,
+ )
wf = tlw.TriggerListenerWorkflow()
event = {"id": "evt-1", "type": "x", "data": {}}
diff --git a/server/tests/test_whatsapp_trigger_canary_producer.py b/server/tests/test_whatsapp_trigger_canary_producer.py
index 2fdaeb1c..e2c3f7c9 100644
--- a/server/tests/test_whatsapp_trigger_canary_producer.py
+++ b/server/tests/test_whatsapp_trigger_canary_producer.py
@@ -92,8 +92,13 @@ class TestWhatsappProducerDualEmit:
},
)
- # WS broadcasts still fire (legacy raw + typed sibling).
- assert broadcaster.broadcast.await_count == 2
+ # Legacy raw frame still goes out so the existing FE message-list
+ # handler at WebSocketContext.tsx (which reads ``data.*``)
+ # keeps working until the Wave 12 D4 follow-up migrates it to
+ # envelope-shape. The duplicate typed-envelope WS broadcast was
+ # dropped in Wave 13 — dispatch.emit broadcasts the envelope on
+ # the same wire key once.
+ assert broadcaster.broadcast.await_count == 1
# And the canary path fires exactly once.
assert len(emit_calls) == 1
@@ -132,10 +137,12 @@ class TestWhatsappProducerDualEmit:
{"chat_id": "[email protected]", "text": "hi"},
)
- # WS broadcasts still fire for the observation channel.
- assert broadcaster.broadcast.await_count == 2
+ # Single legacy raw broadcast for the FE message-list
+ # observation channel — duplicate typed-envelope sibling dropped
+ # in Wave 13.
+ assert broadcaster.broadcast.await_count == 1
- # But NO canary fan-out (no trigger consumes outbound).
+ # No canary fan-out (no trigger consumes outbound).
assert emit_calls == []