Self-improving AI Employees for everyone
Event Framework and Durable Trigger Queues
Current architecture: This document retains the Wave-12 rollout history. Controlled deployments now consolidate push registration, event Signals, and polling activities inside
WorkflowControlWorkflow; see workflow control contract. Listener workflows described in the phase log are legacy compatibility contracts.
Temporal-native event-routing layer for OpenCompany. Implements RFC sections 6.3 (Temporal worker contract) + 6.4 (CloudEvents broadcast contract) from plugin_authoring_rfc.md.
This is the operator and plugin-author reference. The phase log below is historical; the current control and queue contract is documented here and in Temporal architecture. The original design is in the archived execution RFC.
Status (2026-05-15)
Shipped
| Phase | State |
|---|---|
| A1-A9 — Temporal primitives + core CloudEvents envelope compliance | ✅ commit c3dc85a (16 tests; legacy extension-name debt documented below) |
A7 completion — _pop_matching_event helper |
✅ commit 0e835e2 (4 tests) |
B1-B10 — plugin-owned _events.py modules (9 plugin folders) |
✅ commits 7e4ff7b / c4d9428 / da63d73 / de8be88 |
C1 canary (webhookTrigger) — TriggerListenerWorkflow |
✅ commit c24bc62 (25 tests) |
| C1 rollouts — chat / task / telegram / whatsapp | ✅ commits 850cc9d / 0d406b9 / 688f686 / b4db7da |
C1 architecture pivot — plugin-self-registered canary_registry retires tribal frozenset |
✅ commit 688f686 |
C2 canary (googleGmailReceive) — PollingTriggerWorkflow + as_poll_activity |
✅ commit 00dbf10 (10 tests) |
C3 canary (cronScheduler) — Temporal Schedule + plugin-owned CronTriggerWorkflow via SimplePlugin |
✅ commit 9314aff (17 tests) |
C4 sub-piece A — social_provider_registry closes nodes/social→nodes/whatsapp |
✅ commit d1cc33c (8 tests) |
C4 sub-piece B — shutdown_hooks registry closes 2 reaches in main.py lifespan + IdempotentRegistry reload-tolerance fix |
✅ commit 4912239 (10 tests) |
C4 sub-piece C — service_factories registry closes core/container.py top-level imports |
✅ commit 73c5f08 (9 tests) |
D1 — Shared _retry_policies + NodeUserError non-retryable |
✅ commit 751ab94 (11 tests) |
D3 — Visibility admin WS handlers (list_canary_listeners / list_canary_schedules / get_workflow_failure_history) |
✅ commit aebfb35 (13 tests) |
D5 — Auto-gen DEFAULT_TOOL_NAMES from ToolNode ClassVars |
✅ commits a01f590 / 07906f0 / 899771c (78 plugin classes + golden fixture + 3 invariant tests; 75 passed) |
B11 — FE plugin_connection_status envelope handler |
✅ commit 899771c |
D4 — Drop legacy *_status raw frames (status-only round) |
✅ commit 5ea4e90 (whatsapp/android/telegram status retired; FE consumes typed channel) |
| Canary flag default flip + invariant lock | ✅ shipped (default True; the EVENT_FRAMEWORK_ENABLED=false rollback no longer works, see "Rollback (known gap)" below) |
| Wave 13 — EventType SA mismatch fix + canary-only emit path + trigger status lifecycle + polling OOM + template resolution + cancel sweep | ✅ shipped (see "Wave 13 fixes" below) |
Dropped / Deferred / Pending
| Phase | State |
|---|---|
D2 — Custom event_dlq SQLModel table |
❌ dropped (commit 89b15bd, docs only). Temporal Event History + Visibility queries cover the ops-inspection use case; reinventing them would contradict Wave 12's "Temporal-native, no custom infra" thesis. See § "Failure inspection — no separate DLQ table". |
D2b — Retire event_waiter.py Redis-Streams branch |
✅ shipped as Wave 15.3 (see TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md). event_waiter is memory-mode-only now; Temporal owns durable delivery. The in-memory collector still backs canvas-Run + non-canary (Twitter) triggers. |
D4 — Drain the last legacy event_waiter.dispatch dual-emit |
⏳ pending — only nodes/twitter/_events.py remains (twitterReceive is not canary-registered). The whatsapp message/newsletter/history wire keys are now one-way legacy WS frames for FE readers, not dual-emit. |
| WorkflowEnvironment integration smoke test | ⏳ pending — every canary type against an in-process Temporal cluster. Existing unit tests + per-canary producer tests + TestCanaryRegistryCoverage cover the static surface; the integration smoke would catch real-cluster regressions only. |
Settings.event_framework_enabled gates the new dispatch path. Default flipped to True on 2026-05-15 — the Temporal-Signal consumer fan-out is now production-default. With the flag off, services.events.dispatch.emit is a pass-through no-op (logged at DEBUG as event-framework disabled — emit no-op), and DeploymentManager._canary_listener_enabled_for sends canary trigger types back to the in-process collector (trigger_manager.setup_event_trigger).
Rollback (known gap)
EVENT_FRAMEWORK_ENABLED=false is not a working rollback any more. The in-process collector fires only on event_waiter.dispatch(...), and most canary producers no longer call it: their _events.py calls dispatch.emit(...) alone (chat, webhook and email say so in their module docstrings). With the flag off, those deployed triggers never fire, and the in-process WebSocket broadcast that emit also performs stops too. The only remaining event_waiter.dispatch callers are nodes/twitter/_events.py, StatusBroadcaster.send_custom_event, and the plugin WebhookSource intake in services/events/webhook.py. Restoring a rollback would mean a dual-dispatch in emit itself; nothing does that today, so leave the flag on.
Locked by tests/test_event_framework_phase_a.py::TestEventFrameworkEnabledDefault::test_event_framework_enabled_defaults_true (source-introspection check that the Field(default=True, ...) declaration is present) and TestCanaryRegistryCoverage::test_seven_canary_types_registered, which asserts that the seven original canary types are registered. It is a floor, not the full list: later plugins register too. For the live set, run grep -rn "register_canary_trigger_type(" server/nodes.
What this framework does
Push producers build a WorkflowEvent and call dispatch.emit. Routing uses
Temporal Visibility to find running legacy EventType consumers and generation
controllers. A controller advertises ControlEventTypes and matches accepted
Signals against its registered trigger definitions. Polling controllers instead
call provider Activities and enqueue their returned events; cron uses Temporal
Schedules and the plugin's CronTriggerWorkflow.
Visibility is eventually consistent. The dispatch helper logs and suppresses Temporal connection, lookup and individual Signal failures, and returns the envelope rather than a delivery receipt. Its return therefore does not prove that a consumer accepted the event. Once a Signal RPC succeeds, Temporal stores the Signal durably; successful acceptance still does not acknowledge handler processing or graph execution. There is no producer outbox in this path.
Controller queues preserve accepted events across Stop, worker restart and continue-as-new. Deduplication is an application key contract, supplemented by deterministic child Workflow IDs; Workflow ID policies do not deduplicate arbitrary Signal payloads. Existing SQL overflow pages and atomic read receipts are reused for queue spill, without adding an event bus or separate DLQ.
Architecture
Inbound source (FastAPI process)
↓
services/events/dispatch.py:emit(event: WorkflowEvent)
├─→ Temporal Visibility: legacy listeners + generation controllers
├─→ on_event Signal to matching consumers
│ └─→ controller queue → admitted graph child
└─→ status_broadcaster.broadcast() — direct in-process WS fan-out
(skipped with emit(..., broadcast=False))
Webhook bodies are capped at the edge. Every webhook body is read through
services/events/webhook.read_body_capped: the generic /webhook/{path}
route, WebhookSource.handle (Stripe, Dodo Payments, WhatsApp Business) and
Discord interactions. A body larger than WEBHOOK_MAX_BODY_BYTES (default in
.env.template) gets 413 before anything is dispatched, because the body
rides into a workflow's input, which Temporal caps at 2 MB, and the trigger
path copies it several times (RFC-0004 C1). The declared Content-Length
refuses an honest oversized request before it is read; the running total of
what actually arrives is the limit. WebhookSource.handle no longer puts
each event on the source's queue, which nothing read.
Private payloads skip the broadcast. The broadcast reaches every
connected socket, so emit has no default for broadcast: every producer
states whether a client reads the frame (RFC-0005 Phase 0.9). An event whose
payload is someone's content passes broadcast=False and reaches only the
signalled workflows: owner chat (clients learn of a new message from the
identity-only chat.updated), webhooks, Stripe, Dodo Payments, WhatsApp,
WhatsApp Business, Telegram, Discord, email and Twitter. No client read any
of those frames. Delegated-task completion broadcasts a separate
identity-only envelope (the Task Manager panel refreshes on it), and
todos_updated broadcasts because the checklist reads it. The WhatsApp
bridge's outbound event.message_sent is no longer relayed at all.
Workflow-scoped delivery (core dispatch rule, July 2026): an envelope
whose workflow_id field is set by its producer's call site reaches ONLY
that workflow's consumers — _signal_running_consumers appends
AND EventWorkflowId='<id>' to both the listener clause and the controller
clause of the Visibility query (every consumer shape carries that Search
Attribute). Unscoped envelopes (telegram / webhook / gmail — genuinely
broadcast semantics) are unchanged. First producer: chat — the panel is
bound to one workflow (session_id IS its workflow id), and without the
scope one workflow's chat message fired every deployed workflow's
chatTrigger. The scoping DECISION lives in the core call site
(services/chat/handlers.py:handle_send_chat_message; the legacy
"default" session stays unscoped) and the narrowing in core dispatch —
event factories (the chat one is services/chat/events.py, source
opencompany://services/chat) only plumb the field of their own wire
shape. The chat message is emitted with broadcast=False (it carries the
owner's text), and its CloudEvent id is the chat run it started, so the
listener's child run id is predictable; chat run events (chat_run_event)
never ride emit or the broadcast at all: services/chat/hub.py delivers
them only to sockets subscribed to the session
(chat_protocol.md).
Second producer: delegated-task completion (nodes/agent/_events.py,
com.opencompany.agent.task.completed). Without the scope, one workflow's
finished task fired every deployed workflow's taskTrigger (RFC-0004 H11).
The envelope is emitted with broadcast=False, because its data holds the
child agent's answer or error. The task_completed WebSocket frame is a
second envelope of the same type and id whose data is only task_id,
status, team_id and workflow_id, which is all the Task Manager panel
reads to refresh.
Third producer: generic webhooks (nodes/trigger/webhook_trigger/_routes.py,
com.opencompany.webhook.received). deliver_webhook takes its route table
from the running and paused generations' snapshots and answers 404 for a path
none declares, 405 for a method none of its triggers accepts, and 401 when
the trigger's header check fails (expected value read from the saved node
parameters, never from a snapshot or a Temporal input; compared with
hmac.compare_digest; no configured value refuses). Each workflow with an
accepting trigger gets one envelope scoped to it, emitted with
broadcast=False; authorization, cookie, proxy-authorization and the
trigger's own header are removed first. A webhookTrigger with an empty path
matches nothing, and Start refuses one (WEBHOOK_PATH_REQUIRED). Before this
(RFC-0004 C3) the header check was never made, one webhook fired every
deployed workflow's trigger, and headers and body were broadcast to every
socket.
Worker is embedded in the FastAPI process (the main.py lifespan schedules
run_temporal_lifecycle from services/temporal/lifecycle.py as one
asyncio.create_task() named temporal-init; that module owns the connect loop and the
TemporalWorkerManager / TemporalWorkerPool start). Activities
and the WebSocket connection pool share memory + event loop, so the fan-out
to FE clients is a direct in-process call — no Redis Streams hop required.
Search Attributes setup
The framework requires 7 custom Search Attributes on the Temporal
namespace. Registration is idempotent + automatic on Temporal client
connect (services/temporal/client.py:TemporalClientWrapper.connect):
| Attribute | Type | Used for |
|---|---|---|
EventType |
KEYWORD | Visibility query — find consumers by CloudEvents type |
EventSource |
KEYWORD | Routing when same type arrives from multiple sources |
EventWorkflowId |
KEYWORD | Scope events to a OpenCompany workflow_id |
TriggerNodeId |
KEYWORD | Per-trigger event-history queries |
EventTriggerKind |
KEYWORD | Coarse classification (webhook / polling / …) |
EventReceivedAt |
DATETIME | Time-range queries |
ControlEventTypes |
KEYWORD_LIST | The push event types a WorkflowControlWorkflow currently has triggers for (upserted as triggers register); dispatch.emit skips controllers whose list does not contain the event type. Absent on pre-upgrade histories = match-all. |
Declarations live in
services/temporal/search_attributes.py:EVENT_SEARCH_ATTRIBUTES.
Single source of truth — add an entry to that list and registration
picks it up next connect.
Manual registration (if needed)
If the auto-registration on connect fails (e.g. permissions on a managed Temporal Cloud namespace), register manually via the Temporal CLI:
temporal operator search-attribute create \
--namespace default \
--name EventType \
--type Keyword
# ... repeat for the other 6 attributes (ControlEventTypes is --type KeywordList)
Verification
temporal operator search-attribute list --namespace default
Should show the 7 framework attributes alongside Temporal's built-in
default attributes (WorkflowType, WorkflowId, ExecutionStatus, …).
Temporal contract for plugin authors
Event keys and accumulator behavior
The controller borrows durable buffering, stable keys and continuation-state carry from Temporal's Event Accumulator pattern. The complete inactivity-window batching pattern is unsuitable here: triggers process individual events immediately when admission is open. Stop holds the queue; Resume admits it in FIFO order without replaying completed graph work.
New version 1 histories use controller-event-accumulator-v1. Pending keys are
protected separately from the bounded recent-completion cache, including an
event currently being dispatched. Restored overflow pages retain their earlier
FIFO position over a same-key live redelivery. Continue-as-new rechecks arrivals
after every spill and after message handlers finish, then carries or spills the
accepted tail. Reset's close flag prevents rollover from reopening the controller.
Use stable source identities for redelivery of the same occurrence:
| Producer | Event ID |
|---|---|
| Telegram | telegram:{chat_id}:{message_id} |
| Discord gateway message | discord:message:{message_id} |
| Discord interaction | discord:interaction:{interaction_id} |
whatsapp:{direction}:{chat_id_or_sender}:{message_id} |
|
| Chat | Tracked run ID, otherwise saved message ID |
A missing component retains a fresh generated ID rather than collapsing unrelated messages. Provider event factories own these mappings; do not generate random IDs inside a Workflow. Dedup is bounded, not a permanent occurrence ledger. An already-started deterministic child is a successful duplicate admission; transient child-start failures keep the original event pending for retry.
Signal-With-Start is appropriate when creating a collector is part of the producer contract. Here Start owns generation creation and execution-root membership. Recreating a lost version 1 controller via Signal-With-Start would produce an empty registry, so producers signal the existing controller and its loss fails closed. Legacy deployment recovery remains separate.
See workflow messaging for completed control Updates and read-only status Queries. The Signal queue and controller rollover tests are listed in the verification guide.
Activity policy and Stop boundaries
| Class attribute | Purpose | Default |
|---|---|---|
start_to_close_timeout |
Per-attempt budget (one activity execution) | Kind-base default: ActionNode=10m, TriggerNode=24h, ToolNode=10m |
retry_policy: RetryPolicy |
Backoff + max attempts + non-retryable error types | DEFAULT_RETRY — 3 attempts, 1-60s exponential. NodeUserError is auto-non-retryable. |
heartbeat_timeout |
Max idle between activity.heartbeat() calls |
2 minutes |
task_queue |
Worker pool routing | TaskQueue.DEFAULT |
Override only when the kind-base default doesn't fit; an inline comment
explaining why is required (enforced by
tests/test_plugin_contract.py::TestStartToCloseTimeoutOverridesAreCommented).
Ordinary agent tool dispatch carries the resolved plugin's timeout, heartbeat, retry and task-queue policy in prepared tool bindings. Routing also respects the run's frozen worker-pool setting. Node and tool operations remain regular Activities. Stop closes admission for the next operation and lets admitted operations finish under their existing retry policies, including bookkeeping. Resume consumes recorded results and continues from the next pending action.
Heartbeats track liveness, enable cancellation delivery and can record real recoverable progress after failure. A status string cannot checkpoint an opaque HTTP request, subprocess or provider session. Do not add a generic heartbeat cursor or cancel the current tool to implement Stop. See the node authoring contract and long-running Activity guidance.
Worker graceful shutdown
Workers honor SIGTERM with a configurable grace window
(Settings.temporal_graceful_shutdown_seconds, default 30s, override via
env TEMPORAL_GRACEFUL_SHUTDOWN_SECONDS). Activities mid-flight finish
or hand back to the server for retry instead of being killed mid-call.
CloudEvents envelope
Domain/lifecycle broadcasts wrap WorkflowEvent (CloudEvents v1.0;
services/events/envelope.py). High-frequency state projections such as
node_status, output, variables, and log streams remain the explicitly tested
telemetry carve-out; they are reconnect snapshots, not lifecycle occurrences.
New envelopes use the required specversion, id, source, and type
attributes, a reverse-DNS type, and an exact entity subject. Application
scope such as workflow_id, execution_id, and root_execution_id belongs
inside data. CloudEvents requires extension-attribute names to contain only
lowercase ASCII letters and digits, so the historical top-level
workflow_id, trigger_node_id, and correlation_id fields are compatibility
debt and must not be copied into new contracts.
Wire shape: {"type": "<legacy_wire_key>", "data": <WorkflowEvent JSON>}.
The outer type is what FE switches on; the inner envelope carries the
CloudEvents routing identity and dataschema lookup.
Agent capability occurrences
Skill and tool activity uses the agent_capability WebSocket route with a
complete CloudEvent inside data:
source = opencompany://services/agent;type = com.opencompany.agent.(skill|tool).<state>;subject = data.agent_node_id, the exact invoking agent;data.target_node_id, when present, is the exact connected Master Skill or tool node;- Temporal occurrences use a deterministic
id; consumers deduplicate on the CloudEvents(source, id)identity pair.
The browser rejects envelopes whose type/state/subject disagree, ignores late events from an archived root execution, and updates only the stated agent and target nodes. Prompts, tool arguments, instruction/resource bodies, raw results, secrets, and raw exceptions are forbidden from the payload.
Capability telemetry broadcasts directly to WebSocket clients through
StatusBroadcaster.broadcast_agent_capability. It does not use
services.events.dispatch.emit: that path performs Temporal Visibility lookup
and signals trigger consumers, while capability activity is an observation of
an execution already in progress. The paired raw node_status frame is only a
latest-state/reconnect projection; the CloudEvent is the canonical occurrence.
Plugin-owned event factories
Plugin-specific events (e.g. com.opencompany.telegram.message.received)
live in nodes/<plugin>/_events.py. Cross-cutting factories
(credential, oauth_completed, agent_progress, agent_capability, task_completed,
workflow_lifecycle, deployment_snapshot, team_event,
node_parameters_updated) stay in services/events/envelope.py. A core
service that owns its event keeps the factory beside it, so core code never
imports a plugin's _events.py: services/chat_thread.py (chat_updated)
and services/workflow_ops.py (workflow_ops_applied, sent by
broadcast_workflow_ops).
See RFC §6.4 for the classification rule + the canonical
telegram/_events.py example.
UI-only lifecycle events broadcast directly, never through emit
dispatch.emit exists to reach Temporal consumers: it runs a Visibility
ListWorkflowExecutions query to find running listeners, then broadcasts
in-process as a side effect. That query is only worth paying for when some node
type registered the event via register_canary_trigger_type.
The Context conversation event (context.updated, in
nodes/context/_events.py) and the Memory mutation event (memory.updated,
in nodes/tool/simple_memory/_events.py) have no canary consumer, so the
query is guaranteed to match nothing — once per save/mutation. They call
get_status_broadcaster().broadcast({...}) directly instead, which is the same
pattern nodes/telegram/_events.py uses for status. The CloudEvents envelope,
source, type, subject and data are identical either way, so the wire
contract the frontend sees does not change. Normal mode's two events follow
the same rule: employee_lifecycle (com.opencompany.employee.*, in
services/employees/events.py) and approval_lifecycle
(com.opencompany.approval.*, in services/approvals/events.py,
identity only). See normal_mode.md. So do
the Browser plugin's browser_updated, browser_profiles_updated and
browser_runtime (com.opencompany.browser.*, in nodes/browser/_events.py;
identity and state only, never a URL, a page title or cookie data).
Two core events follow it too. chat.updated (com.opencompany.chat.updated,
in services/chat_thread.py) goes out after every chat row insert and clear,
with data {workflow_id, session_id, role} (role null for a clear,
workflow_id null for session "default"): identity only, so Home's thread
and the editor's chat pane refetch through get_chat_messages. The
workflow-ops push workflow_ops_apply (com.opencompany.workflow.ops.applied,
built and sent by services/workflow_ops.broadcast_workflow_ops) is the one
whose frame is the event's flat data,
{workflow_id, caller_node_id, operations, persisted?}, the shape
useWorkflowOpsListener reads.
persisted: true marks a batch the server already saved
(services/workflow_storage/mutate.apply_graph_additions: Turn on Talk, the
Agent Builder); its ops carry the server's ids and full parameter rows, and
editors adopt them without saving. See
workflow_ops_protocol.md.
Rule of thumb: if no register_canary_trigger_type call names your event
type, broadcast it directly. Reach for emit only when a Temporal workflow
has to receive it. tests/nodes/test_context_events.py locks this for Context
by parsing the module's imports (AST, not grep, so the docstring can explain
the reasoning without tripping the assertion).
Context events are emitted at the persistence boundary
save_conversation (services/agent_context/conversation.py) is the one
place every Context writer passes through — the in-process agent loop, the
Temporal LLM activity, and the specialized-provider bridge all persist
through it. So the "conversation advanced" notification is emitted there
rather than at each call site, via a fanout registry in
services/agent_context/listeners.py:
register_conversation_listener(async_fn) # nodes/context/__init__.py
await notify_conversation_saved(workflow_id=..., generation=..., agent_node_id=..., message_count=...)
Same shape as the plugin registries in plugin_system.md, and for the same
reason: the store must never import nodes/. A new writer gets live updates for
free, and no caller carries broadcast code.
Two properties are load-bearing and have tests in
tests/services/agent_context/test_conversation.py:
- After commit. The notification fires only after the durable upsert, so it can never be observed ahead of the state it describes.
- A listener can never fail a save.
notify_conversation_savedswallows and logs at WARNING (visible at the default log level — a silently failing listener is how "the panel never updates live" becomes undiagnosable). Saves run inside the Temporal LLM activity's post-send window, after the provider has been called and billed — a throwing listener there would fail a run over a UI notification.
Clears are deliberately not routed through the store listener: the
clear handler and the Context node's Reset hook dispatch context.updated
themselves (a delete is not a save, and only the caller knows it happened).
Verification
Each Phase-A milestone has a verification command:
| Phase | Check |
|---|---|
| A1 | pytest tests/test_plugin_contract.py::TestStartToCloseTimeoutOverridesAreCommented |
| A2 | python -c "from services.plugin.scaling import RetryPolicy; assert 'NodeUserError' in RetryPolicy().non_retryable_error_types" |
| A3 | python -c "from core.config import Settings; print(Settings().temporal_graceful_shutdown_seconds)" |
| A4 | After Temporal connect: temporal operator search-attribute list | grep -E 'EventType|EventSource|EventWorkflowId|TriggerNodeId|EventTriggerKind|EventReceivedAt|ControlEventTypes' — all 7 lines |
Full test surface landed in Phase A9. Phase B (plugin _events.py
modules), Phase C (Temporal trigger-waiter migration), and Phase D
(admin handlers; the custom DLQ table was dropped — see the status
table and the section below) built on this foundation.
Failure inspection — no separate DLQ table
When a canary listener / polling cycle / cron firing fails after its
RetryPolicy is exhausted, Temporal's own primitives are the ops
inspection surface:
- Visibility list:
client.list_workflows(query="ExecutionStatus='Failed' AND EventWorkflowId='<deployment_workflow_id>'")returns every failed run for a deployment. The same Search Attributes the cancel sweep uses for cleanup (perservices/temporal/search_attributes.py) make this query work. - Failure detail:
client.get_workflow_history(workflow_id, run_id)returns the full Event History, including theActivityTaskFailedevent's error message + stacktrace + each retry attempt timestamp. - Temporal Web UI:
http://localhost:<TEMPORAL_UI_PORT>— the same data, browsable.
This is why Wave 12 explicitly does NOT add a custom event_dlq SQLModel table. Doing so would reinvent the Temporal primitives the rest of the framework was built AROUND, not against. The pre-Temporal services/execution/models.py::DLQEntry for the legacy WorkflowExecutor is a separate concern and stays where it is.
Wave 12 D3 (shipped — see the status table) added thin WS handlers (list_canary_listeners / list_canary_schedules / get_workflow_failure_history) that wrap these Visibility queries for the FE admin surface, so operators can inspect failed runs without leaving the OpenCompany UI.
Wave 13 fixes
Six load-bearing corrections to the Wave 12 canary path. All locked by regression tests.
1. EventType Search Attribute must equal the producer's event.type
Pre-fix the deployment manager registered the legacy snake_case event_type from TriggerConfig (e.g. "chat_message_received") as the EventType SA on every canary listener. But dispatch.emit Visibility-queries by event.type from the CloudEvents envelope (reverse-DNS, e.g. "com.opencompany.chat.message.received"). The strings never matched — the listener started OK, never reacted to incoming events.
Fix:
register_canary_trigger_type(node_type, cloudevent_type)now requires the CloudEvents type as a second arg (canary_registry.py). Re-registering with a divergingcloudevent_typeraisesValueErrorso plugin upgrades that change the envelope shape surface loudly.cloudevent_type_for(node_type)lookup, used byDeploymentManager._start_canary_listenerfor theEventTypeSA value (wasconfig.event_type).WorkflowEvent.task_completedfactory unified to single typecom.opencompany.agent.task.completed(was.succeeded/.failedsplit — broke single-SA listener matching).
Locked by TestCloudEventTypeMatchesSearchAttribute in test_canary_registry.py.
2. Plugin _events.py is canary-only — no more dual-emit
event_waiter.dispatch / broadcaster.send_custom_event calls inside canary-registered plugin _events.py files (chat / webhook / task / telegram / email) had zero consumers in canary-on mode — the deployment manager skips setup_event_trigger when is_canary_trigger_type(...) is True, so no legacy waiter ever registered. Removed.
dispatch.emit(envelope, wire_routing_key=...)is the single delivery path. It signals legacyEventTypeconsumers and running workflow controllers through one Temporal Visibility query, while broadcasting to the frontend on the same wire key. Each controller filters against its durable trigger registry — note this is a match on the CloudEvents type, not on the trigger node's own parameters.The trigger node's own filter is applied in
TriggerListenerWorkflow._spawn_child_run, gated on themachina-trigger-listener-node-filterpatch. That method is the single choke point:WorkflowControlWorkflow._spawn_push_runconstructs aTriggerListenerWorkflowpurely to call it, so both the controller path and the legacy listener path go through one branch. The predicate runs inevaluate_trigger_filter_activityrather than inline, because filter builders live in plugin folders and reach imports the workflow sandbox forbids; the recorded boolean keeps replay deterministic. It receives the CloudEventsdatamember, not the envelope (event_waiter.dispatchunpacks to(event_type, data)and hands filters the inner payload), and fails open — an unknown node type, malformed payload or raising builder all admit the event, because over-firing is recoverable while a dropped trigger event looks like a broken product.Before this,
filter_paramswas carried inlistener_dataand never read, so theEventTypeSearch Attribute was the only narrowing and every event of the right type spawned a run: awebhookTriggerbound to/afired on a POST to/b(all webhooks share one CloudEvents type and are unscoped), ataskTriggerwatching one agent fired on every task, and awhatsappReceivescoped to one group fired on every message. The canvas-Run path applied these filters, so Run and deploy disagreed about what the same node does.listener_data["filter_params"]is the only possible source: the graph snapshot carries no parameters (node.dataholds just the label, andload_persisted_workflow_graph_activityreturns nodes and edges only), so the hot graph re-read cannot supply them. Editing a deployed trigger's filter therefore takes effect on re-deploy — consistent with the trigger node itself, which is pre-executed with the event as its output and never re-reads its parameters at run time.For workflow-control generations, trigger definitions, push-event signals, and polling activities live directly in the generation's
WorkflowControlWorkflow. There are no separate listener workflow runs; only an actual graph execution is started as a child. StandaloneTriggerListenerWorkflow/PollingTriggerWorkflowstarts remain only as a compatibility path for legacy deployments without a controller.google/_events.py:dispatch_gmail_receiveddeleted. Controlled Gmail polling runs as an activity inside the controller;PollingTriggerWorkflowremains the legacy compatibility implementation.whatsapp/_events.pykept the legacy raw frame onwhatsapp_message_receivedfor received messages because the FE message-list handler readsdata.sender(legacy shape) directly — drop blocked on FE migration (D4 follow-up). The duplicate typed-envelope sibling on the same wire key was dropped.twitter/_events.pykeeps both paths — twitter is the only deferred canary plugin (needs PollingTriggerNode subclass refactor first).
3. Trigger node status pulse on every firing
Pre-fix the canary listener broadcast waiting once at deploy and stayed there forever — when an event fired, FE saw downstream nodes light up but no visual signal on the trigger node itself. The legacy services/deployment/triggers.py collector/processor did waiting → idle (Graph executing...) → waiting per event; canary skipped this.
Fix: broadcast_trigger_status_activity (activities.py). The shared trigger-run helper used by the controller and legacy TriggerListenerWorkflow calls it before + after each child spawn:
- Before:
status="idle"withmessage="Graph executing..."data - After:
status="waiting"with the next-event message
The controller's polling path and legacy PollingTriggerWorkflow use the same status lifecycle. Pause overrides the armed waiting projection with an explicit paused state and Resume rearms it.
4. Polling seen set OOM leak
Both polling paths grew seen: Set[str] unboundedly:
as_poll_activityreturnedseen_ids: list(prior_seen | current)every cycle — Temporal payload paid for the union forever._build_poll_coroutinecalledseen.add(msg_id)per emit with no eviction.
At Gmail's ~100 msgs/day / 60s poll, this hit ~36K entries / ~1.4MB just for IDs in a year. Fix: rebase seen = current at end of every cycle in both paths (polling.py). Items the provider no longer reports drop out. Bounded by the provider's natural window size.
Since 1b.18 the activity's seen_ids is current minus the new ids a cycle defers: it walks new ids oldest first (numeric ids as numbers), stops admitting once its events would pass 512,000 bytes of JSON (POLL_RESULT_MAX_BYTES; the first event is always admitted) or once detail fetching has used two thirds of the attempt's deadline, and leaves the rest new for the next cycle; post_emit runs only for emitted events. A single event over the 2 MiB payload limit still fails its cycle.
Semantic note: if a filtered item disappears then reappears (e.g. user marks an email unread again under is:unread), it re-emits. That's the correct semantic for visibility-filtered providers; the old unbounded set suppressed legitimate re-emits.
5. Template resolution against trigger output (canary path)
Pre-fix the canary path's pre-executed trigger output never reached the workflow output store. MachinaWorkflow.run set outputs[node_id] in workflow memory only — never persisted. ParameterResolver._gather_connected_outputs reads from the store, so {{triggerNode.field}} templates in downstream nodes resolved to empty.
The legacy DeploymentManager._execute_from_trigger called _store_output(trigger_node_id, "output_0", trigger_output) explicitly. Canary skipped it.
Fix: new store_node_output_activity (activities.py). MachinaWorkflow.run's pre-executed loop schedules it for every firing trigger (skips non-firing siblings with _trigger_output={"not_triggered": True}). Writes to output_main + output_top + output_0 so any downstream edge handle resolves.
6. DeploymentManager.cancel full status sweep
Pre-fix cancel reset only cron + listener trigger nodes to idle. Downstream agents/tools/actions that were mid-execute when cancel hit stayed in executing forever on FE. Toolbar Start/Stop indicator also stayed at executing=True because no terminal executing=False workflow_status broadcast was emitted.
Fix (manager.py DeploymentManager.cancel):
await broadcaster._clear_stuck_node_statuses(workflow_id, include_waiting=True)— sweeps every node still broadcast asexecuting/waiting. The delegation guard inside_clear_stuck_node_statusesstill protects in-flight fire-and-forget child agents.await broadcaster.update_workflow_status(executing=False, workflow_id=...)— terminal toolbar state. Avoids relying onworkflow_run_ended's counter (can race against in-flightworkflow_run_startedcallers).
Locked by TestCancelSweepsStuckNodeStatuses in test_deployment_canary_listener.py.
References
- Current contract: Workflow control, including admission/drain acknowledgements, compatibility gates and verification.
- RFC:
plugin_authoring_rfc.md - Temporal: Messaging patterns · Python messages · Event Accumulator · Search Attributes · Schedules · Retry Policies
- CloudEvents: v1.0.2 spec