Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
feat(events): Wave 12 Phase A — Temporal-native event framework foundation
trohitgc3dc85ac89b6
diff --git a/docs-internal/event_framework.md b/docs-internal/event_framework.mdnew file mode 100644index 00000000..6e0a8d29--- /dev/null+++ b/docs-internal/event_framework.md@@ -0,0 +1,146 @@+# Event Framework (Wave 12)++Temporal-native event-routing layer for MachinaOs. Implements RFC sections+6.3 (Temporal worker contract) + 6.4 (CloudEvents broadcast contract) from+[plugin_authoring_rfc.md](./plugin_authoring_rfc.md).++This doc is the operator + plugin-author reference. The design rationale ++phase plan lives in `~/.claude/plans/properly-fix-the-tech-dreamy-tarjan.md`.++## What this framework does++Every inbound event (HTTP webhook, Telegram message, Gmail poll result,+task completion, …) becomes a Temporal Signal delivered to whichever+running workflows are waiting on that event type. Routing happens via+the Temporal Visibility API — workflows tag themselves with custom+Search Attributes at start, and the dispatch helper queries+`ListWorkflows(query="EventType='X' AND ExecutionStatus='Running'")`+to find consumers.++Why Temporal: durability, replay safety, server-side dedup+(`WorkflowIDReusePolicy`), and zero custom infrastructure. The framework+adds ~300 LOC on top of Temporal primitives rather than reinventing an+EventBus + event_log + DLQ table.++## Architecture++```+Inbound source (FastAPI process)+ ↓+services/events/dispatch.py:emit(event: WorkflowEvent)+ ├─→ Temporal Visibility query: workflows where EventType=event.type+ ├─→ Signal each matching workflow+ └─→ status_broadcaster.broadcast() — direct in-process WS fan-out+```++Worker is embedded in the FastAPI process (`main.py:211-292`+`TemporalWorkerManager.start()` runs as `asyncio.create_task()`). Activities+and the WebSocket connection pool share memory + event loop, so the fan-out+to FE clients is a direct in-process call — no Redis Streams hop required.++## Search Attributes setup++The framework requires 6 custom Search Attributes on the Temporal+namespace. Registration is **idempotent + automatic on Temporal client+connect** (`services/temporal/client.py:TemporalClientWrapper.connect`):++| Attribute | Type | Used for |+|---|---|---|+| `EventType` | KEYWORD | Visibility query — find consumers by CloudEvents type |+| `EventSource` | KEYWORD | Routing when same type arrives from multiple sources |+| `EventWorkflowId` | KEYWORD | Scope events to a MachinaOs workflow_id |+| `TriggerNodeId` | KEYWORD | Per-trigger event-history queries |+| `EventTriggerKind` | KEYWORD | Coarse classification (webhook / polling / …) |+| `EventReceivedAt` | DATETIME | Time-range queries |++Declarations live in+[`services/temporal/search_attributes.py:EVENT_SEARCH_ATTRIBUTES`](../server/services/temporal/search_attributes.py).+Single source of truth — add an entry to that list and registration+picks it up next connect.++### Manual registration (if needed)++If the auto-registration on connect fails (e.g. permissions on a managed+Temporal Cloud namespace), register manually via the Temporal CLI:++```bash+temporal operator search-attribute create \+ --namespace default \+ --name EventType \+ --type Keyword+# ... repeat for the other 5 attributes+```++### Verification++```bash+temporal operator search-attribute list --namespace default+```++Should show the 6 framework attributes alongside Temporal's built-in+default attributes (`WorkflowType`, `WorkflowId`, `ExecutionStatus`, …).++## Temporal contract for plugin authors++| Class attribute | Purpose | Default |+|---|---|---|+| `start_to_close_timeout` | Per-attempt budget (one activity execution) | Kind-base default: ActionNode=10m, TriggerNode=24h, ToolNode=10m |+| `retry_policy: RetryPolicy` | Backoff + max attempts + non-retryable error types | `DEFAULT_RETRY` — 3 attempts, 1-60s exponential. `NodeUserError` is auto-non-retryable. |+| `heartbeat_timeout` | Max idle between `activity.heartbeat()` calls | 2 minutes |+| `task_queue` | Worker pool routing | `TaskQueue.DEFAULT` |++Override only when the kind-base default doesn't fit; an inline comment+explaining why is required (enforced by+`tests/test_plugin_contract.py::TestStartToCloseTimeoutOverridesAreCommented`).++### Worker graceful shutdown++Workers honor SIGTERM with a configurable grace window+(`Settings.temporal_graceful_shutdown_seconds`, default 30s, override via+env `TEMPORAL_GRACEFUL_SHUTDOWN_SECONDS`). Activities mid-flight finish+or hand back to the server for retry instead of being killed mid-call.++## CloudEvents envelope++Every server→FE broadcast wraps `WorkflowEvent` (CloudEvents v1.0;+`services/events/envelope.py`). Spec-compliant ID, source, type, time,+plus MachinaOs extension attributes (`workflow_id`, `trigger_node_id`,+`correlation_id` — kept in snake_case per the documented internal-naming+rationale at `envelope.py:4-12`).++Wire shape: `{"type": "<legacy_wire_key>", "data": <WorkflowEvent JSON>}`.+The outer `type` is what FE switches on; the inner envelope is what+parses for spec-compliant routing + dataschema lookup.++### Plugin-owned event factories++Plugin-specific events (e.g. `com.machinaos.telegram.message.received`)+live in `nodes/<plugin>/_events.py`. Cross-cutting factories+(`credential`, `oauth_completed`, `agent_progress`, `task_completed`,+`workflow_lifecycle`, `deployment_snapshot`, `team_event`,+`node_parameters_updated`) stay in `services/events/envelope.py`.++See RFC §6.4 for the classification rule + the canonical+`telegram/_events.py` example.++## Verification++Each Phase-A milestone has a verification command:++| Phase | Check |+|---|---|+| A1 | `pytest tests/test_plugin_contract.py::TestStartToCloseTimeoutOverridesAreCommented` |+| A2 | `python -c "from services.plugin.scaling import RetryPolicy; assert 'NodeUserError' in RetryPolicy().non_retryable_error_types"` |+| A3 | `python -c "from core.config import Settings; print(Settings().temporal_graceful_shutdown_seconds)"` |+| A4 | After Temporal connect: `temporal operator search-attribute list \| grep -E 'EventType\|EventSource\|EventWorkflowId\|TriggerNodeId\|EventTriggerKind\|EventReceivedAt'` — all 6 lines |++Full test surface lands in Phase A9. Phase B (plugin `_events.py`+modules) + Phase C (Temporal trigger-waiter migration) + Phase D+(admin handlers + DLQ) build on this foundation.++## References++- Plan: `~/.claude/plans/properly-fix-the-tech-dreamy-tarjan.md`+- RFC: [`plugin_authoring_rfc.md`](./plugin_authoring_rfc.md)+- Temporal: [Search Attributes](https://docs.temporal.io/search-attribute) · [Signals](https://docs.temporal.io/develop/python/message-passing) · [Schedules](https://docs.temporal.io/develop/python/schedules) · [Retry Policies](https://docs.temporal.io/encyclopedia/retry-policies)+- CloudEvents: [v1.0 spec](https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md)diff --git a/server/core/config.py b/server/core/config.pyindex d80d1ca0..50fce7dd 100644--- a/server/core/config.py+++ b/server/core/config.py@@ -69,6 +69,67 @@ class Settings(BaseSettings):temporal_agent_workflow_enabled: bool = Field(default=False, env="TEMPORAL_AGENT_WORKFLOW_ENABLED",)+ # Wave 12 A3: SIGTERM grace window for Temporal workers. Activities+ # mid-flight finish (or hand back to the server for retry) instead of+ # being killed mid-call. Default 30s matches the polling-trigger+ # heartbeat interval — a worker drain completes within one polling+ # cycle. Tune via env when SIGTERM-to-restart latency matters.+ temporal_graceful_shutdown_seconds: int = Field(+ default=30, env="TEMPORAL_GRACEFUL_SHUTDOWN_SECONDS", ge=1,+ )+ # Wave 12 A6: feature flag for the Temporal-native event dispatch+ # path (services/events/dispatch.py:emit). Default off in Phase A+ # so the new dispatch + Signal-based consumer fan-out activates only+ # for opt-in dogfooding before Phase B migrates plugin callsites.+ # Phase A acceptance criterion is enabling this flag on a single+ # canary callsite (e.g. whatsappReceive).+ event_framework_enabled: bool = Field(+ default=False, env="EVENT_FRAMEWORK_ENABLED",+ )++ # Persistence backend for the supervised Temporal cluster.+ # 'sqlite' (dev default) — single ServiceSpec runs `temporal api`.+ # 'postgres' (prod) — pgserver + temporal-server binary with+ # YAML config managed by services/temporal/.+ temporal_backend: Literal["sqlite", "postgres"] = Field(+ default="sqlite", env="TEMPORAL_BACKEND",+ )+ # Internal Temporal service gRPC ports — exposed for the YAML config+ # renderer and the runtime's readiness probe. Defaults match the+ # ports Temporal documents at https://docs.temporal.io/references/configuration+ temporal_frontend_grpc_port: int = Field(+ default=7233, env="TEMPORAL_FRONTEND_GRPC_PORT", ge=1024, le=65535,+ )+ temporal_matching_grpc_port: int = Field(+ default=7235, env="TEMPORAL_MATCHING_GRPC_PORT", ge=1024, le=65535,+ )+ temporal_history_grpc_port: int = Field(+ default=7234, env="TEMPORAL_HISTORY_GRPC_PORT", ge=1024, le=65535,+ )+ temporal_worker_grpc_port: int = Field(+ default=7239, env="TEMPORAL_WORKER_GRPC_PORT", ge=1024, le=65535,+ )+ temporal_bind_local_only: bool = Field(+ default=True, env="TEMPORAL_BIND_LOCAL_ONLY",+ )+ temporal_num_history_shards: int = Field(+ default=4, env="TEMPORAL_NUM_HISTORY_SHARDS", ge=1, le=4096,+ )+ temporal_default_max_conns: int = Field(+ default=20, env="TEMPORAL_DEFAULT_MAX_CONNS", ge=1, le=500,+ )+ temporal_visibility_max_conns: int = Field(+ default=4, env="TEMPORAL_VISIBILITY_MAX_CONNS", ge=1, le=500,+ )+ temporal_max_conn_lifetime: str = Field(+ default="1h", env="TEMPORAL_MAX_CONN_LIFETIME",+ )+ temporal_binary_version: str = Field(+ default="1.31.0", env="TEMPORAL_BINARY_VERSION",+ )+ temporal_postgres_dsn: Optional[str] = Field(+ default=None, env="TEMPORAL_POSTGRES_DSN",+ )# API Keys (all optional, injected at runtime)google_maps_api_key: Optional[str] = Field(default=None, env="GOOGLE_MAPS_API_KEY")diff --git a/server/services/events/dispatch.py b/server/services/events/dispatch.pynew file mode 100644index 00000000..2a959025--- /dev/null+++ b/server/services/events/dispatch.py@@ -0,0 +1,197 @@+"""Wave 12 A6: Temporal-native event dispatch.++One public function — :func:`emit` — takes a :class:`WorkflowEvent` and+routes it to:++1. **Running consumer workflows** via Temporal Visibility query ++ ``Client.signal_workflow``. Workflows tag themselves with the+ ``EventType`` Search Attribute when they're started; this dispatch+ finds them via ``ListWorkflows(query="EventType='X' AND+ ExecutionStatus='Running'")`` and signals each.++2. **In-process FastAPI WebSocket clients** via direct+ ``status_broadcaster.broadcast()`` call. Worker + WS pool share+ memory + event loop, so no IPC hop is needed (audit confirmed —+ ``main.py:211-292``).++No EventDispatchWorkflow, no Redis Streams, no DLQ table. Temporal's+Visibility + Signal API + Event History provide the durability primitives+this function depends on; everything else is a thin pass-through.++Behind ``Settings.event_framework_enabled`` (default off in Phase A).+When disabled, :func:`emit` is a no-op pass-through that returns the+envelope unchanged.+"""++from __future__ import annotations++import asyncio+from typing import Optional++from core.config import Settings+from core.logging import get_logger+from services.events.envelope import WorkflowEvent++logger = get_logger(__name__)+++# Wire-routing key the FastAPI WebSocket layer switches on. Plugin emits+# its own wire key today (``credential_catalogue_updated`` /+# ``agent_progress`` / ``plugin_connection_status`` / etc.). This+# default is the generic CloudEvents-envelope channel used by events+# that don't carry a plugin-specific wire key.+_DEFAULT_WIRE_ROUTING_KEY = "cloudevent"++# Visibility query template — produces a SQL-like List Filter string.+# ``ExecutionStatus`` enum value 'Running' matches Temporal's status+# encoding; see https://docs.temporal.io/list-filter.+_RUNNING_CONSUMERS_QUERY = "EventType='{event_type}' AND ExecutionStatus='Running'"++# Signal handler name on consumer workflows. Plain identifier rather+# than CloudEvents type because Temporal Signal names are Python+# identifiers (cannot contain dots).+_SIGNAL_NAME = "on_event"+++async def emit(+ event: WorkflowEvent,+ *,+ wire_routing_key: Optional[str] = None,+) -> WorkflowEvent:+ """Route ``event`` to running consumer workflows + in-process WS clients.++ Args:+ event: The :class:`WorkflowEvent` to dispatch. CloudEvents+ envelope with a populated ``type`` (the Visibility filter+ keys off this).+ wire_routing_key: Outer WebSocket routing key (the ``type``+ field on the WS frame). Defaults to the generic+ ``cloudevent`` channel; plugin emitters override per their+ existing wire key (e.g. ``"telegram_message_received"``).++ Returns:+ The envelope unchanged — callers may chain.++ Behaviour when ``Settings.event_framework_enabled=False`` (Phase A+ default): pass-through no-op. Logged at DEBUG so opt-in dogfooding+ is observable without flipping the flag globally.+ """+ if not Settings().event_framework_enabled:+ logger.debug(+ f"event-framework disabled — emit no-op for event.id={event.id} "+ f"type={event.type}",+ )+ return event++ # Concurrent fan-out: Visibility query + Signal each consumer, AND+ # broadcast to in-process WS clients. asyncio.gather lets the+ # broadcast happen even if the Temporal Visibility query fails (and+ # vice versa).+ await asyncio.gather(+ _signal_running_consumers(event),+ _broadcast_in_process(event, wire_routing_key or _DEFAULT_WIRE_ROUTING_KEY),+ return_exceptions=False,+ )+ return event+++async def _signal_running_consumers(event: WorkflowEvent) -> None:+ """Find running workflows tagged with ``EventType=event.type`` and+ signal each with ``on_event``.++ Fail-soft: if the Temporal client is unavailable or the Visibility+ query errors, log a warning and continue. The in-process broadcast+ path is independent and still delivers to WS clients.+ """+ try:+ from core.container import container++ wrapper = container.temporal_client()+ except Exception as exc: # noqa: BLE001+ logger.warning(f"emit: container.temporal_client unavailable: {exc}")+ return++ if wrapper is None or wrapper.client is None:+ logger.debug(+ f"emit: Temporal not connected — skipping consumer fan-out for "+ f"event.id={event.id}",+ )+ return++ client = wrapper.client+ query = _RUNNING_CONSUMERS_QUERY.format(event_type=event.type)++ try:+ consumers = []+ async for wf in client.list_workflows(query=query):+ consumers.append(wf)+ except Exception as exc: # noqa: BLE001+ logger.warning(+ f"emit: Visibility query failed (query={query!r}): {exc}"+ )+ return++ if not consumers:+ logger.debug(+ f"emit: 0 consumers for event.type={event.type!r} "+ f"(event.id={event.id})",+ )+ return++ # Signal each consumer concurrently. Per-target failures don't+ # block other consumers (return_exceptions=True surfaces failures+ # as exceptions in the result list).+ signal_results = await asyncio.gather(+ *[+ _signal_one(client, wf.id, event)+ for wf in consumers+ ],+ return_exceptions=True,+ )++ delivered = sum(1 for r in signal_results if not isinstance(r, Exception))+ failed = len(signal_results) - delivered+ logger.info(+ f"emit: signalled {delivered}/{len(consumers)} consumers for "+ f"event.type={event.type!r} (failed={failed}, event.id={event.id})",+ )+++async def _signal_one(client, workflow_id: str, event: WorkflowEvent) -> None:+ """Send the ``on_event`` signal to one workflow.++ Kept as its own coroutine so :func:`_signal_running_consumers` can+ ``asyncio.gather`` independently per-consumer.+ """+ handle = client.get_workflow_handle(workflow_id)+ await handle.signal(_SIGNAL_NAME, event.model_dump(mode="json"))+++async def _broadcast_in_process(event: WorkflowEvent, wire_routing_key: str) -> None:+ """Direct in-process WS fan-out via the status broadcaster.++ Same asyncio event loop as the FastAPI handlers — see ``main.py:+ 211-292`` (TemporalWorkerManager starts as ``asyncio.create_task``).+ Activity → broadcaster is a direct method call against in-memory+ ``Set[WebSocket]``; no IPC.+ """+ try:+ from services.status_broadcaster import get_status_broadcaster++ broadcaster = get_status_broadcaster()+ except Exception as exc: # noqa: BLE001+ logger.warning(f"emit: status_broadcaster unavailable: {exc}")+ return++ try:+ await broadcaster.broadcast({+ "type": wire_routing_key,+ "data": event.model_dump(mode="json"),+ })+ except Exception as exc: # noqa: BLE001+ logger.warning(+ f"emit: WS broadcast failed (wire_key={wire_routing_key!r}): {exc}"+ )+++__all__ = ["emit"]diff --git a/server/services/plugin/scaling.py b/server/services/plugin/scaling.pyindex 759df692..3032a78b 100644--- a/server/services/plugin/scaling.py+++ b/server/services/plugin/scaling.py@@ -51,6 +51,13 @@ class RetryPolicy:"ValidationError","PermissionDeniedError","InvalidParametersError",+ # Wave 12 A2: NodeUserError is the canonical "user-correctable+ # failure" exception (missing input, missing credential, API+ # rejected the payload). Retrying it just burns attempts since+ # the underlying input won't fix itself. Plugins that raise+ # NodeUserError fail fast; the framework surfaces the message+ # to the user / agent loop without traceback noise.+ "NodeUserError",))def to_temporal(self):diff --git a/server/services/temporal/activities.py b/server/services/temporal/activities.pyindex 17c4987a..34f1f0fc 100644--- a/server/services/temporal/activities.py+++ b/server/services/temporal/activities.py@@ -316,3 +316,59 @@ async def create_shared_session(pool_size: int = 100) -> aiohttp.ClientSession:)logger.info(f"Created shared session with pool_size={pool_size}")return session+++# ---------------------------------------------------------------------------+# Wave 12 A8: emit_event_activity+#+# Thin Temporal-activity wrapper around ``services.events.dispatch.emit``.+# Used by ``PollingTriggerWorkflow`` (Phase C2) when a poll cycle produces+# new items — the workflow can't directly do I/O (the dispatch helper+# uses asyncio + a network round-trip to Temporal Visibility), so it+# offloads to this activity.+#+# Activity scheduling timeouts (``start_to_close_timeout``, retry policy,+# heartbeat) are set by the CALLING workflow at scheduling time —+# ``workflow.execute_activity(emit_event_activity, ...,+# start_to_close_timeout=timedelta(seconds=2))``. Defaults are not+# hardcoded here.++async def emit_event_activity(event_payload: Dict[str, Any]) -> Dict[str, Any]:+ """Re-enter :func:`services.events.dispatch.emit` from a workflow context.++ Args:+ event_payload: The serialised :class:`WorkflowEvent` (output of+ ``event.model_dump(mode="json")``). The activity reconstructs+ the envelope and routes it through the dispatch helper.++ Returns:+ Status dict:+ {+ "delivered": True | False, # whether dispatch completed+ "event_id": str,+ "event_type": str,+ }+ """+ from services.events.envelope import WorkflowEvent+ from services.events.dispatch import emit++ try:+ event = WorkflowEvent.model_validate(event_payload)+ except Exception as exc: # noqa: BLE001+ activity.logger.warning(+ f"emit_event_activity: envelope rejected: {exc}"+ )+ return {+ "delivered": False,+ "event_id": event_payload.get("id", "?"),+ "event_type": event_payload.get("type", "?"),+ "error": f"{type(exc).__name__}: {exc}",+ }++ await emit(event)+ return {+ "delivered": True,+ "event_id": event.id,+ "event_type": event.type,+ }diff --git a/server/services/temporal/client.py b/server/services/temporal/client.pyindex d3e96a46..d3a71b29 100644--- a/server/services/temporal/client.py+++ b/server/services/temporal/client.py@@ -71,6 +71,20 @@ class TemporalClientWrapper:self._client = clientprint(f"[Temporal] Connected to {self.server_address}", flush=True)logger.info("Connected to Temporal server")+ # Wave 12 A4: idempotently register the event-framework+ # Search Attributes. Failure here is non-fatal — the+ # framework still works without them, dispatch.emit just+ # falls back to broadcast-only routing instead of+ # signalling consumers.+ try:+ from services.temporal.search_attributes import (+ register_search_attributes,+ )+ await register_search_attributes(self._client, self.namespace)+ except Exception as sa_exc: # noqa: BLE001 — non-fatal+ logger.warning(+ f"Search-attribute registration failed (non-fatal): {sa_exc}"+ )return self._clientexcept Exception as e:print(f"[Temporal] Connection attempt {attempt}/{retries} failed: {e}", flush=True)diff --git a/server/services/temporal/search_attributes.py b/server/services/temporal/search_attributes.pynew file mode 100644index 00000000..cdc8fc36--- /dev/null+++ b/server/services/temporal/search_attributes.py@@ -0,0 +1,181 @@+"""Wave 12 A4: Temporal custom Search Attributes registration.++Defines the 6 Search Attributes the event framework needs for the+Visibility API to route inbound events to running consumer workflows+(see ``services/events/dispatch.py:emit``). Registration is idempotent —+Temporal Server's ``add_search_attributes`` rejects duplicates with an+``AlreadyExists`` error which we treat as success.++Attribute declarations live in one place (:data:`EVENT_SEARCH_ATTRIBUTES`)+as a structured list so callers iterate rather than restate. Adding a+new attribute = add an entry to that list; everything else (registration,+docs, error handling) flows from it.++References:+- https://docs.temporal.io/search-attribute+- https://python.temporal.io/temporalio.client.html#Client.operator_service+"""++from __future__ import annotations++from dataclasses import dataclass+from typing import Sequence++from temporalio.api.enums.v1 import IndexedValueType+from temporalio.api.operatorservice.v1 import (+ AddSearchAttributesRequest,+ ListSearchAttributesRequest,+)+from temporalio.client import Client++from core.logging import get_logger++logger = get_logger(__name__)+++@dataclass(frozen=True)+class SearchAttributeSpec:+ """Declarative spec for one Temporal Search Attribute."""++ name: str+ indexed_type: IndexedValueType+ description: str+++# Single source of truth for the 6 event-framework Search Attributes.+# Anything that names or filters by these attributes (the dispatch+# helper, the Visibility queries in admin handlers, the test suite)+# imports from this list rather than restating the strings.+EVENT_SEARCH_ATTRIBUTES: Sequence[SearchAttributeSpec] = (+ SearchAttributeSpec(+ name="EventType",+ indexed_type=IndexedValueType.INDEXED_VALUE_TYPE_KEYWORD,+ description=(+ "CloudEvents type the workflow is consuming "+ "(e.g. com.machinaos.whatsapp.message.received). Used by "+ "dispatch.emit to find running consumers via Visibility "+ "queries: ListWorkflows(query=f\"EventType='{event.type}' "+ "AND ExecutionStatus='Running'\")."+ ),+ ),+ SearchAttributeSpec(+ name="EventSource",+ indexed_type=IndexedValueType.INDEXED_VALUE_TYPE_KEYWORD,+ description=(+ "CloudEvents source URI fragment "+ "(e.g. machinaos://nodes/whatsapp). Used for routing when "+ "the same type can arrive from multiple sources."+ ),+ ),+ SearchAttributeSpec(+ name="EventWorkflowId",+ indexed_type=IndexedValueType.INDEXED_VALUE_TYPE_KEYWORD,+ description=(+ "MachinaOs workflow_id the event belongs to (distinct from "+ "the Temporal Workflow ID which addresses the workflow run). "+ "Used to scope event delivery to a specific deployment."+ ),+ ),+ SearchAttributeSpec(+ name="TriggerNodeId",+ indexed_type=IndexedValueType.INDEXED_VALUE_TYPE_KEYWORD,+ description=(+ "Trigger node id that emitted or will consume the event. "+ "Used for per-trigger event-history queries and the "+ "list_event_waiters admin handler."+ ),+ ),+ SearchAttributeSpec(+ name="EventTriggerKind",+ indexed_type=IndexedValueType.INDEXED_VALUE_TYPE_KEYWORD,+ description=(+ "High-level event source classification (webhook / polling / "+ "whatsapp / telegram / ...). Coarser-grained than EventType; "+ "useful for ops dashboards."+ ),+ ),+ SearchAttributeSpec(+ name="EventReceivedAt",+ indexed_type=IndexedValueType.INDEXED_VALUE_TYPE_DATETIME,+ description=(+ "When the event entered the framework. Supports time-range "+ "visibility queries (e.g. 'all events received in the last "+ "hour')."+ ),+ ),+)+++def attribute_names() -> Sequence[str]:+ """Return the configured attribute names. Helper for tests + admin code."""+ return tuple(spec.name for spec in EVENT_SEARCH_ATTRIBUTES)+++async def register_search_attributes(client: Client, namespace: str) -> dict:+ """Idempotently register every :data:`EVENT_SEARCH_ATTRIBUTES` entry+ on the given Temporal namespace.++ Returns a status dict keyed by attribute name:+ {name: "registered" | "already_exists" | "error: ..."}++ Safe to call multiple times — already-present attributes are+ detected via a pre-flight ``ListSearchAttributes`` call and skipped.+ Attributes that fail to register surface in the result dict with+ the error string so the caller can decide whether to fail-fast.+ """+ operator = client.service_client.operator_service++ # Pre-flight: which attributes already exist on this namespace?+ try:+ existing_resp = await operator.list_search_attributes(+ ListSearchAttributesRequest(namespace=namespace),+ )+ existing_names = set(existing_resp.custom_attributes.keys())+ except Exception as exc: # noqa: BLE001 — surface every failure+ logger.warning(+ f"Could not list existing Search Attributes on namespace={namespace!r}: {exc}. "+ "Attempting registration anyway; duplicates will be reported as errors."+ )+ existing_names = set()++ status: dict[str, str] = {}+ to_register: dict[str, IndexedValueType] = {+ spec.name: spec.indexed_type+ for spec in EVENT_SEARCH_ATTRIBUTES+ if spec.name not in existing_names+ }+ for name in existing_names & {s.name for s in EVENT_SEARCH_ATTRIBUTES}:+ status[name] = "already_exists"++ if not to_register:+ logger.info(+ f"All {len(EVENT_SEARCH_ATTRIBUTES)} event-framework Search Attributes "+ f"already registered on namespace={namespace!r}",+ )+ return status++ try:+ await operator.add_search_attributes(+ AddSearchAttributesRequest(+ namespace=namespace,+ search_attributes=to_register,+ ),+ )+ except Exception as exc: # noqa: BLE001+ # Whole batch failed — mark each requested attribute with the error.+ err = f"error: {type(exc).__name__}: {exc}"+ for name in to_register:+ status[name] = err+ logger.error(+ f"Failed to register Search Attributes on namespace={namespace!r}: {exc}",+ )+ return status++ for name in to_register:+ status[name] = "registered"++ logger.info(+ f"Registered {len(to_register)} event-framework Search Attributes on "+ f"namespace={namespace!r}: {list(to_register.keys())}",+ )+ return statusdiff --git a/server/services/temporal/worker.py b/server/services/temporal/worker.pyindex 29ad565e..612a3155 100644--- a/server/services/temporal/worker.py+++ b/server/services/temporal/worker.py@@ -16,6 +16,7 @@ References:"""import asyncio+from datetime import timedeltafrom typing import Optionalimport aiohttp@@ -35,6 +36,18 @@ logger = get_logger(__name__)tracer = trace.get_tracer(__name__)+def _graceful_shutdown_timeout() -> timedelta:+ """Wave 12 A3: SIGTERM grace window for Temporal workers.++ Read from ``Settings.temporal_graceful_shutdown_seconds`` at the+ moment each Worker is instantiated (rather than module load) so+ test fixtures + env reloads pick up overrides.+ """+ from core.config import Settings++ return timedelta(seconds=Settings().temporal_graceful_shutdown_seconds)++def create_runtime() -> Runtime:"""Create a Temporal runtime with worker heartbeating disabled.@@ -136,6 +149,7 @@ class TemporalWorkerManager:# Allow concurrent activity execution for parallel branchesmax_concurrent_activities=self.pool_size,max_concurrent_workflow_tasks=10,+ graceful_shutdown_timeout=_graceful_shutdown_timeout(),)logger.info("Registered Temporal activities",@@ -273,6 +287,7 @@ class TemporalWorkerPool:activities=activities,max_concurrent_activities=concurrency,max_concurrent_workflow_tasks=10,+ graceful_shutdown_timeout=_graceful_shutdown_timeout(),)task = asyncio.create_task(worker.run(), name=f"worker-{queue}")self._workers.append(worker)@@ -356,6 +371,7 @@ async def run_standalone_worker(activities=[activities.execute_node_activity], # Pass bound methodmax_concurrent_activities=pool_size,max_concurrent_workflow_tasks=10,+ graceful_shutdown_timeout=_graceful_shutdown_timeout(),)logger.info("Worker running. Press Ctrl+C to stop.")@@ -394,6 +410,7 @@ async def create_worker(activities=[activities.execute_node_activity], # Pass bound methodmax_concurrent_activities=100,max_concurrent_workflow_tasks=10,+ graceful_shutdown_timeout=_GRACEFUL_SHUTDOWN_TIMEOUT,)diff --git a/server/services/temporal/workflow.py b/server/services/temporal/workflow.pyindex 2ef3d673..4d1e594e 100644--- a/server/services/temporal/workflow.py+++ b/server/services/temporal/workflow.py@@ -79,8 +79,75 @@ class MachinaWorkflow:- Per-node retry policies- Config node filtering (tools, memory, services)- Multi-tenant support via tenant_id in context+ - Wave 12 A7: event-framework signal handler — receives Temporal+ Signals from ``services/events/dispatch.py:emit`` and parks+ pending events in ``_matched_events`` for trigger nodes to+ consume via ``workflow.wait_condition(_has_event_matching, ...)``.+ Per-run dedup via ``_seen_event_ids`` (reconstructed+ deterministically from Event History on replay)."""+ def __init__(self) -> None:+ # Wave 12 A7: per-run event-framework state. Populated by the+ # ``on_event`` signal handler; consumed by Phase C1's+ # ``wait_condition`` predicate in the trigger-waiter rewrite.+ # Today these stay empty unless the event-framework feature+ # flag is on AND a producer signals this workflow.+ self._seen_event_ids: Set[str] = set()+ self._matched_events: List[Dict[str, Any]] = []++ @workflow.signal+ async def on_event(self, event_payload: Dict[str, Any]) -> None:+ """Wave 12 A7: receive an event from ``services.events.dispatch.emit``.++ Drops duplicates by ``event.id`` (per-run dedup — the set is+ reconstructed deterministically from Event History on replay).+ Matching events are queued for the workflow body to consume via+ :meth:`_has_event_matching` in a ``wait_condition`` (wired in+ Phase C1).+ """+ event_id = event_payload.get("id")+ if not event_id:+ workflow.logger.warning(+ "on_event: skipping malformed envelope without 'id'"+ )+ return+ if event_id in self._seen_event_ids:+ workflow.logger.debug(+ f"on_event: dedup hit for event.id={event_id}"+ )+ return+ self._seen_event_ids.add(event_id)+ self._matched_events.append(event_payload)+ workflow.logger.info(+ f"on_event: queued event.id={event_id} "+ f"type={event_payload.get('type', '?')} "+ f"(matched={len(self._matched_events)})"+ )++ def _has_event_matching(self, predicate=None) -> bool:+ """Wave 12 A7: ``wait_condition`` predicate for trigger nodes.++ Returns True when at least one queued event matches the optional+ ``predicate(event_payload) -> bool``. Defaults to "any event+ queued" — most trigger nodes filter further by inspecting the+ envelope after pop.++ Used by Phase C1's trigger-waiter rewrite to replace the+ in-process ``asyncio.Future`` waiter:++ await workflow.wait_condition(+ lambda: self._has_event_matching(my_filter),+ timeout=...,+ )+ event = self._pop_matching_event(my_filter)+ """+ if not self._matched_events:+ return False+ if predicate is None:+ return True+ return any(predicate(e) for e in self._matched_events)+@workflow.runasync def run(self, workflow_data: Dict[str, Any]) -> Dict[str, Any]:"""Execute workflow by orchestrating node activities.diff --git a/server/tests/test_event_framework_phase_a.py b/server/tests/test_event_framework_phase_a.pynew file mode 100644index 00000000..37960648--- /dev/null+++ b/server/tests/test_event_framework_phase_a.py@@ -0,0 +1,307 @@+"""Wave 12 Phase A: smoke tests for the event-framework foundation.++Covers (with no live Temporal cluster required):++A1: ``TestStartToCloseTimeoutOverridesAreCommented`` — lives in+ ``test_plugin_contract.py``; not duplicated here.++A2: ``NodeUserError`` is in :class:`RetryPolicy`'s default+ ``non_retryable_error_types``.++A3: ``Settings.temporal_graceful_shutdown_seconds`` is exposed,+ configurable, and consumed by :func:`worker._graceful_shutdown_timeout`.++A4: :data:`EVENT_SEARCH_ATTRIBUTES` is a non-empty structured spec list;+ every entry has a name + indexed_type + description; the+ :func:`attribute_names` helper agrees with the structure.++A6: :func:`services.events.dispatch.emit` is a no-op pass-through when+ ``Settings.event_framework_enabled=False`` (Phase-A default).++A7: :class:`MachinaWorkflow` initializes its event state in ``__init__``;+ the ``on_event`` signal handler dedups by ``event.id`` and queues+ matching events; :meth:`_has_event_matching` implements the+ ``wait_condition`` predicate contract.++A8: :func:`emit_event_activity` validates the envelope payload and+ forwards to :func:`dispatch.emit`.+"""++from __future__ import annotations++import sys+import types+from datetime import timedelta+from unittest.mock import MagicMock++import pytest++# Stub the `machina` package (lives at project root, outside server/) so+# `services.temporal.__init__` → `_runtime.py` → `from machina.tcp import+# probe_tcp_port` doesn't crash collection. Conftest stubs core.* but+# not machina.*.+if "machina" not in sys.modules:+ _machina = types.ModuleType("machina")+ _machina.__path__ = []+ sys.modules["machina"] = _machina+ _machina_tcp = types.ModuleType("machina.tcp")+ _machina_tcp.probe_tcp_port = MagicMock(return_value=False)+ sys.modules["machina.tcp"] = _machina_tcp++from core.config import Settings+from services.events.envelope import WorkflowEvent+from services.plugin.scaling import RetryPolicy+++class TestA2NodeUserErrorNonRetryable:+ """A2: ``NodeUserError`` ships in the default ``non_retryable_error_types``+ so a single ``cls.retry_policy = DEFAULT_RETRY`` picks it up."""++ def test_node_user_error_in_default_non_retryable(self):+ policy = RetryPolicy()+ assert "NodeUserError" in policy.non_retryable_error_types, (+ "RetryPolicy.non_retryable_error_types default must contain "+ "'NodeUserError' so plugin-raised user errors fail fast in "+ "Temporal activities."+ )++ def test_to_temporal_preserves_non_retryable_list(self):+ policy = RetryPolicy()+ # Import-on-call to keep the test independent of Temporal SDK+ # availability at module-import time.+ try:+ temporal_policy = policy.to_temporal()+ except ImportError: # pragma: no cover+ pytest.skip("temporalio SDK not installed in test env")+ assert "NodeUserError" in temporal_policy.non_retryable_error_types+++class TestA3GracefulShutdownTimeout:+ """A3: SIGTERM grace window is config-driven, not hardcoded.++ ``conftest.py`` mocks ``core.config.Settings`` as ``MagicMock`` so+ handler tests don't pay Pydantic startup cost. We patch a real+ ``Settings``-like stub inside this class to verify the contract.+ """++ def test_worker_helper_returns_timedelta(self, monkeypatch):+ from services.temporal import worker as worker_mod++ class _FakeSettings:+ temporal_graceful_shutdown_seconds = 17++ # Patch the lazy import inside worker._graceful_shutdown_timeout.+ # It does `from core.config import Settings`; patch that module+ # entry to return our fake.+ import core.config++ monkeypatch.setattr(core.config, "Settings", lambda: _FakeSettings())++ td = worker_mod._graceful_shutdown_timeout()+ assert isinstance(td, timedelta)+ assert td.total_seconds() == 17+++class TestA4SearchAttributesSpec:+ """A4: Search Attributes are declared structurally, NOT scattered+ as literals through the dispatch + admin code."""++ def test_spec_is_non_empty(self):+ from services.temporal.search_attributes import EVENT_SEARCH_ATTRIBUTES++ assert len(EVENT_SEARCH_ATTRIBUTES) >= 6++ def test_each_spec_complete(self):+ from services.temporal.search_attributes import EVENT_SEARCH_ATTRIBUTES++ for spec in EVENT_SEARCH_ATTRIBUTES:+ assert spec.name, "every search-attribute spec needs a name"+ assert spec.indexed_type, f"{spec.name} missing indexed_type"+ assert spec.description, f"{spec.name} missing description"++ def test_attribute_names_helper_matches_spec(self):+ from services.temporal.search_attributes import (+ EVENT_SEARCH_ATTRIBUTES,+ attribute_names,+ )++ spec_names = tuple(s.name for s in EVENT_SEARCH_ATTRIBUTES)+ assert attribute_names() == spec_names++ def test_required_attributes_present(self):+ """The 6 attributes the plan promises."""+ from services.temporal.search_attributes import attribute_names++ required = {+ "EventType",+ "EventSource",+ "EventWorkflowId",+ "TriggerNodeId",+ "EventTriggerKind",+ "EventReceivedAt",+ }+ assert required.issubset(set(attribute_names()))+++class TestA6DispatchEmitFeatureFlag:+ """A6: ``emit`` is a no-op pass-through when the flag is off."""++ @pytest.mark.asyncio+ async def test_disabled_emit_is_passthrough(self, monkeypatch):+ # Pydantic Settings reads from env; force the flag off for this test.+ monkeypatch.setenv("EVENT_FRAMEWORK_ENABLED", "false")++ from services.events.dispatch import emit++ event = WorkflowEvent(+ source="machinaos://services/test",+ type="com.machinaos.test.disabled",+ )+ returned = await emit(event)+ assert returned is event++ @pytest.mark.asyncio+ async def test_enabled_emit_calls_both_paths(self, monkeypatch):+ """When the flag is on, both signal-fanout and in-process+ broadcast are invoked. Both are patched to record-only."""+ monkeypatch.setenv("EVENT_FRAMEWORK_ENABLED", "true")+ # Force Settings() to re-read by clearing the cached singleton+ # if any; here Settings() builds fresh per call so monkeypatch+ # of env is enough.++ signal_calls = []+ broadcast_calls = []++ async def fake_signal(event):+ signal_calls.append(event)++ async def fake_broadcast(event, wire_key):+ broadcast_calls.append((event, wire_key))++ from services.events import dispatch++ monkeypatch.setattr(dispatch, "_signal_running_consumers", fake_signal)+ monkeypatch.setattr(dispatch, "_broadcast_in_process", fake_broadcast)++ event = WorkflowEvent(+ source="machinaos://services/test",+ type="com.machinaos.test.enabled",+ )+ await dispatch.emit(event, wire_routing_key="custom_wire_key")++ assert len(signal_calls) == 1+ assert signal_calls[0] is event+ assert len(broadcast_calls) == 1+ assert broadcast_calls[0] == (event, "custom_wire_key")+++class TestA7MachinaWorkflowSignalHandler:+ """A7: signal handler dedups + queues; predicate matches.++ ``workflow.logger`` requires a Temporal workflow event loop, which+ we don't spin up in these unit tests. Patch ``temporalio.workflow.logger``+ to a no-op so the handler can run synchronously.+ """++ @pytest.fixture(autouse=True)+ def _patch_workflow_logger(self, monkeypatch):+ from unittest.mock import MagicMock+ from temporalio import workflow as temporal_workflow++ monkeypatch.setattr(temporal_workflow, "logger", MagicMock())++ def test_init_sets_empty_state(self):+ from services.temporal.workflow import MachinaWorkflow++ wf = MachinaWorkflow()+ assert wf._seen_event_ids == set()+ assert wf._matched_events == []++ @pytest.mark.asyncio+ async def test_on_event_dedups_by_id(self):+ from services.temporal.workflow import MachinaWorkflow++ wf = MachinaWorkflow()+ payload = {"id": "evt-1", "type": "com.machinaos.test.x"}++ await wf.on_event(payload)+ await wf.on_event(payload) # duplicate++ assert wf._seen_event_ids == {"evt-1"}+ assert len(wf._matched_events) == 1++ @pytest.mark.asyncio+ async def test_on_event_drops_malformed_envelope(self):+ from services.temporal.workflow import MachinaWorkflow++ wf = MachinaWorkflow()+ await wf.on_event({"type": "com.machinaos.test.x"}) # no 'id'+ assert wf._matched_events == []++ def test_has_event_matching_empty_state(self):+ from services.temporal.workflow import MachinaWorkflow++ wf = MachinaWorkflow()+ assert wf._has_event_matching() is False++ def test_has_event_matching_predicate(self):+ from services.temporal.workflow import MachinaWorkflow++ wf = MachinaWorkflow()+ wf._matched_events.append(+ {"id": "e1", "type": "com.machinaos.whatsapp.message.received"}+ )++ # No-predicate: any queued event matches.+ assert wf._has_event_matching() is True++ # Truthy predicate matches.+ assert wf._has_event_matching(+ lambda e: e["type"].endswith("message.received")+ ) is True++ # Falsy predicate doesn't match.+ assert wf._has_event_matching(+ lambda e: e["type"].endswith(".nope")+ ) is False+++class TestA8EmitEventActivity:+ """A8: activity wrapper validates the envelope and forwards to emit."""++ @pytest.mark.asyncio+ async def test_valid_payload_returns_delivered_true(self, monkeypatch):+ captured = []++ async def fake_emit(event, **kwargs):+ captured.append(event)+ return event++ from services.events import dispatch as dispatch_mod++ monkeypatch.setattr(dispatch_mod, "emit", fake_emit)++ from services.temporal.activities import emit_event_activity++ payload = WorkflowEvent(+ source="machinaos://services/test",+ type="com.machinaos.test.a8",+ ).model_dump(mode="json")++ # Activities decorated with @activity.defn are still callable+ # outside a worker — they're plain async functions with metadata.+ result = await emit_event_activity(payload)++ assert result["delivered"] is True+ assert result["event_type"] == "com.machinaos.test.a8"+ assert len(captured) == 1++ @pytest.mark.asyncio+ async def test_malformed_payload_returns_delivered_false(self):+ from services.temporal.activities import emit_event_activity++ # Missing required `source` and `type` fields → validation fails.+ result = await emit_event_activity({"id": "evt-malformed"})++ assert result["delivered"] is False+ assert "error" in resultdiff --git a/server/tests/test_plugin_contract.py b/server/tests/test_plugin_contract.pyindex a9e2f8c5..e0af6ef3 100644--- a/server/tests/test_plugin_contract.py+++ b/server/tests/test_plugin_contract.py@@ -130,6 +130,70 @@ class TestScalingKnobs:)+class TestStartToCloseTimeoutOverridesAreCommented:+ """Wave 12 — A1: any plugin that overrides ``start_to_close_timeout``+ away from its kind-base default must carry an inline comment in the+ class body explaining why.++ Today every plugin inherits the kind-base default (``ActionNode``=10m,+ ``TriggerNode``=24h, ``ToolNode``=10m). The kind-defaults are the+ declared intent at the kind-base level; per-plugin overrides are the+ exception, not the norm. When an override does ship, the comment+ forces the author to justify the deviation (and gives the next reader+ the reason without git-blame archaeology).++ Why introspect with ``inspect.getsource`` rather than just check+ equality against the kind-default: equality alone can't catch the+ case where a future plugin author copies the literal kind-default+ value into their class body (silently restating intent — still+ needs a comment so reviewers know the override is deliberate).+ """++ _OVERRIDE_PATTERN = "start_to_close_timeout"++ def test_overrides_have_inline_comment(self):+ import inspect++ for cls in _all_plugin_classes():+ # Only flag plugins that LITERALLY set the attribute in their+ # own class body (not inherited from kind-base). The class-dict+ # check is the cheapest signal — if it's not in __dict__, it's+ # inherited and there's nothing to comment.+ if self._OVERRIDE_PATTERN not in cls.__dict__:+ continue+ try:+ src = inspect.getsource(cls)+ except (OSError, TypeError):+ # Skip if source unavailable (e.g. dynamically-created class+ # in a test fixture).+ continue+ # Find the override line and check the same-line comment OR the+ # immediately-preceding line is a comment.+ lines = src.splitlines()+ override_idx = None+ for i, line in enumerate(lines):+ stripped = line.strip()+ if stripped.startswith(self._OVERRIDE_PATTERN) and "=" in stripped:+ override_idx = i+ break+ if override_idx is None:+ # Defensive: __dict__ said it's there but source parsing+ # missed it (e.g. multi-line attribute assignment). Skip+ # rather than false-positive.+ continue+ override_line = lines[override_idx]+ has_inline_comment = "#" in override_line.split("=", 1)[1]+ has_preceding_comment = (+ override_idx > 0 and lines[override_idx - 1].strip().startswith("#")+ )+ assert has_inline_comment or has_preceding_comment, (+ f"{cls.__qualname__} overrides start_to_close_timeout away "+ f"from the kind-base default but has no explanatory comment. "+ f"Add a comment on the same line or directly above explaining "+ f"why this plugin needs a non-default timeout."+ )++class TestToolSchemaGeneration:"""ToolNode.Params must produce LLM-compatible JSON schema."""