Skip to content

Self-improving AI Employees for everyone

feat(events): Wave 12 Phase A — Temporal-native event framework foundation

trohitgc3dc85ac89b6
diff --git a/docs-internal/event_framework.md b/docs-internal/event_framework.md
new file mode 100644
index 00000000..6e0a8d29
--- /dev/null
+++ b/docs-internal/event_framework.md
@@ -0,0 +1,146 @@
+# Event Framework (Wave 12)
+
+Temporal-native event-routing layer for MachinaOs. Implements RFC sections
+6.3 (Temporal worker contract) + 6.4 (CloudEvents broadcast contract) from
+[plugin_authoring_rfc.md](./plugin_authoring_rfc.md).
+
+This doc is the operator + plugin-author reference. The design rationale +
+phase plan lives in `~/.claude/plans/properly-fix-the-tech-dreamy-tarjan.md`.
+
+## What this framework does
+
+Every inbound event (HTTP webhook, Telegram message, Gmail poll result,
+task completion, …) becomes a Temporal Signal delivered to whichever
+running workflows are waiting on that event type. Routing happens via
+the Temporal Visibility API — workflows tag themselves with custom
+Search Attributes at start, and the dispatch helper queries
+`ListWorkflows(query="EventType='X' AND ExecutionStatus='Running'")`
+to find consumers.
+
+Why Temporal: durability, replay safety, server-side dedup
+(`WorkflowIDReusePolicy`), and zero custom infrastructure. The framework
+adds ~300 LOC on top of Temporal primitives rather than reinventing an
+EventBus + event_log + DLQ table.
+
+## Architecture
+
+```
+Inbound source (FastAPI process)
+ ↓
+services/events/dispatch.py:emit(event: WorkflowEvent)
+ ├─→ Temporal Visibility query: workflows where EventType=event.type
+ ├─→ Signal each matching workflow
+ └─→ status_broadcaster.broadcast() — direct in-process WS fan-out
+```
+
+Worker is embedded in the FastAPI process (`main.py:211-292`
+`TemporalWorkerManager.start()` runs as `asyncio.create_task()`). Activities
+and the WebSocket connection pool share memory + event loop, so the fan-out
+to FE clients is a direct in-process call — no Redis Streams hop required.
+
+## Search Attributes setup
+
+The framework requires 6 custom Search Attributes on the Temporal
+namespace. Registration is **idempotent + automatic on Temporal client
+connect** (`services/temporal/client.py:TemporalClientWrapper.connect`):
+
+| Attribute | Type | Used for |
+|---|---|---|
+| `EventType` | KEYWORD | Visibility query — find consumers by CloudEvents type |
+| `EventSource` | KEYWORD | Routing when same type arrives from multiple sources |
+| `EventWorkflowId` | KEYWORD | Scope events to a MachinaOs workflow_id |
+| `TriggerNodeId` | KEYWORD | Per-trigger event-history queries |
+| `EventTriggerKind` | KEYWORD | Coarse classification (webhook / polling / …) |
+| `EventReceivedAt` | DATETIME | Time-range queries |
+
+Declarations live in
+[`services/temporal/search_attributes.py:EVENT_SEARCH_ATTRIBUTES`](../server/services/temporal/search_attributes.py).
+Single source of truth — add an entry to that list and registration
+picks it up next connect.
+
+### Manual registration (if needed)
+
+If the auto-registration on connect fails (e.g. permissions on a managed
+Temporal Cloud namespace), register manually via the Temporal CLI:
+
+```bash
+temporal operator search-attribute create \
+ --namespace default \
+ --name EventType \
+ --type Keyword
+# ... repeat for the other 5 attributes
+```
+
+### Verification
+
+```bash
+temporal operator search-attribute list --namespace default
+```
+
+Should show the 6 framework attributes alongside Temporal's built-in
+default attributes (`WorkflowType`, `WorkflowId`, `ExecutionStatus`, …).
+
+## Temporal contract for plugin authors
+
+| Class attribute | Purpose | Default |
+|---|---|---|
+| `start_to_close_timeout` | Per-attempt budget (one activity execution) | Kind-base default: ActionNode=10m, TriggerNode=24h, ToolNode=10m |
+| `retry_policy: RetryPolicy` | Backoff + max attempts + non-retryable error types | `DEFAULT_RETRY` — 3 attempts, 1-60s exponential. `NodeUserError` is auto-non-retryable. |
+| `heartbeat_timeout` | Max idle between `activity.heartbeat()` calls | 2 minutes |
+| `task_queue` | Worker pool routing | `TaskQueue.DEFAULT` |
+
+Override only when the kind-base default doesn't fit; an inline comment
+explaining why is required (enforced by
+`tests/test_plugin_contract.py::TestStartToCloseTimeoutOverridesAreCommented`).
+
+### Worker graceful shutdown
+
+Workers honor SIGTERM with a configurable grace window
+(`Settings.temporal_graceful_shutdown_seconds`, default 30s, override via
+env `TEMPORAL_GRACEFUL_SHUTDOWN_SECONDS`). Activities mid-flight finish
+or hand back to the server for retry instead of being killed mid-call.
+
+## CloudEvents envelope
+
+Every server→FE broadcast wraps `WorkflowEvent` (CloudEvents v1.0;
+`services/events/envelope.py`). Spec-compliant ID, source, type, time,
+plus MachinaOs extension attributes (`workflow_id`, `trigger_node_id`,
+`correlation_id` — kept in snake_case per the documented internal-naming
+rationale at `envelope.py:4-12`).
+
+Wire shape: `{"type": "<legacy_wire_key>", "data": <WorkflowEvent JSON>}`.
+The outer `type` is what FE switches on; the inner envelope is what
+parses for spec-compliant routing + dataschema lookup.
+
+### Plugin-owned event factories
+
+Plugin-specific events (e.g. `com.machinaos.telegram.message.received`)
+live in `nodes/<plugin>/_events.py`. Cross-cutting factories
+(`credential`, `oauth_completed`, `agent_progress`, `task_completed`,
+`workflow_lifecycle`, `deployment_snapshot`, `team_event`,
+`node_parameters_updated`) stay in `services/events/envelope.py`.
+
+See RFC §6.4 for the classification rule + the canonical
+`telegram/_events.py` example.
+
+## Verification
+
+Each Phase-A milestone has a verification command:
+
+| Phase | Check |
+|---|---|
+| A1 | `pytest tests/test_plugin_contract.py::TestStartToCloseTimeoutOverridesAreCommented` |
+| A2 | `python -c "from services.plugin.scaling import RetryPolicy; assert 'NodeUserError' in RetryPolicy().non_retryable_error_types"` |
+| A3 | `python -c "from core.config import Settings; print(Settings().temporal_graceful_shutdown_seconds)"` |
+| A4 | After Temporal connect: `temporal operator search-attribute list \| grep -E 'EventType\|EventSource\|EventWorkflowId\|TriggerNodeId\|EventTriggerKind\|EventReceivedAt'` — all 6 lines |
+
+Full test surface lands in Phase A9. Phase B (plugin `_events.py`
+modules) + Phase C (Temporal trigger-waiter migration) + Phase D
+(admin handlers + DLQ) build on this foundation.
+
+## References
+
+- Plan: `~/.claude/plans/properly-fix-the-tech-dreamy-tarjan.md`
+- RFC: [`plugin_authoring_rfc.md`](./plugin_authoring_rfc.md)
+- Temporal: [Search Attributes](https://docs.temporal.io/search-attribute) · [Signals](https://docs.temporal.io/develop/python/message-passing) · [Schedules](https://docs.temporal.io/develop/python/schedules) · [Retry Policies](https://docs.temporal.io/encyclopedia/retry-policies)
+- CloudEvents: [v1.0 spec](https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md)
diff --git a/server/core/config.py b/server/core/config.py
index d80d1ca0..50fce7dd 100644
--- a/server/core/config.py
+++ b/server/core/config.py
@@ -69,6 +69,67 @@ class Settings(BaseSettings):
temporal_agent_workflow_enabled: bool = Field(
default=False, env="TEMPORAL_AGENT_WORKFLOW_ENABLED",
)
+ # Wave 12 A3: SIGTERM grace window for Temporal workers. Activities
+ # mid-flight finish (or hand back to the server for retry) instead of
+ # being killed mid-call. Default 30s matches the polling-trigger
+ # heartbeat interval — a worker drain completes within one polling
+ # cycle. Tune via env when SIGTERM-to-restart latency matters.
+ temporal_graceful_shutdown_seconds: int = Field(
+ default=30, env="TEMPORAL_GRACEFUL_SHUTDOWN_SECONDS", ge=1,
+ )
+ # Wave 12 A6: feature flag for the Temporal-native event dispatch
+ # path (services/events/dispatch.py:emit). Default off in Phase A
+ # so the new dispatch + Signal-based consumer fan-out activates only
+ # for opt-in dogfooding before Phase B migrates plugin callsites.
+ # Phase A acceptance criterion is enabling this flag on a single
+ # canary callsite (e.g. whatsappReceive).
+ event_framework_enabled: bool = Field(
+ default=False, env="EVENT_FRAMEWORK_ENABLED",
+ )
+
+ # Persistence backend for the supervised Temporal cluster.
+ # 'sqlite' (dev default) — single ServiceSpec runs `temporal api`.
+ # 'postgres' (prod) — pgserver + temporal-server binary with
+ # YAML config managed by services/temporal/.
+ temporal_backend: Literal["sqlite", "postgres"] = Field(
+ default="sqlite", env="TEMPORAL_BACKEND",
+ )
+ # Internal Temporal service gRPC ports — exposed for the YAML config
+ # renderer and the runtime's readiness probe. Defaults match the
+ # ports Temporal documents at https://docs.temporal.io/references/configuration
+ temporal_frontend_grpc_port: int = Field(
+ default=7233, env="TEMPORAL_FRONTEND_GRPC_PORT", ge=1024, le=65535,
+ )
+ temporal_matching_grpc_port: int = Field(
+ default=7235, env="TEMPORAL_MATCHING_GRPC_PORT", ge=1024, le=65535,
+ )
+ temporal_history_grpc_port: int = Field(
+ default=7234, env="TEMPORAL_HISTORY_GRPC_PORT", ge=1024, le=65535,
+ )
+ temporal_worker_grpc_port: int = Field(
+ default=7239, env="TEMPORAL_WORKER_GRPC_PORT", ge=1024, le=65535,
+ )
+ temporal_bind_local_only: bool = Field(
+ default=True, env="TEMPORAL_BIND_LOCAL_ONLY",
+ )
+ temporal_num_history_shards: int = Field(
+ default=4, env="TEMPORAL_NUM_HISTORY_SHARDS", ge=1, le=4096,
+ )
+ temporal_default_max_conns: int = Field(
+ default=20, env="TEMPORAL_DEFAULT_MAX_CONNS", ge=1, le=500,
+ )
+ temporal_visibility_max_conns: int = Field(
+ default=4, env="TEMPORAL_VISIBILITY_MAX_CONNS", ge=1, le=500,
+ )
+ temporal_max_conn_lifetime: str = Field(
+ default="1h", env="TEMPORAL_MAX_CONN_LIFETIME",
+ )
+ temporal_binary_version: str = Field(
+ default="1.31.0", env="TEMPORAL_BINARY_VERSION",
+ )
+ temporal_postgres_dsn: Optional[str] = Field(
+ default=None, env="TEMPORAL_POSTGRES_DSN",
+ )
# API Keys (all optional, injected at runtime)
google_maps_api_key: Optional[str] = Field(default=None, env="GOOGLE_MAPS_API_KEY")
diff --git a/server/services/events/dispatch.py b/server/services/events/dispatch.py
new file mode 100644
index 00000000..2a959025
--- /dev/null
+++ b/server/services/events/dispatch.py
@@ -0,0 +1,197 @@
+"""Wave 12 A6: Temporal-native event dispatch.
+
+One public function — :func:`emit` — takes a :class:`WorkflowEvent` and
+routes it to:
+
+1. **Running consumer workflows** via Temporal Visibility query +
+ ``Client.signal_workflow``. Workflows tag themselves with the
+ ``EventType`` Search Attribute when they're started; this dispatch
+ finds them via ``ListWorkflows(query="EventType='X' AND
+ ExecutionStatus='Running'")`` and signals each.
+
+2. **In-process FastAPI WebSocket clients** via direct
+ ``status_broadcaster.broadcast()`` call. Worker + WS pool share
+ memory + event loop, so no IPC hop is needed (audit confirmed —
+ ``main.py:211-292``).
+
+No EventDispatchWorkflow, no Redis Streams, no DLQ table. Temporal's
+Visibility + Signal API + Event History provide the durability primitives
+this function depends on; everything else is a thin pass-through.
+
+Behind ``Settings.event_framework_enabled`` (default off in Phase A).
+When disabled, :func:`emit` is a no-op pass-through that returns the
+envelope unchanged.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from typing import Optional
+
+from core.config import Settings
+from core.logging import get_logger
+from services.events.envelope import WorkflowEvent
+
+logger = get_logger(__name__)
+
+
+# Wire-routing key the FastAPI WebSocket layer switches on. Plugin emits
+# its own wire key today (``credential_catalogue_updated`` /
+# ``agent_progress`` / ``plugin_connection_status`` / etc.). This
+# default is the generic CloudEvents-envelope channel used by events
+# that don't carry a plugin-specific wire key.
+_DEFAULT_WIRE_ROUTING_KEY = "cloudevent"
+
+# Visibility query template — produces a SQL-like List Filter string.
+# ``ExecutionStatus`` enum value 'Running' matches Temporal's status
+# encoding; see https://docs.temporal.io/list-filter.
+_RUNNING_CONSUMERS_QUERY = "EventType='{event_type}' AND ExecutionStatus='Running'"
+
+# Signal handler name on consumer workflows. Plain identifier rather
+# than CloudEvents type because Temporal Signal names are Python
+# identifiers (cannot contain dots).
+_SIGNAL_NAME = "on_event"
+
+
+async def emit(
+ event: WorkflowEvent,
+ *,
+ wire_routing_key: Optional[str] = None,
+) -> WorkflowEvent:
+ """Route ``event`` to running consumer workflows + in-process WS clients.
+
+ Args:
+ event: The :class:`WorkflowEvent` to dispatch. CloudEvents
+ envelope with a populated ``type`` (the Visibility filter
+ keys off this).
+ wire_routing_key: Outer WebSocket routing key (the ``type``
+ field on the WS frame). Defaults to the generic
+ ``cloudevent`` channel; plugin emitters override per their
+ existing wire key (e.g. ``"telegram_message_received"``).
+
+ Returns:
+ The envelope unchanged — callers may chain.
+
+ Behaviour when ``Settings.event_framework_enabled=False`` (Phase A
+ default): pass-through no-op. Logged at DEBUG so opt-in dogfooding
+ is observable without flipping the flag globally.
+ """
+ if not Settings().event_framework_enabled:
+ logger.debug(
+ f"event-framework disabled — emit no-op for event.id={event.id} "
+ f"type={event.type}",
+ )
+ return event
+
+ # Concurrent fan-out: Visibility query + Signal each consumer, AND
+ # broadcast to in-process WS clients. asyncio.gather lets the
+ # broadcast happen even if the Temporal Visibility query fails (and
+ # vice versa).
+ await asyncio.gather(
+ _signal_running_consumers(event),
+ _broadcast_in_process(event, wire_routing_key or _DEFAULT_WIRE_ROUTING_KEY),
+ return_exceptions=False,
+ )
+ return event
+
+
+async def _signal_running_consumers(event: WorkflowEvent) -> None:
+ """Find running workflows tagged with ``EventType=event.type`` and
+ signal each with ``on_event``.
+
+ Fail-soft: if the Temporal client is unavailable or the Visibility
+ query errors, log a warning and continue. The in-process broadcast
+ path is independent and still delivers to WS clients.
+ """
+ try:
+ from core.container import container
+
+ wrapper = container.temporal_client()
+ except Exception as exc: # noqa: BLE001
+ logger.warning(f"emit: container.temporal_client unavailable: {exc}")
+ return
+
+ if wrapper is None or wrapper.client is None:
+ logger.debug(
+ f"emit: Temporal not connected — skipping consumer fan-out for "
+ f"event.id={event.id}",
+ )
+ return
+
+ client = wrapper.client
+ query = _RUNNING_CONSUMERS_QUERY.format(event_type=event.type)
+
+ try:
+ consumers = []
+ async for wf in client.list_workflows(query=query):
+ consumers.append(wf)
+ except Exception as exc: # noqa: BLE001
+ logger.warning(
+ f"emit: Visibility query failed (query={query!r}): {exc}"
+ )
+ return
+
+ if not consumers:
+ logger.debug(
+ f"emit: 0 consumers for event.type={event.type!r} "
+ f"(event.id={event.id})",
+ )
+ return
+
+ # Signal each consumer concurrently. Per-target failures don't
+ # block other consumers (return_exceptions=True surfaces failures
+ # as exceptions in the result list).
+ signal_results = await asyncio.gather(
+ *[
+ _signal_one(client, wf.id, event)
+ for wf in consumers
+ ],
+ return_exceptions=True,
+ )
+
+ delivered = sum(1 for r in signal_results if not isinstance(r, Exception))
+ failed = len(signal_results) - delivered
+ logger.info(
+ f"emit: signalled {delivered}/{len(consumers)} consumers for "
+ f"event.type={event.type!r} (failed={failed}, event.id={event.id})",
+ )
+
+
+async def _signal_one(client, workflow_id: str, event: WorkflowEvent) -> None:
+ """Send the ``on_event`` signal to one workflow.
+
+ Kept as its own coroutine so :func:`_signal_running_consumers` can
+ ``asyncio.gather`` independently per-consumer.
+ """
+ handle = client.get_workflow_handle(workflow_id)
+ await handle.signal(_SIGNAL_NAME, event.model_dump(mode="json"))
+
+
+async def _broadcast_in_process(event: WorkflowEvent, wire_routing_key: str) -> None:
+ """Direct in-process WS fan-out via the status broadcaster.
+
+ Same asyncio event loop as the FastAPI handlers — see ``main.py:
+ 211-292`` (TemporalWorkerManager starts as ``asyncio.create_task``).
+ Activity → broadcaster is a direct method call against in-memory
+ ``Set[WebSocket]``; no IPC.
+ """
+ try:
+ from services.status_broadcaster import get_status_broadcaster
+
+ broadcaster = get_status_broadcaster()
+ except Exception as exc: # noqa: BLE001
+ logger.warning(f"emit: status_broadcaster unavailable: {exc}")
+ return
+
+ try:
+ await broadcaster.broadcast({
+ "type": wire_routing_key,
+ "data": event.model_dump(mode="json"),
+ })
+ except Exception as exc: # noqa: BLE001
+ logger.warning(
+ f"emit: WS broadcast failed (wire_key={wire_routing_key!r}): {exc}"
+ )
+
+
+__all__ = ["emit"]
diff --git a/server/services/plugin/scaling.py b/server/services/plugin/scaling.py
index 759df692..3032a78b 100644
--- a/server/services/plugin/scaling.py
+++ b/server/services/plugin/scaling.py
@@ -51,6 +51,13 @@ class RetryPolicy:
"ValidationError",
"PermissionDeniedError",
"InvalidParametersError",
+ # Wave 12 A2: NodeUserError is the canonical "user-correctable
+ # failure" exception (missing input, missing credential, API
+ # rejected the payload). Retrying it just burns attempts since
+ # the underlying input won't fix itself. Plugins that raise
+ # NodeUserError fail fast; the framework surfaces the message
+ # to the user / agent loop without traceback noise.
+ "NodeUserError",
))
def to_temporal(self):
diff --git a/server/services/temporal/activities.py b/server/services/temporal/activities.py
index 17c4987a..34f1f0fc 100644
--- a/server/services/temporal/activities.py
+++ b/server/services/temporal/activities.py
@@ -316,3 +316,59 @@ async def create_shared_session(pool_size: int = 100) -> aiohttp.ClientSession:
)
logger.info(f"Created shared session with pool_size={pool_size}")
return session
+
+
+# ---------------------------------------------------------------------------
+# Wave 12 A8: emit_event_activity
+#
+# Thin Temporal-activity wrapper around ``services.events.dispatch.emit``.
+# Used by ``PollingTriggerWorkflow`` (Phase C2) when a poll cycle produces
+# new items — the workflow can't directly do I/O (the dispatch helper
+# uses asyncio + a network round-trip to Temporal Visibility), so it
+# offloads to this activity.
+#
+# Activity scheduling timeouts (``start_to_close_timeout``, retry policy,
+# heartbeat) are set by the CALLING workflow at scheduling time —
+# ``workflow.execute_activity(emit_event_activity, ...,
+# start_to_close_timeout=timedelta(seconds=2))``. Defaults are not
+# hardcoded here.
+
+async def emit_event_activity(event_payload: Dict[str, Any]) -> Dict[str, Any]:
+ """Re-enter :func:`services.events.dispatch.emit` from a workflow context.
+
+ Args:
+ event_payload: The serialised :class:`WorkflowEvent` (output of
+ ``event.model_dump(mode="json")``). The activity reconstructs
+ the envelope and routes it through the dispatch helper.
+
+ Returns:
+ Status dict:
+ {
+ "delivered": True | False, # whether dispatch completed
+ "event_id": str,
+ "event_type": str,
+ }
+ """
+ from services.events.envelope import WorkflowEvent
+ from services.events.dispatch import emit
+
+ try:
+ event = WorkflowEvent.model_validate(event_payload)
+ except Exception as exc: # noqa: BLE001
+ activity.logger.warning(
+ f"emit_event_activity: envelope rejected: {exc}"
+ )
+ return {
+ "delivered": False,
+ "event_id": event_payload.get("id", "?"),
+ "event_type": event_payload.get("type", "?"),
+ "error": f"{type(exc).__name__}: {exc}",
+ }
+
+ await emit(event)
+ return {
+ "delivered": True,
+ "event_id": event.id,
+ "event_type": event.type,
+ }
diff --git a/server/services/temporal/client.py b/server/services/temporal/client.py
index d3e96a46..d3a71b29 100644
--- a/server/services/temporal/client.py
+++ b/server/services/temporal/client.py
@@ -71,6 +71,20 @@ class TemporalClientWrapper:
self._client = client
print(f"[Temporal] Connected to {self.server_address}", flush=True)
logger.info("Connected to Temporal server")
+ # Wave 12 A4: idempotently register the event-framework
+ # Search Attributes. Failure here is non-fatal — the
+ # framework still works without them, dispatch.emit just
+ # falls back to broadcast-only routing instead of
+ # signalling consumers.
+ try:
+ from services.temporal.search_attributes import (
+ register_search_attributes,
+ )
+ await register_search_attributes(self._client, self.namespace)
+ except Exception as sa_exc: # noqa: BLE001 — non-fatal
+ logger.warning(
+ f"Search-attribute registration failed (non-fatal): {sa_exc}"
+ )
return self._client
except Exception as e:
print(f"[Temporal] Connection attempt {attempt}/{retries} failed: {e}", flush=True)
diff --git a/server/services/temporal/search_attributes.py b/server/services/temporal/search_attributes.py
new file mode 100644
index 00000000..cdc8fc36
--- /dev/null
+++ b/server/services/temporal/search_attributes.py
@@ -0,0 +1,181 @@
+"""Wave 12 A4: Temporal custom Search Attributes registration.
+
+Defines the 6 Search Attributes the event framework needs for the
+Visibility API to route inbound events to running consumer workflows
+(see ``services/events/dispatch.py:emit``). Registration is idempotent —
+Temporal Server's ``add_search_attributes`` rejects duplicates with an
+``AlreadyExists`` error which we treat as success.
+
+Attribute declarations live in one place (:data:`EVENT_SEARCH_ATTRIBUTES`)
+as a structured list so callers iterate rather than restate. Adding a
+new attribute = add an entry to that list; everything else (registration,
+docs, error handling) flows from it.
+
+References:
+- https://docs.temporal.io/search-attribute
+- https://python.temporal.io/temporalio.client.html#Client.operator_service
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Sequence
+
+from temporalio.api.enums.v1 import IndexedValueType
+from temporalio.api.operatorservice.v1 import (
+ AddSearchAttributesRequest,
+ ListSearchAttributesRequest,
+)
+from temporalio.client import Client
+
+from core.logging import get_logger
+
+logger = get_logger(__name__)
+
+
+@dataclass(frozen=True)
+class SearchAttributeSpec:
+ """Declarative spec for one Temporal Search Attribute."""
+
+ name: str
+ indexed_type: IndexedValueType
+ description: str
+
+
+# Single source of truth for the 6 event-framework Search Attributes.
+# Anything that names or filters by these attributes (the dispatch
+# helper, the Visibility queries in admin handlers, the test suite)
+# imports from this list rather than restating the strings.
+EVENT_SEARCH_ATTRIBUTES: Sequence[SearchAttributeSpec] = (
+ SearchAttributeSpec(
+ name="EventType",
+ indexed_type=IndexedValueType.INDEXED_VALUE_TYPE_KEYWORD,
+ description=(
+ "CloudEvents type the workflow is consuming "
+ "(e.g. com.machinaos.whatsapp.message.received). Used by "
+ "dispatch.emit to find running consumers via Visibility "
+ "queries: ListWorkflows(query=f\"EventType='{event.type}' "
+ "AND ExecutionStatus='Running'\")."
+ ),
+ ),
+ SearchAttributeSpec(
+ name="EventSource",
+ indexed_type=IndexedValueType.INDEXED_VALUE_TYPE_KEYWORD,
+ description=(
+ "CloudEvents source URI fragment "
+ "(e.g. machinaos://nodes/whatsapp). Used for routing when "
+ "the same type can arrive from multiple sources."
+ ),
+ ),
+ SearchAttributeSpec(
+ name="EventWorkflowId",
+ indexed_type=IndexedValueType.INDEXED_VALUE_TYPE_KEYWORD,
+ description=(
+ "MachinaOs workflow_id the event belongs to (distinct from "
+ "the Temporal Workflow ID which addresses the workflow run). "
+ "Used to scope event delivery to a specific deployment."
+ ),
+ ),
+ SearchAttributeSpec(
+ name="TriggerNodeId",
+ indexed_type=IndexedValueType.INDEXED_VALUE_TYPE_KEYWORD,
+ description=(
+ "Trigger node id that emitted or will consume the event. "
+ "Used for per-trigger event-history queries and the "
+ "list_event_waiters admin handler."
+ ),
+ ),
+ SearchAttributeSpec(
+ name="EventTriggerKind",
+ indexed_type=IndexedValueType.INDEXED_VALUE_TYPE_KEYWORD,
+ description=(
+ "High-level event source classification (webhook / polling / "
+ "whatsapp / telegram / ...). Coarser-grained than EventType; "
+ "useful for ops dashboards."
+ ),
+ ),
+ SearchAttributeSpec(
+ name="EventReceivedAt",
+ indexed_type=IndexedValueType.INDEXED_VALUE_TYPE_DATETIME,
+ description=(
+ "When the event entered the framework. Supports time-range "
+ "visibility queries (e.g. 'all events received in the last "
+ "hour')."
+ ),
+ ),
+)
+
+
+def attribute_names() -> Sequence[str]:
+ """Return the configured attribute names. Helper for tests + admin code."""
+ return tuple(spec.name for spec in EVENT_SEARCH_ATTRIBUTES)
+
+
+async def register_search_attributes(client: Client, namespace: str) -> dict:
+ """Idempotently register every :data:`EVENT_SEARCH_ATTRIBUTES` entry
+ on the given Temporal namespace.
+
+ Returns a status dict keyed by attribute name:
+ {name: "registered" | "already_exists" | "error: ..."}
+
+ Safe to call multiple times — already-present attributes are
+ detected via a pre-flight ``ListSearchAttributes`` call and skipped.
+ Attributes that fail to register surface in the result dict with
+ the error string so the caller can decide whether to fail-fast.
+ """
+ operator = client.service_client.operator_service
+
+ # Pre-flight: which attributes already exist on this namespace?
+ try:
+ existing_resp = await operator.list_search_attributes(
+ ListSearchAttributesRequest(namespace=namespace),
+ )
+ existing_names = set(existing_resp.custom_attributes.keys())
+ except Exception as exc: # noqa: BLE001 — surface every failure
+ logger.warning(
+ f"Could not list existing Search Attributes on namespace={namespace!r}: {exc}. "
+ "Attempting registration anyway; duplicates will be reported as errors."
+ )
+ existing_names = set()
+
+ status: dict[str, str] = {}
+ to_register: dict[str, IndexedValueType] = {
+ spec.name: spec.indexed_type
+ for spec in EVENT_SEARCH_ATTRIBUTES
+ if spec.name not in existing_names
+ }
+ for name in existing_names & {s.name for s in EVENT_SEARCH_ATTRIBUTES}:
+ status[name] = "already_exists"
+
+ if not to_register:
+ logger.info(
+ f"All {len(EVENT_SEARCH_ATTRIBUTES)} event-framework Search Attributes "
+ f"already registered on namespace={namespace!r}",
+ )
+ return status
+
+ try:
+ await operator.add_search_attributes(
+ AddSearchAttributesRequest(
+ namespace=namespace,
+ search_attributes=to_register,
+ ),
+ )
+ except Exception as exc: # noqa: BLE001
+ # Whole batch failed — mark each requested attribute with the error.
+ err = f"error: {type(exc).__name__}: {exc}"
+ for name in to_register:
+ status[name] = err
+ logger.error(
+ f"Failed to register Search Attributes on namespace={namespace!r}: {exc}",
+ )
+ return status
+
+ for name in to_register:
+ status[name] = "registered"
+
+ logger.info(
+ f"Registered {len(to_register)} event-framework Search Attributes on "
+ f"namespace={namespace!r}: {list(to_register.keys())}",
+ )
+ return status
diff --git a/server/services/temporal/worker.py b/server/services/temporal/worker.py
index 29ad565e..612a3155 100644
--- a/server/services/temporal/worker.py
+++ b/server/services/temporal/worker.py
@@ -16,6 +16,7 @@ References:
"""
import asyncio
+from datetime import timedelta
from typing import Optional
import aiohttp
@@ -35,6 +36,18 @@ logger = get_logger(__name__)
tracer = trace.get_tracer(__name__)
+def _graceful_shutdown_timeout() -> timedelta:
+ """Wave 12 A3: SIGTERM grace window for Temporal workers.
+
+ Read from ``Settings.temporal_graceful_shutdown_seconds`` at the
+ moment each Worker is instantiated (rather than module load) so
+ test fixtures + env reloads pick up overrides.
+ """
+ from core.config import Settings
+
+ return timedelta(seconds=Settings().temporal_graceful_shutdown_seconds)
+
+
def create_runtime() -> Runtime:
"""Create a Temporal runtime with worker heartbeating disabled.
@@ -136,6 +149,7 @@ class TemporalWorkerManager:
# Allow concurrent activity execution for parallel branches
max_concurrent_activities=self.pool_size,
max_concurrent_workflow_tasks=10,
+ graceful_shutdown_timeout=_graceful_shutdown_timeout(),
)
logger.info(
"Registered Temporal activities",
@@ -273,6 +287,7 @@ class TemporalWorkerPool:
activities=activities,
max_concurrent_activities=concurrency,
max_concurrent_workflow_tasks=10,
+ graceful_shutdown_timeout=_graceful_shutdown_timeout(),
)
task = asyncio.create_task(worker.run(), name=f"worker-{queue}")
self._workers.append(worker)
@@ -356,6 +371,7 @@ async def run_standalone_worker(
activities=[activities.execute_node_activity], # Pass bound method
max_concurrent_activities=pool_size,
max_concurrent_workflow_tasks=10,
+ graceful_shutdown_timeout=_graceful_shutdown_timeout(),
)
logger.info("Worker running. Press Ctrl+C to stop.")
@@ -394,6 +410,7 @@ async def create_worker(
activities=[activities.execute_node_activity], # Pass bound method
max_concurrent_activities=100,
max_concurrent_workflow_tasks=10,
+ graceful_shutdown_timeout=_GRACEFUL_SHUTDOWN_TIMEOUT,
)
diff --git a/server/services/temporal/workflow.py b/server/services/temporal/workflow.py
index 2ef3d673..4d1e594e 100644
--- a/server/services/temporal/workflow.py
+++ b/server/services/temporal/workflow.py
@@ -79,8 +79,75 @@ class MachinaWorkflow:
- Per-node retry policies
- Config node filtering (tools, memory, services)
- Multi-tenant support via tenant_id in context
+ - Wave 12 A7: event-framework signal handler — receives Temporal
+ Signals from ``services/events/dispatch.py:emit`` and parks
+ pending events in ``_matched_events`` for trigger nodes to
+ consume via ``workflow.wait_condition(_has_event_matching, ...)``.
+ Per-run dedup via ``_seen_event_ids`` (reconstructed
+ deterministically from Event History on replay).
"""
+ def __init__(self) -> None:
+ # Wave 12 A7: per-run event-framework state. Populated by the
+ # ``on_event`` signal handler; consumed by Phase C1's
+ # ``wait_condition`` predicate in the trigger-waiter rewrite.
+ # Today these stay empty unless the event-framework feature
+ # flag is on AND a producer signals this workflow.
+ self._seen_event_ids: Set[str] = set()
+ self._matched_events: List[Dict[str, Any]] = []
+
+ @workflow.signal
+ async def on_event(self, event_payload: Dict[str, Any]) -> None:
+ """Wave 12 A7: receive an event from ``services.events.dispatch.emit``.
+
+ Drops duplicates by ``event.id`` (per-run dedup — the set is
+ reconstructed deterministically from Event History on replay).
+ Matching events are queued for the workflow body to consume via
+ :meth:`_has_event_matching` in a ``wait_condition`` (wired in
+ Phase C1).
+ """
+ event_id = event_payload.get("id")
+ if not event_id:
+ workflow.logger.warning(
+ "on_event: skipping malformed envelope without 'id'"
+ )
+ return
+ if event_id in self._seen_event_ids:
+ workflow.logger.debug(
+ f"on_event: dedup hit for event.id={event_id}"
+ )
+ return
+ self._seen_event_ids.add(event_id)
+ self._matched_events.append(event_payload)
+ workflow.logger.info(
+ f"on_event: queued event.id={event_id} "
+ f"type={event_payload.get('type', '?')} "
+ f"(matched={len(self._matched_events)})"
+ )
+
+ def _has_event_matching(self, predicate=None) -> bool:
+ """Wave 12 A7: ``wait_condition`` predicate for trigger nodes.
+
+ Returns True when at least one queued event matches the optional
+ ``predicate(event_payload) -> bool``. Defaults to "any event
+ queued" — most trigger nodes filter further by inspecting the
+ envelope after pop.
+
+ Used by Phase C1's trigger-waiter rewrite to replace the
+ in-process ``asyncio.Future`` waiter:
+
+ await workflow.wait_condition(
+ lambda: self._has_event_matching(my_filter),
+ timeout=...,
+ )
+ event = self._pop_matching_event(my_filter)
+ """
+ if not self._matched_events:
+ return False
+ if predicate is None:
+ return True
+ return any(predicate(e) for e in self._matched_events)
+
@workflow.run
async def run(self, workflow_data: Dict[str, Any]) -> Dict[str, Any]:
"""Execute workflow by orchestrating node activities.
diff --git a/server/tests/test_event_framework_phase_a.py b/server/tests/test_event_framework_phase_a.py
new file mode 100644
index 00000000..37960648
--- /dev/null
+++ b/server/tests/test_event_framework_phase_a.py
@@ -0,0 +1,307 @@
+"""Wave 12 Phase A: smoke tests for the event-framework foundation.
+
+Covers (with no live Temporal cluster required):
+
+A1: ``TestStartToCloseTimeoutOverridesAreCommented`` — lives in
+ ``test_plugin_contract.py``; not duplicated here.
+
+A2: ``NodeUserError`` is in :class:`RetryPolicy`'s default
+ ``non_retryable_error_types``.
+
+A3: ``Settings.temporal_graceful_shutdown_seconds`` is exposed,
+ configurable, and consumed by :func:`worker._graceful_shutdown_timeout`.
+
+A4: :data:`EVENT_SEARCH_ATTRIBUTES` is a non-empty structured spec list;
+ every entry has a name + indexed_type + description; the
+ :func:`attribute_names` helper agrees with the structure.
+
+A6: :func:`services.events.dispatch.emit` is a no-op pass-through when
+ ``Settings.event_framework_enabled=False`` (Phase-A default).
+
+A7: :class:`MachinaWorkflow` initializes its event state in ``__init__``;
+ the ``on_event`` signal handler dedups by ``event.id`` and queues
+ matching events; :meth:`_has_event_matching` implements the
+ ``wait_condition`` predicate contract.
+
+A8: :func:`emit_event_activity` validates the envelope payload and
+ forwards to :func:`dispatch.emit`.
+"""
+
+from __future__ import annotations
+
+import sys
+import types
+from datetime import timedelta
+from unittest.mock import MagicMock
+
+import pytest
+
+# Stub the `machina` package (lives at project root, outside server/) so
+# `services.temporal.__init__` → `_runtime.py` → `from machina.tcp import
+# probe_tcp_port` doesn't crash collection. Conftest stubs core.* but
+# not machina.*.
+if "machina" not in sys.modules:
+ _machina = types.ModuleType("machina")
+ _machina.__path__ = []
+ sys.modules["machina"] = _machina
+ _machina_tcp = types.ModuleType("machina.tcp")
+ _machina_tcp.probe_tcp_port = MagicMock(return_value=False)
+ sys.modules["machina.tcp"] = _machina_tcp
+
+from core.config import Settings
+from services.events.envelope import WorkflowEvent
+from services.plugin.scaling import RetryPolicy
+
+
+class TestA2NodeUserErrorNonRetryable:
+ """A2: ``NodeUserError`` ships in the default ``non_retryable_error_types``
+ so a single ``cls.retry_policy = DEFAULT_RETRY`` picks it up."""
+
+ def test_node_user_error_in_default_non_retryable(self):
+ policy = RetryPolicy()
+ assert "NodeUserError" in policy.non_retryable_error_types, (
+ "RetryPolicy.non_retryable_error_types default must contain "
+ "'NodeUserError' so plugin-raised user errors fail fast in "
+ "Temporal activities."
+ )
+
+ def test_to_temporal_preserves_non_retryable_list(self):
+ policy = RetryPolicy()
+ # Import-on-call to keep the test independent of Temporal SDK
+ # availability at module-import time.
+ try:
+ temporal_policy = policy.to_temporal()
+ except ImportError: # pragma: no cover
+ pytest.skip("temporalio SDK not installed in test env")
+ assert "NodeUserError" in temporal_policy.non_retryable_error_types
+
+
+class TestA3GracefulShutdownTimeout:
+ """A3: SIGTERM grace window is config-driven, not hardcoded.
+
+ ``conftest.py`` mocks ``core.config.Settings`` as ``MagicMock`` so
+ handler tests don't pay Pydantic startup cost. We patch a real
+ ``Settings``-like stub inside this class to verify the contract.
+ """
+
+ def test_worker_helper_returns_timedelta(self, monkeypatch):
+ from services.temporal import worker as worker_mod
+
+ class _FakeSettings:
+ temporal_graceful_shutdown_seconds = 17
+
+ # Patch the lazy import inside worker._graceful_shutdown_timeout.
+ # It does `from core.config import Settings`; patch that module
+ # entry to return our fake.
+ import core.config
+
+ monkeypatch.setattr(core.config, "Settings", lambda: _FakeSettings())
+
+ td = worker_mod._graceful_shutdown_timeout()
+ assert isinstance(td, timedelta)
+ assert td.total_seconds() == 17
+
+
+class TestA4SearchAttributesSpec:
+ """A4: Search Attributes are declared structurally, NOT scattered
+ as literals through the dispatch + admin code."""
+
+ def test_spec_is_non_empty(self):
+ from services.temporal.search_attributes import EVENT_SEARCH_ATTRIBUTES
+
+ assert len(EVENT_SEARCH_ATTRIBUTES) >= 6
+
+ def test_each_spec_complete(self):
+ from services.temporal.search_attributes import EVENT_SEARCH_ATTRIBUTES
+
+ for spec in EVENT_SEARCH_ATTRIBUTES:
+ assert spec.name, "every search-attribute spec needs a name"
+ assert spec.indexed_type, f"{spec.name} missing indexed_type"
+ assert spec.description, f"{spec.name} missing description"
+
+ def test_attribute_names_helper_matches_spec(self):
+ from services.temporal.search_attributes import (
+ EVENT_SEARCH_ATTRIBUTES,
+ attribute_names,
+ )
+
+ spec_names = tuple(s.name for s in EVENT_SEARCH_ATTRIBUTES)
+ assert attribute_names() == spec_names
+
+ def test_required_attributes_present(self):
+ """The 6 attributes the plan promises."""
+ from services.temporal.search_attributes import attribute_names
+
+ required = {
+ "EventType",
+ "EventSource",
+ "EventWorkflowId",
+ "TriggerNodeId",
+ "EventTriggerKind",
+ "EventReceivedAt",
+ }
+ assert required.issubset(set(attribute_names()))
+
+
+class TestA6DispatchEmitFeatureFlag:
+ """A6: ``emit`` is a no-op pass-through when the flag is off."""
+
+ @pytest.mark.asyncio
+ async def test_disabled_emit_is_passthrough(self, monkeypatch):
+ # Pydantic Settings reads from env; force the flag off for this test.
+ monkeypatch.setenv("EVENT_FRAMEWORK_ENABLED", "false")
+
+ from services.events.dispatch import emit
+
+ event = WorkflowEvent(
+ source="machinaos://services/test",
+ type="com.machinaos.test.disabled",
+ )
+ returned = await emit(event)
+ assert returned is event
+
+ @pytest.mark.asyncio
+ async def test_enabled_emit_calls_both_paths(self, monkeypatch):
+ """When the flag is on, both signal-fanout and in-process
+ broadcast are invoked. Both are patched to record-only."""
+ monkeypatch.setenv("EVENT_FRAMEWORK_ENABLED", "true")
+ # Force Settings() to re-read by clearing the cached singleton
+ # if any; here Settings() builds fresh per call so monkeypatch
+ # of env is enough.
+
+ signal_calls = []
+ broadcast_calls = []
+
+ async def fake_signal(event):
+ signal_calls.append(event)
+
+ async def fake_broadcast(event, wire_key):
+ broadcast_calls.append((event, wire_key))
+
+ from services.events import dispatch
+
+ monkeypatch.setattr(dispatch, "_signal_running_consumers", fake_signal)
+ monkeypatch.setattr(dispatch, "_broadcast_in_process", fake_broadcast)
+
+ event = WorkflowEvent(
+ source="machinaos://services/test",
+ type="com.machinaos.test.enabled",
+ )
+ await dispatch.emit(event, wire_routing_key="custom_wire_key")
+
+ assert len(signal_calls) == 1
+ assert signal_calls[0] is event
+ assert len(broadcast_calls) == 1
+ assert broadcast_calls[0] == (event, "custom_wire_key")
+
+
+class TestA7MachinaWorkflowSignalHandler:
+ """A7: signal handler dedups + queues; predicate matches.
+
+ ``workflow.logger`` requires a Temporal workflow event loop, which
+ we don't spin up in these unit tests. Patch ``temporalio.workflow.logger``
+ to a no-op so the handler can run synchronously.
+ """
+
+ @pytest.fixture(autouse=True)
+ def _patch_workflow_logger(self, monkeypatch):
+ from unittest.mock import MagicMock
+ from temporalio import workflow as temporal_workflow
+
+ monkeypatch.setattr(temporal_workflow, "logger", MagicMock())
+
+ def test_init_sets_empty_state(self):
+ from services.temporal.workflow import MachinaWorkflow
+
+ wf = MachinaWorkflow()
+ assert wf._seen_event_ids == set()
+ assert wf._matched_events == []
+
+ @pytest.mark.asyncio
+ async def test_on_event_dedups_by_id(self):
+ from services.temporal.workflow import MachinaWorkflow
+
+ wf = MachinaWorkflow()
+ payload = {"id": "evt-1", "type": "com.machinaos.test.x"}
+
+ await wf.on_event(payload)
+ await wf.on_event(payload) # duplicate
+
+ assert wf._seen_event_ids == {"evt-1"}
+ assert len(wf._matched_events) == 1
+
+ @pytest.mark.asyncio
+ async def test_on_event_drops_malformed_envelope(self):
+ from services.temporal.workflow import MachinaWorkflow
+
+ wf = MachinaWorkflow()
+ await wf.on_event({"type": "com.machinaos.test.x"}) # no 'id'
+ assert wf._matched_events == []
+
+ def test_has_event_matching_empty_state(self):
+ from services.temporal.workflow import MachinaWorkflow
+
+ wf = MachinaWorkflow()
+ assert wf._has_event_matching() is False
+
+ def test_has_event_matching_predicate(self):
+ from services.temporal.workflow import MachinaWorkflow
+
+ wf = MachinaWorkflow()
+ wf._matched_events.append(
+ {"id": "e1", "type": "com.machinaos.whatsapp.message.received"}
+ )
+
+ # No-predicate: any queued event matches.
+ assert wf._has_event_matching() is True
+
+ # Truthy predicate matches.
+ assert wf._has_event_matching(
+ lambda e: e["type"].endswith("message.received")
+ ) is True
+
+ # Falsy predicate doesn't match.
+ assert wf._has_event_matching(
+ lambda e: e["type"].endswith(".nope")
+ ) is False
+
+
+class TestA8EmitEventActivity:
+ """A8: activity wrapper validates the envelope and forwards to emit."""
+
+ @pytest.mark.asyncio
+ async def test_valid_payload_returns_delivered_true(self, monkeypatch):
+ captured = []
+
+ async def fake_emit(event, **kwargs):
+ captured.append(event)
+ return event
+
+ from services.events import dispatch as dispatch_mod
+
+ monkeypatch.setattr(dispatch_mod, "emit", fake_emit)
+
+ from services.temporal.activities import emit_event_activity
+
+ payload = WorkflowEvent(
+ source="machinaos://services/test",
+ type="com.machinaos.test.a8",
+ ).model_dump(mode="json")
+
+ # Activities decorated with @activity.defn are still callable
+ # outside a worker — they're plain async functions with metadata.
+ result = await emit_event_activity(payload)
+
+ assert result["delivered"] is True
+ assert result["event_type"] == "com.machinaos.test.a8"
+ assert len(captured) == 1
+
+ @pytest.mark.asyncio
+ async def test_malformed_payload_returns_delivered_false(self):
+ from services.temporal.activities import emit_event_activity
+
+ # Missing required `source` and `type` fields → validation fails.
+ result = await emit_event_activity({"id": "evt-malformed"})
+
+ assert result["delivered"] is False
+ assert "error" in result
diff --git a/server/tests/test_plugin_contract.py b/server/tests/test_plugin_contract.py
index a9e2f8c5..e0af6ef3 100644
--- a/server/tests/test_plugin_contract.py
+++ b/server/tests/test_plugin_contract.py
@@ -130,6 +130,70 @@ class TestScalingKnobs:
)
+class TestStartToCloseTimeoutOverridesAreCommented:
+ """Wave 12 — A1: any plugin that overrides ``start_to_close_timeout``
+ away from its kind-base default must carry an inline comment in the
+ class body explaining why.
+
+ Today every plugin inherits the kind-base default (``ActionNode``=10m,
+ ``TriggerNode``=24h, ``ToolNode``=10m). The kind-defaults are the
+ declared intent at the kind-base level; per-plugin overrides are the
+ exception, not the norm. When an override does ship, the comment
+ forces the author to justify the deviation (and gives the next reader
+ the reason without git-blame archaeology).
+
+ Why introspect with ``inspect.getsource`` rather than just check
+ equality against the kind-default: equality alone can't catch the
+ case where a future plugin author copies the literal kind-default
+ value into their class body (silently restating intent — still
+ needs a comment so reviewers know the override is deliberate).
+ """
+
+ _OVERRIDE_PATTERN = "start_to_close_timeout"
+
+ def test_overrides_have_inline_comment(self):
+ import inspect
+
+ for cls in _all_plugin_classes():
+ # Only flag plugins that LITERALLY set the attribute in their
+ # own class body (not inherited from kind-base). The class-dict
+ # check is the cheapest signal — if it's not in __dict__, it's
+ # inherited and there's nothing to comment.
+ if self._OVERRIDE_PATTERN not in cls.__dict__:
+ continue
+ try:
+ src = inspect.getsource(cls)
+ except (OSError, TypeError):
+ # Skip if source unavailable (e.g. dynamically-created class
+ # in a test fixture).
+ continue
+ # Find the override line and check the same-line comment OR the
+ # immediately-preceding line is a comment.
+ lines = src.splitlines()
+ override_idx = None
+ for i, line in enumerate(lines):
+ stripped = line.strip()
+ if stripped.startswith(self._OVERRIDE_PATTERN) and "=" in stripped:
+ override_idx = i
+ break
+ if override_idx is None:
+ # Defensive: __dict__ said it's there but source parsing
+ # missed it (e.g. multi-line attribute assignment). Skip
+ # rather than false-positive.
+ continue
+ override_line = lines[override_idx]
+ has_inline_comment = "#" in override_line.split("=", 1)[1]
+ has_preceding_comment = (
+ override_idx > 0 and lines[override_idx - 1].strip().startswith("#")
+ )
+ assert has_inline_comment or has_preceding_comment, (
+ f"{cls.__qualname__} overrides start_to_close_timeout away "
+ f"from the kind-base default but has no explanatory comment. "
+ f"Add a comment on the same line or directly above explaining "
+ f"why this plugin needs a non-default timeout."
+ )
+
+
class TestToolSchemaGeneration:
"""ToolNode.Params must produce LLM-compatible JSON schema."""