Skip to content

Self-improving AI Employees for everyone

feat(chat): track each owner message as a run the employee answers

trohitgb940cae55beb
diff --git a/CLAUDE.md b/CLAUDE.md
index 9654d355..50e12f00 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -13,7 +13,8 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire
| **[UI Migration Plan](./docs-internal/ARCHIVE/ui_migration_plan.md)** | *Archived.* antd → shadcn/ui migration completion log (Waves 1–10). Current state lives in [Frontend Architecture](./docs-internal/frontend_architecture.md) and [Theme System](./docs-internal/theme_system.md). |
| **[Temporal Cleanup & Resilience Plan (Waves 15-18)](./docs-internal/ARCHIVE/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md)** | *Archived — all four waves shipped.* 15 dead-code retirement (Redis-Streams branch in event_waiter, APScheduler cron stack, orphaned `connection_status` factory), 16 per-queue task routing (`TemporalWorkerPool`), 17 resilience hardening (cron `catchup_window`, SDK interceptors, `DEPLOYMENT_MODE`, periodic heartbeats; the one-shot LLM-step retry it records was later reverted to unlimited-with-backoff), 18 worker performance tuning. Current settings live in [Temporal Architecture](./docs-internal/TEMPORAL_ARCHITECTURE.md). |
| **[Node Allowlist](./docs-internal/node_allowlist.md)** | Single-config UI visibility — `server/config/node_allowlist.json` controls which nodes / credential categories / skill folders show in the UI. Five lists with two enforcement tiers (mode-gated allowlist + absolute blocklist). `useNodeAllowlist` hook exposes `isVisible` / `isBlocked` / `isAllowed` / `isCredentialCategoryDisabled` / `isSkillFolderDisabled`. Adding a new disabled domain = single JSON edit, no code change. |
-| **[Normal Mode (Home)](./docs-internal/normal_mode.md)** | The owner-facing landing screen: describe a job, an LLM on the server drafts a setup screen (a JSON UI spec the client parses, normalises and renders from a fixed catalogue), and Hire builds, validates, saves and starts one workflow per AI employee (`services/employees/`, graph built from `config/employee_apps.json` plus the owner's skill library and a Canvas node, and checked against `node_allowlist.json` `enabled_nodes`). Covers the screen switch (`app/`, `shellMode` / `ui_shell_mode`, `VITE_NORMAL_MODE=false` kill switch, Home shows light/dark only), the employee page (only their name above the conversation; Pause is in the Workspace header), deleting an employee from the team sidebar (stopped through Reset first, then gone from every tab), the `approvalGate` "ask me before sending" step, run records for "done today" (`machina-run-record-v1`), automatic-pause reasons, Settings (Profile fields; usage-only Billing via `get_employee_usage`; Skills, the user-skills library each new hire copies onto a Skills node; consumer Connectors with `publisher` / `verified`; Plugins, the starter bundles in `hire/starters.json`; the last three share one catalog page), the Workspace dock (an employee's Canvas board, named by the summary's `canvas_node_id`; a live Browser tab with user takeover, shared with Dev mode; Mobile shows the managed local Android phone), the "Web browser" app (read-only under "ask me first" through the tool's `ask_first_params`; while the agent waits in `request_user` the summary's `browser_request` shows Needs you and Help in browser), Talk on the employee page (a thread over the chat session whose id is the workflow id, rows written only through `services/chat_thread.py`; the talk line is chatTrigger → an agent → `chatReply` "Reply in Chat": a chat hire's own agent is its talk agent, other hires get a "Talk with {name}" agent sharing the worker's tools and skills with its own Context, and schedule hires also "Post to Talk"), Turn on Talk (for talk `off`) and Apply (while the summary's `pending_changes` is true) via `enable_employee_talk` / `apply_employee_changes`, each a restart on the saved graph through `services/deployment/restart.py`, self-extension from Talk through the Agent Builder within the Hire rule (`services/employees/policy.py`), the 3D orb, the wire contract and the `employee_lifecycle` / `approval_lifecycle` / `chat.updated` broadcasts. |
+| **[Normal Mode (Home)](./docs-internal/normal_mode.md)** | The owner-facing landing screen: describe a job, an LLM on the server drafts a setup screen (a JSON UI spec the client parses, normalises and renders from a fixed catalogue), and Hire builds, validates, saves and starts one workflow per AI employee (`services/employees/`, graph built from `config/employee_apps.json` plus the owner's skill library and a Canvas node, and checked against `node_allowlist.json` `enabled_nodes`). Covers the screen switch (`app/`, `shellMode` / `ui_shell_mode`, `VITE_NORMAL_MODE=false` kill switch, Home shows light/dark only), the employee page (only their name above the conversation; Pause is in the Workspace header), deleting an employee from the team sidebar (stopped through Reset first, then gone from every tab), the `approvalGate` "ask me before sending" step, run records for "done today" (`machina-run-record-v1`), automatic-pause reasons, Settings (Profile fields; usage-only Billing via `get_employee_usage`; Skills, the user-skills library each new hire copies onto a Skills node; consumer Connectors with `publisher` / `verified`; Plugins, the starter bundles in `hire/starters.json`; the last three share one catalog page), the Workspace dock (an employee's Canvas board, named by the summary's `canvas_node_id`; a live Browser tab with user takeover, shared with Dev mode; Mobile shows the managed local Android phone), the "Web browser" app (read-only under "ask me first" through the tool's `ask_first_params`; while the agent waits in `request_user` the summary's `browser_request` shows Needs you and Help in browser), Talk on the employee page (a thread over the chat session whose id is the workflow id, rows written through `services/chat_thread.py` and, for the owner's messages and their chat runs, `services/chat/ledger.py` (see [Chat Protocol](./docs-internal/chat_protocol.md)); the talk line is chatTrigger → an agent → `chatReply` "Reply in Chat": a chat hire's own agent is its talk agent, other hires get a "Talk with {name}" agent sharing the worker's tools and skills with its own Context, and schedule hires also "Post to Talk"), Turn on Talk (for talk `off`) and Apply (while the summary's `pending_changes` is true) via `enable_employee_talk` / `apply_employee_changes`, each a restart on the saved graph through `services/deployment/restart.py`, self-extension from Talk through the Agent Builder within the Hire rule (`services/employees/policy.py`), the 3D orb, the wire contract and the `employee_lifecycle` / `approval_lifecycle` / `chat.updated` broadcasts. |
+| **[Chat Protocol](./docs-internal/chat_protocol.md)** | The wire contract between the chat runtime (`server/services/chat/`) and the shared chat UI: AG-UI's event model on the existing WebSocket, snake_case, in the CloudEvents envelope. A message a deployed chat trigger will answer starts a **chat run** admitted with it in one reserved write transaction (`Database.reserved_session`, SQLite `BEGIN IMMEDIATE`): one live run per session (the lane, also a partial unique index; a second send answers `run_in_progress`), idempotent `client_message_id`, claimed and finished by the MachinaWorkflow that answers it behind `machina-chat-run-v1` (the run id is trusted only from an event whose source is `opencompany://services/chat`; `run_scope` rides every node context; only the claimant finishes), the reply saved as `a_<run id>` by `chatReply`, every message appended after the session's active leaf (`chat_threads`) so concurrent writes never fork the thread. `chat_run_event` frames reach only sockets that passed `chat_subscribe`'s owner check (`services/chat/hub.py`: per-run `seq`, `hub_epoch`, bounded per-socket queues with a resync), never the status broadcast. A watchdog (`services/chat/watchdog.py`, started by `main.py`) ends runs nothing will finish (`not_delivered`, `timed_out`, `interrupted`). Lists every event, handler, part and error code, including the ones later phases build. |
| **[Theme System](./docs-internal/theme_system.md)** | 12-way visual theme system — 2 base (light, dark) + 5 utopian (renaissance, greek, edo, steampunk, atomic) + 5 dystopian (cyber, wasteland, rot, plague, surveillance) — driven by `<html data-theme>` + per-theme CSS files in `client/src/themes/`. Token taxonomy (surface / fg / border / accent / typography / motion) in **hex + `color-mix()`**; `@theme inline` bridge maps `--color-X: var(--X)` (no `hsl()` wrapper); per-theme files own the colour hex, base.css owns the shared `--tint-*` alpha scale; action role tokens (incl. `-ink` readable text), **per-theme `--code-*` syntax tokens** (tier 6 — the code editor, console/output JSON viewers, and chat code blocks paint in each theme's own palette: keyword→trigger, string→success, number→agent, function→model, on an adaptive dark/light code surface; replaced the global dracula `--prism-*` block + dead `getPrismTokenCSS`; the `OutputPanel` JSON viewer reads the same vars), per-theme `--pulse-keyframe` animation system + `.opencompany-*` helpers in `animations.css`, decorative-layer wrappers (`.app-frame` / `.canvas-host` / `.modal-frame`), per-component decorative ornaments (panel textures + canvas decorations + node pseudo-element overlays + theme-specific keyframes), **canvas-node visual contract via `--node-color` CSS custom property** (no inline `background` / `border` on node components — base.css + per-theme CSS owns visuals; `NodeStyle` helper type at [types/NodeTypes.ts](./client/src/types/NodeTypes.ts) makes the inline custom-prop typecheck-clean), **`--node-pulse-color` separate from `--node-color`** so executing-node glow uses each theme's highest-contrast accent regardless of plugin accent (Cyber neon cyan, Surveillance REC red, Renaissance ultramarine, etc.), **`data-page-hidden` animation pause** (set by `app/usePageActivitySync.ts`, mounted in `AppShell` on both screens, while the tab is hidden or the window is blurred; base.css declares `html[data-page-hidden] *, *::before, *::after { animation-play-state: paused !important }` to prevent compositor stall on tab return), **per-theme icon glyph system** (290 inline SVG glyph strings across 29 keys × 10 themes via [themedGlyphs.ts](./client/src/assets/icons/themedGlyphs.ts) + theme-aware [NodeIcon.tsx](./client/src/assets/icons/NodeIcon.tsx)), **per-theme canvas-grid + custom cursors** via `--canvas-grid` / `--cursor-default` slots, **decorative HTML primitives** (`<SvgFilterDefs>` mounting `#ink-blot` / `#noise` / `#crt` filter IDs at app root, `<DropCap>` wrapper for Renaissance ornament rule), **parameter panel migrated to Tailwind tokens** (no `useAppTheme()` reads; section headers carry the display-typography triplet; raw `<Button>` swapped to `<ActionButton intent>`), **per-theme scrollbar webkit rules** in all 12 themes, 9-event WebAudio sound system (10 packs via `--sound-pack` token + `useSound()` hook + global hover delegate + sonner toast monkey-patch + `withSound()` HOC + `Sounds.unlock()` gesture-unlock for AudioContext autoplay-policy compliance), `@media (prefers-reduced-motion: reduce)` accessibility, 30 ms throttle on `type` / `hover`, migration recipe, anti-patterns. Read this before adding a new theme, migrating a component to the new contract, or adding a canvas-node component. |
| **[Design System Bundle](./docs-internal/design-system/IMPLEMENTATION.md)** | Canonical, vendored design-system reference (extracted from this repo; see its `IMPLEMENTATION.md` for the contract). Tokens are **hex + `color-mix()`** (`tokens/{colors,typography,spacing,motion,fonts,base}.css`) — the format the live codebase standardizes on. Includes 34 reference components (`components/`, inline-style reference only — recreate in Tailwind/shadcn idioms), full-app UI kit (`ui_kits/opencompany/index.html`), 12-theme docs (`guidelines/THEMES.md`), **the merged handoff brief (`HANDOFF.md` — glow-ladder traps, step-edge contract, toolbar layout traps, canvas-layer asymmetries, plus the recorded product amendments)**, and **the panel×theme fidelity target (`reference-mockup/Panel Theme Matrix.dc.html` — 17 panels × 12 themes, open in a browser)**. `reference/themes/` no longer vendors CSS snapshots — it points at the authoritative [client/src/themes/](./client/src/themes/). Copy token values verbatim; do not re-derive by eye. Pairs with [Theme System](./docs-internal/theme_system.md). |
| **[Schema Source of Truth RFC](./docs-internal/ARCHIVE/schema_source_of_truth_rfc.md)** | *Archived — shipped.* Backend is SSOT for node schemas, visual metadata, handlers, palette metadata, icons; `VITE_NODESPEC_BACKEND` defaults on and the frontend node definitions are gone. Plugin pattern: one `BaseNode` subclass in `server/nodes/<group>/<plugin>/__init__.py`. Wire format: `asset:<key>` / `<lib>:<brand>` / URL / emoji. Endpoint: `/api/schemas/nodes/{type}/spec.json`. Current reference: [Plugin System](./docs-internal/plugin_system.md). |
@@ -991,7 +992,7 @@ const [selectedConsoleId, setSelectedConsoleId] = useState<string>('');
```
#### Chat Message Persistence
-Chat messages are persisted to SQLite database and survive server restarts. A workflow's session id is its workflow id (`"default"` with no workflow open), and Home shows the same thread as Talk on the employee's page. Every row is written through [`services/chat_thread.py`](./server/services/chat_thread.py) (`record_chat_message` / `clear_chat_thread`), which stamps the session's live generation and broadcasts the identity-only `chat.updated`; agents answer into the thread through the `chatReply` ("Reply in Chat") node. The thread lives as long as the generation: a workflow Reset (every restart, Home's Apply and Turn on Talk included) clears it through the `chatTrigger` / `chatReply` `reset_execution_state` hook, in the same Reset in which the Context node forgets the conversation, and deleting the workflow deletes it (a workflow-deleted hook the `chatReply` plugin registers).
+Chat messages are persisted to SQLite database and survive server restarts. A workflow's session id is its workflow id (`"default"` with no workflow open), and Home shows the same thread as Talk on the employee's page. Rows are written through [`services/chat_thread.py`](./server/services/chat_thread.py) (`record_chat_message` / `clear_chat_thread`) and, for the owner's messages and the chat runs they start, [`services/chat/ledger.py`](./server/services/chat/ledger.py); both stamp the session's live generation, append the row after the session's active leaf (one chain, `chat_threads`), and broadcast the identity-only `chat.updated`; agents answer into the thread through the `chatReply` ("Reply in Chat") node, as the run's reply when a chat run started it. Runs, their events and the wire contract are in [Chat Protocol](./docs-internal/chat_protocol.md). The thread lives as long as the generation: a workflow Reset (every restart, Home's Apply and Turn on Talk included) clears it through the `chatTrigger` / `chatReply` `reset_execution_state` hook, in the same Reset in which the Context node forgets the conversation, and deleting the workflow deletes it (a workflow-deleted hook the `chatReply` plugin registers).
**Database Model** (`server/models/database.py`):
```python
@@ -1002,21 +1003,32 @@ class ChatMessage(SQLModel, table=True):
execution_id: Optional[str] # the generation (root_execution_id) it was written in
role: str = Field(max_length=20) # 'user' or 'assistant'
message: str = Field(max_length=50000)
+ uid: Optional[str] # stable wire id: m_... (owner), a_<run id> (a run's reply); unique
+ parent_uid: Optional[str] # the message it follows (branches link here)
+ run_id: Optional[str] # the chat run it started or answers (models/chat.py)
+ kind: str # text / report / action / notice
+ status: str # complete / stopped / error
+ parts: Dict; attachments: list; meta: Dict # JSON
created_at: datetime
```
-**WebSocket Handlers** (`server/routers/websocket.py`):
+**WebSocket Handlers** (`server/services/chat/handlers.py`, registered by `services/chat/__init__.py`):
| Handler | Description |
|---------|-------------|
| `send_chat_message` | Save the message and dispatch it to the workflow's chatTriggers. A workflow session answers `delivery: "now"` (running / starting / resuming) or `"queued"` (paused / pausing, runs on Resume); in any other state `not_running`, saving nothing |
| `get_chat_messages` | A session's messages, oldest first: `{id, role, message, timestamp, run_key}` (UTC-offset timestamps; `run_key` = the generation). The latest generation only (none after a Reset), unless `all_generations` (Home's thread; a Reset clears the thread, so only a row from before a Start shows beside the live ones) |
| `clear_chat_messages` | Clear a session's chat, every generation; for a workflow's session its agents forget the conversation too (`chat_thread.clear_chat_session` tells the listeners registered with `register_chat_cleared_listener`: the Context plugin clears the workflow's conversations) |
| `save_chat_message` | Save one message with the given role |
+| `chat_subscribe` / `chat_unsubscribe` | Start / stop receiving a session's `chat_run_event` frames; subscribe answers the live runs' snapshots |
+| `get_chat_run` | One run's snapshot |
+
+Every chat command refuses the internal worker socket, and a workflow's session answers only the workflow's owner (`services/chat/access.py`).
**Database Methods** (`server/core/database.py`):
-- `add_chat_message(session_id, role, message, execution_id=None)` - Add message to database
-- `get_chat_messages(session_id, limit=None, execution_id=None)` - Newest `limit` messages, oldest first, optionally of one generation
-- `clear_chat_messages(session_id)` - Delete all messages for session
+- `add_chat_message(session_id, role, message, execution_id=None, *, uid=None, run_id=None, ...)` - Append a message after the session's active leaf; returns the row (a `uid` already saved returns that row)
+- `append_chat_row(session, ...)` - The same, inside a caller's `reserved_session()` (the run ledger admits a message and its run in one transaction)
+- `get_chat_messages(session_id, limit=None, execution_id=None)` - Newest `limit` messages, oldest first, optionally of one generation (`read_chat_messages` raises instead of returning `[]` on failure)
+- `clear_chat_messages(session_id)` - Delete all messages for session, with its runs, run parts and thread state
#### Console Log Persistence
Console logs are persisted to SQLite database and loaded on page refresh.
@@ -1038,7 +1050,8 @@ Console logs are persisted to SQLite database and loaded on page refresh.
| `client/src/components/ui/ConsolePanel.tsx` | Main panel component with chat/console/terminal tabs |
| `server/models/database.py` | ChatMessage and ConsoleLog SQLModel definitions |
| `server/core/database.py` | Chat message and console log CRUD methods |
-| `server/routers/websocket.py` | WebSocket handlers for chat and console operations |
+| `server/routers/websocket.py` | WebSocket handlers for console operations |
+| `server/services/chat/` | Chat runs, run events and the chat WebSocket handlers ([Chat Protocol](./docs-internal/chat_protocol.md)) |
### Per-Workflow Workspace Directory
Each workflow execution gets a persistent workspace directory where nodes save output files and agents such as Coding Agent access them through connected filesystem tools.
@@ -2503,7 +2516,7 @@ Updated `client_left` and presence handlers use `has_real_android_devices()` ins
| **WhatsApp** | `whatsapp_status`, `whatsapp_connected_phone`, `whatsapp_qr`, `whatsapp_send`, `whatsapp_start`, `whatsapp_restart`, `whatsapp_groups`, `whatsapp_group_info`, `whatsapp_chat_history`, `whatsapp_newsletters`, `whatsapp_rate_limit_get`, `whatsapp_rate_limit_set`, `whatsapp_rate_limit_stats`, `whatsapp_rate_limit_unpause`, `whatsapp_mark_read`, `whatsapp_typing`, `whatsapp_presence`, `whatsapp_stop`, `whatsapp_diagnostics` |
| **Telegram** | `telegram_connect`, `telegram_disconnect`, `telegram_status`, `telegram_send`, `telegram_reconnect`, `telegram_get_me`, `telegram_get_chat` |
| **Workflow Storage** | `save_workflow`, `get_workflow`, `get_all_workflows`, `delete_workflow` |
-| **Chat Messages** | `send_chat_message`, `get_chat_messages`, `clear_chat_messages`, `save_chat_message` |
+| **Chat Messages** | `send_chat_message`, `get_chat_messages`, `clear_chat_messages`, `save_chat_message`, `chat_subscribe`, `chat_unsubscribe`, `get_chat_run` (see [chat_protocol.md](./docs-internal/chat_protocol.md)) |
| **Console/Terminal** | `get_console_logs`, `clear_console_logs`, `get_terminal_logs`, `clear_terminal_logs` |
| **User Skills** | `get_user_skills`, `get_user_skill`, `create_user_skill`, `update_user_skill`, `delete_user_skill` |
| **Built-in Skills** | `get_skill_content`, `save_skill_content`, `scan_skill_folder`, `list_skill_folders` |
@@ -2528,6 +2541,7 @@ Updated `client_left` and presence handlers use `has_real_android_devices()` ins
| `employee_lifecycle` | CloudEvents v1.0 envelope (type=`com.opencompany.employee.{hired,updated,removed}`, subject = workflow id) — Home's team list; `updated` is coalesced to one per second per employee, control changes go out at once |
| `approval_lifecycle` | CloudEvents v1.0 envelope (type=`com.opencompany.approval.{requested,decided,expired,cancelled}`) — identity only, never the draft or the recipient; the card refetches via `list_approvals` |
| `chat.updated` | CloudEvents v1.0 envelope (type=`com.opencompany.chat.updated`, data `{workflow_id, session_id, role}`) — sent directly (not `dispatch.emit`) by `services/chat_thread.py` after every chat insert and clear; identity only, Home's thread and the editor's chat pane refetch via `get_chat_messages` |
+| `chat_run_event` | CloudEvents v1.0 envelope (type=`com.opencompany.chat.run.<suffix>`, id `<run id>:<seq>`, subject = run id) — AG-UI run events, sent by `services/chat/hub.py` only to sockets subscribed to the session through `chat_subscribe`, never on the broadcast; see [chat_protocol.md](./docs-internal/chat_protocol.md) |
| `workflow_lifecycle` | CloudEvents v1.0 envelope for workflow stages (`deployment.*`, `lock.*`, `execution.*`, `imported`, `renamed`, `created`, `deleted`) — other tabs invalidate their workflow lists |
| `deployment_snapshot` | CloudEvents v1.0 envelope (type=`workflow.deployment.snapshot`) — pushed once per WS connect from `broadcaster._send_deployment_snapshot`; lets the FE reconcile stale `deploymentStatus.isRunning=true` after a backend restart wiped `DeploymentManager._deployments`. Empty list is meaningful (forces reset). |
| `variable_update` | Single variable value change |
@@ -2788,7 +2802,7 @@ This function:
- **Typed SDK error → `LLMError` → `NodeUserError`**: Native providers normalize SDK failures into structured `LLMError` values carrying category, retryability, HTTP status, provider code, request ID, and retry-after metadata. `ChatUnifier` translates them at the execution boundary into a user-safe `NodeUserError`; `BaseNode.execute()` logs that at WARN with one line and no stack trace. Unexpected exceptions retain the full-traceback path. In-process agent turns retry only errors marked retryable, while Temporal owns activity retries and disables the inner retry loop.
- **Plugin extraction (Wave 11.I)**: every plugin's WS handlers, OAuth client, FastAPI router, and lifecycle service live entirely under `nodes/<plugin>/`. Eight plugin domains migrated this round (whatsapp / twitter / google-workspace / android / browser / email / code / credential-validation-scaffold for maps+apify+ollama+lmstudio) follow the telegram pattern. `routers/websocket.py` shrunk from ~3,785 to ~2,977 LOC (-808). Three plugin routers (`routers/twitter.py`, `routers/google.py`, `routers/android.py`) moved into `nodes/<plugin>/_router.py` and mount via the plugin-router loop in `main.py`; the explicit `app.include_router(<plugin>.router)` calls and `from routers import <plugin>` imports in `main.py` are gone. **`register_router(router, name=...)` is the new sibling helper to `register_ws_handlers` (same file: `services/ws_handler_registry.py`)** — the generic registry set has grown since (the Plugin System row at the top of this file gives the `grep` for the live list). **`Credential.validate(data) -> dict` + `Credential._probe(api_key) -> ProbeResult` is the new shared validator scaffold** in `services/plugin/credential.py` — replaces the per-router `_SPECIAL_PROVIDER_VALIDATORS` dict; one `_LLMApiKey._probe` method serves all 10 cloud LLM providers, dedicated `_probe` overrides handle Maps + Apify, `_LocalLLM.validate` overrides for Ollama / LM Studio's 2-storage edge case. **`tests/test_plugin_self_containment.py` locks the contract** with 10 invariant classes (forbidden-imports / no-router-outside-nodes / per-plugin self-registration / registry-API sanity / stale-paths-absent / main.py-does-not-mount / WS_HANDLERS-non-empty / plugin-folder-has-node-file / typed-event-factories / package-imports-cleanly); same `inspect.getsource` introspection style as `test_credential_broadcasts.py`. Wire format unchanged — frontend WS message-type strings (`whatsapp_status`, `twitter_oauth_login`, `google_oauth_status`, `android_relay_connect`, etc.) and HTTP route paths (`/api/twitter/callback`, `/api/google/callback`, `/api/android/*`) are byte-identical post-migration.
- **Typed CloudEvents factory pattern for new broadcasts** (b6aecd3): every new server→FE lifecycle broadcast follows the recipe locked by `broadcast_credential_event`, `broadcast_agent_progress`, and `broadcast_agent_capability`. (1) Add a typed factory — convention: `source = "opencompany://services/<area>"`, reverse-DNS `type = "com.opencompany.<area>.<event>"`, and `subject = <exact primary entity id>`. Put snake_case workflow/execution scope inside `data`; CloudEvents extension attribute names MUST be lowercase-alphanumeric, so the historical top-level `workflow_id`/`trigger_node_id`/`correlation_id` fields are compatibility debt and are forbidden on new contracts. (2) Add a broadcaster wrapper that emits `broadcast({type: "<wire_key>", data: event.model_dump(mode="json", exclude_none=True)})`. (3) FE validates `specversion`, ID/source/type, subject ownership, and event-specific payload invariants, deduplicates by `(source,id)`, then writes only the exact scoped entity. (4) Add contract tests in `test_events.py` and keep `tests/test_status_broadcasts.py`'s raw-callsite allowlist empty. Raw `node_status` remains allowed only as the established high-frequency latest-state/reconnect projection paired with a typed lifecycle occurrence; it is not the event of record.
-- **Plugin-folder location for plugin-specific CloudEvents factories** (RFC §6.4): when a typed event is owned by one plugin (telegram, whatsapp, android, email, google, master_skill, webhook_trigger, chat_trigger, agent), the factory + `broadcast_<event>` wrapper live in [`server/nodes/<plugin>/_events.py`](./server/nodes), NOT on `WorkflowEvent` / `StatusBroadcaster` directly. Telegram's [`_events.py`](./server/nodes/telegram/_events.py) is the canonical template — `broadcast_telegram_status` constructs the envelope via a plain factory function and calls `get_status_broadcaster().broadcast(...)` itself. Latest example: [`server/nodes/skill/master_skill/_events.py`](./server/nodes/skill/master_skill/_events.py) (`broadcast_skill_lifecycle("created" \| "updated" \| "deleted" \| "content_saved", name=..., **data_extra)`) — replaces four raw-dict broadcasts in `handle_create_user_skill` / `handle_update_user_skill` / `handle_delete_user_skill` / `handle_save_skill_content` (the last one was previously silent). FE routes wire key `skill_lifecycle` to invalidate `userSkills` + `folderSkills` queries and drop the `skillContent` cache on delete + content_saved so the Master Skill panel refreshes live across every connected client. Cross-cutting events (credential, workflow_lifecycle, agent_progress, agent_capability, node_parameters_updated) keep their `WorkflowEvent` classmethods + `StatusBroadcaster` wrappers — only plugin-scoped events move to the plugin folder. A core service's own event keeps its factory beside it, so core never imports a plugin `_events.py`: `services/workflow_ops.py` (`workflow_ops_apply`) and `services/chat_thread.py` (`chat.updated`).
+- **Plugin-folder location for plugin-specific CloudEvents factories** (RFC §6.4): when a typed event is owned by one plugin (telegram, whatsapp, android, email, google, master_skill, webhook_trigger, agent), the factory + `broadcast_<event>` wrapper live in [`server/nodes/<plugin>/_events.py`](./server/nodes), NOT on `WorkflowEvent` / `StatusBroadcaster` directly. Telegram's [`_events.py`](./server/nodes/telegram/_events.py) is the canonical template — `broadcast_telegram_status` constructs the envelope via a plain factory function and calls `get_status_broadcaster().broadcast(...)` itself. Latest example: [`server/nodes/skill/master_skill/_events.py`](./server/nodes/skill/master_skill/_events.py) (`broadcast_skill_lifecycle("created" \| "updated" \| "deleted" \| "content_saved", name=..., **data_extra)`) — replaces four raw-dict broadcasts in `handle_create_user_skill` / `handle_update_user_skill` / `handle_delete_user_skill` / `handle_save_skill_content` (the last one was previously silent). FE routes wire key `skill_lifecycle` to invalidate `userSkills` + `folderSkills` queries and drop the `skillContent` cache on delete + content_saved so the Master Skill panel refreshes live across every connected client. Cross-cutting events (credential, workflow_lifecycle, agent_progress, agent_capability, node_parameters_updated) keep their `WorkflowEvent` classmethods + `StatusBroadcaster` wrappers — only plugin-scoped events move to the plugin folder. A core service's own event keeps its factory beside it, so core never imports a plugin `_events.py`: `services/workflow_ops.py` (`workflow_ops_apply`), `services/chat_thread.py` (`chat.updated`) and `services/chat/events.py` (the owner's chat message to the chatTrigger listeners, which moved there from the plugin, and `chat_run_event`).
- **Operator-metadata fields in credential panels** (f746ddf): when a credential panel needs a non-credential follow-up field (e.g. Telegram bot token + owner chat id), declare it as a second entry in the provider's `fields` array in `server/config/credential_providers.json`. The first field stays the validate/connect target; subsequent fields render below as plain text inputs via `SecondaryFieldRow` in `ApiKeyPanel.tsx` (shadcn `Input` + `Label` + `<ActionButton intent="save">`). Each field carries an optional `help` slot (`FieldDef.help` / `ServerFieldDef.help`) for always-visible explanatory text under the input. Save writes via `panel.actions.save(field.key, value)` → `auth_service.store_api_key` (which is permissive about provider string). The credential class must declare the new key in `extra_fields` (`Credential` subclass attribute) so the storage layer accepts it. Reference: `telegram_owner_chat_id` field.
- **Telegram owner detection — three layers** (f746ddf): the bot owner is captured by THREE independent paths so the realistic setup flows all converge to a working state. (1) **Explicit field**: secondary `FieldDef` in the credentials modal lets the user paste their Telegram user ID directly (recommended; works without DM'ing the bot). (2) **Pre-poll peek** in `connect()` via `_capture_owner_from_pending_updates()` — calls `bot.get_updates(timeout=0)` BEFORE `start_polling(drop_pending_updates=True)` discards the queue, scans for any historical private DM, captures + persists atomically. (3) **Atomic write-through in `_on_message_received`** — persists FIRST, sets in-memory ONLY on success. Invariant: "in-memory has owner ⇒ DB has owner" so a process restart can re-capture cleanly. Failure logged at `ERROR` with `exc_info=True` (was `WARNING` previously, masking persist failures). The lazy fallback in `telegram_send.py` (read DB → `service.set_owner`) sits on top of all three.
- **Supervisor Job Object loud-failure (cli/tree.py + cli/supervisor.py)** (8a64eb9): the Windows process-tree mechanism depends entirely on a Job Object with `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE` (no fallback — `start_new_session` is POSIX-only). `pywin32>=308` is a hard supervisor dependency on Windows; the declaration in the root `pyproject.toml` (where `opencompany-cli` lives — `cli/pyproject.toml` was removed as a stale duplicate) carries an inline comment so future maintainers don't drop it. `_JobObject.__init__` splits failures into explicit `ImportError` (with reinstall command) and `Exception` (with `repr`) branches — both write to `stderr` instead of swallowing. `_JobObject.add()` verifies enrollment via `win32job.IsProcessInJob(handle, self._handle)` after assignment; on mismatch it queries `IsProcessInJob(handle, None)` so the warning explicitly names the wrapper-job hypothesis (npm / bun / conhost wrappers occasionally place us in their own non-nesting Job). `supervisor._spawn_once` checks `add_to_job` return value and emits a yellow `WARN: pid=N not enrolled in Job Object` per child if False. Without these guards, a stale pywin32 install or a wrapping Job Object silently leaks orphan Python processes on every Ctrl-C and they accumulate to the point of holding SQLite locks that block subsequent backend startup.
diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.md
index 8e04868c..3c71dffa 100644
--- a/docs-internal/TEMPORAL_ARCHITECTURE.md
+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md
@@ -704,7 +704,7 @@ The Temporal binary + persistence are managed in-process by the plugin-folder pa
**Months-long durability contract**: running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. None of the mechanisms below carries a `workflow.patched` marker: the lifetime-cap removal and the poll-interval floor apply unconditionally, including when an older history is replayed.
-- **No lifetime caps on new child runs.** Trigger/cron-spawned `MachinaWorkflow` runs, agent children, and delegated-task runners previously carried 1-2h `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so any pause longer than the cap silently terminated the run (and a timed-out delegated runner skipped its compensation: leaked permit + stuck task row). New executions start children unbounded (no patch marker was retained for this change). The live `workflow.patched` markers are `machina-conditional-edges-v1`, `machina-run-record-v1` and `workspace-node-cancellation-v1` in `services/temporal/workflow.py`, `machina-trigger-listener-node-filter` in `services/temporal/trigger_listener_workflow.py`, and `workspace-task-activity-policy-v1` in `services/temporal/agent_workflow.py`; the list grows, so check it with `grep -rn "workflow.patched(" server/services/temporal`. `machina-run-record-v1` gates the `workflow_runs.record_completion` activity that trigger-spawned runs schedule when they finish, for Normal mode's "done today" (see [normal_mode.md](./normal_mode.md#done-today)). The two `workspace-*` markers are described under [Direct Workspace tasks](#direct-workspace-tasks). Liveness is the activity layer's job: node activities heartbeat every 30s against a 2-minute `heartbeat_timeout`; their `start_to_close` is a generous 24h ceiling, not 10 minutes. The subagent-permit wait uses `PERMIT_WAIT_RETRY` (unlimited attempts) so a queued delegation waits as long as admission takes instead of failing after ~3h.
+- **No lifetime caps on new child runs.** Trigger/cron-spawned `MachinaWorkflow` runs, agent children, and delegated-task runners previously carried 1-2h `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so any pause longer than the cap silently terminated the run (and a timed-out delegated runner skipped its compensation: leaked permit + stuck task row). New executions start children unbounded (no patch marker was retained for this change). The live `workflow.patched` markers are `machina-conditional-edges-v1`, `machina-run-record-v1`, `machina-chat-run-v1` and `workspace-node-cancellation-v1` in `services/temporal/workflow.py`, `machina-trigger-listener-node-filter` in `services/temporal/trigger_listener_workflow.py`, and `workspace-task-activity-policy-v1` in `services/temporal/agent_workflow.py`; the list grows, so check it with `grep -rn "workflow.patched(" server/services/temporal`. `machina-run-record-v1` gates the `workflow_runs.record_completion` activity that trigger-spawned runs schedule when they finish, for Normal mode's "done today" (see [normal_mode.md](./normal_mode.md#done-today)). `machina-chat-run-v1` gates the `chat_run.start` / `chat_run.finish` activities a run spawned by the owner's chat message schedules to claim and finish its chat run; the run id is read only from an event whose source is `opencompany://services/chat` (see [chat_protocol.md](./chat_protocol.md#runs)). The two `workspace-*` markers are described under [Direct Workspace tasks](#direct-workspace-tasks). Liveness is the activity layer's job: node activities heartbeat every 30s against a 2-minute `heartbeat_timeout`; their `start_to_close` is a generous 24h ceiling, not 10 minutes. The subagent-permit wait uses `PERMIT_WAIT_RETRY` (unlimited attempts) so a queued delegation waits as long as admission takes instead of failing after ~3h.
- **History-pressure continue-as-new everywhere.** Temporal terminates any workflow around ~51,200 history events. `WorkflowControlWorkflow` (which multiplexes all of a deployment's triggers into one history) now rolls over on `is_continue_as_new_suggested()` / a 10K-event soft cap, carrying trigger specs, per-trigger provider `seen_ids` (written back into the spec after every poll cycle), queued push events, the bounded dedup baseline, and the control state — a rollover works mid-pause too, since a paused controller still accretes signal history. `TriggerListenerWorkflow`/`PollingTriggerWorkflow` gained the same pressure check (the old `_processed_count >= 16_000` gate was unreachable: real spawns cost ~15-25 events each, and polling counted only emitted events while a quiet mailbox burned ~11 events/cycle — dead in ~3 days at the 60s default). Poll intervals are clamped to a 30s floor (`_MIN_POLL_INTERVAL_S` in `polling_trigger_workflow.py` and `workflow_control_workflow.py`), with no patch gate. Because run ids change on rollover, **controller handles are addressed by workflow id only, never run_id-pinned** (`_controller_handle`, manager `register_trigger`).
- **dispatch.emit controller narrowing.** Controllers advertise their push event types via the `ControlEventTypes` keyword-list Search Attribute (upserted as triggers register); `dispatch.emit` skips controllers with no matching trigger instead of signalling every running controller with every platform event (each unmatched signal was ~4 immutable history events — one busy deployment burned every other controller's rollover budget). Controllers without the attribute (pre-upgrade histories) keep match-all behaviour.
- **Boot-time reconcile** (`reconcile_active_controls_on_boot`, called from the lifecycle module after workers start): runs the lazy `_reconcile_control` over every active control row, converges `starting` rows a crash left behind (controller alive with triggers registered → `running`; alive-but-empty for a graph that declares triggers → `failed` + controller closed; vanished → `failed`), and re-arms the process-local half of running/paused generations from the persisted graph snapshot — DeploymentManager state, in-process collectors for non-canary trigger types, cron pause posture. Idempotent by construction (controller `register_trigger` keyed by listener id, legacy starts use `USE_EXISTING`, cron creation preserves server-owned pause state).
diff --git a/docs-internal/chat_protocol.md b/docs-internal/chat_protocol.md
index a9a2fb56..e9b1fcd6 100644
--- a/docs-internal/chat_protocol.md
+++ b/docs-internal/chat_protocol.md
@@ -10,7 +10,12 @@ AG-UI JSON.
Status: being built on `feature/chat-genui`; the design and the owner's decisions are in the plan recorded at
`~/.claude/plans/analyze-the-new-generative-zazzy-cosmos.md`. Change this document in the same commit as any change
-to the shapes below, and bump `protocol_version` when an existing field changes meaning.
+to the shapes below, and bump `protocol_version` (returned by `get_chat_messages`) when an existing field changes
+meaning.
+
+Built so far: runs and their storage, the lifecycle events (`started`, `finished`, `failed`), subscriptions,
+`get_chat_messages` v2, `get_chat_run`, the watchdog and the MachinaWorkflow patch (see [Runs](#runs)). Every other
+event, handler and part below is the contract the later phases build to.
## Concepts
@@ -42,8 +47,15 @@ internal socket, load the workflow, and compare its owner with the socket's exec
attributes.
- `seq` increases by one per run. The client drops duplicates `(run_id, seq)` and asks for a snapshot on a gap.
- `hub_epoch` changes when the server process restarts. A changed epoch means the client must resync.
-- Each subscriber has a bounded queue. Under pressure the hub merges consecutive text deltas of one segment, never
- drops lifecycle events, and finally sends `custom` `opencompany.resync`.
+- `chat_subscribe` answers with the session's live runs as snapshots, each with the `seq` it reflects. The hub is
+ read before the database, so an event published meanwhile carries a higher `seq` than its snapshot: drop events
+ with `seq` at or below the snapshot's, and buffer events that arrive before the response.
+- Each subscriber has a bounded queue (`hub.subscriber_queue_size` in `server/config/chat_defaults.json`). A socket
+ that falls that far behind has its queue replaced by one `custom` `opencompany.resync` per session it follows
+ (`data: {session_id, hub_epoch, name, value}`, no `run_id`); the client then takes fresh snapshots. Merging
+ consecutive text deltas of one segment comes with streaming.
+- Delivery is in-process: events published by an activity on a worker in another process do not reach sockets here
+ yet (a relay is planned).
- `chat.updated` remains an identity-only broadcast (`{workflow_id, session_id, role}`) that tells every open thread
to refetch `get_chat_messages`.
@@ -89,7 +101,10 @@ streamed.
| `interrupt` | Drafts wait for the owner. `outcome.interrupts: [{id, reason: "tool_call", message, tool_call_id, response_schema, expires_at}]`; `id` is the approval id. |
| `stopped` | The owner pressed Stop. OpenCompany extension; AG-UI defines only `success` and `interrupt`. |
-Never-picked-up runs end with `run.failed` code `not_delivered`; a Reset ends live runs with code `reset`.
+Runs nothing will finish end with `run.failed`: code `not_delivered` (never picked up, or the employee stopped
+first), `timed_out` (running longer than `runs.max_running_s`), `interrupted` (its workflow closed without finishing
+it and no reply was saved; with a saved reply it finishes instead). A Reset ends live runs with code `reset`, the
+owner's Clear with `cleared`, and both delete them.
## Handlers
@@ -100,10 +115,11 @@ All are WebSocket request/response handlers with snake_case payloads. Failures a
| Handler | Request | Response |
|---|---|---|
-| `send_chat_message` | `{session_id, message, role: "user", timestamp?, client_message_id?, attachments?: FileRef[], options?: {web?}, ui_event?: {part_id, element_id, action, params}}` | `{success, message_id, run_id, delivery: "now" \| "queued", timestamp}` |
-| `get_chat_messages` | `{session_id, limit?, all_generations?}` | `{success, messages, thread: {active_leaf_id, revision}, active_runs}` |
-| `chat_subscribe` / `chat_unsubscribe` | `{session_id}` | `{success, hub_epoch, active_runs: [RunSnapshot]}` |
-| `get_chat_run` | `{run_id}` | `{success, run: RunSnapshot}` |
+| `send_chat_message` | `{session_id, message, role: "user", timestamp?, client_message_id?, attachments?: FileRef[], options?: {web?}, ui_event?: {part_id, element_id, action, params}}` | `{success, message_id, run_id, delivery: "now" \| "queued", timestamp}`; `run_id` is null when the message starts no run |
+| `get_chat_messages` | `{session_id, limit?, all_generations?}` | `{success, protocol_version, session_id, messages, thread: {active_leaf_id, revision}, active_runs: [RunSnapshot]}` |
+| `chat_subscribe` | `{session_id}` | `{success, session_id, hub_epoch, active_runs: [RunSnapshot]}` |
+| `chat_unsubscribe` | `{session_id}` | `{success, session_id, hub_epoch}` |
+| `get_chat_run` | `{run_id}` | `{success, run: RunSnapshot}`, or `not_found` |
| `stop_chat_run` | `{run_id}` or `{session_id}` | `{success, run_id, state}` |
| `edit_chat_message` | `{message_id, text, attachments?, expected_revision}` | `{success, message_id, run_id}` |
| `regenerate_chat_reply` | `{message_id, expected_revision}` | `{success, run_id}` |
@@ -112,10 +128,15 @@ All are WebSocket request/response handlers with snake_case payloads. Failures a
| `get_chat_context` | `{session_id}` | `{success, commands, suggestions, capabilities, genui_catalog, limits, ask_first: {value, editable, replies_gated}}` |
| `clear_chat_messages` | `{session_id}` | `{success}`; also clears runs, parts, snapshots, notes and feedback, and makes the employee forget the conversation |
-`send_chat_message` with the same `client_message_id` returns the run the first call created.
+`send_chat_message` with the same `client_message_id` (1 to 100 letters, digits or `_.:-`) returns the message
+and run the first call created, and dispatches nothing again.
-**RunSnapshot** is the state the client reducer would hold after replaying the run's events:
-`{run_id, kind, state, seq, steps, segments: [{message_id, text, final}], activities, interrupts, started_at}`.
+**RunSnapshot** is the state the client reducer would hold after replaying the run's events
+(`server/services/chat/reducer.py` folds them the same way):
+`{run_id, session_id, workflow_id, kind, state, seq, hub_epoch, user_message_id, reply_message_id, parent_run_id,
+created_at, started_at, finished_at, steps, segments: [{message_id, text, final}], activities, interrupts, outcome,
+result, error, error_code}`. `state` is `queued`, `pending`, `running`, `stopping`, `finished`, `error` or `stopped`.
+A snapshot read after a server restart has its steps but no text segments (text is stored with the reply).
### Approvals (`server/services/approvals/handlers.py`)
@@ -147,10 +168,15 @@ summary adds `kind` (`gate` or `tool_call`), `node_type`, `tool_name`, `tool_cal
```
{id, legacy_id, role, kind, text, message, timestamp, run_key, run_id, parent_id, status, attachments,
- parts, feedback, siblings: {index, count, ids}, editable}
+ parts, feedback, siblings: {index, count, ids}, editable, client_message_id?}
```
+- `id` is the message's stable id (`m_…` for the owner's, `a_<run id>` for a run's reply; rows saved before chat
+ runs are `m<number>`). `legacy_id` is the database row number.
- `message` equals `text`; it is kept for older readers.
+- `client_message_id` is echoed on the owner's messages that carried one, so an optimistic row and its saved row
+ keep one key.
+- Until branches land, `siblings` always holds the message alone and `editable` is false.
- `kind`: `text`, `report` (Post to Talk: no parent, not editable), `action` (a button press), `notice`.
- `status`: `complete`, `stopped`, `error`.
- `siblings` are branches for owner messages and versions for replies.
@@ -197,6 +223,33 @@ Approvals are joined live from the approvals store; the part only names them.
json-render renders each element inside its own error boundary, so one failing element renders nothing instead of
breaking the reply.
+## Runs
+
+`server/services/chat/` (never imports `nodes/`):
+
+- **Admission** (`ledger.admit_message`). `send_chat_message` writes the owner's message and its run in one write
+ transaction reserved before its first read (`Database.reserved_session`, SQLite `BEGIN IMMEDIATE`), after checking
+ the lane; a partial unique index on `chat_runs` enforces it too. A message starts a run only when a chat trigger in
+ the deployed graph (the control generation's `graph_snapshot`) accepts its session, judged by that trigger's own
+ filter (`event_waiter.build_filter`). Otherwise it is saved and dispatched without a run, like the editor's
+ `"default"` session, which keeps its unscoped delivery.
+- **Dispatch.** The `chat_message_received` event (`services/chat/events.py`, source `opencompany://services/chat`)
+ has the run id as its CloudEvent id and carries `message_id` and `run_id` in `data`, so the listener's child run
+ id is `<slug>-<trigger label>-<run id>`. It is never broadcast.
+- **Start and finish.** MachinaWorkflow, behind the `machina-chat-run-v1` patch, reads the run id only from an event
+ with that source and type, claims the run (`chat_run.start`: `pending` or `queued` to `running`, recording the
+ Temporal workflow and run ids) once the firing trigger's output is stored, passes `run_scope {run_id, session_id}`
+ to every node context, and finishes it at its single exit (`chat_run.finish`). With several chat triggers in one
+ graph, the first to claim tracks the run and the others run untracked. Only the claimant may finish.
+- **The reply.** `chatReply` with a `run_scope` saves the answer through `ledger.post_reply`: the first reply takes
+ the run's reply id `a_<run id>`, another reply node in the same run `a_<run id>.<n>`, and a retry from the same node
+ saves nothing new. A run whose conversation was reset or cleared posts nothing.
+- **One chain.** Every message is appended after the session's active leaf (`chat_threads`) inside the same reserved
+ transaction, so concurrent writes never fork the thread.
+- **The watchdog** (`services/chat/watchdog.py`, started by `main.py`) sweeps every `runs.watchdog_interval_s` and
+ ends the runs nothing will finish (codes above). A pending run's wait (`runs.pickup_timeout_s`) counts from the
+ later of its creation and the server's start.
+
## Notes to the employee
Things the employee should learn on its next turn are queued in `chat_notes` and prepended to that turn's user
@@ -223,9 +276,12 @@ and server tests alike:
| `run_in_progress` | send, edit, regenerate, switch | A run is live in this session; `run_id` names it. |
| `save_failed` | send | The message could not be saved; nothing was dispatched. |
| `not_running` | send | The employee is not running and cannot queue messages. |
+| `invalid_request` | send, save | A malformed field (`detail` says which): an empty message, a role other than the owner's, or a bad `client_message_id`. |
+| `read_failed` | get_chat_messages, chat_subscribe | The thread could not be read. Never answered as an empty thread. |
+| `not_found` | get_chat_run | No such run. |
| `conflict` | edit, regenerate, switch, set_ask_first | `expected_revision` is stale. |
| `not_editable` | edit, regenerate, decide | The message belongs to an older generation, its prefix was compacted, or an edited argument is not editable. |
| `ui_event_rejected` | send | The element or action does not match the saved spec. |
-| `access_denied` | all | The socket's principal does not own the workflow. |
+| `access_denied` | all | The socket's principal does not own the workflow, or it is the internal worker socket. |
| `too_late` | decide | The Undo or Restore window has passed. |
| `speech_unavailable` | transcribe_audio | No dictation provider has a stored key. |
diff --git a/docs-internal/event_framework.md b/docs-internal/event_framework.md
index 5aed1455..1adef0ad 100644
--- a/docs-internal/event_framework.md
+++ b/docs-internal/event_framework.md
@@ -98,9 +98,16 @@ broadcast semantics) are unchanged. First producer: chat — the panel is
bound to one workflow (`session_id` IS its workflow id), and without the
scope one workflow's chat message fired every deployed workflow's
chatTrigger. The scoping DECISION lives in the core call site
-(`routers/websocket.py:handle_send_chat_message`; the legacy `"default"`
-session stays unscoped) and the narrowing in core dispatch — plugin
-`_events.py` factories only plumb the field of their own wire shape.
+(`services/chat/handlers.py:handle_send_chat_message`; the legacy
+`"default"` session stays unscoped) and the narrowing in core dispatch —
+event factories (the chat one is `services/chat/events.py`, source
+`opencompany://services/chat`) only plumb the field of their own wire
+shape. The chat message is emitted with `broadcast=False` (it carries the
+owner's text), and its CloudEvent id is the chat run it started, so the
+listener's child run id is predictable; chat run events (`chat_run_event`)
+never ride `emit` or the broadcast at all: `services/chat/hub.py` delivers
+them only to sockets subscribed to the session
+([chat_protocol.md](./chat_protocol.md)).
Worker is embedded in the FastAPI process (the `main.py` lifespan schedules
`run_temporal_lifecycle` from `services/temporal/lifecycle.py` as one
diff --git a/docs-internal/node-logic-flows/chat_utility/chatReply.md b/docs-internal/node-logic-flows/chat_utility/chatReply.md
index 6b5d871e..8906637b 100644
--- a/docs-internal/node-logic-flows/chat_utility/chatReply.md
+++ b/docs-internal/node-logic-flows/chat_utility/chatReply.md
@@ -48,8 +48,10 @@ like `console`. It does not carry console's `isConsoleSink` hint.
```ts
// ChatReplyOutput (model_config extra="allow")
{
- posted: boolean; // false when there was nothing to post
- message?: string; // the trimmed text, when posted
+ posted: boolean; // false when there was nothing to post
+ message?: string; // the trimmed text, when posted
+ message_id?: string; // the saved message's id
+ run_id?: string; // the chat run it answered, in a run the owner's message started
}
```
@@ -62,7 +64,12 @@ flowchart TD
C -- yes --> D[Return posted=false: nothing written or broadcast]
C -- no --> E{ctx.workflow_id set?}
E -- no --> F[Raise NodeUserError: save the workflow first]
- E -- yes --> G[services.chat_thread.record_chat_message<br/>session = workflow id, role = assistant]
+ E -- yes --> R{run_scope in the context?}
+ R -- yes --> S{the chat run still exists?}
+ S -- no --> T[Return posted=false: the conversation was reset or cleared]
+ S -- yes --> U[services.chat.ledger.post_reply<br/>id a_run id, linked to the run]
+ U --> J
+ R -- no --> G[services.chat_thread.record_chat_message<br/>session = workflow id, role = assistant]
G --> H{row saved?}
H -- no --> I[Raise RuntimeError: the reply could not be saved]
H -- yes --> J[broadcast chat.updated]
@@ -79,6 +86,15 @@ flowchart TD
nothing to send) posts nothing and succeeds with `posted: false`. The edges
the builders wire into the node carry the same `result.response neq NO_REPLY`
condition, so normally the node does not run then at all.
+- **Chat runs**: in a run the owner's chat message started, MachinaWorkflow
+ passes `run_scope {run_id, session_id}` in the context
+ ([chat_protocol.md → Runs](../../chat_protocol.md#runs)). The reply is then
+ saved through `services.chat.ledger.post_reply` as that run's answer: the
+ first reply takes the run's reply id `a_<run id>`, another reply node in the
+ same run `a_<run id>.<n>`, and a retried step returns the row it saved. When
+ the run no longer exists (a Reset or the owner's Clear removed the
+ conversation it answered) nothing is posted and the node succeeds with
+ `posted: false`.
- **Generation**: `record_chat_message` stamps the row with the latest
control's `root_execution_id` (`chat_thread.chat_execution_id`; none when
nothing was started since the last Reset). The editor's chat pane reads the
@@ -107,8 +123,9 @@ flowchart TD
- **Database writes**: one `chat_messages` row: `session_id` = the workflow id,
`role` = `assistant`, `message` = the trimmed text, `execution_id` = the
- live generation. On a Reset or a workflow delete, every row of the
- workflow's thread is deleted.
+ live generation, appended after the thread's last message; in a chat run,
+ `run_id` and `uid` name the run and its reply. On a Reset or a workflow
+ delete, every row of the workflow's thread is deleted, with its chat runs.
- **Broadcasts**: `chat.updated` (CloudEvents `com.opencompany.chat.updated`,
data `{workflow_id, session_id, role: "assistant"}`), sent directly through
the status broadcaster, not `services.events.dispatch.emit`. Home's thread
@@ -120,7 +137,7 @@ flowchart TD
## External Dependencies
- **Credentials**: none.
-- **Services**: `services.chat_thread` (`record_chat_message`), the database
+- **Services**: `services.chat_thread` (`record_chat_message`), `services.chat.ledger` (`get_run`, `post_reply`), the database
(`services.plugin.deps.get_database`), `StatusBroadcaster`.
- **Python packages**: stdlib only (`json`).
- **Environment variables**: none.
diff --git a/docs-internal/node-logic-flows/workflow_triggers/chatTrigger.md b/docs-internal/node-logic-flows/workflow_triggers/chatTrigger.md
index d29f9ea1..661900a0 100644
--- a/docs-internal/node-logic-flows/workflow_triggers/chatTrigger.md
+++ b/docs-internal/node-logic-flows/workflow_triggers/chatTrigger.md
@@ -12,9 +12,10 @@
Fires when the user sends a chat message: from the editor's chat pane
(Console Panel chat tab), or from Talk on an employee's Home page. The
-producer [`server/nodes/trigger/chat_trigger/_events.py`](../../../server/nodes/trigger/chat_trigger/_events.py)
-emits a CloudEvents `WorkflowEvent` (`type: com.opencompany.chat.message.received`)
-via `dispatch.emit`. `chatTrigger` is canary-registered
+producer [`server/services/chat/events.py`](../../../server/services/chat/events.py)
+emits a CloudEvents `WorkflowEvent` (`type: com.opencompany.chat.message.received`,
+source `opencompany://services/chat`) via `dispatch.emit`, never broadcast to
+sockets. `chatTrigger` is canary-registered
(`register_canary_trigger_type`), so `DeploymentManager` starts a
`TriggerListenerWorkflow` for it; the listener receives the event via Temporal
Signal and spawns a child `MachinaWorkflow` per matching event. Any `chatTrigger`
@@ -53,17 +54,23 @@ trigger that Hire and Turn on Talk add beside another worker) has
### Output payload
-`routers/websocket.py::handle_send_chat_message` builds the event as:
+`services/chat/handlers.py::handle_send_chat_message` builds the event as:
```ts
{
message: string;
timestamp: string; // ISO 8601; the client's, else the server's time (UTC)
session_id: string;
+ message_id: string; // the saved message's id
+ run_id?: string; // the chat run the message started, when one did
}
```
-Wrapped in the standard envelope.
+Wrapped in the standard envelope, whose id is the run id when there is one.
+A run spawned from it claims that chat run (MachinaWorkflow,
+`machina-chat-run-v1`) and passes `run_scope` to every node, so
+[`chatReply`](../chat_utility/chatReply.md) saves its answer as the run's reply.
+See [chat_protocol.md](../../chat_protocol.md#runs).
## Logic Flow
@@ -71,8 +78,8 @@ Wrapped in the standard envelope.
flowchart TD
P[chat pane or Home Talk sends a message] --> V{workflow session with no<br/>controller to read it?}
V -- yes --> X[not_running: nothing saved or sent]
- V -- no --> W[chat_thread.record_chat_message]
- W --> Q[_events.py dispatch.emit<br/>WorkflowEvent com.opencompany.chat.message.received]
+ V -- no --> W[chat ledger: save the message,<br/>and its run when a chat trigger answers the session]
+ W --> Q[services/chat/events.py dispatch.emit<br/>WorkflowEvent com.opencompany.chat.message.received]
Q --> R[TriggerListenerWorkflow receives via Temporal Signal]
R --> S[ChatTriggerNode.build_filter:<br/>if session_id != 'default' require exact match]
S -- match --> T[spawn child MachinaWorkflow<br/>trigger pre-executed with event payload]
@@ -95,16 +102,17 @@ flowchart TD
## Side Effects
- **Database writes**: none while it fires. (`send_chat_message` keeps the
- message in the session's thread first, through
- `services/chat_thread.record_chat_message`, which stamps the live generation
- and broadcasts `chat.updated`.) On a workflow Reset its
+ message in the session's thread first, with its chat run, through
+ `services/chat/ledger.admit_message`, stamped with the live generation, and
+ announces `chat.updated`.) On a workflow Reset its
`reset_execution_state` clears the workflow's own thread (session = the
workflow id, never a custom `session_id` another workflow may share),
as `chatReply`'s does: the conversation ended with the generation. It
matters for a graph with a trigger and no reply yet, such as the one
Turn on Talk resets before its new reply node runs.
- **Broadcasts**: the producer emits a CloudEvents `WorkflowEvent` via
- `dispatch.emit` (Temporal Signal fan-out + in-process WS broadcast). The
+ `dispatch.emit` (Temporal Signal fan-out only: `broadcast=False`, since the
+ envelope carries the owner's text). The
`TriggerListenerWorkflow` emits firing-pulse status via
`broadcast_trigger_status_activity` before/after each child spawn.
- **External API calls**: none.
@@ -124,8 +132,10 @@ flowchart TD
per trigger is the only way to scope messages to a specific node.
- When multiple `chatTrigger` nodes exist with the same session ID, all of
them fire for a matching message, so one message starts one run per
- trigger. Hire and Turn on Talk never add a second trigger on a workflow's
- own session: a chat hire's "Chat" trigger is already its talk line.
+ trigger; the first run to start claims the chat run and the others run
+ untracked. Hire and Turn on Talk never add a second trigger on a
+ workflow's own session: a chat hire's "Chat" trigger is already its talk
+ line.
- The handler has no timeout; it waits forever until an event arrives or
the run is cancelled.
- For a workflow's session, `send_chat_message` first reads the latest
@@ -134,7 +144,9 @@ flowchart TD
and starts a run on Resume (`delivery: "queued"`). In any other state the
handler answers `not_running` and neither saves nor dispatches the message.
Session `'default'` is saved and dispatched as before, so a message there is
- stored even when no `chatTrigger` is waiting.
+ stored even when no `chatTrigger` is waiting. A workflow's message starts a
+ chat run only when a deployed `chatTrigger` accepts its session; while a run
+ is live the next message is refused with `run_in_progress`.
## Related
diff --git a/docs-internal/normal_mode.md b/docs-internal/normal_mode.md
index 27b845c3..e7682422 100644
--- a/docs-internal/normal_mode.md
+++ b/docs-internal/normal_mode.md
@@ -19,7 +19,7 @@ up in Home too, with details derived from their graph.
| Setup-screen pipeline (parse, normalise, render) | [client/src/features/home/genui/](../client/src/features/home/genui/) |
| Employees service (list, setup, hire, start, run records) | [server/services/employees/](../server/services/employees/) |
| Approvals (the "ask me first" step) | [server/services/approvals/](../server/services/approvals/), node [approvalGate](./node-logic-flows/workflow_triggers/approvalGate.md) |
-| Talk (the chat thread and the talk line) | [server/services/chat_thread.py](../server/services/chat_thread.py), [services/employees/talk.py](../server/services/employees/talk.py), node [chatReply](./node-logic-flows/chat_utility/chatReply.md) |
+| Talk (the chat thread and the talk line) | [server/services/chat_thread.py](../server/services/chat_thread.py), chat runs in [server/services/chat/](../server/services/chat/) ([chat_protocol.md](./chat_protocol.md)), [services/employees/talk.py](../server/services/employees/talk.py), node [chatReply](./node-logic-flows/chat_utility/chatReply.md) |
| Growing a saved employee (Turn on Talk, the Agent Builder, Apply) | [server/services/graph_build.py](../server/services/graph_build.py), [services/employees/policy.py](../server/services/employees/policy.py), [services/workflow_storage/mutate.py](../server/services/workflow_storage/mutate.py), [services/deployment/restart.py](../server/services/deployment/restart.py), node [agentBuilder](./node-logic-flows/ai_tools/agentBuilder.md) |
| Built-in skills offered to new hires (Settings > Skills > Discover) | [server/skills/employee/](../server/skills/employee/) |
@@ -390,17 +390,24 @@ Connect an AI model action.
The owner talks to an employee on its page. The conversation is the chat
session whose id is the employee's workflow id, the thread the editor's chat
-pane also shows for that workflow (there, only the live generation). Every
-row goes through
-[services/chat_thread.py](../server/services/chat_thread.py): the owner's
-messages from `send_chat_message`, answers and reports from the
-[chatReply](./node-logic-flows/chat_utility/chatReply.md) ("Reply in Chat")
-node. Each row is stamped with the live generation, and each insert or clear
-is announced as `chat.updated`. Clearing the chat in the editor's chat pane
+pane also shows for that workflow (there, only the live generation). The
+owner's messages come in through `send_chat_message`
+([services/chat/handlers.py](../server/services/chat/handlers.py)), answers
+and reports from the [chatReply](./node-logic-flows/chat_utility/chatReply.md)
+("Reply in Chat") node. Each row is stamped with the live generation and
+appended to one chain, and each insert or clear is announced as
+`chat.updated`. Clearing the chat in the editor's chat pane
(`clear_chat_messages`, `chat_thread.clear_chat_session`) deletes the thread
and, through the Context plugin's chat-cleared listener, every conversation
of the workflow, so the employee starts over with the chat.
+A message the talk line will answer starts a **chat run**, saved with it: one
+run at a time per conversation (a second send while one is live answers
+`run_in_progress`), claimed and finished by the workflow run that answers it,
+with the reply saved as the run's (`a_<run id>`). Sockets that subscribe
+(`chat_subscribe`) receive the run's events; a watchdog ends runs nothing
+will finish. The wire contract is [chat_protocol.md](./chat_protocol.md).
+
### On the employee's page
[employee/EmployeeTalk.tsx](../client/src/features/home/employee/EmployeeTalk.tsx)
@@ -710,8 +717,8 @@ WebSocket requests (snake_case; failures come back as `success: false` with an
| `start_employee` | `{workflow_id, expected_revision, idempotency_key}` | as `start_workflow` |
| `enable_employee_talk` | `{workflow_id, idempotency_key}` | `{employee}`. Errors: `invalid_request`, `not_found`, `unsupported`, `conflict` (a start, pause, resume or reset is under way, or the graph changed meanwhile), `restart_failed`; the last three carry `employee` too |
| `apply_employee_changes` | `{workflow_id, idempotency_key}` | `{employee}`: running ends running, paused or failed ends ready, ready is left alone. Errors: `invalid_request`, `not_found`, `conflict`, `restart_failed` (the last two with `employee`) |
-| `send_chat_message` | `{message, role: "user", session_id: <workflow_id>, timestamp}` | `{timestamp, delivery}`: `"now"` while running, starting or resuming; `"queued"` while paused or pausing (it runs on Resume). In any other state `not_running`, and nothing is saved or sent. Session `"default"` works as before, with no `delivery` |
-| `get_chat_messages` | `{session_id, limit?, all_generations?}` | `{messages}`, oldest first, each `{id, role, message, timestamp, run_key}`. Timestamps carry their UTC offset; `run_key` is the generation the row was written in. Without `all_generations`, only the latest generation's rows (none after a Reset; every row when the workflow was never started) |
+| `send_chat_message` | `{message, role: "user", session_id: <workflow_id>, timestamp, client_message_id?}` | `{timestamp, delivery, message_id, run_id}`: `"now"` while running, starting or resuming; `"queued"` while paused or pausing (it runs on Resume). In any other state `not_running`, and nothing is saved or sent; `run_in_progress` (with the live `run_id`) while a run is live. `run_id` is null when no deployed chat trigger answers the session. Session `"default"` works as before, with no `delivery` |
+| `get_chat_messages` | `{session_id, limit?, all_generations?}` | `{messages, thread, active_runs}`, messages oldest first, each `{id, role, message, timestamp, run_key, ...}` (the full shape is in [chat_protocol.md](./chat_protocol.md#messages)). Timestamps carry their UTC offset; `run_key` is the generation the row was written in. Without `all_generations`, only the latest generation's rows (none after a Reset; every row when the workflow was never started). A failed read answers `read_failed`, never an empty thread |
| `list_approvals` | `{workflow_id?, status?, limit <= 100}` | `{approvals, counts, server_time}` |
| `decide_approval` | `{approval_id, decision, text?, subject?, decision_key}` | `{approval, will_send_on_resume}` |
| `delete_workflow` | `{workflow_id}` | `{workflow_id, contexts_archived, context_archives_pending}`; `DELETE /api/database/workflows/{id}` is the same handler. Error `workflow_shutdown_failed` (with `detail`) when stopping the employee failed: nothing was deleted |
@@ -743,7 +750,8 @@ history). Talk and growing a saved employee: `tests/services/employees/`
(`test_talk.py`, `test_enable_talk.py`, `test_apply_changes.py`,
`test_policy.py`, and `test_builder_snapshot.py` against
`tests/fixtures/employee_builder_snapshot.json`),
-`tests/services/test_chat_thread.py`, `tests/nodes/test_chat_reply.py`,
+`tests/services/test_chat_thread.py`, `tests/services/chat/`,
+`tests/temporal/test_machina_chat_run.py`, `tests/nodes/test_chat_reply.py`,
`tests/services/test_graph_build.py`, `tests/services/test_graph_additions.py`,
`tests/services/test_graph_listeners.py`,
`tests/services/test_deployment_restart.py`,
diff --git a/docs-internal/status_broadcaster.md b/docs-internal/status_broadcaster.md
index 7f758dd0..35e7c7fd 100644
--- a/docs-internal/status_broadcaster.md
+++ b/docs-internal/status_broadcaster.md
@@ -81,9 +81,14 @@ Message loop:
Frontend unmounts or logs out
|
v
-StatusBroadcaster.disconnect(ws) -> remove from _connections
+StatusBroadcaster.disconnect(ws) -> remove from _connections, then call each
+ register_disconnect_listener(fn) listener
```
+`register_disconnect_listener(fn)` (module-level, idempotent) is for services that keep per-socket state:
+`fn(websocket)` runs after the socket leaves `_connections`, synchronously, and must not block. Chat registers one so a
+closed socket's run subscriptions are dropped (`services/chat/__init__.py`).
+
Auto-reconnect is handled by `WebSocketContext.tsx` through PartySocket's `ReconnectingWebSocket` (`partysocket/ws`), with jittered exponential backoff (`MIN_DELAY_MS` / `MAX_DELAY_MS` / `GROW_FACTOR` in `client/src/lib/connectionConfig.ts`). Requests queued while disconnected are sent after the next open; requests already in flight are rejected on close and never replayed. A remote close, even with code 1000, keeps the loop retrying; only the client's own `disposeConnection` (logout or unmount) closes with 1000 and stops it. The full client-side rules are in [frontend_architecture.md → Real-time](./frontend_architecture.md#real-time). A 100ms mount delay avoids React Strict Mode double-connect in dev.
## Handler Registry
@@ -120,7 +125,7 @@ Live total = `len(MESSAGE_HANDLERS) + len(get_ws_handlers())` -- the `MESSAGE_HA
| WhatsApp | `whatsapp_status`, `whatsapp_qr`, `whatsapp_send`, `whatsapp_chat_history`, `whatsapp_newsletters`, `whatsapp_diagnostics`, ... |
| Telegram | `telegram_connect`, `telegram_disconnect`, `telegram_status`, `telegram_send`, `telegram_get_me`, `telegram_get_chat` |
| Workflow storage | `save_workflow`, `get_workflow`, `get_all_workflows`, `delete_workflow` |
-| Chat messages | `send_chat_message`, `get_chat_messages`, `clear_chat_messages`, `get_chat_sessions` |
+| Chat messages | `send_chat_message`, `get_chat_messages`, `chat_subscribe`, `chat_unsubscribe`, `get_chat_run`, `clear_chat_messages`, `save_chat_message` (`services/chat/handlers.py`; run events reach only subscribed sockets, see [chat_protocol.md](./chat_protocol.md)) |
| Console / terminal | `get_console_logs`, `clear_console_logs`, `get_terminal_logs`, `clear_terminal_logs` |
| Skills | `get_skill_content`, `save_skill_content`, `get_user_skills`, `create_user_skill`, `scan_skill_folder` |
| Memory | `clear_memory`, `reset_skill`, `configure_compaction`, `get_compaction_stats` |
diff --git a/server/config/chat_defaults.json b/server/config/chat_defaults.json
new file mode 100644
index 00000000..dc364f71
--- /dev/null
+++ b/server/config/chat_defaults.json
@@ -0,0 +1,10 @@
+{
+ "runs": {
+ "pickup_timeout_s": 120,
+ "max_running_s": 86400,
+ "watchdog_interval_s": 30
+ },
+ "hub": {
+ "subscriber_queue_size": 512
+ }
+}
diff --git a/server/core/database.py b/server/core/database.py
index ba954afb..95968491 100644
--- a/server/core/database.py
+++ b/server/core/database.py
@@ -58,6 +58,11 @@ from models.employees import ( # noqa: F401 - registers SQLModel tables
from models.approvals import ( # noqa: F401 - registers SQLModel tables
ApprovalRequest,
)
+from models.chat import ( # noqa: F401 - registers SQLModel tables
+ ChatRun,
+ ChatRunPart,
+ ChatThread,
+)
from models.cache import CacheEntry # SQLite-backed cache for Redis alternative
from core.logging import get_logger
@@ -132,6 +137,7 @@ class Database:
await self._migrate_agent_teams()
await self._migrate_workflow_controls()
await self._migrate_generation_scoped_runtime_data()
+ await self._migrate_chat_messages()
logger.info("Database initialized successfully")
@@ -436,6 +442,21 @@ class Database:
else:
await asyncio.shield(_teardown_session(session, rollback=False))
+ @asynccontextmanager
+ async def reserved_session(self):
+ """A session whose write transaction is reserved before its first read.
+
+ SQLite's deferred transactions let two writers read the same rows
+ before either writes; ``BEGIN IMMEDIATE`` takes the write reservation
+ first, so a whole read/modify/write runs alone (as in
+ :meth:`run_runtime_mutation`). The caller commits; leaving the block
+ without committing rolls back.
+ """
+ async with self.get_session() as session:
+ if self.engine is not None and self.engine.dialect.name == "sqlite":
+ await session.execute(text("BEGIN IMMEDIATE"))
+ yield session
+
async def _migrate_workflow_controls(self):
"""Backfill control-plane columns when upgrading an early preview DB."""
try:
@@ -503,6 +524,50 @@ class Database:
except Exception as exc:
logger.warning(f"Generation runtime-data migration check failed: {exc}")
+ async def _migrate_chat_messages(self):
+ """Give older chat rows the columns chat runs need.
+
+ Rows written before this migration get a ``uid`` (``m<id>``) and a
+ ``parent_uid`` chaining each session's rows in order, the shape every
+ new row is written in. The backfill runs once: only in the startup
+ that adds the ``uid`` column, so a session's first message or a
+ report (both legitimately parentless) never gains a parent later.
+ """
+ try:
+ async with self.engine.begin() as conn:
+ result = await conn.execute(text("PRAGMA table_info(chat_messages)"))
+ columns = {row[1] for row in result.fetchall()}
+ if not columns:
+ return
+ additions = {
+ "uid": "VARCHAR(64)",
+ "parent_uid": "VARCHAR(64)",
+ "run_id": "VARCHAR(64)",
+ "kind": "VARCHAR(20) DEFAULT 'text'",
+ "status": "VARCHAR(20) DEFAULT 'complete'",
+ "parts": "JSON DEFAULT '{}'",
+ "attachments": "JSON DEFAULT '[]'",
+ "meta": "JSON DEFAULT '{}'",
+ }
+ backfill = "uid" not in columns
+ for column, definition in additions.items():
+ if column not in columns:
+ await conn.execute(text(f"ALTER TABLE chat_messages ADD COLUMN {column} {definition}"))
+ if backfill:
+ await conn.execute(text("UPDATE chat_messages SET uid = 'm' || id WHERE uid IS NULL"))
+ await conn.execute(text(
+ "UPDATE chat_messages SET parent_uid = ("
+ " SELECT prev.uid FROM chat_messages AS prev"
+ " WHERE prev.session_id = chat_messages.session_id"
+ " AND (prev.created_at < chat_messages.created_at"
+ " OR (prev.created_at = chat_messages.created_at AND prev.id < chat_messages.id))"
+ " ORDER BY prev.created_at DESC, prev.id DESC LIMIT 1)"
+ ))
+ await conn.execute(text("CREATE UNIQUE INDEX IF NOT EXISTS ux_chat_messages_uid ON chat_messages(uid)"))
+ await conn.execute(text("CREATE INDEX IF NOT EXISTS ix_chat_messages_run_id ON chat_messages(run_id)"))
+ except Exception as exc:
+ logger.warning(f"Chat message migration check failed: {exc}")
+
async def run_runtime_mutation(
self,
*,
@@ -1356,62 +1421,164 @@ class Database:
# Chat Messages (Console Panel persistence)
# ============================================================================
+ async def append_chat_row(
+ self,
+ session: AsyncSession,
+ *,
+ session_id: str,
+ role: str,
+ message: str,
+ execution_id: Optional[str] = None,
+ uid: Optional[str] = None,
+ run_id: Optional[str] = None,
+ kind: str = "text",
+ status: str = "complete",
+ parts: Optional[Dict[str, Any]] = None,
+ attachments: Optional[List[Dict[str, Any]]] = None,
+ meta: Optional[Dict[str, Any]] = None,
+ ) -> ChatMessage:
+ """Append a message to the end of a session's shown path, inside the
+ caller's :meth:`reserved_session` (the caller commits).
+
+ The new row follows the thread's active leaf and becomes the leaf, so
+ every message joins one chain: two writers at once would otherwise
+ both follow the same leaf and fork it, and a fork hides one branch. A
+ session written before threads existed follows its newest message.
+ """
+ from uuid import uuid4
+
+ thread = await session.get(ChatThread, session_id)
+ if thread is None:
+ newest = await session.execute(
+ select(ChatMessage.uid)
+ .where(ChatMessage.session_id == session_id)
+ .order_by(ChatMessage.created_at.desc(), ChatMessage.id.desc())
+ .limit(1)
+ )
+ parent_uid = newest.scalar_one_or_none()
+ thread = ChatThread(session_id=session_id)
+ session.add(thread)
+ else:
+ parent_uid = thread.active_leaf_uid
+ row = ChatMessage(
+ session_id=session_id, role=role, message=message,
+ execution_id=execution_id,
+ uid=uid or f"m_{uuid4().hex}",
+ parent_uid=parent_uid,
+ run_id=run_id,
+ kind=kind,
+ status=status,
+ parts=parts or {},
+ attachments=attachments or [],
+ meta=meta or {},
+ )
+ session.add(row)
+ thread.active_leaf_uid = row.uid
+ thread.revision = (thread.revision or 0) + 1
+ thread.updated_at = datetime.now(timezone.utc)
+ await session.flush()
+ return row
+
async def add_chat_message(
self, session_id: str, role: str, message: str,
execution_id: Optional[str] = None,
- ) -> bool:
- """Add a chat message to the console panel history."""
+ *,
+ uid: Optional[str] = None,
+ run_id: Optional[str] = None,
+ kind: str = "text",
+ status: str = "complete",
+ parts: Optional[Dict[str, Any]] = None,
+ attachments: Optional[List[Dict[str, Any]]] = None,
+ meta: Optional[Dict[str, Any]] = None,
+ ) -> Optional[Dict[str, Any]]:
+ """Append a chat message (:meth:`append_chat_row`) and return it as
+ :meth:`chat_row` shapes it, or None when it could not be saved.
+
+ A ``uid`` already in the session returns the stored row unchanged, so
+ a retried write (an activity attempt, a resent message) saves once.
+ """
try:
- async with self.get_session() as session:
- chat_msg = ChatMessage(
- session_id=session_id, role=role, message=message,
- execution_id=execution_id,
+ async with self.reserved_session() as session:
+ if uid is not None:
+ found = await session.execute(select(ChatMessage).where(ChatMessage.uid == uid))
+ existing = found.scalar_one_or_none()
+ if existing is not None:
+ if existing.session_id != session_id:
+ raise ValueError("message id belongs to another chat")
+ return self.chat_row(existing)
+ row = await self.append_chat_row(
+ session,
+ session_id=session_id, role=role, message=message, execution_id=execution_id,
+ uid=uid, run_id=run_id, kind=kind, status=status,
+ parts=parts, attachments=attachments, meta=meta,
)
- session.add(chat_msg)
await session.commit()
logger.debug(f"[Chat] Added {role} message to session '{session_id}'")
- return True
+ return self.chat_row(row)
except Exception as e:
logger.error("Failed to add chat message", session_id=session_id, error=str(e))
- return False
+ return None
+
+ @staticmethod
+ def chat_row(message: ChatMessage) -> Dict[str, Any]:
+ """A chat row as the handlers read it. The timestamp carries its UTC
+ offset (SQLite returns the stored UTC time naive)."""
+ moment = message.created_at
+ if moment is not None and moment.tzinfo is None:
+ moment = moment.replace(tzinfo=timezone.utc)
+ return {
+ "id": message.id,
+ "uid": message.uid,
+ "parent_uid": message.parent_uid,
+ "run_id": message.run_id,
+ "role": message.role,
+ "message": message.message,
+ "kind": message.kind or "text",
+ "status": message.status or "complete",
+ "parts": message.parts or {},
+ "attachments": message.attachments or [],
+ "meta": message.meta or {},
+ "timestamp": moment.isoformat() if moment is not None else None,
+ "execution_id": message.execution_id,
+ }
async def get_chat_messages(
self, session_id: str, limit: Optional[int] = None,
execution_id: Optional[str] = None,
) -> List[Dict[str, Any]]:
"""Chat messages for a session, oldest first: the newest ``limit``
- when given, of one generation when ``execution_id`` is. Each is
- ``{id, role, message, timestamp, execution_id}``; the timestamp
- carries its UTC offset (SQLite returns the stored UTC time naive)."""
+ when given, of one generation when ``execution_id`` is. Rows are
+ shaped by :meth:`chat_row`. A read that fails returns ``[]``; use
+ :meth:`read_chat_messages` where a failure must be told apart from an
+ empty thread."""
try:
- async with self.get_session() as session:
- stmt = select(ChatMessage).where(ChatMessage.session_id == session_id)
- if execution_id is not None:
- stmt = stmt.where(ChatMessage.execution_id == execution_id)
- stmt = stmt.order_by(ChatMessage.created_at.desc(), ChatMessage.id.desc())
- if limit and limit > 0:
- stmt = stmt.limit(limit)
-
- result = await session.execute(stmt)
- messages = list(reversed(result.scalars().all()))
-
- def utc(moment: Optional[datetime]) -> Optional[str]:
- if moment is None:
- return None
- return (moment if moment.tzinfo else moment.replace(tzinfo=timezone.utc)).isoformat()
-
- return [
- {"id": m.id, "role": m.role, "message": m.message, "timestamp": utc(m.created_at), "execution_id": m.execution_id}
- for m in messages
- ]
-
+ return await self.read_chat_messages(session_id, limit, execution_id=execution_id)
except Exception as e:
logger.error("Failed to get chat messages", session_id=session_id, error=str(e))
return []
+ async def read_chat_messages(
+ self, session_id: str, limit: Optional[int] = None,
+ execution_id: Optional[str] = None,
+ ) -> List[Dict[str, Any]]:
+ """:meth:`get_chat_messages`, raising when the read fails."""
+ async with self.get_session() as session:
+ stmt = select(ChatMessage).where(ChatMessage.session_id == session_id)
+ if execution_id is not None:
+ stmt = stmt.where(ChatMessage.execution_id == execution_id)
+ stmt = stmt.order_by(ChatMessage.created_at.desc(), ChatMessage.id.desc())
+ if limit and limit > 0:
+ stmt = stmt.limit(limit)
+
+ result = await session.execute(stmt)
+ return [self.chat_row(m) for m in reversed(result.scalars().all())]
+
async def clear_chat_messages(self, session_id: str) -> int:
- """Clear all chat messages for a session. Returns count deleted."""
+ """Clear a session's chat: its messages, and the runs, run parts and
+ thread state that belong to them. Returns the messages deleted."""
+ from sqlalchemy import delete as sa_delete
+
try:
async with self.get_session() as session:
stmt = select(ChatMessage).where(ChatMessage.session_id == session_id)
@@ -1422,6 +1589,11 @@ class Database:
for message in messages:
await session.delete(message)
+ run_ids = select(ChatRun.run_id).where(ChatRun.session_id == session_id)
+ await session.execute(sa_delete(ChatRunPart).where(ChatRunPart.run_id.in_(run_ids)))
+ await session.execute(sa_delete(ChatRun).where(ChatRun.session_id == session_id))
+ await session.execute(sa_delete(ChatThread).where(ChatThread.session_id == session_id))
+
await session.commit()
logger.info(f"[Chat] Cleared {count} messages from session '{session_id}'")
return count
diff --git a/server/main.py b/server/main.py
index 651df1df..bd4492e0 100644
--- a/server/main.py
+++ b/server/main.py
@@ -166,6 +166,11 @@ async def lifespan(app: FastAPI):
# runs on workflow delete, and the employee_lifecycle summary builder.
import services.employees # noqa: F401
+ # services/chat/__init__.py self-registers the chat commands
+ # (send_chat_message / get_chat_messages / chat_subscribe / ...) and the
+ # disconnect listener that drops a closed socket's run subscriptions.
+ import services.chat # noqa: F401
+
# Wave 13.8: services/pricing_handlers.py self-registers the 3
# pricing handlers (get_pricing_config / save_pricing_config /
# get_api_usage_summary). Flat module (sibling to services/pricing.py)
@@ -188,6 +193,13 @@ async def lifespan(app: FastAPI):
await container.cache().startup()
_startup_log("Database + cache started")
+ # Ends chat runs nothing will finish (never picked up, or their run
+ # closed without finishing them). Its first sweep runs now.
+ from services.chat.watchdog import ChatRunWatchdog
+
+ chat_watchdog = ChatRunWatchdog(container.database())
+ chat_watchdog.start()
+
# Initialize credentials database (creates tables if not exist)
credentials_db = container.credentials_database()
salt = await credentials_db.initialize()
@@ -503,6 +515,9 @@ async def lifespan(app: FastAPI):
logger.debug("[CLI MCP] lifespan shutdown: %s", exc)
_startup_log("Lifespan shutdown: CLI MCP lifespan exited")
+ await chat_watchdog.stop()
+ _startup_log("Lifespan shutdown: chat run watchdog stopped")
+
# Drain cached native SDK clients before credentials/database teardown.
await container.chat_unifier().aclose()
_startup_log("Lifespan shutdown: LLM clients closed")
diff --git a/server/models/chat.py b/server/models/chat.py
new file mode 100644
index 00000000..efb6b9ef
--- /dev/null
+++ b/server/models/chat.py
@@ -0,0 +1,98 @@
+"""Chat runs: the answer an employee works on after each owner message.
+
+A run is admitted in the same transaction as the owner's message
+(``services/chat/ledger.py``). It moves ``queued`` (the employee is paused)
+or ``pending`` -> ``running`` -> ``finished`` | ``error`` | ``stopped``, each
+step a compare-and-swap on ``state``. ``kind`` says what started it:
+``message`` (the owner wrote), ``edit``, ``regenerate``, ``action`` (a button
+in generated UI) or ``resume`` (an approved send executing later).
+
+**The lane.** At most one non-terminal run per session, resume runs aside,
+enforced by a partial unique index: two runs that loaded the same stored
+conversation would each save over the other's (docs-internal/
+agent_context_flow.md). A second message while one is live is refused with
+``run_in_progress``; the composer shows Stop instead of Send meanwhile.
+
+``ChatRunPart`` collects what a run's tools produce for its reply (generated
+UI, sources, artifacts, approvals), keyed so an activity retry writes the
+same row again. ``ChatThread`` holds a session's active leaf: the message the
+shown path ends at, which branches will move.
+"""
+
+from __future__ import annotations
+
+from datetime import datetime, timezone
+from typing import Any, Dict, List, Optional
+
+from sqlalchemy import JSON, Column, DateTime, Index, UniqueConstraint, text
+from sqlmodel import Field, SQLModel
+
+
+def _utcnow() -> datetime:
+ return datetime.now(timezone.utc)
+
+
+#: States that hold the lane.
+LIVE_STATES = ("queued", "pending", "running", "stopping")
+#: States a run ends in.
+TERMINAL_STATES = ("finished", "error", "stopped")
+
+_LANE_WHERE = "state IN ('queued', 'pending', 'running', 'stopping') AND kind != 'resume'"
+
+
+class ChatThread(SQLModel, table=True):
+ __tablename__ = "chat_threads"
+
+ session_id: str = Field(primary_key=True, max_length=255)
+ active_leaf_uid: Optional[str] = Field(default=None, max_length=64)
+ revision: int = Field(default=0)
+ updated_at: datetime = Field(default_factory=_utcnow, sa_column=Column(DateTime(timezone=True), nullable=False))
+
+
+class ChatRun(SQLModel, table=True):
+ __tablename__ = "chat_runs"
+ __table_args__ = (Index("ux_chat_runs_lane", "session_id", unique=True, sqlite_where=text(_LANE_WHERE)),)
+
+ run_id: str = Field(primary_key=True, max_length=64)
+ session_id: str = Field(index=True, max_length=255)
+ workflow_id: Optional[str] = Field(default=None, index=True, max_length=255)
+ #: The generation's root execution id, like ``chat_messages.execution_id``.
+ run_key: Optional[str] = Field(default=None, max_length=255)
+ kind: str = Field(default="message", max_length=20)
+ state: str = Field(default="pending", index=True, max_length=20)
+ user_message_uid: Optional[str] = Field(default=None, max_length=64)
+ reply_message_uid: Optional[str] = Field(default=None, max_length=64)
+ parent_run_id: Optional[str] = Field(default=None, max_length=64)
+ #: Per-message choices the owner made (``web``).
+ options: Dict[str, Any] = Field(default_factory=dict, sa_column=Column(JSON))
+ #: Working steps as they finished, for "Worked for Ns · N steps" on reload.
+ steps: List[Dict[str, Any]] = Field(default_factory=list, sa_column=Column(JSON))
+ #: How a finished run ended: ``success``, ``stopped`` or ``interrupt``
+ #: (drafts wait for the owner). None until it ends, and for errors.
+ outcome: Optional[str] = Field(default=None, max_length=20)
+ #: The terminal event's ``result`` (``reply_message_id`` or ``no_reply``).
+ result: Dict[str, Any] = Field(default_factory=dict, sa_column=Column(JSON))
+ error: Optional[str] = Field(default=None, max_length=500)
+ error_code: Optional[str] = Field(default=None, max_length=60)
+ temporal_workflow_id: Optional[str] = Field(default=None, max_length=500)
+ temporal_run_id: Optional[str] = Field(default=None, max_length=255)
+ created_at: datetime = Field(default_factory=_utcnow, sa_column=Column(DateTime(timezone=True), nullable=False))
+ started_at: Optional[datetime] = Field(default=None, sa_column=Column(DateTime(timezone=True), nullable=True))
+ finished_at: Optional[datetime] = Field(default=None, sa_column=Column(DateTime(timezone=True), nullable=True))
+
+
+class ChatRunPart(SQLModel, table=True):
+ __tablename__ = "chat_run_parts"
+ __table_args__ = (UniqueConstraint("run_id", "key", name="uq_chat_run_parts_key"),)
+
+ id: Optional[int] = Field(default=None, primary_key=True)
+ run_id: str = Field(index=True, max_length=64)
+ #: ``ui``, ``sources``, ``artifacts``, ``approvals``, ``followups``.
+ kind: str = Field(max_length=20)
+ #: Unique within the run; a retried activity writes the same key again.
+ key: str = Field(max_length=255)
+ payload: Dict[str, Any] = Field(default_factory=dict, sa_column=Column(JSON))
+ created_at: datetime = Field(default_factory=_utcnow, sa_column=Column(DateTime(timezone=True), nullable=False))
+
+
+__all__ = ["ChatRun", "ChatRunPart", "ChatThread", "LIVE_STATES", "TERMINAL_STATES"]
diff --git a/server/models/database.py b/server/models/database.py
index 95adc602..ad92ccc4 100644
--- a/server/models/database.py
+++ b/server/models/database.py
@@ -3,7 +3,7 @@
from datetime import datetime, timezone
from typing import Optional, Dict, Any
from sqlmodel import SQLModel, Field, Column, DateTime, JSON
-from sqlalchemy import UniqueConstraint, func
+from sqlalchemy import Index, UniqueConstraint, func
class NodeParameter(SQLModel, table=True):
@@ -215,15 +215,33 @@ class ChatMessage(SQLModel, table=True):
Stores user and assistant messages from the chat panel (Console Panel chat section).
These are separate from ConversationMessage which stores AI agent memory.
+
+ ``uid`` is the message's stable id on the wire (``m_…`` for the owner's,
+ ``a_<run id>`` for a run's reply). ``parent_uid`` links a message to the
+ one it follows, so edits can branch (models/chat.py holds the runs and
+ each session's active leaf). ``parts`` is the structured content a reply
+ gathered (steps, generated UI, sources, ...); ``message`` stays the text.
"""
__tablename__ = "chat_messages"
+ # Named like core/database.py's migration creates it on older databases.
+ __table_args__ = (Index("ux_chat_messages_uid", "uid", unique=True),)
id: Optional[int] = Field(default=None, primary_key=True)
session_id: str = Field(default="default", index=True, max_length=255)
execution_id: Optional[str] = Field(default=None, index=True, max_length=255)
role: str = Field(max_length=20) # 'user' or 'assistant'
message: str = Field(max_length=50000) # Large content support
+ uid: Optional[str] = Field(default=None, max_length=64)
+ parent_uid: Optional[str] = Field(default=None, max_length=64)
+ run_id: Optional[str] = Field(default=None, index=True, max_length=64)
+ #: ``text``; ``report`` (Post to Talk); ``action`` (a button press).
+ kind: str = Field(default="text", max_length=20)
+ #: ``complete``, ``stopped`` (the owner pressed Stop), ``error``.
+ status: str = Field(default="complete", max_length=20)
+ parts: Dict[str, Any] = Field(default_factory=dict, sa_column=Column(JSON))
+ attachments: list = Field(default_factory=list, sa_column=Column(JSON))
+ meta: Dict[str, Any] = Field(default_factory=dict, sa_column=Column(JSON))
created_at: datetime = Field(
default_factory=lambda: datetime.now(timezone.utc), sa_column=Column(DateTime(timezone=True), server_default=func.now())
)
diff --git a/server/nodes/chat/chat_reply/__init__.py b/server/nodes/chat/chat_reply/__init__.py
index ec37495f..4e2063ab 100644
--- a/server/nodes/chat/chat_reply/__init__.py
+++ b/server/nodes/chat/chat_reply/__init__.py
@@ -9,6 +9,13 @@ which stamps the live generation and announces ``chat.updated``.
A message that is empty, or exactly NO_REPLY (the agent had nothing to
say), posts nothing.
+In a run the owner's chat message started, the reply is that chat run's
+answer (``run_scope``, set by MachinaWorkflow): it is saved under the run's
+reply id through ``services/chat/ledger.py``, so a retried step saves it
+once, and the run reports it when it finishes. A run whose conversation was
+reset or cleared meanwhile posts nothing: its answer belongs to a thread
+that no longer exists.
+
The thread ends with the workflow's generation: a Reset (every restart)
clears it through ``reset_execution_state``, as the Context node forgets
the conversation in the same Reset, and deleting the workflow deletes it
@@ -52,6 +59,9 @@ class ChatReplyOutput(BaseModel):
#: False when there was nothing to post.
posted: bool = False
message: Optional[str] = None
+ #: The saved message's id, and the chat run it answers (if any).
+ message_id: Optional[str] = None
+ run_id: Optional[str] = None
model_config = ConfigDict(extra="allow")
@@ -81,9 +91,30 @@ class ChatReplyNode(ActionNode):
return ChatReplyOutput(posted=False)
if not ctx.workflow_id:
raise NodeUserError("Reply in Chat posts to the workflow's chat: save the workflow first.")
- if not await record_chat_message(get_database(), ctx.workflow_id, "assistant", text):
+ database = get_database()
+ scope = ctx.raw.get("run_scope") if isinstance(ctx.raw, dict) else None
+ run_id = scope.get("run_id") if isinstance(scope, dict) else None
+ if isinstance(run_id, str) and run_id:
+ return await self._reply_to_run(database, ctx, run_id, text)
+ saved = await record_chat_message(database, ctx.workflow_id, "assistant", text)
+ if not saved:
raise RuntimeError("The reply could not be saved to the chat")
- return ChatReplyOutput(posted=True, message=text)
+ return ChatReplyOutput(posted=True, message=text, message_id=_message_id(saved))
+
+ async def _reply_to_run(self, database, ctx: NodeContext, run_id: str, text: str) -> ChatReplyOutput:
+ from services.chat import ledger
+ from services.chat_thread import announce_chat_updated, chat_execution_id
+
+ run = await ledger.get_run(database, run_id)
+ if run is None or run.session_id != ctx.workflow_id:
+ # Reset or Clear removed the conversation this run answered.
+ return ChatReplyOutput(posted=False, run_id=run_id)
+ control = await database.get_latest_workflow_control(run.session_id)
+ saved = await ledger.post_reply(
+ database, run=run, node_id=ctx.node_id, text=text, execution_id=chat_execution_id(control)
+ )
+ await announce_chat_updated(run.session_id, "assistant")
+ return ChatReplyOutput(posted=True, message=text, message_id=saved.get("uid"), run_id=run_id)
@classmethod
async def reset_execution_state(
@@ -106,6 +137,10 @@ class ChatReplyNode(ActionNode):
return {"reset": bool(cleared), "cleared_chat_messages": cleared}
+def _message_id(saved: Any) -> Optional[str]:
+ return saved.get("uid") if isinstance(saved, dict) else None
+
+
# The thread belongs to its workflow: deleting the workflow deletes it.
from services.chat_thread import clear_chat_thread # noqa: E402
from services.workflow_storage.hooks import register_workflow_deleted_hook # noqa: E402
diff --git a/server/nodes/trigger/chat_trigger/__init__.py b/server/nodes/trigger/chat_trigger/__init__.py
index 313d682f..5e78417c 100644
--- a/server/nodes/trigger/chat_trigger/__init__.py
+++ b/server/nodes/trigger/chat_trigger/__init__.py
@@ -9,6 +9,12 @@ A Reset clears the workflow's chat thread (``reset_execution_state``), as
Talk resets on that graph before adding the reply) must not carry its old
messages into the next generation.
+The owner's message reaches it from ``services/chat/handlers.py``
+(``send_chat_message``), which saves it and signals it through
+``services/chat/events.py``. Its output names the saved message
+(``message_id``) and, when the message started a chat run, the run
+(``run_id``): the run that answers it is that chat run.
+
Replaces:
- ``nodes/triggers.py:chatTrigger`` metadata-only registration.
- ``event_waiter.build_chat_filter`` stays wired until the generic
@@ -37,6 +43,8 @@ class ChatTriggerOutput(BaseModel):
message: Optional[str] = None
timestamp: Optional[str] = None
session_id: Optional[str] = None
+ message_id: Optional[str] = None
+ run_id: Optional[str] = None
model_config = ConfigDict(extra="allow")
@@ -94,7 +102,7 @@ class ChatTriggerNode(TriggerNode):
# Wave 12 C1 rollout #1: opt this trigger into the TriggerListenerWorkflow
# consumer path. Producer side: dispatch_chat_message_received in
-# ./_events.py calls dispatch.emit unconditionally; the canary registry
-# tells DeploymentManager to start a listener for this type. See
+# services/chat/events.py calls dispatch.emit; the canary registry tells
+# DeploymentManager to start a listener for this type. See
# services/deployment/canary_registry.py.
register_canary_trigger_type(ChatTriggerNode.type, "com.opencompany.chat.message.received")
diff --git a/server/nodes/trigger/chat_trigger/_events.py b/server/nodes/trigger/chat_trigger/_events.py
deleted file mode 100644
index fd358d3e..00000000
--- a/server/nodes/trigger/chat_trigger/_events.py
+++ /dev/null
@@ -1,73 +0,0 @@
-"""CloudEvents factory + dispatch wrapper for chat_trigger.
-
-Per RFC plugin_authoring_rfc.md §6.4: plugin-specific factories live in
-the plugin folder.
-
-Delivery: single ``dispatch.emit`` call routes events to running
-:class:`TriggerListenerWorkflow` consumers via Temporal Visibility.
-chatTrigger is canary-registered (see ``nodes/trigger/chat_trigger/__init__.py``)
-so the deployment manager skips ``setup_event_trigger`` and the legacy
-``event_waiter.dispatch`` path has zero consumers — removed.
-
-It is never broadcast. The envelope carries the owner's message text, and
-the WebSocket broadcast reaches every connected socket; no client handled
-it anyway. Clients learn about a new message from the identity-only
-``chat.updated`` that ``services/chat_thread.py`` sends when it is saved.
-"""
-
-from __future__ import annotations
-
-from typing import Any, Mapping, Optional
-
-from services.events.envelope import WorkflowEvent
-
-
-# Outer wire-routing key. Matches ``ChatTriggerNode.event_type`` and the
-# FE WS channel; the inner envelope carries
-# ``com.opencompany.chat.message.received``.
-_WIRE_ROUTING_KEY = "chat_message_received"
-
-
-def chat_message_received(
- event_data: Mapping[str, Any],
- *,
- workflow_id: Optional[str] = None,
-) -> WorkflowEvent:
- """Incoming chat message envelope. ``subject`` is the session_id so
- consumers can route per-conversation.
-
- ``workflow_id`` is carried verbatim onto the envelope — the scoping
- DECISION belongs to the core call site (routers/websocket.py) and
- the scoped-delivery rule to core dispatch; this factory only plumbs
- the field of its own wire shape.
- """
- payload = dict(event_data)
- session_id = payload.get("session_id")
- return WorkflowEvent(
- source="opencompany://nodes/chat_trigger",
- type="com.opencompany.chat.message.received",
- subject=str(session_id) if session_id else None,
- workflow_id=workflow_id,
- data=payload,
- )
-
-
-async def dispatch_chat_message_received(
- event_data: Mapping[str, Any],
- *,
- workflow_id: Optional[str] = None,
-) -> None:
- """Dispatch an incoming chat message via the canary CloudEvents path."""
- from services.events.dispatch import emit
-
- await emit(
- chat_message_received(dict(event_data), workflow_id=workflow_id),
- wire_routing_key=_WIRE_ROUTING_KEY,
- broadcast=False,
- )
-
-
-__all__ = [
- "chat_message_received",
- "dispatch_chat_message_received",
-]
diff --git a/server/routers/websocket.py b/server/routers/websocket.py
index 8ef7cadf..f25cf1eb 100644
--- a/server/routers/websocket.py
+++ b/server/routers/websocket.py
@@ -15,7 +15,6 @@ import asyncio
import uuid
import weakref
from typing import Dict, Any, Callable, Awaitable, Optional, Set
-from datetime import datetime, timezone
from fastapi import APIRouter, WebSocket, WebSocketDisconnect
from services.status_broadcaster import get_status_broadcaster
@@ -455,6 +454,11 @@ async def handle_execute_node(data: Dict[str, Any], websocket: WebSocket) -> Dic
)
if key in data
}
+ # A chat run's scope links what a node produces to the owner's chat, so
+ # only the worker relaying MachinaWorkflow's context may set it.
+ socket_scope = getattr(websocket, "scope", None) or {}
+ if "run_scope" in data and isinstance(socket_scope, dict) and socket_scope.get("path") == "/ws/internal":
+ invocation_extras["run_scope"] = data["run_scope"]
user_id = execution_principal(data, websocket)
await broadcaster.update_node_status(
@@ -1071,113 +1075,11 @@ async def handle_get_ai_models(data: Dict[str, Any], websocket: WebSocket) -> Di
# ============================================================================
-# Chat Message Handler (for chatTrigger nodes)
+# Chat — moved to services/chat/handlers.py with the chat run runtime
# ============================================================================
-
-
-@ws_handler("message")
-async def handle_send_chat_message(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:
- """Send a chat message to a workflow's chatTrigger nodes, and keep it in
- the workflow's thread (services/chat_thread.py).
-
- A workflow's session (``session_id`` = its id) takes a message only
- while a deployment would read it: the response's ``delivery`` is
- ``"now"`` while it runs, ``"queued"`` while it is paused (it runs on
- Resume). Otherwise the answer is ``not_running`` and nothing is saved
- or sent. Session ``"default"`` (no workflow open) is saved and
- broadcast to every deployment, as it always was.
- """
- # Wave 12 B7: route through chat_trigger plugin _events.py wrapper —
- # the chat_message_received event wire shape lives with the plugin,
- # not in this central WS router.
- from nodes.trigger.chat_trigger._events import dispatch_chat_message_received
- from services.chat_thread import delivery_for, record_chat_message
-
- message = data["message"]
- role = data.get("role", "user")
- session_id = data.get("session_id", "default")
- timestamp = data.get("timestamp") or datetime.now(timezone.utc).isoformat()
-
- # Chat is inherently workflow-scoped: the panel is bound to ONE
- # workflow and its ``session_id`` IS that workflow's id ("default"
- # only when no workflow is open — that legacy session keeps broadcast
- # semantics). The scope rides the envelope's ``workflow_id`` so
- # ``dispatch.emit`` signals only this workflow's consumers via the
- # EventWorkflowId Search Attribute; without it, one workflow's chat
- # message fired EVERY deployed workflow's chatTrigger. The scoping
- # decision lives here (core router) and the narrowing in core
- # dispatch — the plugin factory only carries the field.
- workflow_scope = session_id if session_id and session_id != "default" else None
-
- database = container.database()
- delivery = None
- if workflow_scope is not None:
- delivery = delivery_for(await database.get_latest_workflow_control(session_id))
- if delivery is None:
- return {"success": False, "error": "not_running"}
-
- # Kept in the thread, stamped with the live generation.
- await record_chat_message(database, session_id, role, message)
-
- # Build event data matching chatTrigger output schema
- event_data = {"message": message, "timestamp": timestamp, "session_id": session_id}
-
- # Dispatch via canary CloudEvents path — Visibility-query Signal
- # fan-out to running TriggerListenerWorkflow consumers + in-process
- # WS broadcast on ``chat_message_received``.
- await dispatch_chat_message_received(event_data, workflow_id=workflow_scope)
-
- logger.info(f"[ChatMessage] Dispatched canary event for session={session_id}")
-
- response = {"success": True, "message": "Chat message sent", "timestamp": timestamp}
- if delivery is not None:
- response["delivery"] = delivery
- return response
-
-
-@ws_handler()
-async def handle_get_chat_messages(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:
- """A session's messages, oldest first, the newest ``limit`` of them:
- ``{id, role, message, timestamp, run_key}``. ``run_key`` is the
- generation the message was written in. The live generation's only,
- unless ``all_generations`` (Home's thread, which also shows a message
- from before a Start). A Reset clears a workflow's thread (the chat
- nodes' Reset hook), so generations rarely mix."""
- session_id = data.get("session_id", "default")
- limit = data.get("limit") # Optional limit
-
- database = container.database()
- if data.get("all_generations"):
- rows = await database.get_chat_messages(session_id, limit)
- else:
- control = await database.get_latest_workflow_control(session_id)
- if control is not None and control.status == "reset":
- rows = []
- else:
- rows = await database.get_chat_messages(
- session_id, limit,
- execution_id=control.root_execution_id if control is not None else None,
- )
- messages = [
- {"id": row["id"], "role": row["role"], "message": row["message"], "timestamp": row["timestamp"], "run_key": row["execution_id"]}
- for row in rows
- ]
-
- return {"success": True, "messages": messages, "session_id": session_id}
-
-
-@ws_handler()
-async def handle_clear_chat_messages(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:
- """Clear a session's chat, every generation of it. For a workflow's
- session the agent forgets the conversation too
- (``services.chat_thread.clear_chat_session``: the Context plugin clears
- the workflow's conversations), so it starts over with the chat."""
- from services.chat_thread import clear_chat_session
-
- session_id = data.get("session_id", "default")
- count = await clear_chat_session(container.database(), session_id)
-
- return {"success": True, "message": f"Cleared {count} chat messages", "cleared_count": count}
+# send_chat_message / get_chat_messages / chat_subscribe / chat_unsubscribe /
+# get_chat_run / clear_chat_messages / save_chat_message self-register via
+# services/chat/__init__.py.
@ws_handler()
@@ -1244,26 +1146,6 @@ async def handle_clear_console_logs(data: Dict[str, Any], websocket: WebSocket)
}
-@ws_handler("message", "role")
-async def handle_save_chat_message(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:
- """Save a single chat message (used for assistant responses)."""
- from services.chat_thread import record_chat_message
-
- session_id = data.get("session_id", "default")
- success = await record_chat_message(container.database(), session_id, data["role"], data["message"])
-
- return {"success": success, "message": "Chat message saved" if success else "Failed to save chat message"}
-
-
-@ws_handler()
-async def handle_get_chat_sessions(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:
- """Get list of all chat sessions."""
- database = container.database()
- sessions = await database.get_chat_sessions()
-
- return {"success": True, "sessions": sessions}
-
-
# ============================================================================
# Terminal Logs Handlers
# ============================================================================
@@ -1533,11 +1415,7 @@ MESSAGE_HANDLERS: Dict[str, MessageHandler] = {
# via _resolve_handler() defined above.
# Workflow storage operations — extracted to services/workflow_storage/handlers.py
# (Wave 13.7); registered via ws_handler_registry on package import.
- # Chat message (for chatTrigger nodes)
- "send_chat_message": handle_send_chat_message,
- "get_chat_messages": handle_get_chat_messages,
- "clear_chat_messages": handle_clear_chat_messages,
- "save_chat_message": handle_save_chat_message,
+ # Chat commands live in services/chat/handlers.py (registered on import).
# Console logs (for Console nodes)
"get_console_logs": handle_get_console_logs,
"clear_console_logs": handle_clear_console_logs,
diff --git a/server/services/chat/__init__.py b/server/services/chat/__init__.py
new file mode 100644
index 00000000..4480e551
--- /dev/null
+++ b/server/services/chat/__init__.py
@@ -0,0 +1,40 @@
+"""Chat runs: the conversation between the owner and a workflow, as runs.
+
+The wire contract is docs-internal/chat_protocol.md. This package never
+imports ``nodes/``.
+
+- ``ledger``: admits a run with the owner's message, starts and finishes it,
+ and the watchdog's sweep ends runs nothing will.
+- ``hub``: delivers run events to the sockets subscribed to a session.
+- ``reducer``: a run's snapshot, folded from its events.
+- ``events``: the CloudEvents (the owner's message to the listeners, run
+ events).
+- ``access``: who may use a session.
+- ``handlers``: the WebSocket commands.
+- ``activities``: the Temporal activities MachinaWorkflow starts and
+ finishes runs through.
+- ``watchdog``: the periodic sweep main.py runs.
+
+Thread rows (and ``chat.updated``) stay with ``services/chat_thread.py``.
+
+Side-effect import (from main.py) registers the WebSocket commands and the
+disconnect listener that drops a closed socket's subscriptions.
+"""
+
+from __future__ import annotations
+
+from services.status_broadcaster import register_disconnect_listener as _register_disconnect_listener
+from services.ws_handler_registry import register_ws_handlers as _register_ws_handlers
+
+from .handlers import WS_HANDLERS as _CHAT_WS_HANDLERS
+from .hub import get_chat_hub
+
+
+def _drop_socket(websocket) -> None:
+ get_chat_hub().drop_socket(websocket)
+
+
+_register_ws_handlers(_CHAT_WS_HANDLERS)
+_register_disconnect_listener(_drop_socket)
+
+__all__ = ["get_chat_hub"]
diff --git a/server/services/chat/access.py b/server/services/chat/access.py
new file mode 100644
index 00000000..68113ebd
--- /dev/null
+++ b/server/services/chat/access.py
@@ -0,0 +1,61 @@
+"""Who may use a chat session.
+
+A workflow's session (its id) answers only the workflow's owner, compared
+with the socket's principal as the canvas and Memory panels do, and the
+internal worker socket may use none of chat's commands. The editor's
+``"default"`` session belongs to no workflow. A session that names no saved
+workflow is still scoped to that id: its messages reach no other workflow.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any, Optional
+
+from services.authz.ws_surface import execution_principal
+from services.chat_thread import DEFAULT_SESSION
+
+
+class ChatAccessDenied(Exception):
+ """The socket may not use this session."""
+
+ def __init__(self) -> None:
+ super().__init__("access_denied")
+
+
+@dataclass(frozen=True)
+class ChatScope:
+ session_id: str
+ #: The workflow the session belongs to; None for ``"default"``.
+ workflow_id: Optional[str]
+
+
+def _owner_of(saved: Any) -> str:
+ data = getattr(saved, "data", None)
+ if data is None and isinstance(saved, dict):
+ data = saved.get("data", saved)
+ return str(data.get("owner_id") or "") if isinstance(data, dict) else ""
+
+
+def session_id_of(data: Any) -> str:
+ """The request's session, ``"default"`` when it names none."""
+ value = data.get("session_id") if isinstance(data, dict) else None
+ text = str(value).strip() if value is not None else ""
+ return text or DEFAULT_SESSION
+
+
+async def authorize_session(database: Any, websocket: Any, session_id: str) -> ChatScope:
+ """The scope a socket may use, or :class:`ChatAccessDenied`."""
+ scope = getattr(websocket, "scope", None)
+ if isinstance(scope, dict) and scope.get("path") == "/ws/internal":
+ raise ChatAccessDenied()
+ if session_id == DEFAULT_SESSION:
+ return ChatScope(session_id=session_id, workflow_id=None)
+ saved = await database.get_workflow(session_id)
+ owner = _owner_of(saved) if saved is not None else ""
+ if owner and owner != execution_principal({}, websocket):
+ raise ChatAccessDenied()
+ return ChatScope(session_id=session_id, workflow_id=session_id)
+
+
+__all__ = ["ChatAccessDenied", "ChatScope", "authorize_session", "session_id_of"]
diff --git a/server/services/chat/activities.py b/server/services/chat/activities.py
new file mode 100644
index 00000000..de78bb66
--- /dev/null
+++ b/server/services/chat/activities.py
@@ -0,0 +1,66 @@
+"""The Temporal activities a chat run starts and finishes through.
+
+MachinaWorkflow schedules them, behind the ``machina-chat-run-v1`` patch,
+for a run that the owner's chat message spawned (services/temporal/
+workflow.py). Both are safe to retry: the ledger accepts a repeat from the
+workflow that claimed the run and publishes nothing twice. A failure raises,
+so Temporal retries; when the retries run out the workflow carries on and
+the watchdog ends the run.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict
+
+from temporalio import activity
+
+
[email protected](name="chat_run.start")
+async def start_chat_run_activity(payload: Dict[str, Any]) -> Dict[str, Any]:
+ """Claim the run for the calling workflow.
+
+ Payload ``{run_id, temporal_workflow_id, temporal_run_id}``. Returns
+ ``{claimed}``: False when another workflow claimed it first (several
+ chat triggers in one graph) or it can no longer start (it was stopped,
+ or the watchdog ended it); that workflow then runs without tracking.
+ """
+ from core.container import container
+ from services.chat import ledger
+
+ run = await ledger.start_run(
+ container.database(),
+ run_id=str(payload.get("run_id") or ""),
+ temporal_workflow_id=str(payload.get("temporal_workflow_id") or ""),
+ temporal_run_id=payload.get("temporal_run_id"),
+ )
+ return {"claimed": run is not None}
+
+
[email protected](name="chat_run.finish")
+async def finish_chat_run_activity(payload: Dict[str, Any]) -> Dict[str, Any]:
+ """End the run the calling workflow claimed.
+
+ Payload ``{run_id, temporal_workflow_id, temporal_run_id, success,
+ error?, code?, hint?, requires_user_action?}``. Returns ``{state}``, the
+ run's state afterwards, or None when the caller does not hold it.
+ """
+ from core.container import container
+ from services.chat import ledger
+
+ run = await ledger.finish_run(
+ container.database(),
+ run_id=str(payload.get("run_id") or ""),
+ temporal_workflow_id=str(payload.get("temporal_workflow_id") or ""),
+ temporal_run_id=payload.get("temporal_run_id"),
+ success=bool(payload.get("success")),
+ error=payload.get("error"),
+ code=payload.get("code"),
+ hint=payload.get("hint"),
+ requires_user_action=payload.get("requires_user_action"),
+ )
+ return {"state": run.state if run is not None else None}
+
+
+CHAT_RUN_ACTIVITIES = (start_chat_run_activity, finish_chat_run_activity)
+
+__all__ = ["CHAT_RUN_ACTIVITIES", "finish_chat_run_activity", "start_chat_run_activity"]
diff --git a/server/services/chat/config.py b/server/services/chat/config.py
new file mode 100644
index 00000000..2a4c45b4
--- /dev/null
+++ b/server/services/chat/config.py
@@ -0,0 +1,27 @@
+"""Chat runtime settings (``config/chat_defaults.json``), loaded once."""
+
+from __future__ import annotations
+
+import json
+from functools import lru_cache
+from pathlib import Path
+from typing import Any, Mapping
+
+CONFIG_PATH = Path(__file__).resolve().parents[2] / "config" / "chat_defaults.json"
+
+
+@lru_cache(maxsize=1)
+def load_chat_config() -> Mapping[str, Any]:
+ return json.loads(CONFIG_PATH.read_text(encoding="utf-8"))
+
+
+def runs_setting(name: str) -> float:
+ """``runs.<name>``: seconds, from the config file."""
+ return float(load_chat_config()["runs"][name])
+
+
+def hub_setting(name: str) -> int:
+ return int(load_chat_config()["hub"][name])
+
+
+__all__ = ["CONFIG_PATH", "hub_setting", "load_chat_config", "runs_setting"]
diff --git a/server/services/chat/events.py b/server/services/chat/events.py
new file mode 100644
index 00000000..6a0f4caf
--- /dev/null
+++ b/server/services/chat/events.py
@@ -0,0 +1,156 @@
+"""Chat's CloudEvents: the owner's message to a workflow, and run events.
+
+**The owner's message.** ``chat_message_received`` carries a message to the
+workflow's chatTrigger listeners through ``dispatch.emit`` (Temporal Signals,
+narrowed to the workflow by the envelope's ``workflow_id``). It is never
+broadcast: it carries the owner's text, and a broadcast reaches every socket;
+clients learn of new messages from the identity-only ``chat.updated``. When
+the message started a run, the event id is the run id, so the run a listener
+spawns has a predictable id (``<slug>-<trigger label>-<run id>``) and a
+re-sent event starts nothing twice. ``data.run_id`` names the run;
+MachinaWorkflow trusts it only from this module's source and type
+(``services/temporal/workflow.py``).
+
+**Run events.** ``chat_run_event`` frames tell a session's subscribers how a
+run moves (``services/chat/hub.py``). AG-UI's event model: the type is
+``com.opencompany.chat.run.<suffix>``, the id ``<run id>:<seq>``, the subject
+the run id, and the scope (``workflow_id``, ``session_id``, ``run_id``, ``seq``,
+``hub_epoch``) sits in ``data``, never in top-level extension attributes.
+docs-internal/chat_protocol.md lists every suffix and its fields.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict, Mapping, Optional
+
+from services.events.envelope import WorkflowEvent
+
+SOURCE = "opencompany://services/chat"
+MESSAGE_RECEIVED_TYPE = "com.opencompany.chat.message.received"
+RUN_TYPE_PREFIX = "com.opencompany.chat.run."
+#: WebSocket ``type`` of a run event frame.
+RUN_WIRE_KEY = "chat_run_event"
+#: ``dispatch.emit``'s wire routing key for the owner's message; matches
+#: ``ChatTriggerNode.event_type``.
+MESSAGE_WIRE_ROUTING_KEY = "chat_message_received"
+
+RUN_SUFFIXES = frozenset(
+ {
+ "started",
+ "finished",
+ "failed",
+ "step.started",
+ "step.finished",
+ "text.started",
+ "text.content",
+ "text.ended",
+ "tool_call.started",
+ "tool_call.args",
+ "tool_call.ended",
+ "tool_call.result",
+ "activity.snapshot",
+ "activity.delta",
+ "reasoning.started",
+ "reasoning.ended",
+ "custom",
+ }
+)
+#: Suffixes after which a run publishes nothing more.
+TERMINAL_SUFFIXES = frozenset({"finished", "failed"})
+
+
+def chat_message_received(
+ event_data: Mapping[str, Any],
+ *,
+ workflow_id: Optional[str] = None,
+ event_id: Optional[str] = None,
+) -> WorkflowEvent:
+ """The owner's message as the chatTrigger listeners receive it.
+ ``subject`` is the session. ``workflow_id`` scopes delivery to one
+ workflow's listeners; the caller decides it."""
+ payload = dict(event_data)
+ session_id = payload.get("session_id")
+ fields: Dict[str, Any] = {}
+ if event_id:
+ fields["id"] = event_id
+ return WorkflowEvent(
+ source=SOURCE,
+ type=MESSAGE_RECEIVED_TYPE,
+ subject=str(session_id) if session_id else None,
+ workflow_id=workflow_id,
+ data=payload,
+ **fields,
+ )
+
+
+async def dispatch_chat_message_received(
+ event_data: Mapping[str, Any],
+ *,
+ workflow_id: Optional[str] = None,
+ event_id: Optional[str] = None,
+) -> None:
+ """Signal the message to the listeners, without a broadcast."""
+ from services.events.dispatch import emit
+
+ await emit(
+ chat_message_received(event_data, workflow_id=workflow_id, event_id=event_id),
+ wire_routing_key=MESSAGE_WIRE_ROUTING_KEY,
+ broadcast=False,
+ )
+
+
+def chat_run_event(
+ *,
+ suffix: str,
+ run_id: str,
+ seq: int,
+ hub_epoch: str,
+ workflow_id: Optional[str],
+ session_id: str,
+ fields: Optional[Mapping[str, Any]] = None,
+) -> WorkflowEvent:
+ """One run event. ``fields`` are the suffix's own fields; the scope is
+ added here and cannot be overridden by them."""
+ if suffix not in RUN_SUFFIXES:
+ raise ValueError(f"not a chat run event: {suffix!r}")
+ data: Dict[str, Any] = {**dict(fields or {})}
+ data.update(
+ {
+ "workflow_id": workflow_id,
+ "session_id": session_id,
+ "run_id": run_id,
+ "seq": seq,
+ "hub_epoch": hub_epoch,
+ }
+ )
+ return WorkflowEvent(
+ id=f"{run_id}:{seq}",
+ source=SOURCE,
+ type=f"{RUN_TYPE_PREFIX}{suffix}",
+ subject=run_id,
+ data=data,
+ )
+
+
+def run_event_suffix(event_type: str) -> Optional[str]:
+ """``started`` for ``com.opencompany.chat.run.started``; None for any
+ other type."""
+ if not event_type.startswith(RUN_TYPE_PREFIX):
+ return None
+ suffix = event_type[len(RUN_TYPE_PREFIX):]
+ return suffix if suffix in RUN_SUFFIXES else None
+
+
+__all__ = [
+ "MESSAGE_RECEIVED_TYPE",
+ "MESSAGE_WIRE_ROUTING_KEY",
+ "RUN_SUFFIXES",
+ "RUN_TYPE_PREFIX",
+ "RUN_WIRE_KEY",
+ "SOURCE",
+ "TERMINAL_SUFFIXES",
+ "chat_message_received",
+ "chat_run_event",
+ "dispatch_chat_message_received",
+ "run_event_suffix",
+]
diff --git a/server/services/chat/handlers.py b/server/services/chat/handlers.py
new file mode 100644
index 00000000..264f5715
--- /dev/null
+++ b/server/services/chat/handlers.py
@@ -0,0 +1,330 @@
+"""Chat's WebSocket commands (docs-internal/chat_protocol.md).
+
+Moved here from ``routers/websocket.py`` with the run runtime. Every command
+that names a session checks the socket first (``services/chat/access.py``).
+
+``send_chat_message`` saves the owner's message and dispatches it to the
+workflow's chat triggers. A workflow's session takes a message only while
+its deployment would read it (``delivery``: ``"now"`` while it runs,
+``"queued"`` while it is paused); otherwise nothing is saved and the answer
+is ``not_running``. When a deployed chat trigger will answer the session,
+the message starts a run, admitted in the same transaction
+(``services/chat/ledger.py``), and the response names it. The editor's
+``"default"`` session is saved and dispatched to every deployment, without a
+run, as it always was.
+"""
+
+from __future__ import annotations
+
+from datetime import datetime, timezone
+from typing import Any, Dict, List, Optional
+
+from fastapi import WebSocket
+
+from core.container import container
+from core.logging import get_logger
+from services.chat import ledger, reducer
+from services.chat.access import ChatAccessDenied, authorize_session, session_id_of
+from services.chat.events import MESSAGE_WIRE_ROUTING_KEY, dispatch_chat_message_received
+from services.chat.hub import get_chat_hub
+from services.chat_thread import (
+ DEFAULT_SESSION,
+ announce_chat_updated,
+ chat_execution_id,
+ clear_chat_session,
+ delivery_for,
+ record_chat_message,
+)
+from services.ws_handler_registry import ws_handler
+
+logger = get_logger(__name__)
+
+PROTOCOL_VERSION = 1
+_DENIED = {"success": False, "error": "access_denied"}
+
+
+def _wire_message(row: Dict[str, Any]) -> Dict[str, Any]:
+ """A stored message as ``get_chat_messages`` sends it."""
+ message_id = row.get("uid") or f"m{row.get('id')}"
+ wire: Dict[str, Any] = {
+ "id": message_id,
+ "legacy_id": row.get("id"),
+ "role": row.get("role"),
+ "kind": row.get("kind") or "text",
+ "text": row.get("message"),
+ "message": row.get("message"),
+ "timestamp": row.get("timestamp"),
+ "run_key": row.get("execution_id"),
+ "run_id": row.get("run_id"),
+ "parent_id": row.get("parent_uid"),
+ "status": row.get("status") or "complete",
+ "attachments": row.get("attachments") or [],
+ "parts": row.get("parts") or {},
+ "feedback": None,
+ "siblings": {"index": 0, "count": 1, "ids": [message_id]},
+ # Editing arrives with branches; until then no message is editable.
+ "editable": False,
+ }
+ client_message_id = (row.get("meta") or {}).get("client_message_id")
+ if client_message_id:
+ wire["client_message_id"] = client_message_id
+ return wire
+
+
+async def run_snapshots(database: Any, session_id: str) -> List[Dict[str, Any]]:
+ """Snapshots of a session's live runs. The hub is read before the rows,
+ so an event published in between carries a newer ``seq`` than the
+ snapshot and still reaches a subscriber."""
+ hub = get_chat_hub()
+ live = hub.session_live(session_id)
+ rows = await ledger.live_runs(database, session_id)
+ return [{**reducer.merge_snapshot(reducer.snapshot_from_row(run), live.get(run.run_id)), "hub_epoch": hub.epoch} for run in rows]
+
+
+async def _thread_state(database: Any, session_id: str) -> Dict[str, Any]:
+ from models.chat import ChatThread
+
+ async with database.get_session() as session:
+ thread = await session.get(ChatThread, session_id)
+ return {
+ "active_leaf_id": thread.active_leaf_uid if thread is not None else None,
+ "revision": thread.revision if thread is not None else 0,
+ }
+
+
+async def _answers_session(database: Any, control: Any, session_id: str) -> bool:
+ """Whether a chat trigger in the deployed graph accepts this session, so
+ a run started for the message will be picked up. Decided from the
+ trigger's own filter (``event_waiter.build_filter``), never from a node
+ type name."""
+ from services import event_waiter
+
+ graph = getattr(control, "graph_snapshot", None) or {}
+ for node in graph.get("nodes") or []:
+ node_type = str(node.get("type") or "")
+ config = event_waiter.get_trigger_config(node_type)
+ if config is None or config.event_type != MESSAGE_WIRE_ROUTING_KEY:
+ continue
+ if (node.get("data") or {}).get("disabled"):
+ continue
+ params = await database.get_node_parameters(str(node.get("id") or "")) or {}
+ try:
+ if event_waiter.build_filter(node_type, params)({"session_id": session_id, "message": ""}):
+ return True
+ except Exception: # noqa: BLE001 - a broken filter admits, as the listener's does
+ return True
+ return False
+
+
+@ws_handler("message")
+async def handle_send_chat_message(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:
+ """Save the owner's message, start its run, and dispatch it. Answers
+ ``{message_id, run_id, delivery, timestamp}``; ``run_in_progress`` (with
+ the live ``run_id``), ``not_running``, ``save_failed``, ``access_denied``
+ or ``invalid_request`` otherwise."""
+ message = data.get("message")
+ if not isinstance(message, str) or not message.strip():
+ return {"success": False, "error": "invalid_request", "detail": "message must be text"}
+ if data.get("role", "user") != "user":
+ return {"success": False, "error": "invalid_request", "detail": "send_chat_message sends the owner's messages"}
+ session_id = session_id_of(data)
+ timestamp = data.get("timestamp") or datetime.now(timezone.utc).isoformat()
+ database = container.database()
+ try:
+ scope = await authorize_session(database, websocket, session_id)
+ except ChatAccessDenied:
+ return dict(_DENIED)
+
+ # The scope rides the envelope's ``workflow_id``, so ``dispatch.emit``
+ # signals only this workflow's listeners (without it one workflow's chat
+ # fired every deployed chat trigger). The editor's "default" session
+ # keeps its unscoped delivery.
+ control = None
+ delivery: Optional[str] = None
+ if scope.workflow_id is not None:
+ control = await database.get_latest_workflow_control(session_id)
+ delivery = delivery_for(control)
+ if delivery is None:
+ return {"success": False, "error": "not_running"}
+ track = scope.workflow_id is not None and await _answers_session(database, control, session_id)
+
+ try:
+ admission = await ledger.admit_message(
+ database,
+ session_id=session_id,
+ workflow_id=scope.workflow_id,
+ execution_id=chat_execution_id(control) if scope.workflow_id is not None else None,
+ text=message,
+ track=track,
+ state="queued" if delivery == "queued" else "pending",
+ client_message_id=data.get("client_message_id"),
+ )
+ except ledger.RunInProgress as exc:
+ return {"success": False, "error": "run_in_progress", "run_id": exc.run.run_id}
+ except ValueError as exc:
+ return {"success": False, "error": "invalid_request", "detail": str(exc)}
+ except Exception:
+ logger.warning("Chat message could not be saved", session_id=session_id, exc_info=True)
+ return {"success": False, "error": "save_failed"}
+
+ row, run = admission.message, admission.run
+ if admission.created:
+ await announce_chat_updated(session_id, "user")
+ event_data: Dict[str, Any] = {
+ "message": message,
+ "timestamp": timestamp,
+ "session_id": session_id,
+ "message_id": row["uid"],
+ }
+ if run is not None:
+ event_data["run_id"] = run.run_id
+ await dispatch_chat_message_received(
+ event_data,
+ workflow_id=scope.workflow_id,
+ event_id=run.run_id if run is not None else None,
+ )
+ logger.info("Chat message dispatched", session_id=session_id, run_id=run.run_id if run is not None else None)
+
+ response: Dict[str, Any] = {
+ "success": True,
+ "message": "Chat message sent",
+ "timestamp": timestamp,
+ "message_id": row["uid"],
+ "run_id": run.run_id if run is not None else None,
+ }
+ if delivery is not None:
+ response["delivery"] = delivery
+ return response
+
+
+@ws_handler()
+async def handle_get_chat_messages(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:
+ """A session's messages, oldest first, the newest ``limit`` of them, with
+ the thread's state and its live runs. The live generation's messages
+ only, unless ``all_generations`` (Home's thread, which also shows a
+ message from before a Start). A read that fails answers ``read_failed``,
+ never an empty thread."""
+ session_id = session_id_of(data)
+ limit = data.get("limit")
+ database = container.database()
+ try:
+ await authorize_session(database, websocket, session_id)
+ except ChatAccessDenied:
+ return dict(_DENIED)
+ try:
+ if data.get("all_generations"):
+ rows = await database.read_chat_messages(session_id, limit)
+ else:
+ control = await database.get_latest_workflow_control(session_id)
+ if control is not None and control.status == "reset":
+ rows = []
+ else:
+ rows = await database.read_chat_messages(
+ session_id, limit,
+ execution_id=control.root_execution_id if control is not None else None,
+ )
+ thread = await _thread_state(database, session_id)
+ active_runs = await run_snapshots(database, session_id)
+ except Exception:
+ logger.warning("Chat messages could not be read", session_id=session_id, exc_info=True)
+ return {"success": False, "error": "read_failed", "session_id": session_id}
+ return {
+ "success": True,
+ "protocol_version": PROTOCOL_VERSION,
+ "session_id": session_id,
+ "messages": [_wire_message(row) for row in rows],
+ "thread": thread,
+ "active_runs": active_runs,
+ }
+
+
+@ws_handler()
+async def handle_chat_subscribe(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:
+ """Send the session's run events to this socket from now on. Answers
+ the hub's epoch and the live runs' snapshots; events after a snapshot
+ carry a higher ``seq`` than it."""
+ session_id = session_id_of(data)
+ database = container.database()
+ try:
+ await authorize_session(database, websocket, session_id)
+ except ChatAccessDenied:
+ return dict(_DENIED)
+ hub = get_chat_hub()
+ hub.subscribe(websocket, session_id)
+ try:
+ active_runs = await run_snapshots(database, session_id)
+ except Exception:
+ logger.warning("Chat runs could not be read", session_id=session_id, exc_info=True)
+ return {"success": False, "error": "read_failed", "session_id": session_id, "hub_epoch": hub.epoch}
+ return {"success": True, "session_id": session_id, "hub_epoch": hub.epoch, "active_runs": active_runs}
+
+
+@ws_handler()
+async def handle_chat_unsubscribe(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:
+ """Stop sending the session's run events to this socket."""
+ session_id = session_id_of(data)
+ hub = get_chat_hub()
+ hub.unsubscribe(websocket, session_id)
+ return {"success": True, "session_id": session_id, "hub_epoch": hub.epoch}
+
+
+@ws_handler("run_id")
+async def handle_get_chat_run(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:
+ """One run's snapshot, live or ended."""
+ database = container.database()
+ run = await ledger.get_run(database, str(data["run_id"]))
+ if run is None:
+ return {"success": False, "error": "not_found"}
+ try:
+ await authorize_session(database, websocket, run.session_id)
+ except ChatAccessDenied:
+ return dict(_DENIED)
+ hub = get_chat_hub()
+ live = hub.live_snapshot(run.run_id)
+ run = await ledger.get_run(database, run.run_id) or run
+ snapshot = reducer.merge_snapshot(reducer.snapshot_from_row(run), live)
+ return {"success": True, "run": {**snapshot, "hub_epoch": hub.epoch}}
+
+
+@ws_handler()
+async def handle_clear_chat_messages(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:
+ """Clear a session's chat, every generation of it, with its runs. For a
+ workflow's session the agent forgets the conversation too
+ (``services.chat_thread.clear_chat_session``)."""
+ session_id = session_id_of(data)
+ database = container.database()
+ try:
+ await authorize_session(database, websocket, session_id)
+ except ChatAccessDenied:
+ return dict(_DENIED)
+ count = await clear_chat_session(database, session_id)
+ return {"success": True, "message": f"Cleared {count} chat messages", "cleared_count": count}
+
+
+@ws_handler("message", "role")
+async def handle_save_chat_message(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:
+ """Save one message with the given role, without dispatching it."""
+ role = data["role"]
+ if role not in ("user", "assistant"):
+ return {"success": False, "error": "invalid_request", "detail": "role is user or assistant"}
+ session_id = session_id_of(data)
+ database = container.database()
+ try:
+ await authorize_session(database, websocket, session_id)
+ except ChatAccessDenied:
+ return dict(_DENIED)
+ saved = await record_chat_message(database, session_id, role, data["message"])
+ return {"success": bool(saved), "message": "Chat message saved" if saved else "Failed to save chat message"}
+
+
+WS_HANDLERS = {
+ "send_chat_message": handle_send_chat_message,
+ "get_chat_messages": handle_get_chat_messages,
+ "chat_subscribe": handle_chat_subscribe,
+ "chat_unsubscribe": handle_chat_unsubscribe,
+ "get_chat_run": handle_get_chat_run,
+ "clear_chat_messages": handle_clear_chat_messages,
+ "save_chat_message": handle_save_chat_message,
+}
+
+__all__ = ["DEFAULT_SESSION", "PROTOCOL_VERSION", "WS_HANDLERS", "run_snapshots"]
diff --git a/server/services/chat/hub.py b/server/services/chat/hub.py
new file mode 100644
index 00000000..9ba8aa8f
--- /dev/null
+++ b/server/services/chat/hub.py
@@ -0,0 +1,266 @@
+"""Live run events for the sockets that asked for them.
+
+A socket subscribes to a chat session (``chat_subscribe``, once the handler
+has authorized it) and from then on receives that session's
+``chat_run_event`` frames, in order, from its own writer task, so a slow
+socket never holds up the publisher or any other socket. Run events never
+go through ``StatusBroadcaster.broadcast``, which reaches every socket.
+
+- ``seq`` counts a run's events from 1 within this process. ``epoch`` names
+ the process: after a restart the counters start over and the epoch
+ changes, which tells a client to take a fresh snapshot.
+- An event whose ``event_key`` the run already published is dropped, so a
+ retried activity publishes once. Nothing is published for a run after its
+ terminal event.
+- The hub keeps each live run's snapshot (``services/chat/reducer.py``), so a
+ client that subscribes mid-run gets what the earlier events built.
+- Each socket queues at most ``hub.subscriber_queue_size`` frames
+ (``config/chat_defaults.json``). A socket that falls that far behind has
+ its queue replaced by one ``opencompany.resync`` frame per session it
+ follows: the client takes a fresh snapshot instead of reading a stream
+ with a hole in it.
+
+Delivery is in-process. An activity on a worker in another process
+publishes into that process's hub, which no socket here reads.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from collections import OrderedDict, deque
+from typing import Any, Deque, Dict, Mapping, Optional, Set
+from uuid import uuid4
+
+import orjson
+from starlette.websockets import WebSocketState
+
+from core.logging import get_logger
+from services.chat import reducer
+from services.chat.events import RUN_WIRE_KEY, SOURCE, TERMINAL_SUFFIXES, chat_run_event
+from services.events.envelope import WorkflowEvent
+
+logger = get_logger(__name__)
+
+#: Ended runs remembered so a late publish for one is dropped.
+_ENDED_RUNS_KEPT = 2048
+
+
+def _open(websocket: Any) -> bool:
+ return (
+ getattr(websocket, "client_state", WebSocketState.CONNECTED) is WebSocketState.CONNECTED
+ and getattr(websocket, "application_state", WebSocketState.CONNECTED) is WebSocketState.CONNECTED
+ )
+
+
+class _Subscriber:
+ """One socket's subscriptions and its writer."""
+
+ def __init__(self, hub: "ChatRunHub", websocket: Any) -> None:
+ self.hub = hub
+ self.websocket = websocket
+ self.sessions: Set[str] = set()
+ self.frames: Deque[str] = deque()
+ self.wake = asyncio.Event()
+ self.task: Optional[asyncio.Task] = None
+
+ def offer(self, frame: str) -> None:
+ if len(self.frames) >= self.hub.queue_size:
+ self.frames.clear()
+ for session_id in sorted(self.sessions):
+ self.frames.append(self.hub.resync_frame(session_id))
+ logger.info("Chat subscriber fell behind; sent resync", sessions=len(self.sessions))
+ self.frames.append(frame)
+ self.wake.set()
+ if self.task is None or self.task.done():
+ self.task = asyncio.get_running_loop().create_task(self._write(), name="chat-hub-writer")
+
+ async def _write(self) -> None:
+ try:
+ while True:
+ await self.wake.wait()
+ self.wake.clear()
+ while self.frames:
+ frame = self.frames.popleft()
+ if not _open(self.websocket):
+ self.hub.drop_socket(self.websocket)
+ return
+ await self.websocket.send_text(frame)
+ except asyncio.CancelledError:
+ raise
+ except Exception as exc: # noqa: BLE001 - the socket is gone; forget it
+ logger.debug("Chat hub send failed; dropping the socket", error=str(exc))
+ self.hub.drop_socket(self.websocket)
+
+ def close(self) -> None:
+ self.frames.clear()
+ if self.task is not None and not self.task.done() and self.task is not asyncio.current_task():
+ self.task.cancel()
+
+
+class ChatRunHub:
+ def __init__(self, *, queue_size: Optional[int] = None) -> None:
+ if queue_size is None:
+ from services.chat.config import hub_setting
+
+ queue_size = hub_setting("subscriber_queue_size")
+ self.queue_size = max(1, int(queue_size))
+ self.epoch = f"e_{uuid4().hex[:12]}"
+ # Keyed by the socket object itself, never ``id()``: a reused id
+ # would hand a new socket an old one's sessions.
+ self._subscribers: Dict[Any, _Subscriber] = {}
+ self._sessions: Dict[str, Set[Any]] = {}
+ self._seq: Dict[str, int] = {}
+ self._keys: Dict[str, Set[str]] = {}
+ self._live: Dict[str, Dict[str, Any]] = {}
+ self._ended: "OrderedDict[str, None]" = OrderedDict()
+
+ # ---- subscriptions -------------------------------------------------
+
+ def subscribe(self, websocket: Any, session_id: str) -> None:
+ subscriber = self._subscribers.get(websocket)
+ if subscriber is None:
+ subscriber = _Subscriber(self, websocket)
+ self._subscribers[websocket] = subscriber
+ subscriber.sessions.add(session_id)
+ self._sessions.setdefault(session_id, set()).add(websocket)
+
+ def unsubscribe(self, websocket: Any, session_id: Optional[str] = None) -> None:
+ """Stop sending one session (or, without ``session_id``, any) to
+ the socket."""
+ subscriber = self._subscribers.get(websocket)
+ if subscriber is None:
+ return
+ sessions = [session_id] if session_id is not None else list(subscriber.sessions)
+ for session in sessions:
+ subscriber.sessions.discard(session)
+ members = self._sessions.get(session)
+ if members is not None:
+ members.discard(websocket)
+ if not members:
+ del self._sessions[session]
+ if not subscriber.sessions:
+ subscriber.close()
+ del self._subscribers[websocket]
+
+ def drop_socket(self, websocket: Any) -> None:
+ """Forget a closed socket (the status broadcaster calls this on
+ disconnect)."""
+ self.unsubscribe(websocket)
+
+ def subscribed(self, websocket: Any, session_id: str) -> bool:
+ subscriber = self._subscribers.get(websocket)
+ return subscriber is not None and session_id in subscriber.sessions
+
+ # ---- runs ----------------------------------------------------------
+
+ def current_seq(self, run_id: str) -> int:
+ return self._seq.get(run_id, 0)
+
+ def live_snapshot(self, run_id: str) -> Optional[Dict[str, Any]]:
+ snapshot = self._live.get(run_id)
+ return dict(snapshot) if snapshot is not None else None
+
+ def session_live(self, session_id: str) -> Dict[str, Dict[str, Any]]:
+ """The live snapshots of a session's runs, by run id. Read it before
+ reading the runs' rows: an event published in between then reaches a
+ subscriber with a ``seq`` above its snapshot's, never below."""
+ return {run_id: dict(snapshot) for run_id, snapshot in self._live.items() if snapshot.get("session_id") == session_id}
+
+ def ended(self, run_id: str) -> bool:
+ return run_id in self._ended
+
+ def publish(
+ self,
+ *,
+ run_id: str,
+ session_id: str,
+ workflow_id: Optional[str],
+ suffix: str,
+ fields: Optional[Mapping[str, Any]] = None,
+ event_key: Optional[str] = None,
+ ) -> Optional[WorkflowEvent]:
+ """Publish one run event to the session's subscribers. Returns the
+ event, or None when it was dropped (a repeated ``event_key``, or the
+ run already ended). Never raises for delivery problems: a socket that
+ cannot be written to is dropped."""
+ if run_id in self._ended:
+ return None
+ keys = self._keys.setdefault(run_id, set())
+ if event_key is not None:
+ if event_key in keys:
+ return None
+ keys.add(event_key)
+ seq = self._seq.get(run_id, 0) + 1
+ self._seq[run_id] = seq
+ event = chat_run_event(
+ suffix=suffix,
+ run_id=run_id,
+ seq=seq,
+ hub_epoch=self.epoch,
+ workflow_id=workflow_id,
+ session_id=session_id,
+ fields=fields,
+ )
+ payload = event.model_dump(mode="json", exclude_none=True)
+ self._live[run_id] = reducer.apply_event(self._live.get(run_id) or reducer.empty_snapshot(run_id), payload)
+ if suffix in TERMINAL_SUFFIXES:
+ self._end(run_id)
+ self._fan_out(session_id, orjson.dumps({"type": RUN_WIRE_KEY, "data": payload}).decode())
+ return event
+
+ def _end(self, run_id: str) -> None:
+ self._seq.pop(run_id, None)
+ self._keys.pop(run_id, None)
+ self._live.pop(run_id, None)
+ self._ended[run_id] = None
+ while len(self._ended) > _ENDED_RUNS_KEPT:
+ self._ended.popitem(last=False)
+
+ def _fan_out(self, session_id: str, frame: str) -> None:
+ members = self._sessions.get(session_id)
+ if not members:
+ return
+ try:
+ asyncio.get_running_loop()
+ except RuntimeError:
+ return
+ for websocket in list(members):
+ subscriber = self._subscribers.get(websocket)
+ if subscriber is not None and session_id in subscriber.sessions:
+ subscriber.offer(frame)
+
+ def resync_frame(self, session_id: str) -> str:
+ event = WorkflowEvent(
+ id=f"resync:{uuid4().hex}",
+ source=SOURCE,
+ type="com.opencompany.chat.run.custom",
+ subject=session_id,
+ data={
+ "session_id": session_id,
+ "hub_epoch": self.epoch,
+ "name": "opencompany.resync",
+ "value": {},
+ },
+ )
+ return orjson.dumps({"type": RUN_WIRE_KEY, "data": event.model_dump(mode="json", exclude_none=True)}).decode()
+
+
+_hub: Optional[ChatRunHub] = None
+
+
+def get_chat_hub() -> ChatRunHub:
+ """The process's hub."""
+ global _hub
+ if _hub is None:
+ _hub = ChatRunHub()
+ return _hub
+
+
+def reset_chat_hub_for_tests(hub: Optional[ChatRunHub] = None) -> Optional[ChatRunHub]:
+ """Swap the process hub (tests). Returns the previous one."""
+ global _hub
+ previous, _hub = _hub, hub
+ return previous
+
+
+__all__ = ["ChatRunHub", "get_chat_hub", "reset_chat_hub_for_tests"]
diff --git a/server/services/chat/ledger.py b/server/services/chat/ledger.py
new file mode 100644
index 00000000..ac044935
--- /dev/null
+++ b/server/services/chat/ledger.py
@@ -0,0 +1,532 @@
+"""Chat runs: admitted with the owner's message, started and finished by the
+run that answers it, and ended by the watchdog when nothing will.
+
+**Admission** (``admit_message``) writes the owner's message and its run in
+one reserved transaction (``Database.reserved_session``), after checking the
+lane: one live run per session, resume runs aside, also enforced by the
+partial unique index on ``chat_runs``. A message re-sent with the same
+``client_message_id`` returns the first send's message and run instead of
+writing a second one; the id maps to a message id that only this session can
+produce (``client_message_uid``).
+
+**Start** (``start_run``) moves ``pending``/``queued`` to ``running`` and
+records the Temporal workflow that claimed it. Several chatTrigger nodes in
+one graph each spawn a run for the same message: the first claims the chat
+run, the others run untracked. **Finish** (``finish_run``) is accepted only
+from the claimant. A retried start or finish publishes nothing new: events
+carry an ``event_key`` the hub publishes once.
+
+**The watchdog** (``sweep``) ends runs nothing will end: a pending run never
+picked up (``not_delivered``, counted from the later of its creation and
+this process's start, so a restart does not expire runs it held up), a
+queued run whose deployment stopped or did not pick it up after resuming, a
+running run past ``runs.max_running_s`` (``timed_out``), and a running run
+whose Temporal workflow closed without finishing it: finished when its reply
+was saved, ``interrupted`` otherwise.
+
+Every transition commits before its event is published, so a client that
+reads the run after an event never sees an older state than the event.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import re
+from dataclasses import dataclass
+from datetime import datetime, timedelta, timezone
+from typing import Any, Awaitable, Callable, Dict, List, Optional
+
+from sqlalchemy import update
+from sqlalchemy.exc import IntegrityError
+from sqlmodel import select
+
+from core.logging import get_logger
+from models.chat import LIVE_STATES, TERMINAL_STATES, ChatRun
+from models.database import ChatMessage
+from services.chat.config import runs_setting
+from services.chat.hub import get_chat_hub
+from services.chat_thread import delivery_for
+
+logger = get_logger(__name__)
+
+PROCESS_STARTED = datetime.now(timezone.utc)
+
+_CLIENT_MESSAGE_ID = re.compile(r"^[A-Za-z0-9_.:-]{1,100}$")
+#: How long a closed Temporal workflow may leave its run open before the
+#: watchdog ends it (its finish activity may still be on its way).
+_CLOSED_GRACE = timedelta(seconds=60)
+
+#: ``await status(temporal_workflow_id, temporal_run_id)`` -> True when the
+#: workflow has closed, False while it runs, None when unknown.
+TemporalStatus = Callable[[str, Optional[str]], Awaitable[Optional[bool]]]
+
+
+class RunInProgress(Exception):
+ """The session's lane is held by ``run``."""
+
+ def __init__(self, run: ChatRun) -> None:
+ super().__init__("run_in_progress")
+ self.run = run
+
+
+@dataclass(frozen=True)
+class Admission:
+ #: The owner's message, as ``Database.chat_row`` shapes it.
+ message: Dict[str, Any]
+ #: Its run, or None for a message that starts none.
+ run: Optional[ChatRun]
+ #: False when ``client_message_id`` matched an earlier send.
+ created: bool
+
+
+def _utcnow() -> datetime:
+ return datetime.now(timezone.utc)
+
+
+def _aware(moment: Optional[datetime]) -> Optional[datetime]:
+ if moment is None:
+ return None
+ return moment if moment.tzinfo else moment.replace(tzinfo=timezone.utc)
+
+
+def new_run_id() -> str:
+ from uuid import uuid4
+
+ return f"r_{uuid4().hex}"
+
+
+def reply_uid(run_id: str) -> str:
+ """The id a run's reply is saved under."""
+ return f"a_{run_id}"
+
+
+def client_message_uid(session_id: str, client_message_id: Any) -> str:
+ """The message id for a client's ``client_message_id`` in a session.
+ Hashed with the session, so a client cannot name another chat's
+ message."""
+ if not isinstance(client_message_id, str) or not _CLIENT_MESSAGE_ID.match(client_message_id):
+ raise ValueError("client_message_id must be 1 to 100 letters, digits or the characters _.:-")
+ digest = hashlib.sha256(f"{session_id}\x00{client_message_id}".encode()).hexdigest()[:40]
+ return f"m_{digest}"
+
+
+# ---- reads -------------------------------------------------------------
+
+
+async def get_run(database: Any, run_id: str) -> Optional[ChatRun]:
+ async with database.get_session() as session:
+ return await session.get(ChatRun, run_id)
+
+
+async def lane_run(database: Any, session_id: str) -> Optional[ChatRun]:
+ """The run holding the session's lane, if any."""
+ async with database.get_session() as session:
+ result = await session.execute(
+ select(ChatRun)
+ .where(ChatRun.session_id == session_id, ChatRun.state.in_(LIVE_STATES), ChatRun.kind != "resume")
+ .limit(1)
+ )
+ return result.scalar_one_or_none()
+
+
+async def live_runs(database: Any, session_id: Optional[str] = None) -> List[ChatRun]:
+ """Live runs, oldest first: one session's, or every session's."""
+ async with database.get_session() as session:
+ query = select(ChatRun).where(ChatRun.state.in_(LIVE_STATES))
+ if session_id is not None:
+ query = query.where(ChatRun.session_id == session_id)
+ result = await session.execute(query.order_by(ChatRun.created_at, ChatRun.run_id))
+ return list(result.scalars().all())
+
+
+async def _message_exists(database: Any, uid: Optional[str]) -> bool:
+ if not uid:
+ return False
+ async with database.get_session() as session:
+ result = await session.execute(select(ChatMessage.id).where(ChatMessage.uid == uid))
+ return result.scalar_one_or_none() is not None
+
+
+# ---- admission ---------------------------------------------------------
+
+
+async def admit_message(
+ database: Any,
+ *,
+ session_id: str,
+ workflow_id: Optional[str],
+ execution_id: Optional[str],
+ text: str,
+ track: bool,
+ state: str = "pending",
+ kind: str = "message",
+ message_kind: str = "text",
+ client_message_id: Optional[str] = None,
+ options: Optional[Dict[str, Any]] = None,
+ attachments: Optional[List[Dict[str, Any]]] = None,
+) -> Admission:
+ """Save the owner's message and, when ``track``, the run that will answer
+ it. Raises :class:`RunInProgress` when the lane is held, ``ValueError``
+ for a malformed ``client_message_id``."""
+ if state not in ("pending", "queued"):
+ raise ValueError(f"a run is admitted pending or queued, not {state!r}")
+ uid = client_message_uid(session_id, client_message_id) if client_message_id is not None else None
+ meta = {"client_message_id": client_message_id} if client_message_id is not None else {}
+ async with database.reserved_session() as session:
+ if uid is not None:
+ found = (await session.execute(select(ChatMessage).where(ChatMessage.uid == uid))).scalar_one_or_none()
+ if found is not None:
+ run = await session.get(ChatRun, found.run_id) if found.run_id else None
+ return Admission(message=database.chat_row(found), run=run, created=False)
+ run_id: Optional[str] = None
+ if track:
+ held = await session.execute(
+ select(ChatRun)
+ .where(ChatRun.session_id == session_id, ChatRun.state.in_(LIVE_STATES), ChatRun.kind != "resume")
+ .limit(1)
+ )
+ holder = held.scalar_one_or_none()
+ if holder is not None:
+ # Detached first: the rollback on the way out would expire it.
+ session.expunge(holder)
+ raise RunInProgress(holder)
+ run_id = new_run_id()
+ row = await database.append_chat_row(
+ session,
+ session_id=session_id,
+ role="user",
+ message=text,
+ execution_id=execution_id,
+ uid=uid,
+ run_id=run_id,
+ kind=message_kind,
+ attachments=attachments,
+ meta=meta,
+ )
+ run: Optional[ChatRun] = None
+ if run_id is not None:
+ run = ChatRun(
+ run_id=run_id,
+ session_id=session_id,
+ workflow_id=workflow_id,
+ run_key=execution_id,
+ kind=kind,
+ state=state,
+ user_message_uid=row.uid,
+ reply_message_uid=reply_uid(run_id),
+ options=dict(options or {}),
+ )
+ session.add(run)
+ try:
+ await session.commit()
+ except IntegrityError:
+ # Without a write reservation (another database) the lane index
+ # is what refuses the second run.
+ await session.rollback()
+ holder = await lane_run(database, session_id)
+ if holder is not None:
+ raise RunInProgress(holder)
+ raise
+ return Admission(message=database.chat_row(row), run=run, created=True)
+
+
+async def post_reply(
+ database: Any,
+ *,
+ run: ChatRun,
+ node_id: str,
+ text: str,
+ execution_id: Optional[str],
+) -> Dict[str, Any]:
+ """Save a reply to the run's thread and return the row.
+
+ The run's first reply takes the run's reply id (``a_<run id>``), the id
+ its events name, so a client can match the answer it watched to the one
+ saved. Another reply node in the same run gets ``a_<run id>.<n>``. A
+ retry from the same node returns the row it saved. Serialized by the
+ write reservation, so two reply nodes at once cannot take the same id.
+ """
+ async with database.reserved_session() as session:
+ result = await session.execute(
+ select(ChatMessage)
+ .where(ChatMessage.run_id == run.run_id, ChatMessage.role == "assistant")
+ .order_by(ChatMessage.id)
+ )
+ replies = list(result.scalars().all())
+ for reply in replies:
+ if (reply.meta or {}).get("node_id") == node_id:
+ return database.chat_row(reply)
+ uid = run.reply_message_uid or reply_uid(run.run_id)
+ if replies:
+ uid = f"{uid}.{len(replies) + 1}"
+ row = await database.append_chat_row(
+ session,
+ session_id=run.session_id,
+ role="assistant",
+ message=text,
+ execution_id=execution_id,
+ uid=uid,
+ run_id=run.run_id,
+ meta={"node_id": node_id},
+ )
+ await session.commit()
+ return database.chat_row(row)
+
+
+# ---- transitions -------------------------------------------------------
+
+
+def _publish(run: ChatRun, suffix: str, fields: Dict[str, Any], event_key: str) -> None:
+ try:
+ get_chat_hub().publish(
+ run_id=run.run_id,
+ session_id=run.session_id,
+ workflow_id=run.workflow_id,
+ suffix=suffix,
+ fields=fields,
+ event_key=event_key,
+ )
+ except Exception: # noqa: BLE001 - the row is the record; an event is a courtesy
+ logger.warning("Chat run event could not be published", run_id=run.run_id, suffix=suffix, exc_info=True)
+
+
+def _iso(moment: Optional[datetime]) -> Optional[str]:
+ moment = _aware(moment)
+ return moment.isoformat() if moment is not None else None
+
+
+def publish_started(run: ChatRun) -> None:
+ fields: Dict[str, Any] = {
+ "kind": run.kind,
+ "reply_message_id": run.reply_message_uid,
+ "started_at": _iso(run.started_at),
+ }
+ if run.user_message_uid:
+ fields["user_message_id"] = run.user_message_uid
+ if run.parent_run_id:
+ fields["parent_run_id"] = run.parent_run_id
+ _publish(run, "started", fields, "started")
+
+
+def publish_terminal(run: ChatRun) -> None:
+ started, finished = _aware(run.started_at), _aware(run.finished_at)
+ duration_ms = int((finished - started).total_seconds() * 1000) if started and finished else 0
+ result = dict(run.result or {})
+ if run.state == "error":
+ fields: Dict[str, Any] = {"message": run.error or "The run failed", "code": run.error_code or "run_failed"}
+ for key in ("hint", "requires_user_action"):
+ if result.get(key) is not None:
+ fields[key] = result[key]
+ _publish(run, "failed", fields, "terminal")
+ return
+ outcome: Dict[str, Any] = {"type": run.outcome or "success"}
+ interrupts = result.pop("interrupts", None)
+ if run.outcome == "interrupt":
+ outcome["interrupts"] = list(interrupts or [])
+ _publish(
+ run,
+ "finished",
+ {"outcome": outcome, "result": result, "duration_ms": duration_ms, "step_count": len(run.steps or [])},
+ "terminal",
+ )
+
+
+async def start_run(
+ database: Any,
+ *,
+ run_id: str,
+ temporal_workflow_id: str,
+ temporal_run_id: Optional[str],
+) -> Optional[ChatRun]:
+ """Claim a run for the Temporal workflow that answers it. Returns the
+ run when this workflow holds it (also on a retry), None when another
+ workflow claimed it first or it can no longer start."""
+ async with database.get_session() as session:
+ await session.execute(
+ update(ChatRun)
+ .where(ChatRun.run_id == run_id, ChatRun.state.in_(("pending", "queued")))
+ .values(
+ state="running",
+ started_at=_utcnow(),
+ temporal_workflow_id=temporal_workflow_id,
+ temporal_run_id=temporal_run_id,
+ )
+ )
+ await session.commit()
+ run = await get_run(database, run_id)
+ if (
+ run is None
+ or run.state not in ("running", "stopping")
+ or run.temporal_workflow_id != temporal_workflow_id
+ or run.temporal_run_id != temporal_run_id
+ ):
+ return None
+ publish_started(run)
+ return run
+
+
+async def _settle(database: Any, run: ChatRun, values: Dict[str, Any]) -> Optional[ChatRun]:
+ """Move ``run`` from the state it was read in to a terminal one, then
+ publish it. None when the run moved meanwhile."""
+ async with database.get_session() as session:
+ result = await session.execute(
+ update(ChatRun).where(ChatRun.run_id == run.run_id, ChatRun.state == run.state).values(finished_at=_utcnow(), **values)
+ )
+ await session.commit()
+ if not result.rowcount:
+ return None
+ settled = await get_run(database, run.run_id)
+ if settled is not None and settled.state in TERMINAL_STATES:
+ publish_terminal(settled)
+ return settled
+
+
+async def _success_values(database: Any, run: ChatRun, outcome: str) -> Dict[str, Any]:
+ if await _message_exists(database, run.reply_message_uid):
+ result: Dict[str, Any] = {"reply_message_id": run.reply_message_uid}
+ else:
+ result = {"no_reply": True}
+ return {"state": "stopped" if outcome == "stopped" else "finished", "outcome": outcome, "result": result}
+
+
+async def finish_run(
+ database: Any,
+ *,
+ run_id: str,
+ temporal_workflow_id: str,
+ temporal_run_id: Optional[str],
+ success: bool,
+ error: Optional[str] = None,
+ code: Optional[str] = None,
+ hint: Optional[str] = None,
+ requires_user_action: Optional[bool] = None,
+) -> Optional[ChatRun]:
+ """End a run its claimant answered. A run stopped meanwhile ends
+ ``stopped``. Returns the ended run, or None when the caller does not
+ hold it."""
+ run = await get_run(database, run_id)
+ if run is None or run.temporal_workflow_id != temporal_workflow_id or run.temporal_run_id != temporal_run_id:
+ return None
+ if run.state in TERMINAL_STATES:
+ publish_terminal(run)
+ return run
+ if run.state not in ("running", "stopping"):
+ return None
+ if run.state == "stopping":
+ values = await _success_values(database, run, "stopped")
+ elif success:
+ values = await _success_values(database, run, "success")
+ else:
+ details = {key: value for key, value in (("hint", hint), ("requires_user_action", requires_user_action)) if value is not None}
+ values = {
+ "state": "error",
+ "error": (error or "The run failed")[:500],
+ "error_code": (code or "run_failed")[:60],
+ "result": details,
+ }
+ return await _settle(database, run, values)
+
+
+async def fail_run(database: Any, run: ChatRun, *, code: str, message: str) -> Optional[ChatRun]:
+ """End a live run with an error. None when it moved meanwhile."""
+ if run.state not in LIVE_STATES:
+ return None
+ return await _settle(database, run, {"state": "error", "error": message[:500], "error_code": code[:60], "result": {}})
+
+
+def publish_cleared(runs: List[ChatRun], *, code: str) -> None:
+ """Tell subscribers that the runs ended with their conversation (a Reset
+ or the owner's Clear deleted them)."""
+ message = "The conversation was reset." if code == "reset" else "The conversation was cleared."
+ for run in runs:
+ _publish(run, "failed", {"message": message, "code": code}, "terminal")
+
+
+# ---- the watchdog ------------------------------------------------------
+
+
+async def sweep(
+ database: Any,
+ *,
+ now: Optional[datetime] = None,
+ process_started: datetime = PROCESS_STARTED,
+ temporal_status: Optional[TemporalStatus] = None,
+) -> List[str]:
+ """End the live runs nothing will end. Returns their ids."""
+ now = now or _utcnow()
+ pickup = timedelta(seconds=runs_setting("pickup_timeout_s"))
+ longest = timedelta(seconds=runs_setting("max_running_s"))
+ ended: List[str] = []
+ for run in await live_runs(database):
+ try:
+ closed = await _sweep_one(database, run, now, process_started, pickup, longest, temporal_status)
+ except Exception: # noqa: BLE001 - one bad row must not stop the sweep
+ logger.warning("Chat run sweep failed for a run", run_id=run.run_id, exc_info=True)
+ continue
+ if closed is not None:
+ ended.append(closed.run_id)
+ if ended:
+ logger.info("Ended chat runs nothing would finish", count=len(ended))
+ return ended
+
+
+async def _sweep_one(
+ database: Any,
+ run: ChatRun,
+ now: datetime,
+ process_started: datetime,
+ pickup: timedelta,
+ longest: timedelta,
+ temporal_status: Optional[TemporalStatus],
+) -> Optional[ChatRun]:
+ created = max(_aware(run.created_at) or now, process_started)
+ if run.state == "pending":
+ if now - created > pickup:
+ return await fail_run(database, run, code="not_delivered", message="The employee did not pick up this message.")
+ return None
+ if run.state == "queued":
+ control = await database.get_latest_workflow_control(run.workflow_id) if run.workflow_id else None
+ delivery = delivery_for(control)
+ if delivery == "queued":
+ return None
+ if delivery == "now":
+ resumed = max(created, _aware(getattr(control, "updated_at", None)) or created)
+ if now - resumed > pickup:
+ return await fail_run(database, run, code="not_delivered", message="The employee did not pick up this message.")
+ return None
+ return await fail_run(database, run, code="not_delivered", message="The employee stopped before reading this message.")
+ started = _aware(run.started_at) or created
+ if now - started > longest:
+ return await fail_run(database, run, code="timed_out", message="The employee took too long to answer.")
+ if temporal_status is None or not run.temporal_workflow_id or now - started < _CLOSED_GRACE:
+ return None
+ if await temporal_status(run.temporal_workflow_id, run.temporal_run_id) is not True:
+ return None
+ current = await get_run(database, run.run_id)
+ if current is None or current.state not in ("running", "stopping"):
+ return None
+ if await _message_exists(database, current.reply_message_uid):
+ return await _settle(database, current, await _success_values(database, current, "stopped" if current.state == "stopping" else "success"))
+ return await fail_run(database, current, code="interrupted", message="The run ended before it answered.")
+
+
+__all__ = [
+ "Admission",
+ "PROCESS_STARTED",
+ "RunInProgress",
+ "admit_message",
+ "client_message_uid",
+ "fail_run",
+ "finish_run",
+ "get_run",
+ "lane_run",
+ "live_runs",
+ "new_run_id",
+ "post_reply",
+ "publish_cleared",
+ "publish_started",
+ "publish_terminal",
+ "reply_uid",
+ "start_run",
+ "sweep",
+]
diff --git a/server/services/chat/reducer.py b/server/services/chat/reducer.py
new file mode 100644
index 00000000..a431907e
--- /dev/null
+++ b/server/services/chat/reducer.py
@@ -0,0 +1,190 @@
+"""A run's snapshot: what a client needs to draw a run without its events.
+
+``snapshot_from_row`` shapes a ``ChatRun`` row; ``apply_event`` folds one
+run event into a snapshot the way the client folds the stream, so a client
+that subscribes mid-run starts from the same state the events before would
+have built. The hub keeps one live snapshot per run it publishes for
+(``services/chat/hub.py``); ``merge_snapshot`` lays it over the stored row.
+
+Lifecycle, step and text events are folded. Activity events (generated UI,
+approvals, sources) are not yet: nothing publishes them before generated UI
+lands, and folding them needs the RFC 6902 patch code that comes with it.
+"""
+
+from __future__ import annotations
+
+from copy import deepcopy
+from datetime import datetime, timezone
+from typing import Any, Dict, List, Mapping, Optional
+
+from services.chat.events import run_event_suffix
+
+#: Run states a client draws as working.
+LIVE_STATES = frozenset({"queued", "pending", "running", "stopping"})
+
+
+def _iso(moment: Optional[datetime]) -> Optional[str]:
+ if moment is None:
+ return None
+ if moment.tzinfo is None:
+ moment = moment.replace(tzinfo=timezone.utc)
+ return moment.isoformat()
+
+
+def empty_snapshot(run_id: str) -> Dict[str, Any]:
+ return {
+ "run_id": run_id,
+ "session_id": None,
+ "workflow_id": None,
+ "kind": "message",
+ "state": "pending",
+ "seq": 0,
+ "user_message_id": None,
+ "reply_message_id": None,
+ "parent_run_id": None,
+ "created_at": None,
+ "started_at": None,
+ "finished_at": None,
+ "steps": [],
+ "segments": [],
+ "activities": [],
+ "interrupts": [],
+ "outcome": None,
+ "result": {},
+ "error": None,
+ "error_code": None,
+ }
+
+
+def snapshot_from_row(run: Any) -> Dict[str, Any]:
+ """A stored run as a snapshot. Steps come from the row (saved as they
+ finish); text is not stored until the reply is, so a run read after a
+ restart has no segments."""
+ snapshot = empty_snapshot(run.run_id)
+ outcome = {"type": run.outcome} if run.outcome else None
+ if outcome is not None and run.outcome == "interrupt":
+ outcome["interrupts"] = list((run.result or {}).get("interrupts") or [])
+ snapshot.update(
+ {
+ "session_id": run.session_id,
+ "workflow_id": run.workflow_id,
+ "kind": run.kind,
+ "state": run.state,
+ "user_message_id": run.user_message_uid,
+ "reply_message_id": run.reply_message_uid,
+ "parent_run_id": run.parent_run_id,
+ "created_at": _iso(run.created_at),
+ "started_at": _iso(run.started_at),
+ "finished_at": _iso(run.finished_at),
+ "steps": [dict(step) for step in (run.steps or [])],
+ "outcome": outcome,
+ "result": {key: value for key, value in (run.result or {}).items() if key != "interrupts"},
+ "error": run.error,
+ "error_code": run.error_code,
+ }
+ )
+ return snapshot
+
+
+def merge_snapshot(stored: Mapping[str, Any], live: Optional[Mapping[str, Any]]) -> Dict[str, Any]:
+ """The stored run with the hub's live view laid over it. The stored state
+ wins once it has ended: the row commits before its terminal event goes
+ out, so the row can be ahead of the hub, never behind it."""
+ merged = deepcopy(dict(stored))
+ if not live:
+ return merged
+ for key in ("seq", "steps", "segments", "activities", "interrupts"):
+ merged[key] = deepcopy(live.get(key, merged.get(key)))
+ if stored.get("state") in LIVE_STATES and live.get("state") not in (None, "pending"):
+ merged["state"] = live["state"]
+ return merged
+
+
+def _step(snapshot: Dict[str, Any], step_id: str) -> Dict[str, Any]:
+ for step in snapshot["steps"]:
+ if step.get("step_id") == step_id:
+ return step
+ step: Dict[str, Any] = {"step_id": step_id}
+ snapshot["steps"].append(step)
+ return step
+
+
+def _segment(snapshot: Dict[str, Any], message_id: str) -> Dict[str, Any]:
+ for segment in snapshot["segments"]:
+ if segment.get("message_id") == message_id:
+ return segment
+ segment: Dict[str, Any] = {"message_id": message_id, "text": "", "final": None}
+ snapshot["segments"].append(segment)
+ return segment
+
+
+def apply_event(snapshot: Mapping[str, Any], event: Mapping[str, Any]) -> Dict[str, Any]:
+ """Fold one ``chat_run_event`` envelope (as a dict) into ``snapshot``.
+
+ Returns a new snapshot; the input is not modified. An event whose ``seq``
+ is not newer than the snapshot's is a duplicate and changes nothing."""
+ data = event.get("data") or {}
+ suffix = run_event_suffix(str(event.get("type") or ""))
+ seq = int(data.get("seq") or 0)
+ if suffix is None or seq <= int(snapshot.get("seq") or 0):
+ return dict(snapshot)
+ out: Dict[str, Any] = deepcopy(dict(snapshot))
+ out["seq"] = seq
+ for key in ("session_id", "workflow_id"):
+ if data.get(key) is not None:
+ out[key] = data[key]
+
+ if suffix == "started":
+ out["state"] = "running"
+ for key in ("kind", "parent_run_id", "user_message_id", "reply_message_id", "started_at"):
+ if data.get(key) is not None:
+ out[key] = data[key]
+ elif suffix == "finished":
+ outcome = dict(data.get("outcome") or {"type": "success"})
+ out["outcome"] = outcome
+ out["result"] = dict(data.get("result") or {})
+ out["interrupts"] = list(outcome.get("interrupts") or [])
+ out["state"] = "stopped" if outcome.get("type") == "stopped" else "finished"
+ elif suffix == "failed":
+ out["state"] = "error"
+ out["error"] = data.get("message")
+ out["error_code"] = data.get("code")
+ elif suffix == "step.started":
+ step = _step(out, str(data.get("step_id")))
+ step.update({"name": data.get("step_name"), "state": "running"})
+ if data.get("icon"):
+ step["icon"] = data["icon"]
+ elif suffix == "step.finished":
+ step = _step(out, str(data.get("step_id")))
+ step.update({"name": data.get("step_name") or step.get("name"), "state": data.get("state") or "done"})
+ for key in ("detail", "duration_ms", "narration"):
+ if data.get(key) is not None:
+ step[key] = data[key]
+ elif suffix == "text.started":
+ _segment(out, str(data.get("message_id")))
+ elif suffix == "text.content":
+ segment = _segment(out, str(data.get("message_id")))
+ segment["text"] += str(data.get("delta") or "")
+ elif suffix == "text.ended":
+ segment = _segment(out, str(data.get("message_id")))
+ segment["final"] = bool(data.get("final"))
+ elif suffix == "custom":
+ name = data.get("name")
+ value = data.get("value") or {}
+ if name == "opencompany.segment_discarded":
+ discarded = value.get("message_id")
+ out["segments"] = [segment for segment in out["segments"] if segment.get("message_id") != discarded]
+ elif name == "opencompany.stopping":
+ out["state"] = "stopping"
+ return out
+
+
+def replay(run_id: str, events: List[Mapping[str, Any]]) -> Dict[str, Any]:
+ """Fold a run's events in order, from an empty snapshot."""
+ snapshot = empty_snapshot(run_id)
+ for event in events:
+ snapshot = apply_event(snapshot, event)
+ return snapshot
+
+
+__all__ = ["LIVE_STATES", "apply_event", "empty_snapshot", "merge_snapshot", "replay", "snapshot_from_row"]
diff --git a/server/services/chat/watchdog.py b/server/services/chat/watchdog.py
new file mode 100644
index 00000000..ebc98d10
--- /dev/null
+++ b/server/services/chat/watchdog.py
@@ -0,0 +1,78 @@
+"""The chat run watchdog: ``ledger.sweep`` every ``runs.watchdog_interval_s``
+(config/chat_defaults.json).
+
+main.py starts it once the database is up and stops it at shutdown. The
+first sweep runs at once, so a run a crash left open ends once the server
+is back; a pending run's wait counts from this process's start, so the
+restart itself does not expire it.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from typing import Any, Optional
+
+from core.logging import get_logger
+from services.chat import ledger
+from services.chat.config import runs_setting
+
+logger = get_logger(__name__)
+
+
+async def temporal_workflow_closed(workflow_id: str, run_id: Optional[str]) -> Optional[bool]:
+ """Whether the Temporal workflow has closed: True once it is no longer
+ running (or Temporal no longer knows it), False while it runs, None when
+ Temporal cannot be asked."""
+ try:
+ from core.container import container
+
+ wrapper = container.temporal_client()
+ except Exception: # noqa: BLE001 - Temporal is optional
+ return None
+ client = getattr(wrapper, "client", None) if wrapper is not None else None
+ if client is None:
+ return None
+ from temporalio.client import WorkflowExecutionStatus
+ from temporalio.service import RPCError, RPCStatusCode
+
+ try:
+ description = await client.get_workflow_handle(workflow_id, run_id=run_id).describe()
+ except RPCError as exc:
+ return True if exc.status == RPCStatusCode.NOT_FOUND else None
+ except Exception: # noqa: BLE001 - unknown is not closed
+ return None
+ return description.status != WorkflowExecutionStatus.RUNNING
+
+
+class ChatRunWatchdog:
+ def __init__(self, database: Any, *, interval: Optional[float] = None) -> None:
+ self.database = database
+ self.interval = float(interval if interval is not None else runs_setting("watchdog_interval_s"))
+ self._task: Optional[asyncio.Task] = None
+
+ def start(self) -> None:
+ if self._task is None or self._task.done():
+ self._task = asyncio.get_running_loop().create_task(self._loop(), name="chat-run-watchdog")
+
+ async def stop(self) -> None:
+ task, self._task = self._task, None
+ if task is None or task.done():
+ return
+ task.cancel()
+ try:
+ await task
+ except asyncio.CancelledError:
+ pass
+
+ async def _loop(self) -> None:
+ while True:
+ try:
+ await ledger.sweep(self.database, temporal_status=temporal_workflow_closed)
+ except asyncio.CancelledError:
+ raise
+ except Exception: # noqa: BLE001 - the next sweep tries again
+ logger.warning("Chat run sweep failed", exc_info=True)
+ await asyncio.sleep(self.interval)
+
+
+__all__ = ["ChatRunWatchdog", "temporal_workflow_closed"]
diff --git a/server/services/chat_thread.py b/server/services/chat_thread.py
index a4e58c19..9ded1787 100644
--- a/server/services/chat_thread.py
+++ b/server/services/chat_thread.py
@@ -14,13 +14,18 @@ Talk, and the editor shows it in its chat pane.
- ``record_chat_message``: add a row stamped with the session's live
generation (``root_execution_id``, as ``chat_execution_id`` says; the
editor's chat reads one generation, so a row without it would never show
- there), then announce it.
-- ``clear_chat_thread``: delete the session's rows, every generation, then
- announce it when there were any.
-- ``clear_chat_session``: the owner's Clear. The thread goes, and the
- listeners registered with ``register_chat_cleared_listener`` forget what
- it held: the Context plugin clears the workflow's conversations, so the
- agent starts over with the chat.
+ there), then announce it. A reply names its chat run (``run_id``) and is
+ saved under the run's reply id, so a retried save writes it once. The
+ owner's messages are written with their runs by ``services/chat/ledger.py``,
+ which announces them the same way.
+- ``clear_chat_thread``: delete the session's rows, every generation, and
+ its chat runs, then announce it when there were any. Subscribers watching
+ a run that was still live are told it ended (``reset``).
+- ``clear_chat_session``: the owner's Clear. The thread goes (live runs end
+ with ``cleared``), and the listeners registered with
+ ``register_chat_cleared_listener`` forget what it held: the Context plugin
+ clears the workflow's conversations, so the agent starts over with the
+ chat.
A workflow's thread lives as long as its generation and its workflow. A
Reset (every restart, Home's Apply and Turn on Talk included) clears it
@@ -43,7 +48,7 @@ query that matches nothing on every message (the same reason as
from __future__ import annotations
-from typing import Any, Awaitable, Callable, List, Optional
+from typing import Any, Awaitable, Callable, Dict, List, Optional
from core.logging import get_logger
from services.events.envelope import WorkflowEvent
@@ -100,7 +105,9 @@ def chat_updated(*, session_id: str, role: Optional[str]) -> WorkflowEvent:
)
-async def _announce(session_id: str, role: Optional[str]) -> None:
+async def announce_chat_updated(session_id: str, role: Optional[str]) -> None:
+ """Send ``chat.updated`` for a session: a message was added (``role``)
+ or the thread was cleared (``role`` None). Never raises."""
from services.status_broadcaster import get_status_broadcaster
event = chat_updated(session_id=session_id, role=role)
@@ -110,24 +117,48 @@ async def _announce(session_id: str, role: Optional[str]) -> None:
logger.warning("chat.updated broadcast failed", session_id=session_id, exc_info=True)
-async def record_chat_message(database: Any, session_id: str, role: str, message: str) -> bool:
+async def record_chat_message(
+ database: Any,
+ session_id: str,
+ role: str,
+ message: str,
+ *,
+ uid: Optional[str] = None,
+ run_id: Optional[str] = None,
+ kind: str = "text",
+ status: str = "complete",
+ parts: Optional[Dict[str, Any]] = None,
+) -> Optional[Dict[str, Any]]:
"""Add ``message`` to the thread, in the session's live generation, and
- announce it. False when the row could not be saved (nothing is
- announced)."""
+ announce it. Returns the saved row, or None when it could not be saved
+ (nothing is announced). A ``uid`` already saved returns that row; the
+ repeated announcement only makes clients read the thread again."""
control = await database.get_latest_workflow_control(session_id)
- saved = await database.add_chat_message(session_id, role, message, execution_id=chat_execution_id(control))
+ saved = await database.add_chat_message(
+ session_id, role, message, execution_id=chat_execution_id(control),
+ uid=uid, run_id=run_id, kind=kind, status=status, parts=parts,
+ )
if saved:
- await _announce(session_id, role)
- return bool(saved)
+ await announce_chat_updated(session_id, role)
+ return saved or None
+
+async def clear_chat_thread(database: Any, session_id: str, *, reason: str = "reset") -> int:
+ """Delete the thread, every generation of it, with its chat runs, and
+ announce it when there was anything to delete (a Reset runs one clear
+ per chat node). Runs still live end for their subscribers with
+ ``reason`` as the error code. Returns the messages deleted."""
+ from services.chat import ledger
-async def clear_chat_thread(database: Any, session_id: str) -> int:
- """Delete the thread, every generation of it, and announce it when there
- was anything to delete (a Reset runs one clear per chat node). Returns
- the rows deleted."""
+ try:
+ live = await ledger.live_runs(database, session_id)
+ except Exception:
+ logger.warning("Could not read live chat runs before a clear", session_id=session_id, exc_info=True)
+ live = []
count = await database.clear_chat_messages(session_id)
+ ledger.publish_cleared(live, code=reason)
if count:
- await _announce(session_id, None)
+ await announce_chat_updated(session_id, None)
return count
@@ -145,7 +176,7 @@ async def clear_chat_session(database: Any, session_id: str) -> int:
session has listeners to tell (``"default"`` is no workflow's). A
failing listener is logged and never fails the clear. Returns the rows
deleted."""
- count = await clear_chat_thread(database, session_id)
+ count = await clear_chat_thread(database, session_id, reason="cleared")
if session_id == DEFAULT_SESSION:
return count
for listener in list(_CLEARED_LISTENERS):
@@ -165,6 +196,7 @@ __all__ = [
"DEFAULT_SESSION",
"WIRE_KEY",
"ChatClearedListener",
+ "announce_chat_updated",
"chat_execution_id",
"chat_updated",
"clear_chat_session",
diff --git a/server/services/plugin/base.py b/server/services/plugin/base.py
index d1be9cf4..b083af18 100644
--- a/server/services/plugin/base.py
+++ b/server/services/plugin/base.py
@@ -1150,6 +1150,9 @@ class BaseNode:
"context_execution_id",
"context_session_id",
"data_scope_id",
+ # The chat run this run answers (MachinaWorkflow sets it
+ # for a run the owner's chat message started).
+ "run_scope",
):
if key in context:
extras[key] = context[key]
diff --git a/server/services/status_broadcaster.py b/server/services/status_broadcaster.py
index feb38ca1..2a4de622 100644
--- a/server/services/status_broadcaster.py
+++ b/server/services/status_broadcaster.py
@@ -184,10 +184,16 @@ class StatusBroadcaster:
)
async def disconnect(self, websocket: WebSocket):
- """Remove a WebSocket connection."""
+ """Remove a WebSocket connection, then tell the disconnect listeners
+ (``register_disconnect_listener``)."""
async with self._lock:
self._connections.discard(websocket)
logger.info(f"[StatusBroadcaster] Client disconnected. Total: {len(self._connections)}")
+ for listener in list(_DISCONNECT_LISTENERS):
+ try:
+ listener(websocket)
+ except Exception:
+ logger.warning("[StatusBroadcaster] Disconnect listener failed", exc_info=True)
async def _refresh_all_services(self):
"""Fan out plugin-registered refresh callbacks.
@@ -1344,6 +1350,18 @@ def register_service_refresh(callback: _ServiceRefreshCallback) -> None:
_SERVICE_REFRESH_FANOUT.register(callback)
+_DisconnectListener = _typing.Callable[[WebSocket], None]
+_DISCONNECT_LISTENERS: _typing.List[_DisconnectListener] = []
+_DISCONNECT_FANOUT: _IdempotentList[_DisconnectListener] = _IdempotentList("disconnect", items=_DISCONNECT_LISTENERS)
+
+
+def register_disconnect_listener(listener: _DisconnectListener) -> None:
+ """Call ``listener(websocket)`` after a status socket disconnects, so a
+ service that keeps per-socket state (chat's run subscriptions) can drop
+ it. Synchronous and must not block. Idempotent on re-import."""
+ _DISCONNECT_FANOUT.register(listener)
+
+
# Global singleton instance
_broadcaster: Optional[StatusBroadcaster] = None
diff --git a/server/services/temporal/activities.py b/server/services/temporal/activities.py
index 64339bd1..d331775a 100644
--- a/server/services/temporal/activities.py
+++ b/server/services/temporal/activities.py
@@ -233,6 +233,7 @@ class NodeExecutionActivities:
"agent_iteration",
"tool_call_index",
"tool_call_id",
+ "run_scope",
):
if key in context:
message[key] = context[key]
diff --git a/server/services/temporal/agent_workflow.py b/server/services/temporal/agent_workflow.py
index cc6dee8d..6156ab83 100644
--- a/server/services/temporal/agent_workflow.py
+++ b/server/services/temporal/agent_workflow.py
@@ -261,6 +261,9 @@ _INHERITED_SCOPE_KEYS = (
"user_id",
"workspace_dir",
"temporal_worker_pool_enabled",
+ # The chat run the root run answers (services/chat/), so a tool call
+ # or a delegated agent can attach what it produces to that answer.
+ "run_scope",
)
diff --git a/server/services/temporal/worker.py b/server/services/temporal/worker.py
index 2d97c778..3ef70d79 100644
--- a/server/services/temporal/worker.py
+++ b/server/services/temporal/worker.py
@@ -226,6 +226,7 @@ class TemporalWorkerManager:
)
from services.temporal.agent_activities import collect_agent_activities
from services.temporal.workspace_task_activities import reset_workspace_task_runtime
+ from services.chat.activities import CHAT_RUN_ACTIVITIES
from services.temporal.activities import (
broadcast_trigger_status_activity,
evaluate_trigger_filter_activity,
@@ -282,6 +283,7 @@ class TemporalWorkerManager:
pause_workflow_on_failure_activity,
record_run_completion_activity,
reset_workspace_task_runtime,
+ *CHAT_RUN_ACTIVITIES,
store_node_output_activity,
*per_type,
*agent_activities,
@@ -741,6 +743,7 @@ async def run_standalone_worker(
)
from services.temporal.agent_activities import collect_agent_activities
from services.temporal.workspace_task_activities import reset_workspace_task_runtime
+ from services.chat.activities import CHAT_RUN_ACTIVITIES
from services.temporal.agent_workflow import (
AgentWorkflow,
DelegatedTaskWorkflow,
@@ -767,6 +770,7 @@ async def run_standalone_worker(
pause_workflow_on_failure_activity,
record_run_completion_activity,
reset_workspace_task_runtime,
+ *CHAT_RUN_ACTIVITIES,
store_node_output_activity,
*registered_plugin_activities,
*registered_agent_activities,
@@ -824,6 +828,7 @@ async def create_worker(
)
from services.temporal.agent_activities import collect_agent_activities
from services.temporal.workspace_task_activities import reset_workspace_task_runtime
+ from services.chat.activities import CHAT_RUN_ACTIVITIES
from services.temporal.agent_workflow import (
AgentWorkflow,
DelegatedTaskWorkflow,
@@ -849,6 +854,7 @@ async def create_worker(
pause_workflow_on_failure_activity,
record_run_completion_activity,
reset_workspace_task_runtime,
+ *CHAT_RUN_ACTIVITIES,
store_node_output_activity,
*registered_plugin_activities,
*registered_agent_activities,
diff --git a/server/services/temporal/workflow.py b/server/services/temporal/workflow.py
index fbf6bce9..5166d54a 100644
--- a/server/services/temporal/workflow.py
+++ b/server/services/temporal/workflow.py
@@ -42,6 +42,17 @@ CONDITIONAL_EDGES_PATCH = "machina-conditional-edges-v1"
# it (which end without that activity) still replay.
RUN_RECORD_PATCH = "machina-run-record-v1"
+# A run that the owner's chat message spawned claims its chat run when it
+# starts and finishes it at its end (services/chat/ledger.py), so Talk and
+# the editor's chat follow the answer while it is worked on. Gated: the two
+# activities are new commands.
+CHAT_RUN_PATCH = "machina-chat-run-v1"
+# The source and type services/chat/events.py gives the owner's message. A
+# chat run id is trusted only from an event with both, never from what
+# another producer put in its event data.
+CHAT_MESSAGE_SOURCE = "opencompany://services/chat"
+CHAT_MESSAGE_TYPE = "com.opencompany.chat.message.received"
+
# Config handles - nodes connecting via these are config nodes (not executed)
# AI Agent handles: input-context, input-tools, input-model, input-task, input-teammates
# Zeenie handles: input-skill, input-tools
@@ -304,6 +315,75 @@ class MachinaWorkflow:
except Exception as exc: # noqa: BLE001 — cosmetic relative to the run result
workflow.logger.warning(f"run-record activity failed (non-fatal): {exc}")
+ @staticmethod
+ def _chat_run_id(nodes: List[Dict[str, Any]]) -> Optional[str]:
+ """The chat run the firing trigger's event started, or None.
+
+ Read from the event envelope the trigger listener attached to the
+ trigger's output, and only when the chat service produced it. Depends
+ on the workflow input alone, so it is deterministic."""
+ for node in nodes:
+ if not node.get("_pre_executed"):
+ continue
+ output = node.get("_trigger_output")
+ if not isinstance(output, dict) or output.get("not_triggered"):
+ continue
+ envelope = output.get("_event_envelope")
+ if not isinstance(envelope, dict):
+ continue
+ if envelope.get("source") != CHAT_MESSAGE_SOURCE or envelope.get("type") != CHAT_MESSAGE_TYPE:
+ continue
+ data = envelope.get("data")
+ run_id = data.get("run_id") if isinstance(data, dict) else None
+ if isinstance(run_id, str) and run_id:
+ return run_id
+ return None
+
+ async def _start_chat_run(self, run_id: str, session_id: str) -> Optional[Dict[str, Any]]:
+ """Claim the chat run; returns the run scope node contexts carry, or
+ None when this run does not hold it (another chat trigger in the
+ graph claimed it, or it ended) and runs untracked."""
+ info = workflow.info()
+ try:
+ claimed = await workflow.execute_activity(
+ "chat_run.start",
+ {"run_id": run_id, "temporal_workflow_id": info.workflow_id, "temporal_run_id": info.run_id},
+ start_to_close_timeout=timedelta(seconds=30),
+ retry_policy=DEFAULT_ACTIVITY_RETRY,
+ )
+ except Exception as exc: # noqa: BLE001 — the answer matters more than its tracking
+ workflow.logger.warning(f"chat run start failed (running untracked): {exc}")
+ return None
+ if not (isinstance(claimed, dict) and claimed.get("claimed")):
+ return None
+ return {"run_id": run_id, "session_id": session_id}
+
+ async def _finish_chat_run(self, run_scope: Dict[str, Any], success: bool, errors: List[Dict]) -> None:
+ """End the claimed chat run with the run's outcome. Non-fatal: the
+ watchdog ends a run whose finish never arrived."""
+ info = workflow.info()
+ payload: Dict[str, Any] = {
+ "run_id": run_scope["run_id"],
+ "temporal_workflow_id": info.workflow_id,
+ "temporal_run_id": info.run_id,
+ "success": success,
+ }
+ if errors:
+ failure = errors[0] or {}
+ payload["error"] = str(failure.get("error") or "run_failed")[:500]
+ for key in ("hint", "requires_user_action"):
+ if failure.get(key) is not None:
+ payload[key] = failure[key]
+ try:
+ await workflow.execute_activity(
+ "chat_run.finish",
+ payload,
+ start_to_close_timeout=timedelta(seconds=30),
+ retry_policy=DEFAULT_ACTIVITY_RETRY,
+ )
+ except Exception as exc: # noqa: BLE001 — cosmetic relative to the run result
+ workflow.logger.warning(f"chat run finish failed (non-fatal): {exc}")
+
@workflow.signal
async def on_event(self, event_payload: Dict[str, Any]) -> None:
"""Wave 12 A7: receive an event from ``services.events.dispatch.emit``.
@@ -490,6 +570,15 @@ class MachinaWorkflow:
workflow.logger.info(f"Pre-executed: {pre_executed_count}, To execute: {len(node_map) - pre_executed_count}")
+ # The owner's chat message started a chat run: claim it. The patch
+ # check comes after the eligibility test, which reads only the
+ # workflow input, so it is deterministic.
+ run_scope: Optional[Dict[str, Any]] = None
+ chat_run_id = self._chat_run_id(nodes)
+ if chat_run_id is not None and workflow.patched(CHAT_RUN_PATCH):
+ await self._wait_until_resumed()
+ run_scope = await self._start_chat_run(chat_run_id, session_id)
+
# 5. Retry policy for node activities. Wave 12 D1: imported
# shared constant declares ``non_retryable_error_types`` so
# NodeUserError fails fast instead of burning 3 retries on
@@ -596,6 +685,8 @@ class MachinaWorkflow:
)
if "user_id" in workflow_data:
context["user_id"] = workflow_data.get("user_id")
+ if run_scope is not None:
+ context["run_scope"] = dict(run_scope)
context["temporal_worker_pool_enabled"] = bool(
frozen_routing.get("worker_pool_enabled")
)
@@ -722,6 +813,8 @@ class MachinaWorkflow:
# Build final result
success = len(errors) == 0 and len(completed) == len(node_map)
+ if run_scope is not None:
+ await self._finish_chat_run(run_scope, success, errors)
if errors:
await self._pause_deployment_on_failure(workflow_data, nodes, errors)
await self._record_run_completion(workflow_data, nodes, success)
diff --git a/server/tests/nodes/test_chat_reply.py b/server/tests/nodes/test_chat_reply.py
index fd6fd6d4..70653a10 100644
--- a/server/tests/nodes/test_chat_reply.py
+++ b/server/tests/nodes/test_chat_reply.py
@@ -1,6 +1,8 @@
"""Reply in Chat (``chatReply``): posts an answer to the thread of the
workflow it runs in, as the assistant, in the live generation, and says so
-on ``chat.updated``; posts nothing for an empty answer or NO_REPLY."""
+on ``chat.updated``; posts nothing for an empty answer or NO_REPLY. In a run
+the owner's chat message started (``run_scope``), the answer is that chat
+run's reply (tests/services/chat/ cover the ledger itself)."""
from __future__ import annotations
@@ -22,20 +24,26 @@ def thread(harness):
frames.append(message)
harness.database.get_latest_workflow_control = AsyncMock(return_value=SimpleNamespace(status="running", root_execution_id="gen-2"))
- harness.database.add_chat_message = AsyncMock(return_value=True)
+ harness.database.add_chat_message = AsyncMock(return_value={"uid": "m_1", "message": "saved"})
with patch("services.status_broadcaster.get_status_broadcaster", return_value=Broadcaster()):
yield SimpleNamespace(database=harness.database, frames=frames)
-async def _reply(harness, message, *, workflow_id="wf-1"):
- return await harness.execute("chatReply", {"message": message}, context=harness.build_context(workflow_id=workflow_id))
+async def _reply(harness, message, *, workflow_id="wf-1", run_scope=None):
+ context = harness.build_context(workflow_id=workflow_id)
+ if run_scope is not None:
+ context["run_scope"] = run_scope
+ return await harness.execute("chatReply", {"message": message}, context=context)
async def test_the_answer_lands_in_the_workflows_thread(harness, thread):
result = await _reply(harness, " Booked you for 3pm. ")
harness.assert_envelope(result, success=True)
- assert result["result"] == {"posted": True, "message": "Booked you for 3pm."}
- thread.database.add_chat_message.assert_awaited_once_with("wf-1", "assistant", "Booked you for 3pm.", execution_id="gen-2")
+ assert result["result"] == {"posted": True, "message": "Booked you for 3pm.", "message_id": "m_1", "run_id": None}
+ thread.database.add_chat_message.assert_awaited_once_with(
+ "wf-1", "assistant", "Booked you for 3pm.", execution_id="gen-2",
+ uid=None, run_id=None, kind="text", status="complete", parts=None,
+ )
[frame] = thread.frames
assert frame["type"] == "chat.updated"
assert frame["data"]["type"] == "com.opencompany.chat.updated"
@@ -66,12 +74,42 @@ async def test_it_needs_a_saved_workflow(harness, thread):
async def test_a_failed_save_is_an_error_and_announces_nothing(harness, thread):
- thread.database.add_chat_message = AsyncMock(return_value=False)
+ thread.database.add_chat_message = AsyncMock(return_value=None)
result = await _reply(harness, "hello")
harness.assert_envelope(result, success=False)
assert thread.frames == []
+async def test_in_a_chat_run_the_answer_is_the_runs_reply(harness, thread):
+ run = SimpleNamespace(run_id="r_1", session_id="wf-1", reply_message_uid="a_r_1")
+ with (
+ patch("services.chat.ledger.get_run", AsyncMock(return_value=run)) as get_run,
+ patch("services.chat.ledger.post_reply", AsyncMock(return_value={"uid": "a_r_1"})) as post_reply,
+ ):
+ result = await _reply(harness, "Booked.", run_scope={"run_id": "r_1", "session_id": "wf-1"})
+ harness.assert_envelope(result, success=True)
+ assert result["result"] == {"posted": True, "message": "Booked.", "message_id": "a_r_1", "run_id": "r_1"}
+ assert get_run.await_args.args[1] == "r_1"
+ kwargs = post_reply.await_args.kwargs
+ assert (kwargs["run"], kwargs["text"], kwargs["execution_id"]) == (run, "Booked.", "gen-2")
+ assert kwargs["node_id"]
+ thread.database.add_chat_message.assert_not_awaited()
+ assert [frame["data"]["data"]["role"] for frame in thread.frames] == ["assistant"]
+
+
+async def test_a_run_whose_conversation_was_cleared_posts_nothing(harness, thread):
+ with (
+ patch("services.chat.ledger.get_run", AsyncMock(return_value=None)),
+ patch("services.chat.ledger.post_reply", AsyncMock()) as post_reply,
+ ):
+ result = await _reply(harness, "Too late.", run_scope={"run_id": "r_gone", "session_id": "wf-1"})
+ harness.assert_envelope(result, success=True)
+ assert result["result"] == {"posted": False, "message": None, "message_id": None, "run_id": "r_gone"}
+ post_reply.assert_not_awaited()
+ thread.database.add_chat_message.assert_not_awaited()
+ assert thread.frames == []
+
+
def test_it_is_a_sink_the_hire_builder_may_use():
import nodes # noqa: F401
from services.node_allowlist import CONFIG_PATH, NodeAllowlistService
diff --git a/server/tests/services/chat/__init__.py b/server/tests/services/chat/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/server/tests/services/chat/_helpers.py b/server/tests/services/chat/_helpers.py
new file mode 100644
index 00000000..719a7acb
--- /dev/null
+++ b/server/tests/services/chat/_helpers.py
@@ -0,0 +1,68 @@
+"""Helpers for the chat run tests."""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+from typing import Any, Dict, List, Optional
+
+from starlette.websockets import WebSocketState
+
+from models.database import WorkflowControlExecution
+
+
+class FakeSocket:
+ """A status socket: records what the hub writes to it."""
+
+ def __init__(self, *, path: str = "/ws/status", user_id: Optional[str] = None) -> None:
+ self.scope = {"path": path}
+ self.state = SimpleNamespace(user_id=user_id)
+ self.client_state = WebSocketState.CONNECTED
+ self.application_state = WebSocketState.CONNECTED
+ self.sent: List[str] = []
+
+ async def send_text(self, text: str) -> None:
+ self.sent.append(text)
+
+ def events(self) -> List[Dict[str, Any]]:
+ import json
+
+ return [json.loads(text)["data"] for text in self.sent]
+
+
+async def add_control(
+ database,
+ workflow_id: str,
+ status: str,
+ *,
+ generation: int = 1,
+ nodes: Optional[List[Dict[str, Any]]] = None,
+):
+ """A control generation, its graph holding ``nodes``."""
+ async with database.get_session() as session:
+ session.add(
+ WorkflowControlExecution(
+ id=f"{workflow_id}-{generation}",
+ workflow_id=workflow_id,
+ generation=generation,
+ execution_id=f"gen-{generation}",
+ root_execution_id=f"gen-{generation}",
+ graph_hash="0" * 64,
+ graph_snapshot={"nodes": list(nodes or []), "edges": []},
+ status=status,
+ idempotency_key=f"k-{generation}",
+ )
+ )
+ await session.commit()
+
+
+def chat_trigger(workflow_id: str, **data: Any) -> Dict[str, Any]:
+ return {"id": f"{workflow_id}:chatTrigger:1", "type": "chatTrigger", "data": {"label": "Chat", **data}}
+
+
+async def talking(database, workflow_id: str = "wf", status: str = "running", generation: int = 1) -> None:
+ """A deployed workflow whose chat trigger answers its own session."""
+ await add_control(database, workflow_id, status, generation=generation, nodes=[chat_trigger(workflow_id)])
+
+
+def chat_updates(frames) -> List[Dict[str, Any]]:
+ return [frame["data"]["data"] for frame in frames if frame["type"] == "chat.updated"]
diff --git a/server/tests/services/chat/conftest.py b/server/tests/services/chat/conftest.py
new file mode 100644
index 00000000..c6b61487
--- /dev/null
+++ b/server/tests/services/chat/conftest.py
@@ -0,0 +1,85 @@
+"""Fixtures for the chat run tests: a real database on a throwaway file, the
+chat handlers' container pointed at it, a fresh hub, and the broadcaster,
+dispatch and Clear listeners captured."""
+
+from __future__ import annotations
+
+import importlib.util
+import sys
+import uuid
+from pathlib import Path
+from types import SimpleNamespace
+from typing import Any, Dict, List
+
+import pytest
+
+
+async def database(tmp_path: Path):
+ """A real ``core.database.Database`` (the root conftest stubs the module,
+ so the real one is loaded privately)."""
+ module_name = f"tests._chat_database_{uuid.uuid4().hex}"
+ spec = importlib.util.spec_from_file_location(module_name, Path(__file__).resolve().parents[3] / "core" / "database.py")
+ assert spec is not None and spec.loader is not None
+ module = importlib.util.module_from_spec(spec)
+ sys.modules[module_name] = module
+ spec.loader.exec_module(module)
+ db_path = tmp_path / f"chat-{uuid.uuid4().hex}.db"
+ db = module.Database(
+ SimpleNamespace(
+ database_url=f"sqlite+aiosqlite:///{db_path.as_posix()}",
+ database_echo=False,
+ database_pool_size=5,
+ database_max_overflow=5,
+ )
+ )
+ await db.startup()
+ try:
+ yield db
+ finally:
+ await db.shutdown()
+ sys.modules.pop(module_name, None)
+
+
+def hub():
+ from services.chat.hub import ChatRunHub, reset_chat_hub_for_tests
+
+ fresh = ChatRunHub(queue_size=8)
+ previous = reset_chat_hub_for_tests(fresh)
+ try:
+ yield fresh
+ finally:
+ reset_chat_hub_for_tests(previous)
+
+
+def frames(monkeypatch):
+ """Everything the status broadcaster would have sent to every socket."""
+ import services.status_broadcaster as status_broadcaster
+
+ sent: List[Dict[str, Any]] = []
+
+ class Broadcaster:
+ async def broadcast(self, message):
+ sent.append(message)
+
+ monkeypatch.setattr(status_broadcaster, "get_status_broadcaster", lambda: Broadcaster())
+ return sent
+
+
+def chat(monkeypatch, database, hub, frames):
+ """The chat handlers on the test database, with the dispatch captured."""
+ import services.chat.handlers as handlers
+ from services import chat_thread
+
+ dispatched: List[Dict[str, Any]] = []
+
+ async def dispatch(event_data, *, workflow_id=None, event_id=None):
+ dispatched.append({"data": dict(event_data), "workflow_id": workflow_id, "event_id": event_id})
+
+ monkeypatch.setattr(handlers, "container", SimpleNamespace(database=lambda: database))
+ monkeypatch.setattr(handlers, "dispatch_chat_message_received", dispatch)
+ monkeypatch.setattr(chat_thread, "_CLEARED_LISTENERS", [])
+ return SimpleNamespace(database=database, hub=hub, frames=frames, dispatched=dispatched, handlers=handlers)
diff --git a/server/tests/services/chat/test_handlers.py b/server/tests/services/chat/test_handlers.py
new file mode 100644
index 00000000..f0a53a83
--- /dev/null
+++ b/server/tests/services/chat/test_handlers.py
@@ -0,0 +1,160 @@
+"""Chat's WebSocket commands (services/chat/handlers.py) with runs: a message
+a deployed chat trigger will answer starts a run named in the response and
+dispatched as the event id; a second send while it is live is refused; a
+re-sent message returns the first send; reads return the v2 shape with the
+thread and live runs; subscribing returns snapshots and delivers events; and
+only the workflow's owner may use its session."""
+
+from __future__ import annotations
+
+import asyncio
+
+from services.chat import ledger
+from tests.services.chat._helpers import FakeSocket, add_control, chat_trigger, chat_updates, talking
+
+
+async def send(chat, message="Book Saturday", session_id="wf", socket=None, **extra):
+ return await chat.handlers.handle_send_chat_message({"message": message, "session_id": session_id, "timestamp": "t", **extra}, socket)
+
+
+async def test_a_message_the_employee_will_answer_starts_a_run(chat):
+ await talking(chat.database)
+ result = await send(chat)
+ assert result["success"] is True and result["delivery"] == "now"
+ run = await ledger.get_run(chat.database, result["run_id"])
+ assert run.state == "pending" and run.user_message_uid == result["message_id"]
+ [dispatched] = chat.dispatched
+ # The run id is the event id, so the run a listener spawns has a predictable id.
+ assert dispatched["event_id"] == result["run_id"]
+ assert dispatched["workflow_id"] == "wf"
+ assert dispatched["data"] == {
+ "message": "Book Saturday",
+ "timestamp": "t",
+ "session_id": "wf",
+ "message_id": result["message_id"],
+ "run_id": result["run_id"],
+ }
+ assert chat_updates(chat.frames) == [{"workflow_id": "wf", "session_id": "wf", "role": "user"}]
+
+
+async def test_a_paused_employee_queues_the_run(chat):
+ await talking(chat.database, status="paused")
+ result = await send(chat)
+ assert result["delivery"] == "queued"
+ assert (await ledger.get_run(chat.database, result["run_id"])).state == "queued"
+
+
+async def test_a_second_message_while_a_run_is_live_is_refused(chat):
+ await talking(chat.database)
+ first = await send(chat)
+ second = await send(chat, "And Sunday?")
+ assert second == {"success": False, "error": "run_in_progress", "run_id": first["run_id"]}
+ assert len(chat.dispatched) == 1
+ assert [row["message"] for row in await chat.database.get_chat_messages("wf")] == ["Book Saturday"]
+
+
+async def test_a_resent_message_returns_the_first_send_and_dispatches_once(chat):
+ await talking(chat.database)
+ first = await send(chat, client_message_id="c-1")
+ again = await send(chat, client_message_id="c-1")
+ assert (again["message_id"], again["run_id"]) == (first["message_id"], first["run_id"])
+ assert len(chat.dispatched) == 1
+ bad = await send(chat, client_message_id="not ok")
+ assert bad["error"] == "invalid_request"
+
+
+async def test_no_run_when_no_deployed_chat_trigger_answers_the_session(chat):
+ # A graph without a chat trigger, then one whose trigger listens to
+ # another session, then a disabled one: the message is saved, no run.
+ await add_control(chat.database, "wf", "running", nodes=[{"id": "wf:console:1", "type": "console", "data": {}}])
+ assert (await send(chat))["run_id"] is None
+ await add_control(chat.database, "wf2", "running", nodes=[chat_trigger("wf2")])
+ await chat.database.save_node_parameters("wf2:chatTrigger:1", {"session_id": "elsewhere"})
+ assert (await send(chat, session_id="wf2"))["run_id"] is None
+ await add_control(chat.database, "wf3", "running", nodes=[chat_trigger("wf3", disabled=True)])
+ assert (await send(chat, session_id="wf3"))["run_id"] is None
+ assert len(await chat.database.get_chat_messages("wf")) == 1
+
+
+async def test_the_owner_writes_only_their_own_messages(chat):
+ await talking(chat.database)
+ assert (await send(chat, role="assistant"))["error"] == "invalid_request"
+ assert (await send(chat, message=" "))["error"] == "invalid_request"
+ assert chat.dispatched == []
+
+
+async def test_only_the_workflows_owner_may_use_its_chat(chat):
+ await talking(chat.database)
+ await chat.database.save_workflow("wf", "Salon", "Salon_1", {"nodes": [], "edges": [], "owner_id": "alice"})
+ stranger = FakeSocket(user_id="mallory")
+ owner = FakeSocket(user_id="alice")
+ worker = FakeSocket(path="/ws/internal")
+ for socket in (stranger, worker):
+ assert (await send(chat, socket=socket))["error"] == "access_denied"
+ assert (await chat.handlers.handle_get_chat_messages({"session_id": "wf"}, socket))["error"] == "access_denied"
+ assert (await chat.handlers.handle_chat_subscribe({"session_id": "wf"}, socket))["error"] == "access_denied"
+ assert (await chat.handlers.handle_clear_chat_messages({"session_id": "wf"}, socket))["error"] == "access_denied"
+ assert chat.dispatched == []
+ assert (await send(chat, socket=owner))["success"] is True
+
+
+async def test_reading_the_thread_returns_the_v2_shape(chat):
+ await talking(chat.database)
+ sent = await send(chat, client_message_id="c-7")
+ result = await chat.handlers.handle_get_chat_messages({"session_id": "wf", "all_generations": True}, None)
+ assert result["success"] is True and result["protocol_version"] == 1
+ [message] = result["messages"]
+ assert message["id"] == sent["message_id"] and message["run_id"] == sent["run_id"]
+ assert (message["text"], message["message"], message["kind"], message["status"]) == ("Book Saturday",) * 2 + ("text", "complete")
+ assert message["client_message_id"] == "c-7" and message["parent_id"] is None
+ assert message["siblings"] == {"index": 0, "count": 1, "ids": [message["id"]]}
+ assert result["thread"] == {"active_leaf_id": message["id"], "revision": 1}
+ [run] = result["active_runs"]
+ assert (run["run_id"], run["state"], run["user_message_id"], run["seq"]) == (sent["run_id"], "pending", message["id"], 0)
+
+
+async def test_a_failed_read_is_never_an_empty_thread(chat, monkeypatch):
+ async def broken(*_args, **_kwargs):
+ raise RuntimeError("database is locked")
+
+ monkeypatch.setattr(chat.database, "read_chat_messages", broken)
+ result = await chat.handlers.handle_get_chat_messages({"session_id": "wf", "all_generations": True}, None)
+ assert result == {"success": False, "error": "read_failed", "session_id": "wf"}
+
+
+async def test_subscribing_returns_snapshots_and_then_delivers_events(chat):
+ await talking(chat.database)
+ sent = await send(chat)
+ socket = FakeSocket()
+ subscribed = await chat.handlers.handle_chat_subscribe({"session_id": "wf"}, socket)
+ assert subscribed["hub_epoch"] == chat.hub.epoch
+ assert [run["run_id"] for run in subscribed["active_runs"]] == [sent["run_id"]]
+
+ await ledger.start_run(chat.database, run_id=sent["run_id"], temporal_workflow_id="tw", temporal_run_id="tr")
+ await asyncio.sleep(0)
+ [event] = socket.events()
+ assert event["type"] == "com.opencompany.chat.run.started" and event["data"]["seq"] == 1
+
+ snapshot = await chat.handlers.handle_get_chat_run({"run_id": sent["run_id"]}, socket)
+ assert (snapshot["run"]["state"], snapshot["run"]["seq"]) == ("running", 1)
+
+ await chat.handlers.handle_chat_unsubscribe({"session_id": "wf"}, socket)
+ await ledger.finish_run(chat.database, run_id=sent["run_id"], temporal_workflow_id="tw", temporal_run_id="tr", success=True)
+ await asyncio.sleep(0)
+ assert len(socket.events()) == 1
+ assert (await chat.handlers.handle_get_chat_run({"run_id": "r_unknown"}, socket))["error"] == "not_found"
+
+
+async def test_the_owners_clear_ends_a_live_run(chat):
+ await talking(chat.database)
+ sent = await send(chat)
+ socket = FakeSocket()
+ await chat.handlers.handle_chat_subscribe({"session_id": "wf"}, socket)
+ cleared = await chat.handlers.handle_clear_chat_messages({"session_id": "wf"}, socket)
+ assert cleared["cleared_count"] == 1
+ await asyncio.sleep(0)
+ [event] = socket.events()
+ assert (event["data"]["run_id"], event["data"]["code"]) == (sent["run_id"], "cleared")
+ assert await ledger.get_run(chat.database, sent["run_id"]) is None
+ # The lane is free: a new conversation starts.
+ assert (await send(chat, "Hello again"))["run_id"] is not None
diff --git a/server/tests/services/chat/test_hub.py b/server/tests/services/chat/test_hub.py
new file mode 100644
index 00000000..38b42129
--- /dev/null
+++ b/server/tests/services/chat/test_hub.py
@@ -0,0 +1,208 @@
+"""Run events (services/chat/hub.py, reducer.py, events.py): delivered only to
+the sockets subscribed to the session, in order, one ``seq`` per run, once
+per ``event_key``, nothing after the terminal event, a resync for a socket
+that falls behind; and the snapshot folded from a run's events matches the
+protocol fixture."""
+
+from __future__ import annotations
+
+import asyncio
+import json
+from pathlib import Path
+
+import pytest
+from starlette.websockets import WebSocketState
+
+from services.chat import reducer
+from services.chat.events import RUN_SUFFIXES, chat_run_event, run_event_suffix
+from services.chat.hub import ChatRunHub
+from tests.services.chat._helpers import FakeSocket
+
+FIXTURE = Path(__file__).resolve().parents[4] / "client" / "src" / "features" / "chat" / "__fixtures__" / "saturday-booking.events.json"
+
+
+def publish(hub, suffix, *, run_id="r_1", session_id="wf", **fields):
+ event_key = fields.pop("event_key", None)
+ return hub.publish(run_id=run_id, session_id=session_id, workflow_id=session_id, suffix=suffix, fields=fields, event_key=event_key)
+
+
+async def drained():
+ for _ in range(3):
+ await asyncio.sleep(0)
+
+
+async def test_a_subscriber_gets_its_sessions_events_in_order():
+ hub = ChatRunHub(queue_size=16)
+ mine, other = FakeSocket(), FakeSocket()
+ hub.subscribe(mine, "wf")
+ hub.subscribe(other, "another")
+ publish(hub, "started", kind="message")
+ publish(hub, "step.started", step_id="s1", step_name="Checking")
+ publish(hub, "finished", outcome={"type": "success"}, result={"no_reply": True})
+ await drained()
+ events = mine.events()
+ assert [event["type"] for event in events] == [
+ "com.opencompany.chat.run.started",
+ "com.opencompany.chat.run.step.started",
+ "com.opencompany.chat.run.finished",
+ ]
+ assert [event["data"]["seq"] for event in events] == [1, 2, 3]
+ assert [event["id"] for event in events] == ["r_1:1", "r_1:2", "r_1:3"]
+ assert all(event["subject"] == "r_1" and event["data"]["hub_epoch"] == hub.epoch for event in events)
+ # Scope rides in data, never as top-level extension attributes.
+ assert all("workflow_id" not in event for event in events)
+ assert other.sent == []
+
+
+async def test_repeated_keys_and_events_after_the_end_are_dropped():
+ hub = ChatRunHub(queue_size=16)
+ socket = FakeSocket()
+ hub.subscribe(socket, "wf")
+ assert publish(hub, "started", event_key="started") is not None
+ assert publish(hub, "started", event_key="started") is None
+ publish(hub, "failed", message="no", code="x", event_key="terminal")
+ assert publish(hub, "step.started", step_id="late", step_name="Late") is None
+ assert hub.ended("r_1") and hub.live_snapshot("r_1") is None and hub.current_seq("r_1") == 0
+ await drained()
+ assert [event["data"]["seq"] for event in socket.events()] == [1, 2]
+
+
+async def test_unsubscribing_and_disconnecting_stop_delivery():
+ hub = ChatRunHub(queue_size=16)
+ socket = FakeSocket()
+ hub.subscribe(socket, "wf")
+ hub.subscribe(socket, "second")
+ hub.unsubscribe(socket, "wf")
+ assert not hub.subscribed(socket, "wf") and hub.subscribed(socket, "second")
+ publish(hub, "started")
+ publish(hub, "started", run_id="r_2", session_id="second")
+ await drained()
+ assert [event["data"]["run_id"] for event in socket.events()] == ["r_2"]
+ hub.drop_socket(socket)
+ assert not hub.subscribed(socket, "second")
+
+
+async def test_a_closed_socket_is_dropped_when_written_to():
+ hub = ChatRunHub(queue_size=16)
+ socket = FakeSocket()
+ hub.subscribe(socket, "wf")
+ socket.client_state = WebSocketState.DISCONNECTED
+ publish(hub, "started")
+ await drained()
+ assert socket.sent == [] and not hub.subscribed(socket, "wf")
+
+
+async def test_a_socket_that_falls_behind_gets_a_resync():
+ hub = ChatRunHub(queue_size=3)
+ socket = FakeSocket()
+ hub.subscribe(socket, "wf")
+ # No await between publishes: the writer cannot drain.
+ for n in range(5):
+ publish(hub, "step.started", step_id=f"s{n}", step_name="Step")
+ await drained()
+ events = socket.events()
+ assert events[0]["data"]["name"] == "opencompany.resync"
+ assert events[0]["data"]["session_id"] == "wf"
+ # What follows the resync is newer than anything it replaced.
+ assert [event["data"]["seq"] for event in events[1:]] == [4, 5]
+
+
+async def test_the_disconnect_listener_drops_subscriptions(monkeypatch, hub):
+ import services.chat # noqa: F401 - registers the listener
+ from services.status_broadcaster import StatusBroadcaster
+
+ socket = FakeSocket()
+ hub.subscribe(socket, "wf")
+ await StatusBroadcaster().disconnect(socket)
+ assert not hub.subscribed(socket, "wf")
+
+
+async def test_a_session_snapshot_reads_the_live_runs():
+ hub = ChatRunHub(queue_size=16)
+ publish(hub, "started", kind="message", reply_message_id="a_r_1")
+ publish(hub, "text.started", message_id="r_1.1.1")
+ publish(hub, "text.content", message_id="r_1.1.1", delta="Hel")
+ publish(hub, "text.content", message_id="r_1.1.1", delta="lo")
+ publish(hub, "started", run_id="r_9", session_id="elsewhere")
+ live = hub.session_live("wf")
+ assert list(live) == ["r_1"]
+ assert live["r_1"]["seq"] == 4 and live["r_1"]["state"] == "running"
+ assert live["r_1"]["segments"] == [{"message_id": "r_1.1.1", "text": "Hello", "final": None}]
+
+
+# ----- events -----
+
+
+def test_run_events_are_only_the_protocols():
+ with pytest.raises(ValueError):
+ chat_run_event(suffix="exploded", run_id="r", seq=1, hub_epoch="e", workflow_id=None, session_id="s")
+ assert run_event_suffix("com.opencompany.chat.run.step.finished") == "step.finished"
+ assert run_event_suffix("com.opencompany.chat.updated") is None
+ # Fields can never override the scope.
+ event = chat_run_event(suffix="started", run_id="r", seq=3, hub_epoch="e", workflow_id="w", session_id="s", fields={"seq": 99, "run_id": "x"})
+ assert (event.data["seq"], event.data["run_id"]) == (3, "r")
+
+
+def test_the_protocol_doc_lists_every_suffix():
+ doc = (Path(__file__).resolve().parents[4] / "docs-internal" / "chat_protocol.md").read_text(encoding="utf-8")
+ for suffix in RUN_SUFFIXES:
+ assert f"`{suffix}`" in doc, suffix
+
+
+# ----- the reducer against the protocol fixture -----
+
+
+def fixture_events():
+ frames = json.loads(FIXTURE.read_text(encoding="utf-8"))
+ return [frame["data"] for frame in frames]
+
+
+def test_the_fixture_run_folds_to_its_final_state():
+ events = [event for event in fixture_events() if event["subject"] == "r_8f2a1c"]
+ snapshot = reducer.replay("r_8f2a1c", events)
+ assert snapshot["state"] == "finished" and snapshot["seq"] == 29
+ assert snapshot["outcome"]["type"] == "interrupt"
+ assert [interrupt["id"] for interrupt in snapshot["interrupts"]] == ["ap_1"]
+ assert [step["state"] for step in snapshot["steps"]] == ["done", "done", "done"]
+ assert snapshot["steps"][0]["name"] == "Checked Google Calendar"
+ [segment] = snapshot["segments"]
+ assert segment["final"] is True and segment["text"].startswith("Saturday is fairly full")
+ assert (snapshot["user_message_id"], snapshot["reply_message_id"]) == ("m_owner_1", "a_r_8f2a1c")
+
+
+def test_every_prefix_folds_and_duplicates_change_nothing():
+ events = [event for event in fixture_events() if event["subject"] == "r_8f2a1c"]
+ snapshot = reducer.empty_snapshot("r_8f2a1c")
+ for event in events:
+ snapshot = reducer.apply_event(snapshot, event)
+ assert reducer.apply_event(snapshot, event) == snapshot
+
+
+def test_the_resume_run_folds_on_its_own():
+ events = [event for event in fixture_events() if event["subject"] == "r_9b3d2e"]
+ snapshot = reducer.replay("r_9b3d2e", events)
+ assert (snapshot["kind"], snapshot["parent_run_id"], snapshot["state"]) == ("resume", "r_8f2a1c", "finished")
+
+
+def test_a_discarded_segment_and_stopping_fold():
+ snapshot = reducer.empty_snapshot("r")
+ for seq, (suffix, data) in enumerate(
+ [
+ ("started", {}),
+ ("text.started", {"message_id": "r.1.1"}),
+ ("text.content", {"message_id": "r.1.1", "delta": "draft"}),
+ ("custom", {"name": "opencompany.segment_discarded", "value": {"message_id": "r.1.1"}}),
+ ("custom", {"name": "opencompany.stopping", "value": {}}),
+ ],
+ start=1,
+ ):
+ snapshot = reducer.apply_event(snapshot, {"type": f"com.opencompany.chat.run.{suffix}", "data": {"seq": seq, **data}})
+ assert snapshot["segments"] == [] and snapshot["state"] == "stopping"
+
+
+def test_the_stored_run_wins_once_it_has_ended():
+ stored = {**reducer.empty_snapshot("r"), "state": "finished", "seq": 0}
+ live = {**reducer.empty_snapshot("r"), "state": "running", "seq": 7, "steps": [{"step_id": "s"}]}
+ merged = reducer.merge_snapshot(stored, live)
+ assert merged["state"] == "finished" and merged["seq"] == 7 and merged["steps"] == [{"step_id": "s"}]
+ assert reducer.merge_snapshot({**stored, "state": "pending"}, live)["state"] == "running"
diff --git a/server/tests/services/chat/test_ledger.py b/server/tests/services/chat/test_ledger.py
new file mode 100644
index 00000000..a74b12c4
--- /dev/null
+++ b/server/tests/services/chat/test_ledger.py
@@ -0,0 +1,383 @@
+"""Chat runs in the database (services/chat/ledger.py): admitted with the
+owner's message, one live run per session, claimed by one Temporal workflow,
+finished only by it, replies saved under the run's reply id, every message
+appended to one chain, and the watchdog ending runs nothing will."""
+
+from __future__ import annotations
+
+import asyncio
+from datetime import datetime, timedelta, timezone
+
+import pytest
+from sqlalchemy import text
+
+from models.chat import ChatRun
+from services.chat import ledger
+from tests.services.chat._helpers import add_control
+
+
+async def admit(database, session_id="wf", *, text_="Book Saturday", track=True, state="pending", client_message_id=None):
+ return await ledger.admit_message(
+ database,
+ session_id=session_id,
+ workflow_id=session_id,
+ execution_id="gen-1",
+ text=text_,
+ track=track,
+ state=state,
+ client_message_id=client_message_id,
+ )
+
+
+async def runs_of(database, session_id="wf"):
+ async with database.get_session() as session:
+ from sqlmodel import select
+
+ result = await session.execute(select(ChatRun).where(ChatRun.session_id == session_id))
+ return list(result.scalars().all())
+
+
+# ----- admission -----
+
+
+async def test_a_message_and_its_run_are_admitted_together(database, hub):
+ admission = await admit(database)
+ run = admission.run
+ assert admission.created is True
+ assert run.run_id.startswith("r_") and run.state == "pending" and run.kind == "message"
+ assert run.user_message_uid == admission.message["uid"]
+ assert run.reply_message_uid == f"a_{run.run_id}"
+ assert (run.workflow_id, run.run_key) == ("wf", "gen-1")
+ [row] = await database.get_chat_messages("wf")
+ assert (row["role"], row["message"], row["run_id"], row["execution_id"]) == ("user", "Book Saturday", run.run_id, "gen-1")
+
+
+async def test_one_live_run_per_session(database, hub):
+ first = await admit(database)
+ with pytest.raises(ledger.RunInProgress) as refused:
+ await admit(database, text_="And Sunday?")
+ assert refused.value.run.run_id == first.run.run_id
+ # Nothing of the refused send was written.
+ assert [row["message"] for row in await database.get_chat_messages("wf")] == ["Book Saturday"]
+ # Another session has its own lane.
+ assert (await admit(database, "other")).run is not None
+
+
+async def test_the_lane_index_refuses_a_second_live_run(database):
+ await admit(database)
+ with pytest.raises(Exception):
+ async with database.get_session() as session:
+ session.add(ChatRun(run_id="r_sneaky", session_id="wf", state="running", kind="message"))
+ await session.commit()
+ # A resume run (an approved send executing) never holds the lane.
+ async with database.get_session() as session:
+ session.add(ChatRun(run_id="r_resume", session_id="wf", state="running", kind="resume"))
+ await session.commit()
+
+
+async def test_concurrent_sends_admit_one_run(database, hub):
+ results = await asyncio.gather(*(admit(database, text_=f"message {n}") for n in range(5)), return_exceptions=True)
+ admitted = [result for result in results if isinstance(result, ledger.Admission)]
+ refused = [result for result in results if isinstance(result, ledger.RunInProgress)]
+ assert len(admitted) == 1 and len(refused) == 4
+ assert len(await runs_of(database)) == 1
+
+
+async def test_a_resent_message_returns_the_first_send(database, hub):
+ first = await admit(database, client_message_id="c-1")
+ again = await admit(database, client_message_id="c-1")
+ assert again.created is False
+ assert again.message["uid"] == first.message["uid"]
+ assert again.run.run_id == first.run.run_id
+ assert len(await database.get_chat_messages("wf")) == 1
+ # The same client id in another chat names another message.
+ elsewhere = await admit(database, "other", client_message_id="c-1")
+ assert elsewhere.message["uid"] != first.message["uid"]
+
+
[email protected]("bad", ["", "a b", "x" * 101, 7, "../etc"])
+async def test_a_malformed_client_message_id_is_refused(database, bad):
+ with pytest.raises(ValueError):
+ await admit(database, client_message_id=bad)
+ assert await database.get_chat_messages("wf") == []
+
+
+async def test_an_untracked_message_starts_no_run(database):
+ admission = await admit(database, track=False)
+ assert admission.run is None and admission.message["run_id"] is None
+ assert await runs_of(database) == []
+
+
+# ----- one chain -----
+
+
+async def test_every_message_follows_the_last_one(database):
+ for n in range(3):
+ await database.add_chat_message("wf", "user", f"m{n}")
+ rows = await database.get_chat_messages("wf")
+ assert rows[0]["parent_uid"] is None
+ assert [row["parent_uid"] for row in rows[1:]] == [row["uid"] for row in rows[:-1]]
+ async with database.get_session() as session:
+ from models.chat import ChatThread
+
+ thread = await session.get(ChatThread, "wf")
+ assert (thread.active_leaf_uid, thread.revision) == (rows[-1]["uid"], 3)
+
+
+async def test_concurrent_writes_never_fork_the_chain(database):
+ await asyncio.gather(*(database.add_chat_message("wf", "assistant", f"m{n}") for n in range(10)))
+ rows = await database.get_chat_messages("wf")
+ parents = [row["parent_uid"] for row in rows]
+ assert len(set(parents)) == 10, "two messages followed the same one"
+ assert sum(parent is None for parent in parents) == 1
+
+
+async def test_a_message_id_from_another_chat_is_refused(database):
+ await database.add_chat_message("wf", "user", "mine", uid="m_shared")
+ assert await database.add_chat_message("other", "user", "theirs", uid="m_shared") is None
+ assert await database.get_chat_messages("other") == []
+
+
+async def test_older_rows_are_migrated_into_one_chain(database):
+ """A database from before chat runs: the migration adds the columns and
+ chains each session's rows in order, once."""
+ async with database.engine.begin() as conn:
+ await conn.execute(text("DROP TABLE chat_messages"))
+ await conn.execute(
+ text(
+ "CREATE TABLE chat_messages (id INTEGER PRIMARY KEY, session_id VARCHAR(255), execution_id VARCHAR(255),"
+ " role VARCHAR(20), message VARCHAR(50000), created_at DATETIME)"
+ )
+ )
+ for n, (session_id, moment) in enumerate([("wf", "2026-01-01 10:00:00"), ("other", "2026-01-01 10:00:01"), ("wf", "2026-01-01 10:00:02")], start=1):
+ await conn.execute(
+ text("INSERT INTO chat_messages (id, session_id, role, message, created_at) VALUES (:id, :s, 'user', :m, :t)"),
+ {"id": n, "s": session_id, "m": f"old {n}", "t": moment},
+ )
+ await database._migrate_chat_messages()
+ await database._migrate_chat_messages() # runs again at every startup; changes nothing
+ rows = await database.get_chat_messages("wf")
+ assert [(row["uid"], row["parent_uid"]) for row in rows] == [("m1", None), ("m3", "m1")]
+ assert [(row["kind"], row["status"], row["parts"]) for row in rows] == [("text", "complete", {})] * 2
+ # A new message continues the migrated chain.
+ added = await database.add_chat_message("wf", "user", "new")
+ assert added["parent_uid"] == "m3"
+
+
+# ----- start and finish -----
+
+
+async def started(database, admission, workflow_id="tw-1", run_id="tr-1"):
+ return await ledger.start_run(database, run_id=admission.run.run_id, temporal_workflow_id=workflow_id, temporal_run_id=run_id)
+
+
+async def test_the_first_workflow_claims_the_run(database, hub):
+ admission = await admit(database)
+ run = await started(database, admission)
+ assert run.state == "running" and run.started_at is not None
+ # A retry by the same workflow still holds it; another trigger's run does not.
+ assert (await started(database, admission)) is not None
+ assert (await started(database, admission, workflow_id="tw-2")) is None
+
+
+async def test_a_queued_run_starts_when_the_employee_resumes(database, hub):
+ admission = await admit(database, state="queued")
+ assert admission.run.state == "queued"
+ assert (await started(database, admission)).state == "running"
+
+
+async def test_finishing_with_a_reply_names_it(database, hub):
+ admission = await admit(database)
+ run = await started(database, admission)
+ reply = await ledger.post_reply(database, run=run, node_id="wf:chatReply:1", text="Booked.", execution_id="gen-1")
+ assert reply["uid"] == run.reply_message_uid and reply["run_id"] == run.run_id
+ finished = await ledger.finish_run(database, run_id=run.run_id, temporal_workflow_id="tw-1", temporal_run_id="tr-1", success=True)
+ assert (finished.state, finished.outcome, finished.result) == ("finished", "success", {"reply_message_id": run.reply_message_uid})
+ assert finished.finished_at is not None
+
+
+async def test_finishing_without_a_reply_says_so(database, hub):
+ admission = await admit(database)
+ await started(database, admission)
+ finished = await ledger.finish_run(database, run_id=admission.run.run_id, temporal_workflow_id="tw-1", temporal_run_id="tr-1", success=True)
+ assert finished.result == {"no_reply": True}
+
+
+async def test_a_failed_run_keeps_its_error(database, hub):
+ admission = await admit(database)
+ await started(database, admission)
+ failed = await ledger.finish_run(
+ database, run_id=admission.run.run_id, temporal_workflow_id="tw-1", temporal_run_id="tr-1",
+ success=False, error="Calendar said no", hint="Reconnect Google", requires_user_action=True,
+ )
+ assert (failed.state, failed.error, failed.error_code) == ("error", "Calendar said no", "run_failed")
+ assert failed.result == {"hint": "Reconnect Google", "requires_user_action": True}
+ # The lane is free again.
+ assert (await admit(database, text_="Try again")).run is not None
+
+
+async def test_only_the_claimant_finishes(database, hub):
+ admission = await admit(database)
+ await started(database, admission)
+ assert await ledger.finish_run(database, run_id=admission.run.run_id, temporal_workflow_id="tw-2", temporal_run_id="tr-1", success=True) is None
+ assert (await ledger.get_run(database, admission.run.run_id)).state == "running"
+
+
+async def test_a_retried_finish_changes_nothing(database, hub):
+ admission = await admit(database)
+ await started(database, admission)
+ args = dict(run_id=admission.run.run_id, temporal_workflow_id="tw-1", temporal_run_id="tr-1")
+ first = await ledger.finish_run(database, success=True, **args)
+ again = await ledger.finish_run(database, success=False, error="late", **args)
+ assert again.state == first.state == "finished" and again.error is None
+
+
+async def test_a_second_reply_node_gets_its_own_id_and_a_retry_saves_once(database, hub):
+ admission = await admit(database)
+ run = await started(database, admission)
+ first = await ledger.post_reply(database, run=run, node_id="n1", text="One.", execution_id="gen-1")
+ second = await ledger.post_reply(database, run=run, node_id="n2", text="Two.", execution_id="gen-1")
+ retry = await ledger.post_reply(database, run=run, node_id="n1", text="One (retried).", execution_id="gen-1")
+ assert (first["uid"], second["uid"], retry["uid"]) == (run.reply_message_uid, f"{run.reply_message_uid}.2", run.reply_message_uid)
+ assert retry["message"] == "One."
+ assert [row["message"] for row in await database.get_chat_messages("wf")] == ["Book Saturday", "One.", "Two."]
+
+
+async def test_two_reply_nodes_at_once_never_share_an_id(database, hub):
+ admission = await admit(database)
+ run = await started(database, admission)
+ rows = await asyncio.gather(*(ledger.post_reply(database, run=run, node_id=f"n{n}", text=f"{n}", execution_id="gen-1") for n in range(4)))
+ assert len({row["uid"] for row in rows}) == 4
+
+
+# ----- events -----
+
+
+async def test_transitions_publish_once_each(database, hub):
+ from tests.services.chat._helpers import FakeSocket
+
+ socket = FakeSocket()
+ hub.subscribe(socket, "wf")
+ admission = await admit(database)
+ await started(database, admission)
+ await started(database, admission) # a retried start publishes nothing new
+ args = dict(run_id=admission.run.run_id, temporal_workflow_id="tw-1", temporal_run_id="tr-1", success=True)
+ await ledger.finish_run(database, **args)
+ await ledger.finish_run(database, **args)
+ await asyncio.sleep(0)
+ events = socket.events()
+ assert [event["type"].rsplit(".", 1)[-1] for event in events] == ["started", "finished"]
+ started_event, finished_event = events
+ assert started_event["data"]["user_message_id"] == admission.message["uid"]
+ assert started_event["data"]["reply_message_id"] == admission.run.reply_message_uid
+ assert [event["data"]["seq"] for event in events] == [1, 2]
+ assert finished_event["data"]["outcome"] == {"type": "success"}
+ assert finished_event["data"]["result"] == {"no_reply": True}
+
+
+# ----- the watchdog -----
+
+
+NOW = datetime(2026, 10, 3, 12, 0, tzinfo=timezone.utc)
+LONG_AGO = NOW - timedelta(days=2)
+
+
+async def age(database, run_id, **moments):
+ from sqlalchemy import update
+
+ async with database.get_session() as session:
+ await session.execute(update(ChatRun).where(ChatRun.run_id == run_id).values(**moments))
+ await session.commit()
+
+
+async def sweep(database, **kwargs):
+ kwargs.setdefault("process_started", LONG_AGO)
+ return await ledger.sweep(database, now=NOW, **kwargs)
+
+
+async def test_a_run_never_picked_up_ends_as_not_delivered(database, hub):
+ admission = await admit(database)
+ await age(database, admission.run.run_id, created_at=NOW - timedelta(minutes=5))
+ assert await sweep(database) == [admission.run.run_id]
+ run = await ledger.get_run(database, admission.run.run_id)
+ assert (run.state, run.error_code) == ("error", "not_delivered")
+
+
+async def test_a_restart_gives_a_pending_run_its_full_wait_again(database, hub):
+ admission = await admit(database)
+ await age(database, admission.run.run_id, created_at=NOW - timedelta(minutes=5))
+ assert await sweep(database, process_started=NOW - timedelta(seconds=10)) == []
+
+
+async def test_a_queued_run_waits_for_resume_and_ends_when_the_employee_stops(database, hub):
+ admission = await admit(database, state="queued")
+ await age(database, admission.run.run_id, created_at=LONG_AGO)
+ await add_control(database, "wf", "paused")
+ assert await sweep(database) == []
+ async with database.engine.begin() as conn:
+ await conn.execute(text("UPDATE workflow_control_executions SET status = 'reset'"))
+ assert await sweep(database) == [admission.run.run_id]
+
+
+async def test_a_run_past_its_longest_ends_as_timed_out(database, hub):
+ admission = await admit(database)
+ await started(database, admission)
+ await age(database, admission.run.run_id, started_at=LONG_AGO)
+ assert await sweep(database) == [admission.run.run_id]
+ assert (await ledger.get_run(database, admission.run.run_id)).error_code == "timed_out"
+
+
[email protected](("replied", "state", "code"), [(True, "finished", None), (False, "error", "interrupted")])
+async def test_a_run_whose_workflow_closed_without_finishing_is_ended(database, hub, replied, state, code):
+ admission = await admit(database)
+ run = await started(database, admission)
+ await age(database, run.run_id, started_at=NOW - timedelta(minutes=5))
+ if replied:
+ await ledger.post_reply(database, run=run, node_id="n", text="Done.", execution_id="gen-1")
+ asked = []
+
+ async def closed(workflow_id, temporal_run_id):
+ asked.append((workflow_id, temporal_run_id))
+ return True
+
+ assert await sweep(database, temporal_status=closed) == [run.run_id]
+ assert asked == [("tw-1", "tr-1")]
+ ended = await ledger.get_run(database, run.run_id)
+ assert (ended.state, ended.error_code) == (state, code)
+
+
+async def test_a_running_workflow_and_an_unknown_answer_keep_the_run(database, hub):
+ admission = await admit(database)
+ run = await started(database, admission)
+ await age(database, run.run_id, started_at=NOW - timedelta(minutes=5))
+
+ async def running(*_):
+ return False
+
+ async def unknown(*_):
+ return None
+
+ assert await sweep(database, temporal_status=running) == []
+ assert await sweep(database, temporal_status=unknown) == []
+ # A workflow that only just started is not asked at all.
+ await age(database, run.run_id, started_at=NOW - timedelta(seconds=5))
+
+ async def never(*_):
+ raise AssertionError("asked too early")
+
+ assert await sweep(database, temporal_status=never) == []
+
+
+async def test_clearing_ends_live_runs_for_their_subscribers(database, hub, frames):
+ from services import chat_thread
+ from tests.services.chat._helpers import FakeSocket
+
+ socket = FakeSocket()
+ hub.subscribe(socket, "wf")
+ admission = await admit(database)
+ assert await chat_thread.clear_chat_thread(database, "wf", reason="cleared") == 1
+ await asyncio.sleep(0)
+ [event] = socket.events()
+ assert event["type"] == "com.opencompany.chat.run.failed"
+ assert (event["data"]["run_id"], event["data"]["code"]) == (admission.run.run_id, "cleared")
+ assert await runs_of(database) == []
diff --git a/server/tests/services/chat/test_watchdog.py b/server/tests/services/chat/test_watchdog.py
new file mode 100644
index 00000000..2ffe5f96
--- /dev/null
+++ b/server/tests/services/chat/test_watchdog.py
@@ -0,0 +1,93 @@
+"""The chat run watchdog (services/chat/watchdog.py): sweeps at once when it
+starts and then every interval, survives a failing sweep, stops cleanly, and
+asks Temporal whether a run's workflow has closed."""
+
+from __future__ import annotations
+
+import asyncio
+from types import SimpleNamespace
+
+import pytest
+
+from services.chat import watchdog
+
+
+async def test_it_sweeps_at_once_and_keeps_sweeping_after_a_failure(monkeypatch):
+ sweeps = []
+
+ async def sweep(database, *, temporal_status):
+ sweeps.append((database, temporal_status))
+ if len(sweeps) == 1:
+ raise RuntimeError("database is locked")
+ return []
+
+ monkeypatch.setattr(watchdog.ledger, "sweep", sweep)
+ dog = watchdog.ChatRunWatchdog("db", interval=0.01)
+ dog.start()
+ dog.start() # starting twice runs one loop
+ for _ in range(50):
+ if len(sweeps) >= 3:
+ break
+ await asyncio.sleep(0.01)
+ await dog.stop()
+ await dog.stop() # stopping twice is harmless
+ assert len(sweeps) >= 3
+ assert all(database == "db" and status is watchdog.temporal_workflow_closed for database, status in sweeps)
+ count = len(sweeps)
+ await asyncio.sleep(0.05)
+ assert len(sweeps) == count, "the loop kept running after stop"
+
+
+def _container(monkeypatch, client):
+ import core.container as container_module
+
+ wrapper = SimpleNamespace(client=client) if client is not None else None
+ monkeypatch.setattr(container_module, "container", SimpleNamespace(temporal_client=lambda: wrapper))
+
+
+async def test_without_temporal_the_answer_is_unknown(monkeypatch):
+ _container(monkeypatch, None)
+ assert await watchdog.temporal_workflow_closed("w", "r") is None
+
+
[email protected](("status", "closed"), [("RUNNING", False), ("COMPLETED", True), ("TERMINATED", True), ("CANCELED", True)])
+async def test_a_workflow_is_closed_once_it_stops_running(monkeypatch, status, closed):
+ from temporalio.client import WorkflowExecutionStatus
+
+ asked = []
+
+ class Handle:
+ async def describe(self):
+ return SimpleNamespace(status=getattr(WorkflowExecutionStatus, status))
+
+ class Client:
+ def get_workflow_handle(self, workflow_id, run_id=None):
+ asked.append((workflow_id, run_id))
+ return Handle()
+
+ _container(monkeypatch, Client())
+ assert await watchdog.temporal_workflow_closed("w", "r") is closed
+ assert asked == [("w", "r")]
+
+
+async def test_a_workflow_temporal_no_longer_knows_is_closed(monkeypatch):
+ from temporalio.service import RPCError, RPCStatusCode
+
+ class Handle:
+ def __init__(self, status):
+ self.status = status
+
+ async def describe(self):
+ raise RPCError("nope", self.status, b"")
+
+ class Client:
+ status = RPCStatusCode.NOT_FOUND
+
+ def get_workflow_handle(self, workflow_id, run_id=None):
+ return Handle(self.status)
+
+ client = Client()
+ _container(monkeypatch, client)
+ assert await watchdog.temporal_workflow_closed("w", "r") is True
+ client.status = RPCStatusCode.UNAVAILABLE
+ assert await watchdog.temporal_workflow_closed("w", "r") is None
diff --git a/server/tests/services/test_chat_thread.py b/server/tests/services/test_chat_thread.py
index 2d9548ff..7378a4a5 100644
--- a/server/tests/services/test_chat_thread.py
+++ b/server/tests/services/test_chat_thread.py
@@ -1,9 +1,10 @@
"""A workflow's chat thread (services/chat_thread.py) and the chat
-WebSocket handlers built on it: rows carry the live generation, every
-insert and every clear that removed rows is announced on ``chat.updated``,
-a message reaches a workflow only while a deployment would read it, the
-generation is on every row, a Reset clears the thread through the chat
-nodes' Reset hook, and deleting the workflow deletes it."""
+WebSocket handlers built on it (services/chat/handlers.py): rows carry the
+live generation, every insert and every clear that removed rows is announced
+on ``chat.updated``, a message reaches a workflow only while a deployment
+would read it, the generation is on every row, a Reset clears the thread
+through the chat nodes' Reset hook, and deleting the workflow deletes it.
+Chat runs have their own tests in tests/services/chat/."""
from __future__ import annotations
@@ -16,10 +17,9 @@ from types import SimpleNamespace
import pytest
-import nodes.trigger.chat_trigger._events as chat_events
+import services.chat.handlers as chat_handlers
import services.status_broadcaster as status_broadcaster
from models.database import WorkflowControlExecution
-from routers import websocket as ws_router
from services import chat_thread
@@ -59,15 +59,15 @@ def frames(monkeypatch):
@pytest.fixture
def router(monkeypatch, database, frames):
- """The router's chat handlers on the test database, with the chat
- trigger dispatch captured."""
+ """The chat handlers on the test database, with the chat trigger
+ dispatch captured."""
dispatched: list = []
- async def dispatch(event_data, *, workflow_id=None):
+ async def dispatch(event_data, *, workflow_id=None, event_id=None):
dispatched.append((dict(event_data), workflow_id))
- monkeypatch.setattr(ws_router, "container", SimpleNamespace(database=lambda: database))
- monkeypatch.setattr(chat_events, "dispatch_chat_message_received", dispatch)
+ monkeypatch.setattr(chat_handlers, "container", SimpleNamespace(database=lambda: database))
+ monkeypatch.setattr(chat_handlers, "dispatch_chat_message_received", dispatch)
# The Clear's listeners are whatever a test registers, never the plugins'.
monkeypatch.setattr(chat_thread, "_CLEARED_LISTENERS", [])
return SimpleNamespace(database=database, frames=frames, dispatched=dispatched)
@@ -123,7 +123,8 @@ def test_rows_belong_to_the_live_generation():
async def test_a_recorded_message_is_stamped_and_announced(database, frames):
await control(database, "7", "running", generation=2)
- assert await chat_thread.record_chat_message(database, "7", "assistant", "Done.") is True
+ saved = await chat_thread.record_chat_message(database, "7", "assistant", "Done.")
+ assert saved["message"] == "Done." and saved["uid"].startswith("m_")
[row] = await database.get_chat_messages("7")
assert (row["role"], row["message"], row["execution_id"]) == ("assistant", "Done.", "gen-2")
[frame] = frames
@@ -198,7 +199,7 @@ async def test_history_is_the_newest_rows_oldest_first_in_utc(database):
async def send(message, session_id):
- return await ws_router.handle_send_chat_message({"message": message, "session_id": session_id, "timestamp": "t"}, None)
+ return await chat_handlers.handle_send_chat_message({"message": message, "session_id": session_id, "timestamp": "t"}, None)
async def test_a_workflow_that_is_not_running_takes_no_message(router):
@@ -216,13 +217,16 @@ async def test_a_message_goes_now_or_waits_for_resume(router, status, delivery):
assert result["success"] is True and result["delivery"] == delivery and result["timestamp"] == "t"
[row] = await router.database.get_chat_messages("7")
assert (row["role"], row["message"], row["execution_id"]) == ("user", "Book Tuesday", "gen-3")
+ assert result["message_id"] == row["uid"]
+ # No chat trigger answers this graph, so the message starts no run.
+ assert result["run_id"] is None
# Only this workflow's triggers hear it.
- assert router.dispatched == [({"message": "Book Tuesday", "timestamp": "t", "session_id": "7"}, "7")]
+ assert router.dispatched == [({"message": "Book Tuesday", "timestamp": "t", "session_id": "7", "message_id": row["uid"]}, "7")]
assert updates(router.frames) == [{"workflow_id": "7", "session_id": "7", "role": "user"}]
async def test_the_default_session_keeps_its_old_behaviour(router):
- result = await ws_router.handle_send_chat_message({"message": "hi"}, None)
+ result = await chat_handlers.handle_send_chat_message({"message": "hi"}, None)
assert result["success"] is True and "delivery" not in result
assert datetime.fromisoformat(result["timestamp"]).utcoffset() is not None
assert [row["message"] for row in await router.database.get_chat_messages("default")] == ["hi"]
@@ -236,17 +240,20 @@ async def test_history_is_one_generation_unless_all_are_asked_for(router):
await control(router.database, "7", "running", generation=2)
await router.database.add_chat_message("7", "user", "after it", execution_id="gen-2")
- live = await ws_router.handle_get_chat_messages({"session_id": "7"}, None)
+ live = await chat_handlers.handle_get_chat_messages({"session_id": "7"}, None)
assert [message["message"] for message in live["messages"]] == ["after it"]
- everything = await ws_router.handle_get_chat_messages({"session_id": "7", "all_generations": True, "limit": 200}, None)
+ everything = await chat_handlers.handle_get_chat_messages({"session_id": "7", "all_generations": True, "limit": 200}, None)
assert [(m["role"], m["message"], m["run_key"]) for m in everything["messages"]] == [
("user", "before the restart", "gen-1"),
("assistant", "an answer", "gen-1"),
("user", "after it", "gen-2"),
]
- assert all(isinstance(m["id"], int) and datetime.fromisoformat(m["timestamp"]).utcoffset() is not None for m in everything["messages"])
- newest = await ws_router.handle_get_chat_messages({"session_id": "7", "all_generations": True, "limit": 2}, None)
+ assert all(
+ m["id"].startswith("m_") and isinstance(m["legacy_id"], int) and datetime.fromisoformat(m["timestamp"]).utcoffset() is not None
+ for m in everything["messages"]
+ )
+ newest = await chat_handlers.handle_get_chat_messages({"session_id": "7", "all_generations": True, "limit": 2}, None)
assert [m["message"] for m in newest["messages"]] == ["an answer", "after it"]
@@ -254,8 +261,8 @@ async def test_after_a_reset_the_live_read_is_empty(router):
# A row a Reset left (its graph had no chat node to clear it).
await router.database.add_chat_message("7", "user", "kept", execution_id="gen-1")
await control(router.database, "7", "reset", generation=1)
- assert (await ws_router.handle_get_chat_messages({"session_id": "7"}, None))["messages"] == []
- assert len((await ws_router.handle_get_chat_messages({"session_id": "7", "all_generations": True}, None))["messages"]) == 1
+ assert (await chat_handlers.handle_get_chat_messages({"session_id": "7"}, None))["messages"] == []
+ assert len((await chat_handlers.handle_get_chat_messages({"session_id": "7", "all_generations": True}, None))["messages"]) == 1
async def test_the_owners_clear_lets_the_agent_forget_too(router):
@@ -272,23 +279,23 @@ async def test_the_owners_clear_lets_the_agent_forget_too(router):
chat_thread.register_chat_cleared_listener(forget) # registering twice is a no-op
await router.database.add_chat_message("7", "user", "hello", execution_id="gen-1")
- cleared = await ws_router.handle_clear_chat_messages({"session_id": "7"}, None)
+ cleared = await chat_handlers.handle_clear_chat_messages({"session_id": "7"}, None)
assert cleared["cleared_count"] == 1
assert await router.database.get_chat_messages("7") == []
# A failing listener never fails the clear, nor stops the next one.
assert heard == [(router.database, "7")]
# The editor's "default" chat is no workflow's: nothing to forget.
- await ws_router.handle_clear_chat_messages({"session_id": "default"}, None)
+ await chat_handlers.handle_clear_chat_messages({"session_id": "default"}, None)
assert heard == [(router.database, "7")]
async def test_save_and_clear_go_through_the_thread(router):
await control(router.database, "7", "running", generation=4)
- saved = await ws_router.handle_save_chat_message({"message": "An answer", "role": "assistant", "session_id": "7"}, None)
+ saved = await chat_handlers.handle_save_chat_message({"message": "An answer", "role": "assistant", "session_id": "7"}, None)
assert saved["success"] is True
[row] = await router.database.get_chat_messages("7")
assert row["execution_id"] == "gen-4"
- cleared = await ws_router.handle_clear_chat_messages({"session_id": "7"}, None)
+ cleared = await chat_handlers.handle_clear_chat_messages({"session_id": "7"}, None)
assert cleared["cleared_count"] == 1
assert [update["role"] for update in updates(router.frames)] == ["assistant", None]
diff --git a/server/tests/temporal/test_machina_chat_run.py b/server/tests/temporal/test_machina_chat_run.py
new file mode 100644
index 00000000..1dfffcec
--- /dev/null
+++ b/server/tests/temporal/test_machina_chat_run.py
@@ -0,0 +1,335 @@
+"""MachinaWorkflow claims and finishes the chat run the owner's message
+started (``chat_run.start`` / ``chat_run.finish``, behind
+``machina-chat-run-v1``), and every node of the run carries its scope.
+
+Two layers, like the run-record gate:
+
+- the real ``run()`` body with activity dispatch faked: a run is claimed
+ only from an event the chat service produced; a claimed run's nodes carry
+ ``run_scope`` and the run is finished with the outcome; an unclaimed run
+ (another chat trigger got it) runs untracked; a closed patch (a pre-patch
+ history) schedules neither activity;
+- an SDK replay gate against Temporal's time-skipping test server: histories
+ recorded with the patch closed and open both replay, and forcing the patch
+ open on the old history fails replay.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import subprocess
+import sys
+from pathlib import Path
+from types import SimpleNamespace
+from typing import Any, Dict, List
+from unittest.mock import MagicMock
+from uuid import uuid4
+
+import pytest
+
+import nodes # noqa: F401 -- populate plugin registry
+
+ROUTING = {"version": 1, "agent_workflow_enabled": False, "per_type_dispatch_enabled": False, "worker_pool_enabled": False}
+SOURCE = "opencompany://services/chat"
+TYPE = "com.opencompany.chat.message.received"
+
+
+def _graph(*, source: str = SOURCE, event_type: str = TYPE, run_id: Any = "r_1", spawned: bool = True) -> Dict[str, Any]:
+ trigger: Dict[str, Any] = {"id": "wf-cr:chatTrigger:1", "type": "chatTrigger", "data": {"label": "Chat"}}
+ if spawned:
+ data = {"message": "Book Saturday", "session_id": "wf-cr", "run_id": run_id}
+ trigger.update(
+ {
+ "_pre_executed": True,
+ "_trigger_output": {**data, "_event_envelope": {"id": str(run_id), "source": source, "type": event_type, "data": data}},
+ }
+ )
+ return {
+ "nodes": [trigger, {"id": "wf-cr:console:1", "type": "console", "data": {"label": "Log"}}],
+ "edges": [{"id": "e1", "source": trigger["id"], "target": "wf-cr:console:1", "targetHandle": "input-main"}],
+ "workflow_id": "wf-cr",
+ "session_id": "wf-cr",
+ "execution_id": "wf-cr:execution:1",
+ "generation": 2,
+ "_temporal_routing_v1": ROUTING,
+ }
+
+
+# ----- the run() body with fakes -----
+
+
+def fakes(monkeypatch):
+ from temporalio import workflow as temporal_workflow
+
+ from services.temporal.workflow import CHAT_RUN_PATCH, MachinaWorkflow
+
+ state = SimpleNamespace(
+ activities=[],
+ contexts=[],
+ node_success=True,
+ claimed=True,
+ open_patches={CHAT_RUN_PATCH, "machina-conditional-edges-v1", "machina-run-record-v1"},
+ asked=[],
+ )
+ monkeypatch.setattr(temporal_workflow, "logger", MagicMock())
+
+ def patched(patch_id):
+ state.asked.append(patch_id)
+ return patch_id in state.open_patches
+
+ monkeypatch.setattr(temporal_workflow, "patched", patched)
+ monkeypatch.setattr(temporal_workflow, "info", lambda: SimpleNamespace(workflow_id="wf-cr-chat-r_1", run_id="run-1"))
+
+ def start_activity(name, **kwargs):
+ context = kwargs["args"][0]
+ state.contexts.append(context)
+ future = asyncio.get_event_loop().create_future()
+ future.set_result({"success": state.node_success, "node_id": context["node_id"], "result": {}, "error": None if state.node_success else "boom"})
+ return future
+
+ async def execute_activity(name, payload=None, **_kwargs):
+ state.activities.append((name, payload))
+ if name == "chat_run.start":
+ return {"claimed": state.claimed}
+ return None
+
+ monkeypatch.setattr(temporal_workflow, "start_activity", start_activity)
+ monkeypatch.setattr(temporal_workflow, "execute_activity", execute_activity)
+ monkeypatch.setattr(
+ MachinaWorkflow,
+ "_resolve_dispatch",
+ lambda self, node_type, **_kwargs: {"kind": "activity", "name": f"node.{node_type}.v1", "queue": None},
+ )
+ return state
+
+
+async def _run(graph):
+ from services.temporal.workflow import MachinaWorkflow
+
+ return await asyncio.wait_for(MachinaWorkflow().run(graph), timeout=5.0)
+
+
+def _names(state) -> List[str]:
+ return [name for name, _ in state.activities]
+
+
+def _payloads(state, name) -> List[Dict[str, Any]]:
+ return [payload for activity, payload in state.activities if activity == name]
+
+
+async def test_a_chat_message_run_is_claimed_scoped_and_finished(fakes):
+ await _run(_graph())
+ assert _payloads(fakes, "chat_run.start") == [{"run_id": "r_1", "temporal_workflow_id": "wf-cr-chat-r_1", "temporal_run_id": "run-1"}]
+ assert _payloads(fakes, "chat_run.finish") == [
+ {"run_id": "r_1", "temporal_workflow_id": "wf-cr-chat-r_1", "temporal_run_id": "run-1", "success": True}
+ ]
+ assert [context["run_scope"] for context in fakes.contexts] == [{"run_id": "r_1", "session_id": "wf-cr"}]
+ # Claimed before the first node is scheduled; finished before the run is recorded.
+ names = _names(fakes)
+ assert names.index("chat_run.finish") < names.index("workflow_runs.record_completion")
+
+
+async def test_a_failed_run_finishes_with_its_error(fakes):
+ fakes.node_success = False
+ await _run(_graph())
+ [finish] = _payloads(fakes, "chat_run.finish")
+ assert finish["success"] is False and finish["error"] == "boom"
+
+
+async def test_an_unclaimed_run_runs_untracked(fakes):
+ fakes.claimed = False
+ await _run(_graph())
+ assert _payloads(fakes, "chat_run.finish") == []
+ assert all("run_scope" not in context for context in fakes.contexts)
+
+
+ "graph",
+ [
+ _graph(source="opencompany://nodes/webhook"),
+ _graph(event_type="com.opencompany.webhook.received"),
+ _graph(run_id=None),
+ _graph(run_id=42),
+ _graph(spawned=False),
+ ],
+)
+async def test_a_run_id_is_trusted_only_from_the_chat_service(fakes, graph):
+ from services.temporal.workflow import CHAT_RUN_PATCH
+
+ await _run(graph)
+ assert "chat_run.start" not in _names(fakes)
+ assert CHAT_RUN_PATCH not in fakes.asked
+
+
+async def test_a_pre_patch_history_schedules_neither_activity(fakes):
+ from services.temporal.workflow import CHAT_RUN_PATCH
+
+ fakes.open_patches.discard(CHAT_RUN_PATCH)
+ await _run(_graph())
+ assert "chat_run.start" not in _names(fakes) and "chat_run.finish" not in _names(fakes)
+ assert all("run_scope" not in context for context in fakes.contexts)
+
+
+def test_every_worker_registers_the_activities():
+ source = (Path(__file__).parents[2] / "services" / "temporal" / "worker.py").read_text(encoding="utf-8")
+ assert source.count("*CHAT_RUN_ACTIVITIES") == 3
+ from services.chat.activities import CHAT_RUN_ACTIVITIES
+
+ names = sorted(getattr(fn, "__temporal_activity_definition").name for fn in CHAT_RUN_ACTIVITIES)
+ assert names == ["chat_run.finish", "chat_run.start"]
+
+
+def test_the_scope_reaches_every_node_path():
+ """``run_scope`` must survive each hop a node's context takes: the
+ per-type activity extras, the legacy activity's message, the agent
+ workflow's inherited keys, and the internal socket's execute handler."""
+ import inspect
+
+ from routers import websocket as ws_router
+ from services.plugin import base
+ from services.temporal import activities, agent_workflow
+
+ assert "run_scope" in agent_workflow._INHERITED_SCOPE_KEYS
+ assert '"run_scope"' in inspect.getsource(base)
+ assert '"run_scope"' in inspect.getsource(activities.NodeExecutionActivities)
+ handler = inspect.getsource(ws_router.handle_execute_node)
+ assert '"run_scope"' in handler and "/ws/internal" in handler
+
+
+async def test_the_activities_go_through_the_ledger(monkeypatch):
+ import core.container as container_module
+ import services.chat.ledger as ledger
+
+ from services.chat.activities import finish_chat_run_activity, start_chat_run_activity
+
+ calls = []
+
+ async def start_run(database, **kwargs):
+ calls.append(("start", kwargs))
+ return SimpleNamespace(state="running")
+
+ async def finish_run(database, **kwargs):
+ calls.append(("finish", kwargs))
+ return None
+
+ monkeypatch.setattr(ledger, "start_run", start_run)
+ monkeypatch.setattr(ledger, "finish_run", finish_run)
+ monkeypatch.setattr(container_module, "container", SimpleNamespace(database=lambda: "db"))
+ assert await start_chat_run_activity({"run_id": "r", "temporal_workflow_id": "w", "temporal_run_id": "t"}) == {"claimed": True}
+ assert await finish_chat_run_activity({"run_id": "r", "temporal_workflow_id": "w", "temporal_run_id": "t", "success": True}) == {"state": None}
+ assert calls[0] == ("start", {"run_id": "r", "temporal_workflow_id": "w", "temporal_run_id": "t"})
+ assert calls[1][1]["success"] is True and calls[1][1]["error"] is None
+
+
+# ----- the SDK replay gate (run in a subprocess, like the run-record gate) -----
+
+TASK_QUEUE = "machina-chat-run-replay-gate"
+
+
+async def _replay_gate() -> None:
+ from temporalio import activity
+ from temporalio import workflow as temporal_workflow
+ from temporalio.api.enums.v1 import EventType
+ from temporalio.client import WorkflowHistory
+ from temporalio.testing import WorkflowEnvironment
+ from temporalio.worker import Replayer, Worker
+
+ from services.temporal.workflow import CHAT_RUN_PATCH, MachinaWorkflow
+
+ contexts: List[Dict[str, Any]] = []
+
+ @activity.defn(name="execute_node_activity")
+ async def execute_node(context: Dict[str, Any]) -> Dict[str, Any]:
+ contexts.append(context)
+ return {"success": True, "node_id": context["node_id"], "result": {"logged": True}}
+
+ @activity.defn(name="store_node_output_activity")
+ async def store_output(_payload: Dict[str, Any]) -> None:
+ return None
+
+ @activity.defn(name="workflow_control.pause_on_failure")
+ async def pause(_payload: Dict[str, Any]) -> Dict[str, Any]:
+ return {"paused": False}
+
+ @activity.defn(name="workflow_runs.record_completion")
+ async def record(_payload: Dict[str, Any]) -> Dict[str, Any]:
+ return {"recorded": True}
+
+ @activity.defn(name="chat_run.start")
+ async def start(_payload: Dict[str, Any]) -> Dict[str, Any]:
+ return {"claimed": True}
+
+ @activity.defn(name="chat_run.finish")
+ async def finish(_payload: Dict[str, Any]) -> Dict[str, Any]:
+ return {"state": "finished"}
+
+ real_patched = temporal_workflow.patched
+
+ def scheduled(history: WorkflowHistory) -> List[str]:
+ return [
+ event.activity_task_scheduled_event_attributes.activity_type.name
+ for event in history.events
+ if event.event_type == EventType.EVENT_TYPE_ACTIVITY_TASK_SCHEDULED
+ ]
+
+ async def run_once(environment, *, patch_open: bool) -> WorkflowHistory:
+ if not patch_open:
+ temporal_workflow.patched = lambda patch_id: False if patch_id == CHAT_RUN_PATCH else real_patched(patch_id)
+ try:
+ handle = await environment.client.start_workflow(MachinaWorkflow.run, _graph(), id=f"machina-cr-{uuid4()}", task_queue=TASK_QUEUE)
+ result = await handle.result()
+ assert result["success"] is True
+ return await handle.fetch_history()
+ finally:
+ temporal_workflow.patched = real_patched
+
+ async with await WorkflowEnvironment.start_time_skipping() as environment:
+ async with Worker(
+ environment.client,
+ task_queue=TASK_QUEUE,
+ workflows=[MachinaWorkflow],
+ activities=[execute_node, store_output, pause, record, start, finish],
+ ):
+ before = await run_once(environment, patch_open=False)
+ assert all("run_scope" not in context for context in contexts)
+ contexts.clear()
+ after = await run_once(environment, patch_open=True)
+ assert [context.get("run_scope") for context in contexts] == [{"run_id": "r_1", "session_id": "wf-cr"}]
+
+ assert "chat_run.start" not in scheduled(before) and "chat_run.finish" not in scheduled(before)
+ assert "chat_run.start" in scheduled(after) and "chat_run.finish" in scheduled(after)
+
+ replayer = Replayer(workflows=[MachinaWorkflow])
+ for history in (before, after):
+ captured = WorkflowHistory.from_json(history.workflow_id, history.to_json())
+ replay = await replayer.replay_workflow(captured, raise_on_replay_failure=False)
+ assert replay.replay_failure is None, replay.replay_failure
+
+ # Forcing only this patch open on the old history must fail replay: the
+ # gate is what keeps old histories working. Every other patch stays real.
+ temporal_workflow.patched = lambda patch_id: True if patch_id == CHAT_RUN_PATCH else real_patched(patch_id)
+ try:
+ captured = WorkflowHistory.from_json(before.workflow_id, before.to_json())
+ replay = await Replayer(workflows=[MachinaWorkflow]).replay_workflow(captured, raise_on_replay_failure=False)
+ assert replay.replay_failure is not None
+ finally:
+ temporal_workflow.patched = real_patched
+
+
+def test_histories_before_and_after_the_patch_replay() -> None:
+ """Run the SDK gate in a clean process with valid Windows I/O handles."""
+ completed = subprocess.run(
+ [sys.executable, "-m", "tests.temporal.test_machina_chat_run"],
+ cwd=Path(__file__).parents[2],
+ stdin=subprocess.DEVNULL,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ timeout=180,
+ check=False,
+ )
+ assert completed.returncode == 0, completed.stdout.decode(errors="replace")[-3000:]
+
+
+if __name__ == "__main__":
+ asyncio.run(_replay_gate())
diff --git a/server/tests/test_chat_trigger_canary_producer.py b/server/tests/test_chat_trigger_canary_producer.py
index 7c9b6d90..83aab996 100644
--- a/server/tests/test_chat_trigger_canary_producer.py
+++ b/server/tests/test_chat_trigger_canary_producer.py
@@ -1,16 +1,15 @@
-"""chatTrigger producer canary-emit invariant.
-
-Locks the contract: ``nodes.trigger.chat_trigger._events.dispatch_chat_message_received``
-routes through the canary CloudEvents path
-(:func:`services.events.dispatch.emit`) ONLY. The legacy
-``event_waiter.dispatch`` path was removed in Wave 13 — chatTrigger is
-canary-registered, the deployment manager skips ``setup_event_trigger``,
-and the legacy collector has zero consumers in production.
-
-Same regex-introspection invariant style as
-``tests/test_credential_broadcasts.py`` — source-level assertions catch
-the wire contract drifting without paying the cost of standing up
-Temporal in CI.
+"""The owner's chat message reaches chatTrigger listeners through the canary
+path only.
+
+Locks the contract: ``services.chat.events.dispatch_chat_message_received``
+routes through :func:`services.events.dispatch.emit` and never through the
+legacy ``event_waiter.dispatch`` (removed in Wave 13: chatTrigger is
+canary-registered and the legacy collector has no consumers). The envelope
+carries the owner's text, so it is never broadcast to every socket.
+
+Source-level assertions, in the style of
+``tests/test_credential_broadcasts.py``, catch the wire contract drifting
+without standing up Temporal.
"""
from __future__ import annotations
@@ -38,41 +37,26 @@ _EVENT_WAITER_DISPATCH_PATTERN = re.compile(r"event_waiter\.dispatch\s*\(")
_EVENTS_EMIT_PATTERN = re.compile(r"\bemit\s*\(")
-class TestChatTriggerProducerCanaryEmit:
- """Producer wrapper emits via the canary CloudEvents path only."""
+class TestChatMessageProducerCanaryEmit:
+ """The producer emits via the canary CloudEvents path only."""
def test_dispatcher_is_async(self):
- from nodes.trigger.chat_trigger._events import dispatch_chat_message_received
+ from services.chat.events import dispatch_chat_message_received
- assert inspect.iscoroutinefunction(dispatch_chat_message_received), (
- "dispatch_chat_message_received must be async — it awaits " "services.events.dispatch.emit."
- )
+ assert inspect.iscoroutinefunction(dispatch_chat_message_received)
def test_dispatcher_uses_canary_path_only(self):
- from nodes.trigger.chat_trigger import _events
-
- src = inspect.getsource(_events.dispatch_chat_message_received)
+ from services.chat import events
- assert _EVENTS_EMIT_PATTERN.search(src), (
- "dispatch_chat_message_received must call "
- "services.events.dispatch.emit(envelope, ...) — the canary "
- "CloudEvents path Signals running TriggerListenerWorkflow "
- "consumers AND broadcasts to FE on the chat_message_received "
- "wire key."
- )
+ src = inspect.getsource(events.dispatch_chat_message_received)
+ assert _EVENTS_EMIT_PATTERN.search(src), "must call services.events.dispatch.emit(envelope, ...)"
assert not _EVENT_WAITER_DISPATCH_PATTERN.search(src), (
- "dispatch_chat_message_received must NOT call "
- "event_waiter.dispatch — chatTrigger is canary-registered, "
- "the legacy collector path has zero consumers, and that "
- "call was removed in Wave 13. Reintroducing it would "
- "double-dispatch through dead infrastructure."
+ "must NOT call event_waiter.dispatch: chatTrigger is canary-registered and the legacy collector has no consumers"
)
@pytest.mark.asyncio
async def test_runtime_emits_canary_envelope(self, monkeypatch):
- """Invoking the dispatcher calls dispatch.emit with the right
- envelope. The legacy event_waiter is not touched."""
- from nodes.trigger.chat_trigger import _events
+ from services.chat import events
from services.events import dispatch as dispatch_mod
emit_calls: List[Any] = []
@@ -83,36 +67,30 @@ class TestChatTriggerProducerCanaryEmit:
monkeypatch.setattr(dispatch_mod, "emit", fake_emit)
- result = await _events.dispatch_chat_message_received(
- {
- "message": "hello",
- "session_id": "sess-1",
- "timestamp": "2026-05-14T00:00:00",
- }
+ result = await events.dispatch_chat_message_received(
+ {"message": "hello", "session_id": "sess-1", "timestamp": "2026-05-14T00:00:00"}
)
- # No return value — canary-only emit doesn't carry a waiter count.
assert result is None
-
- assert len(emit_calls) == 1
- event = emit_calls[0]["event"]
+ [call] = emit_calls
+ event = call["event"]
assert event.type == "com.opencompany.chat.message.received"
+ assert event.source == "opencompany://services/chat"
assert event.subject == "sess-1"
- assert emit_calls[0]["wire_routing_key"] == "chat_message_received"
+ assert call["wire_routing_key"] == "chat_message_received"
# The envelope carries the owner's text; it reaches the signalled
# workflows only, never every connected socket.
- assert emit_calls[0]["broadcast"] is False
+ assert call["broadcast"] is False
# No scope passed -> unscoped envelope (broadcast semantics).
assert event.workflow_id is None
@pytest.mark.asyncio
- async def test_workflow_scope_rides_the_envelope_verbatim(self, monkeypatch):
- """The factory plumbs ``workflow_id`` onto the envelope without
- any decision logic — the scoping RULE lives at the core call
- site (routers/websocket.py) and the scoped-delivery narrowing in
- core dispatch. Without the scope, one workflow's chat message
- fired every deployed workflow's chatTrigger."""
- from nodes.trigger.chat_trigger import _events
+ async def test_scope_and_run_id_ride_the_envelope_verbatim(self, monkeypatch):
+ """The factory plumbs ``workflow_id`` and the event id without any
+ decision logic: the scoping rule lives in the send handler and the
+ narrowing in core dispatch. The event id is the run id, so the run
+ a listener spawns has a predictable id."""
+ from services.chat import events
from services.events import dispatch as dispatch_mod
emit_calls: List[Any] = []
@@ -123,27 +101,39 @@ class TestChatTriggerProducerCanaryEmit:
monkeypatch.setattr(dispatch_mod, "emit", fake_emit)
- await _events.dispatch_chat_message_received(
- {"message": "hi", "session_id": "wf-1", "timestamp": "t"},
+ await events.dispatch_chat_message_received(
+ {"message": "hi", "session_id": "wf-1", "timestamp": "t", "run_id": "r_1"},
workflow_id="wf-1",
+ event_id="r_1",
)
- assert emit_calls[0]["event"].workflow_id == "wf-1"
+ event = emit_calls[0]["event"]
+ assert event.workflow_id == "wf-1"
+ assert event.id == "r_1"
+ assert event.data["run_id"] == "r_1"
- def test_router_decides_the_scope_not_the_plugin(self):
- """The chat WS handler (core) derives the workflow scope from the
- session and the plugin factory carries it verbatim — execution
- logic stays out of the node folder."""
- from nodes.trigger.chat_trigger import _events
- from routers import websocket as ws_router
+ def test_the_send_handler_decides_the_scope_not_the_factory(self):
+ from services.chat import access, events, handlers
- factory_src = inspect.getsource(_events.chat_message_received)
- assert '"default"' not in factory_src, (
- "The plugin factory must not embed the session!='default' "
- "scoping decision — that rule lives in "
- "routers/websocket.py:handle_send_chat_message."
- )
+ factory_src = inspect.getsource(events.chat_message_received)
+ assert '"default"' not in factory_src, "the factory must not embed the session != 'default' scoping rule"
+
+ handler_src = inspect.getsource(handlers.handle_send_chat_message)
+ assert "workflow_id=scope.workflow_id" in handler_src
+ assert "session_id == DEFAULT_SESSION" in inspect.getsource(access.authorize_session)
+
+ def test_the_workflow_trusts_the_same_source_and_type(self):
+ """MachinaWorkflow reads a chat run id only from an event this
+ module produced; the two must agree."""
+ from services.chat import events
+ from services.temporal import workflow
+
+ assert workflow.CHAT_MESSAGE_SOURCE == events.SOURCE
+ assert workflow.CHAT_MESSAGE_TYPE == events.MESSAGE_RECEIVED_TYPE
+
+ def test_the_trigger_listens_on_the_same_type(self):
+ import nodes # noqa: F401 - registers the trigger
+ from services.chat import events
+ from services.deployment.canary_registry import cloudevent_type_for
- handler_src = inspect.getsource(ws_router.handle_send_chat_message)
- assert "workflow_id=workflow_scope" in handler_src
- assert 'session_id != "default"' in handler_src
+ assert cloudevent_type_for("chatTrigger") == events.MESSAGE_RECEIVED_TYPE