Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
feat(chat): track each owner message as a run the employee answers
trohitgb940cae55beb
diff --git a/CLAUDE.md b/CLAUDE.mdindex 9654d355..50e12f00 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -13,7 +13,8 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire| **[UI Migration Plan](./docs-internal/ARCHIVE/ui_migration_plan.md)** | *Archived.* antd → shadcn/ui migration completion log (Waves 1–10). Current state lives in [Frontend Architecture](./docs-internal/frontend_architecture.md) and [Theme System](./docs-internal/theme_system.md). || **[Temporal Cleanup & Resilience Plan (Waves 15-18)](./docs-internal/ARCHIVE/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md)** | *Archived — all four waves shipped.* 15 dead-code retirement (Redis-Streams branch in event_waiter, APScheduler cron stack, orphaned `connection_status` factory), 16 per-queue task routing (`TemporalWorkerPool`), 17 resilience hardening (cron `catchup_window`, SDK interceptors, `DEPLOYMENT_MODE`, periodic heartbeats; the one-shot LLM-step retry it records was later reverted to unlimited-with-backoff), 18 worker performance tuning. Current settings live in [Temporal Architecture](./docs-internal/TEMPORAL_ARCHITECTURE.md). || **[Node Allowlist](./docs-internal/node_allowlist.md)** | Single-config UI visibility — `server/config/node_allowlist.json` controls which nodes / credential categories / skill folders show in the UI. Five lists with two enforcement tiers (mode-gated allowlist + absolute blocklist). `useNodeAllowlist` hook exposes `isVisible` / `isBlocked` / `isAllowed` / `isCredentialCategoryDisabled` / `isSkillFolderDisabled`. Adding a new disabled domain = single JSON edit, no code change. |-| **[Normal Mode (Home)](./docs-internal/normal_mode.md)** | The owner-facing landing screen: describe a job, an LLM on the server drafts a setup screen (a JSON UI spec the client parses, normalises and renders from a fixed catalogue), and Hire builds, validates, saves and starts one workflow per AI employee (`services/employees/`, graph built from `config/employee_apps.json` plus the owner's skill library and a Canvas node, and checked against `node_allowlist.json` `enabled_nodes`). Covers the screen switch (`app/`, `shellMode` / `ui_shell_mode`, `VITE_NORMAL_MODE=false` kill switch, Home shows light/dark only), the employee page (only their name above the conversation; Pause is in the Workspace header), deleting an employee from the team sidebar (stopped through Reset first, then gone from every tab), the `approvalGate` "ask me before sending" step, run records for "done today" (`machina-run-record-v1`), automatic-pause reasons, Settings (Profile fields; usage-only Billing via `get_employee_usage`; Skills, the user-skills library each new hire copies onto a Skills node; consumer Connectors with `publisher` / `verified`; Plugins, the starter bundles in `hire/starters.json`; the last three share one catalog page), the Workspace dock (an employee's Canvas board, named by the summary's `canvas_node_id`; a live Browser tab with user takeover, shared with Dev mode; Mobile shows the managed local Android phone), the "Web browser" app (read-only under "ask me first" through the tool's `ask_first_params`; while the agent waits in `request_user` the summary's `browser_request` shows Needs you and Help in browser), Talk on the employee page (a thread over the chat session whose id is the workflow id, rows written only through `services/chat_thread.py`; the talk line is chatTrigger → an agent → `chatReply` "Reply in Chat": a chat hire's own agent is its talk agent, other hires get a "Talk with {name}" agent sharing the worker's tools and skills with its own Context, and schedule hires also "Post to Talk"), Turn on Talk (for talk `off`) and Apply (while the summary's `pending_changes` is true) via `enable_employee_talk` / `apply_employee_changes`, each a restart on the saved graph through `services/deployment/restart.py`, self-extension from Talk through the Agent Builder within the Hire rule (`services/employees/policy.py`), the 3D orb, the wire contract and the `employee_lifecycle` / `approval_lifecycle` / `chat.updated` broadcasts. |+| **[Normal Mode (Home)](./docs-internal/normal_mode.md)** | The owner-facing landing screen: describe a job, an LLM on the server drafts a setup screen (a JSON UI spec the client parses, normalises and renders from a fixed catalogue), and Hire builds, validates, saves and starts one workflow per AI employee (`services/employees/`, graph built from `config/employee_apps.json` plus the owner's skill library and a Canvas node, and checked against `node_allowlist.json` `enabled_nodes`). Covers the screen switch (`app/`, `shellMode` / `ui_shell_mode`, `VITE_NORMAL_MODE=false` kill switch, Home shows light/dark only), the employee page (only their name above the conversation; Pause is in the Workspace header), deleting an employee from the team sidebar (stopped through Reset first, then gone from every tab), the `approvalGate` "ask me before sending" step, run records for "done today" (`machina-run-record-v1`), automatic-pause reasons, Settings (Profile fields; usage-only Billing via `get_employee_usage`; Skills, the user-skills library each new hire copies onto a Skills node; consumer Connectors with `publisher` / `verified`; Plugins, the starter bundles in `hire/starters.json`; the last three share one catalog page), the Workspace dock (an employee's Canvas board, named by the summary's `canvas_node_id`; a live Browser tab with user takeover, shared with Dev mode; Mobile shows the managed local Android phone), the "Web browser" app (read-only under "ask me first" through the tool's `ask_first_params`; while the agent waits in `request_user` the summary's `browser_request` shows Needs you and Help in browser), Talk on the employee page (a thread over the chat session whose id is the workflow id, rows written through `services/chat_thread.py` and, for the owner's messages and their chat runs, `services/chat/ledger.py` (see [Chat Protocol](./docs-internal/chat_protocol.md)); the talk line is chatTrigger → an agent → `chatReply` "Reply in Chat": a chat hire's own agent is its talk agent, other hires get a "Talk with {name}" agent sharing the worker's tools and skills with its own Context, and schedule hires also "Post to Talk"), Turn on Talk (for talk `off`) and Apply (while the summary's `pending_changes` is true) via `enable_employee_talk` / `apply_employee_changes`, each a restart on the saved graph through `services/deployment/restart.py`, self-extension from Talk through the Agent Builder within the Hire rule (`services/employees/policy.py`), the 3D orb, the wire contract and the `employee_lifecycle` / `approval_lifecycle` / `chat.updated` broadcasts. |+| **[Chat Protocol](./docs-internal/chat_protocol.md)** | The wire contract between the chat runtime (`server/services/chat/`) and the shared chat UI: AG-UI's event model on the existing WebSocket, snake_case, in the CloudEvents envelope. A message a deployed chat trigger will answer starts a **chat run** admitted with it in one reserved write transaction (`Database.reserved_session`, SQLite `BEGIN IMMEDIATE`): one live run per session (the lane, also a partial unique index; a second send answers `run_in_progress`), idempotent `client_message_id`, claimed and finished by the MachinaWorkflow that answers it behind `machina-chat-run-v1` (the run id is trusted only from an event whose source is `opencompany://services/chat`; `run_scope` rides every node context; only the claimant finishes), the reply saved as `a_<run id>` by `chatReply`, every message appended after the session's active leaf (`chat_threads`) so concurrent writes never fork the thread. `chat_run_event` frames reach only sockets that passed `chat_subscribe`'s owner check (`services/chat/hub.py`: per-run `seq`, `hub_epoch`, bounded per-socket queues with a resync), never the status broadcast. A watchdog (`services/chat/watchdog.py`, started by `main.py`) ends runs nothing will finish (`not_delivered`, `timed_out`, `interrupted`). Lists every event, handler, part and error code, including the ones later phases build. || **[Theme System](./docs-internal/theme_system.md)** | 12-way visual theme system — 2 base (light, dark) + 5 utopian (renaissance, greek, edo, steampunk, atomic) + 5 dystopian (cyber, wasteland, rot, plague, surveillance) — driven by `<html data-theme>` + per-theme CSS files in `client/src/themes/`. Token taxonomy (surface / fg / border / accent / typography / motion) in **hex + `color-mix()`**; `@theme inline` bridge maps `--color-X: var(--X)` (no `hsl()` wrapper); per-theme files own the colour hex, base.css owns the shared `--tint-*` alpha scale; action role tokens (incl. `-ink` readable text), **per-theme `--code-*` syntax tokens** (tier 6 — the code editor, console/output JSON viewers, and chat code blocks paint in each theme's own palette: keyword→trigger, string→success, number→agent, function→model, on an adaptive dark/light code surface; replaced the global dracula `--prism-*` block + dead `getPrismTokenCSS`; the `OutputPanel` JSON viewer reads the same vars), per-theme `--pulse-keyframe` animation system + `.opencompany-*` helpers in `animations.css`, decorative-layer wrappers (`.app-frame` / `.canvas-host` / `.modal-frame`), per-component decorative ornaments (panel textures + canvas decorations + node pseudo-element overlays + theme-specific keyframes), **canvas-node visual contract via `--node-color` CSS custom property** (no inline `background` / `border` on node components — base.css + per-theme CSS owns visuals; `NodeStyle` helper type at [types/NodeTypes.ts](./client/src/types/NodeTypes.ts) makes the inline custom-prop typecheck-clean), **`--node-pulse-color` separate from `--node-color`** so executing-node glow uses each theme's highest-contrast accent regardless of plugin accent (Cyber neon cyan, Surveillance REC red, Renaissance ultramarine, etc.), **`data-page-hidden` animation pause** (set by `app/usePageActivitySync.ts`, mounted in `AppShell` on both screens, while the tab is hidden or the window is blurred; base.css declares `html[data-page-hidden] *, *::before, *::after { animation-play-state: paused !important }` to prevent compositor stall on tab return), **per-theme icon glyph system** (290 inline SVG glyph strings across 29 keys × 10 themes via [themedGlyphs.ts](./client/src/assets/icons/themedGlyphs.ts) + theme-aware [NodeIcon.tsx](./client/src/assets/icons/NodeIcon.tsx)), **per-theme canvas-grid + custom cursors** via `--canvas-grid` / `--cursor-default` slots, **decorative HTML primitives** (`<SvgFilterDefs>` mounting `#ink-blot` / `#noise` / `#crt` filter IDs at app root, `<DropCap>` wrapper for Renaissance ornament rule), **parameter panel migrated to Tailwind tokens** (no `useAppTheme()` reads; section headers carry the display-typography triplet; raw `<Button>` swapped to `<ActionButton intent>`), **per-theme scrollbar webkit rules** in all 12 themes, 9-event WebAudio sound system (10 packs via `--sound-pack` token + `useSound()` hook + global hover delegate + sonner toast monkey-patch + `withSound()` HOC + `Sounds.unlock()` gesture-unlock for AudioContext autoplay-policy compliance), `@media (prefers-reduced-motion: reduce)` accessibility, 30 ms throttle on `type` / `hover`, migration recipe, anti-patterns. Read this before adding a new theme, migrating a component to the new contract, or adding a canvas-node component. || **[Design System Bundle](./docs-internal/design-system/IMPLEMENTATION.md)** | Canonical, vendored design-system reference (extracted from this repo; see its `IMPLEMENTATION.md` for the contract). Tokens are **hex + `color-mix()`** (`tokens/{colors,typography,spacing,motion,fonts,base}.css`) — the format the live codebase standardizes on. Includes 34 reference components (`components/`, inline-style reference only — recreate in Tailwind/shadcn idioms), full-app UI kit (`ui_kits/opencompany/index.html`), 12-theme docs (`guidelines/THEMES.md`), **the merged handoff brief (`HANDOFF.md` — glow-ladder traps, step-edge contract, toolbar layout traps, canvas-layer asymmetries, plus the recorded product amendments)**, and **the panel×theme fidelity target (`reference-mockup/Panel Theme Matrix.dc.html` — 17 panels × 12 themes, open in a browser)**. `reference/themes/` no longer vendors CSS snapshots — it points at the authoritative [client/src/themes/](./client/src/themes/). Copy token values verbatim; do not re-derive by eye. Pairs with [Theme System](./docs-internal/theme_system.md). || **[Schema Source of Truth RFC](./docs-internal/ARCHIVE/schema_source_of_truth_rfc.md)** | *Archived — shipped.* Backend is SSOT for node schemas, visual metadata, handlers, palette metadata, icons; `VITE_NODESPEC_BACKEND` defaults on and the frontend node definitions are gone. Plugin pattern: one `BaseNode` subclass in `server/nodes/<group>/<plugin>/__init__.py`. Wire format: `asset:<key>` / `<lib>:<brand>` / URL / emoji. Endpoint: `/api/schemas/nodes/{type}/spec.json`. Current reference: [Plugin System](./docs-internal/plugin_system.md). |@@ -991,7 +992,7 @@ const [selectedConsoleId, setSelectedConsoleId] = useState<string>('');```#### Chat Message Persistence-Chat messages are persisted to SQLite database and survive server restarts. A workflow's session id is its workflow id (`"default"` with no workflow open), and Home shows the same thread as Talk on the employee's page. Every row is written through [`services/chat_thread.py`](./server/services/chat_thread.py) (`record_chat_message` / `clear_chat_thread`), which stamps the session's live generation and broadcasts the identity-only `chat.updated`; agents answer into the thread through the `chatReply` ("Reply in Chat") node. The thread lives as long as the generation: a workflow Reset (every restart, Home's Apply and Turn on Talk included) clears it through the `chatTrigger` / `chatReply` `reset_execution_state` hook, in the same Reset in which the Context node forgets the conversation, and deleting the workflow deletes it (a workflow-deleted hook the `chatReply` plugin registers).+Chat messages are persisted to SQLite database and survive server restarts. A workflow's session id is its workflow id (`"default"` with no workflow open), and Home shows the same thread as Talk on the employee's page. Rows are written through [`services/chat_thread.py`](./server/services/chat_thread.py) (`record_chat_message` / `clear_chat_thread`) and, for the owner's messages and the chat runs they start, [`services/chat/ledger.py`](./server/services/chat/ledger.py); both stamp the session's live generation, append the row after the session's active leaf (one chain, `chat_threads`), and broadcast the identity-only `chat.updated`; agents answer into the thread through the `chatReply` ("Reply in Chat") node, as the run's reply when a chat run started it. Runs, their events and the wire contract are in [Chat Protocol](./docs-internal/chat_protocol.md). The thread lives as long as the generation: a workflow Reset (every restart, Home's Apply and Turn on Talk included) clears it through the `chatTrigger` / `chatReply` `reset_execution_state` hook, in the same Reset in which the Context node forgets the conversation, and deleting the workflow deletes it (a workflow-deleted hook the `chatReply` plugin registers).**Database Model** (`server/models/database.py`):```python@@ -1002,21 +1003,32 @@ class ChatMessage(SQLModel, table=True):execution_id: Optional[str] # the generation (root_execution_id) it was written inrole: str = Field(max_length=20) # 'user' or 'assistant'message: str = Field(max_length=50000)+ uid: Optional[str] # stable wire id: m_... (owner), a_<run id> (a run's reply); unique+ parent_uid: Optional[str] # the message it follows (branches link here)+ run_id: Optional[str] # the chat run it started or answers (models/chat.py)+ kind: str # text / report / action / notice+ status: str # complete / stopped / error+ parts: Dict; attachments: list; meta: Dict # JSONcreated_at: datetime```-**WebSocket Handlers** (`server/routers/websocket.py`):+**WebSocket Handlers** (`server/services/chat/handlers.py`, registered by `services/chat/__init__.py`):| Handler | Description ||---------|-------------|| `send_chat_message` | Save the message and dispatch it to the workflow's chatTriggers. A workflow session answers `delivery: "now"` (running / starting / resuming) or `"queued"` (paused / pausing, runs on Resume); in any other state `not_running`, saving nothing || `get_chat_messages` | A session's messages, oldest first: `{id, role, message, timestamp, run_key}` (UTC-offset timestamps; `run_key` = the generation). The latest generation only (none after a Reset), unless `all_generations` (Home's thread; a Reset clears the thread, so only a row from before a Start shows beside the live ones) || `clear_chat_messages` | Clear a session's chat, every generation; for a workflow's session its agents forget the conversation too (`chat_thread.clear_chat_session` tells the listeners registered with `register_chat_cleared_listener`: the Context plugin clears the workflow's conversations) || `save_chat_message` | Save one message with the given role |+| `chat_subscribe` / `chat_unsubscribe` | Start / stop receiving a session's `chat_run_event` frames; subscribe answers the live runs' snapshots |+| `get_chat_run` | One run's snapshot |++Every chat command refuses the internal worker socket, and a workflow's session answers only the workflow's owner (`services/chat/access.py`).**Database Methods** (`server/core/database.py`):-- `add_chat_message(session_id, role, message, execution_id=None)` - Add message to database-- `get_chat_messages(session_id, limit=None, execution_id=None)` - Newest `limit` messages, oldest first, optionally of one generation-- `clear_chat_messages(session_id)` - Delete all messages for session+- `add_chat_message(session_id, role, message, execution_id=None, *, uid=None, run_id=None, ...)` - Append a message after the session's active leaf; returns the row (a `uid` already saved returns that row)+- `append_chat_row(session, ...)` - The same, inside a caller's `reserved_session()` (the run ledger admits a message and its run in one transaction)+- `get_chat_messages(session_id, limit=None, execution_id=None)` - Newest `limit` messages, oldest first, optionally of one generation (`read_chat_messages` raises instead of returning `[]` on failure)+- `clear_chat_messages(session_id)` - Delete all messages for session, with its runs, run parts and thread state#### Console Log PersistenceConsole logs are persisted to SQLite database and loaded on page refresh.@@ -1038,7 +1050,8 @@ Console logs are persisted to SQLite database and loaded on page refresh.| `client/src/components/ui/ConsolePanel.tsx` | Main panel component with chat/console/terminal tabs || `server/models/database.py` | ChatMessage and ConsoleLog SQLModel definitions || `server/core/database.py` | Chat message and console log CRUD methods |-| `server/routers/websocket.py` | WebSocket handlers for chat and console operations |+| `server/routers/websocket.py` | WebSocket handlers for console operations |+| `server/services/chat/` | Chat runs, run events and the chat WebSocket handlers ([Chat Protocol](./docs-internal/chat_protocol.md)) |### Per-Workflow Workspace DirectoryEach workflow execution gets a persistent workspace directory where nodes save output files and agents such as Coding Agent access them through connected filesystem tools.@@ -2503,7 +2516,7 @@ Updated `client_left` and presence handlers use `has_real_android_devices()` ins| **WhatsApp** | `whatsapp_status`, `whatsapp_connected_phone`, `whatsapp_qr`, `whatsapp_send`, `whatsapp_start`, `whatsapp_restart`, `whatsapp_groups`, `whatsapp_group_info`, `whatsapp_chat_history`, `whatsapp_newsletters`, `whatsapp_rate_limit_get`, `whatsapp_rate_limit_set`, `whatsapp_rate_limit_stats`, `whatsapp_rate_limit_unpause`, `whatsapp_mark_read`, `whatsapp_typing`, `whatsapp_presence`, `whatsapp_stop`, `whatsapp_diagnostics` || **Telegram** | `telegram_connect`, `telegram_disconnect`, `telegram_status`, `telegram_send`, `telegram_reconnect`, `telegram_get_me`, `telegram_get_chat` || **Workflow Storage** | `save_workflow`, `get_workflow`, `get_all_workflows`, `delete_workflow` |-| **Chat Messages** | `send_chat_message`, `get_chat_messages`, `clear_chat_messages`, `save_chat_message` |+| **Chat Messages** | `send_chat_message`, `get_chat_messages`, `clear_chat_messages`, `save_chat_message`, `chat_subscribe`, `chat_unsubscribe`, `get_chat_run` (see [chat_protocol.md](./docs-internal/chat_protocol.md)) || **Console/Terminal** | `get_console_logs`, `clear_console_logs`, `get_terminal_logs`, `clear_terminal_logs` || **User Skills** | `get_user_skills`, `get_user_skill`, `create_user_skill`, `update_user_skill`, `delete_user_skill` || **Built-in Skills** | `get_skill_content`, `save_skill_content`, `scan_skill_folder`, `list_skill_folders` |@@ -2528,6 +2541,7 @@ Updated `client_left` and presence handlers use `has_real_android_devices()` ins| `employee_lifecycle` | CloudEvents v1.0 envelope (type=`com.opencompany.employee.{hired,updated,removed}`, subject = workflow id) — Home's team list; `updated` is coalesced to one per second per employee, control changes go out at once || `approval_lifecycle` | CloudEvents v1.0 envelope (type=`com.opencompany.approval.{requested,decided,expired,cancelled}`) — identity only, never the draft or the recipient; the card refetches via `list_approvals` || `chat.updated` | CloudEvents v1.0 envelope (type=`com.opencompany.chat.updated`, data `{workflow_id, session_id, role}`) — sent directly (not `dispatch.emit`) by `services/chat_thread.py` after every chat insert and clear; identity only, Home's thread and the editor's chat pane refetch via `get_chat_messages` |+| `chat_run_event` | CloudEvents v1.0 envelope (type=`com.opencompany.chat.run.<suffix>`, id `<run id>:<seq>`, subject = run id) — AG-UI run events, sent by `services/chat/hub.py` only to sockets subscribed to the session through `chat_subscribe`, never on the broadcast; see [chat_protocol.md](./docs-internal/chat_protocol.md) || `workflow_lifecycle` | CloudEvents v1.0 envelope for workflow stages (`deployment.*`, `lock.*`, `execution.*`, `imported`, `renamed`, `created`, `deleted`) — other tabs invalidate their workflow lists || `deployment_snapshot` | CloudEvents v1.0 envelope (type=`workflow.deployment.snapshot`) — pushed once per WS connect from `broadcaster._send_deployment_snapshot`; lets the FE reconcile stale `deploymentStatus.isRunning=true` after a backend restart wiped `DeploymentManager._deployments`. Empty list is meaningful (forces reset). || `variable_update` | Single variable value change |@@ -2788,7 +2802,7 @@ This function:- **Typed SDK error → `LLMError` → `NodeUserError`**: Native providers normalize SDK failures into structured `LLMError` values carrying category, retryability, HTTP status, provider code, request ID, and retry-after metadata. `ChatUnifier` translates them at the execution boundary into a user-safe `NodeUserError`; `BaseNode.execute()` logs that at WARN with one line and no stack trace. Unexpected exceptions retain the full-traceback path. In-process agent turns retry only errors marked retryable, while Temporal owns activity retries and disables the inner retry loop.- **Plugin extraction (Wave 11.I)**: every plugin's WS handlers, OAuth client, FastAPI router, and lifecycle service live entirely under `nodes/<plugin>/`. Eight plugin domains migrated this round (whatsapp / twitter / google-workspace / android / browser / email / code / credential-validation-scaffold for maps+apify+ollama+lmstudio) follow the telegram pattern. `routers/websocket.py` shrunk from ~3,785 to ~2,977 LOC (-808). Three plugin routers (`routers/twitter.py`, `routers/google.py`, `routers/android.py`) moved into `nodes/<plugin>/_router.py` and mount via the plugin-router loop in `main.py`; the explicit `app.include_router(<plugin>.router)` calls and `from routers import <plugin>` imports in `main.py` are gone. **`register_router(router, name=...)` is the new sibling helper to `register_ws_handlers` (same file: `services/ws_handler_registry.py`)** — the generic registry set has grown since (the Plugin System row at the top of this file gives the `grep` for the live list). **`Credential.validate(data) -> dict` + `Credential._probe(api_key) -> ProbeResult` is the new shared validator scaffold** in `services/plugin/credential.py` — replaces the per-router `_SPECIAL_PROVIDER_VALIDATORS` dict; one `_LLMApiKey._probe` method serves all 10 cloud LLM providers, dedicated `_probe` overrides handle Maps + Apify, `_LocalLLM.validate` overrides for Ollama / LM Studio's 2-storage edge case. **`tests/test_plugin_self_containment.py` locks the contract** with 10 invariant classes (forbidden-imports / no-router-outside-nodes / per-plugin self-registration / registry-API sanity / stale-paths-absent / main.py-does-not-mount / WS_HANDLERS-non-empty / plugin-folder-has-node-file / typed-event-factories / package-imports-cleanly); same `inspect.getsource` introspection style as `test_credential_broadcasts.py`. Wire format unchanged — frontend WS message-type strings (`whatsapp_status`, `twitter_oauth_login`, `google_oauth_status`, `android_relay_connect`, etc.) and HTTP route paths (`/api/twitter/callback`, `/api/google/callback`, `/api/android/*`) are byte-identical post-migration.- **Typed CloudEvents factory pattern for new broadcasts** (b6aecd3): every new server→FE lifecycle broadcast follows the recipe locked by `broadcast_credential_event`, `broadcast_agent_progress`, and `broadcast_agent_capability`. (1) Add a typed factory — convention: `source = "opencompany://services/<area>"`, reverse-DNS `type = "com.opencompany.<area>.<event>"`, and `subject = <exact primary entity id>`. Put snake_case workflow/execution scope inside `data`; CloudEvents extension attribute names MUST be lowercase-alphanumeric, so the historical top-level `workflow_id`/`trigger_node_id`/`correlation_id` fields are compatibility debt and are forbidden on new contracts. (2) Add a broadcaster wrapper that emits `broadcast({type: "<wire_key>", data: event.model_dump(mode="json", exclude_none=True)})`. (3) FE validates `specversion`, ID/source/type, subject ownership, and event-specific payload invariants, deduplicates by `(source,id)`, then writes only the exact scoped entity. (4) Add contract tests in `test_events.py` and keep `tests/test_status_broadcasts.py`'s raw-callsite allowlist empty. Raw `node_status` remains allowed only as the established high-frequency latest-state/reconnect projection paired with a typed lifecycle occurrence; it is not the event of record.-- **Plugin-folder location for plugin-specific CloudEvents factories** (RFC §6.4): when a typed event is owned by one plugin (telegram, whatsapp, android, email, google, master_skill, webhook_trigger, chat_trigger, agent), the factory + `broadcast_<event>` wrapper live in [`server/nodes/<plugin>/_events.py`](./server/nodes), NOT on `WorkflowEvent` / `StatusBroadcaster` directly. Telegram's [`_events.py`](./server/nodes/telegram/_events.py) is the canonical template — `broadcast_telegram_status` constructs the envelope via a plain factory function and calls `get_status_broadcaster().broadcast(...)` itself. Latest example: [`server/nodes/skill/master_skill/_events.py`](./server/nodes/skill/master_skill/_events.py) (`broadcast_skill_lifecycle("created" \| "updated" \| "deleted" \| "content_saved", name=..., **data_extra)`) — replaces four raw-dict broadcasts in `handle_create_user_skill` / `handle_update_user_skill` / `handle_delete_user_skill` / `handle_save_skill_content` (the last one was previously silent). FE routes wire key `skill_lifecycle` to invalidate `userSkills` + `folderSkills` queries and drop the `skillContent` cache on delete + content_saved so the Master Skill panel refreshes live across every connected client. Cross-cutting events (credential, workflow_lifecycle, agent_progress, agent_capability, node_parameters_updated) keep their `WorkflowEvent` classmethods + `StatusBroadcaster` wrappers — only plugin-scoped events move to the plugin folder. A core service's own event keeps its factory beside it, so core never imports a plugin `_events.py`: `services/workflow_ops.py` (`workflow_ops_apply`) and `services/chat_thread.py` (`chat.updated`).+- **Plugin-folder location for plugin-specific CloudEvents factories** (RFC §6.4): when a typed event is owned by one plugin (telegram, whatsapp, android, email, google, master_skill, webhook_trigger, agent), the factory + `broadcast_<event>` wrapper live in [`server/nodes/<plugin>/_events.py`](./server/nodes), NOT on `WorkflowEvent` / `StatusBroadcaster` directly. Telegram's [`_events.py`](./server/nodes/telegram/_events.py) is the canonical template — `broadcast_telegram_status` constructs the envelope via a plain factory function and calls `get_status_broadcaster().broadcast(...)` itself. Latest example: [`server/nodes/skill/master_skill/_events.py`](./server/nodes/skill/master_skill/_events.py) (`broadcast_skill_lifecycle("created" \| "updated" \| "deleted" \| "content_saved", name=..., **data_extra)`) — replaces four raw-dict broadcasts in `handle_create_user_skill` / `handle_update_user_skill` / `handle_delete_user_skill` / `handle_save_skill_content` (the last one was previously silent). FE routes wire key `skill_lifecycle` to invalidate `userSkills` + `folderSkills` queries and drop the `skillContent` cache on delete + content_saved so the Master Skill panel refreshes live across every connected client. Cross-cutting events (credential, workflow_lifecycle, agent_progress, agent_capability, node_parameters_updated) keep their `WorkflowEvent` classmethods + `StatusBroadcaster` wrappers — only plugin-scoped events move to the plugin folder. A core service's own event keeps its factory beside it, so core never imports a plugin `_events.py`: `services/workflow_ops.py` (`workflow_ops_apply`), `services/chat_thread.py` (`chat.updated`) and `services/chat/events.py` (the owner's chat message to the chatTrigger listeners, which moved there from the plugin, and `chat_run_event`).- **Operator-metadata fields in credential panels** (f746ddf): when a credential panel needs a non-credential follow-up field (e.g. Telegram bot token + owner chat id), declare it as a second entry in the provider's `fields` array in `server/config/credential_providers.json`. The first field stays the validate/connect target; subsequent fields render below as plain text inputs via `SecondaryFieldRow` in `ApiKeyPanel.tsx` (shadcn `Input` + `Label` + `<ActionButton intent="save">`). Each field carries an optional `help` slot (`FieldDef.help` / `ServerFieldDef.help`) for always-visible explanatory text under the input. Save writes via `panel.actions.save(field.key, value)` → `auth_service.store_api_key` (which is permissive about provider string). The credential class must declare the new key in `extra_fields` (`Credential` subclass attribute) so the storage layer accepts it. Reference: `telegram_owner_chat_id` field.- **Telegram owner detection — three layers** (f746ddf): the bot owner is captured by THREE independent paths so the realistic setup flows all converge to a working state. (1) **Explicit field**: secondary `FieldDef` in the credentials modal lets the user paste their Telegram user ID directly (recommended; works without DM'ing the bot). (2) **Pre-poll peek** in `connect()` via `_capture_owner_from_pending_updates()` — calls `bot.get_updates(timeout=0)` BEFORE `start_polling(drop_pending_updates=True)` discards the queue, scans for any historical private DM, captures + persists atomically. (3) **Atomic write-through in `_on_message_received`** — persists FIRST, sets in-memory ONLY on success. Invariant: "in-memory has owner ⇒ DB has owner" so a process restart can re-capture cleanly. Failure logged at `ERROR` with `exc_info=True` (was `WARNING` previously, masking persist failures). The lazy fallback in `telegram_send.py` (read DB → `service.set_owner`) sits on top of all three.- **Supervisor Job Object loud-failure (cli/tree.py + cli/supervisor.py)** (8a64eb9): the Windows process-tree mechanism depends entirely on a Job Object with `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE` (no fallback — `start_new_session` is POSIX-only). `pywin32>=308` is a hard supervisor dependency on Windows; the declaration in the root `pyproject.toml` (where `opencompany-cli` lives — `cli/pyproject.toml` was removed as a stale duplicate) carries an inline comment so future maintainers don't drop it. `_JobObject.__init__` splits failures into explicit `ImportError` (with reinstall command) and `Exception` (with `repr`) branches — both write to `stderr` instead of swallowing. `_JobObject.add()` verifies enrollment via `win32job.IsProcessInJob(handle, self._handle)` after assignment; on mismatch it queries `IsProcessInJob(handle, None)` so the warning explicitly names the wrapper-job hypothesis (npm / bun / conhost wrappers occasionally place us in their own non-nesting Job). `supervisor._spawn_once` checks `add_to_job` return value and emits a yellow `WARN: pid=N not enrolled in Job Object` per child if False. Without these guards, a stale pywin32 install or a wrapping Job Object silently leaks orphan Python processes on every Ctrl-C and they accumulate to the point of holding SQLite locks that block subsequent backend startup.diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.mdindex 8e04868c..3c71dffa 100644--- a/docs-internal/TEMPORAL_ARCHITECTURE.md+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md@@ -704,7 +704,7 @@ The Temporal binary + persistence are managed in-process by the plugin-folder pa**Months-long durability contract**: running and paused deployments survive backend restarts, are never auto-terminated, and keep executing for months. None of the mechanisms below carries a `workflow.patched` marker: the lifetime-cap removal and the poll-interval floor apply unconditionally, including when an older history is replayed.-- **No lifetime caps on new child runs.** Trigger/cron-spawned `MachinaWorkflow` runs, agent children, and delegated-task runners previously carried 1-2h `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so any pause longer than the cap silently terminated the run (and a timed-out delegated runner skipped its compensation: leaked permit + stuck task row). New executions start children unbounded (no patch marker was retained for this change). The live `workflow.patched` markers are `machina-conditional-edges-v1`, `machina-run-record-v1` and `workspace-node-cancellation-v1` in `services/temporal/workflow.py`, `machina-trigger-listener-node-filter` in `services/temporal/trigger_listener_workflow.py`, and `workspace-task-activity-policy-v1` in `services/temporal/agent_workflow.py`; the list grows, so check it with `grep -rn "workflow.patched(" server/services/temporal`. `machina-run-record-v1` gates the `workflow_runs.record_completion` activity that trigger-spawned runs schedule when they finish, for Normal mode's "done today" (see [normal_mode.md](./normal_mode.md#done-today)). The two `workspace-*` markers are described under [Direct Workspace tasks](#direct-workspace-tasks). Liveness is the activity layer's job: node activities heartbeat every 30s against a 2-minute `heartbeat_timeout`; their `start_to_close` is a generous 24h ceiling, not 10 minutes. The subagent-permit wait uses `PERMIT_WAIT_RETRY` (unlimited attempts) so a queued delegation waits as long as admission takes instead of failing after ~3h.+- **No lifetime caps on new child runs.** Trigger/cron-spawned `MachinaWorkflow` runs, agent children, and delegated-task runners previously carried 1-2h `execution_timeout`/`run_timeout` — Temporal's timeout timers keep ticking through a cooperative pause, so any pause longer than the cap silently terminated the run (and a timed-out delegated runner skipped its compensation: leaked permit + stuck task row). New executions start children unbounded (no patch marker was retained for this change). The live `workflow.patched` markers are `machina-conditional-edges-v1`, `machina-run-record-v1`, `machina-chat-run-v1` and `workspace-node-cancellation-v1` in `services/temporal/workflow.py`, `machina-trigger-listener-node-filter` in `services/temporal/trigger_listener_workflow.py`, and `workspace-task-activity-policy-v1` in `services/temporal/agent_workflow.py`; the list grows, so check it with `grep -rn "workflow.patched(" server/services/temporal`. `machina-run-record-v1` gates the `workflow_runs.record_completion` activity that trigger-spawned runs schedule when they finish, for Normal mode's "done today" (see [normal_mode.md](./normal_mode.md#done-today)). `machina-chat-run-v1` gates the `chat_run.start` / `chat_run.finish` activities a run spawned by the owner's chat message schedules to claim and finish its chat run; the run id is read only from an event whose source is `opencompany://services/chat` (see [chat_protocol.md](./chat_protocol.md#runs)). The two `workspace-*` markers are described under [Direct Workspace tasks](#direct-workspace-tasks). Liveness is the activity layer's job: node activities heartbeat every 30s against a 2-minute `heartbeat_timeout`; their `start_to_close` is a generous 24h ceiling, not 10 minutes. The subagent-permit wait uses `PERMIT_WAIT_RETRY` (unlimited attempts) so a queued delegation waits as long as admission takes instead of failing after ~3h.- **History-pressure continue-as-new everywhere.** Temporal terminates any workflow around ~51,200 history events. `WorkflowControlWorkflow` (which multiplexes all of a deployment's triggers into one history) now rolls over on `is_continue_as_new_suggested()` / a 10K-event soft cap, carrying trigger specs, per-trigger provider `seen_ids` (written back into the spec after every poll cycle), queued push events, the bounded dedup baseline, and the control state — a rollover works mid-pause too, since a paused controller still accretes signal history. `TriggerListenerWorkflow`/`PollingTriggerWorkflow` gained the same pressure check (the old `_processed_count >= 16_000` gate was unreachable: real spawns cost ~15-25 events each, and polling counted only emitted events while a quiet mailbox burned ~11 events/cycle — dead in ~3 days at the 60s default). Poll intervals are clamped to a 30s floor (`_MIN_POLL_INTERVAL_S` in `polling_trigger_workflow.py` and `workflow_control_workflow.py`), with no patch gate. Because run ids change on rollover, **controller handles are addressed by workflow id only, never run_id-pinned** (`_controller_handle`, manager `register_trigger`).- **dispatch.emit controller narrowing.** Controllers advertise their push event types via the `ControlEventTypes` keyword-list Search Attribute (upserted as triggers register); `dispatch.emit` skips controllers with no matching trigger instead of signalling every running controller with every platform event (each unmatched signal was ~4 immutable history events — one busy deployment burned every other controller's rollover budget). Controllers without the attribute (pre-upgrade histories) keep match-all behaviour.- **Boot-time reconcile** (`reconcile_active_controls_on_boot`, called from the lifecycle module after workers start): runs the lazy `_reconcile_control` over every active control row, converges `starting` rows a crash left behind (controller alive with triggers registered → `running`; alive-but-empty for a graph that declares triggers → `failed` + controller closed; vanished → `failed`), and re-arms the process-local half of running/paused generations from the persisted graph snapshot — DeploymentManager state, in-process collectors for non-canary trigger types, cron pause posture. Idempotent by construction (controller `register_trigger` keyed by listener id, legacy starts use `USE_EXISTING`, cron creation preserves server-owned pause state).diff --git a/docs-internal/chat_protocol.md b/docs-internal/chat_protocol.mdindex a9a2fb56..e9b1fcd6 100644--- a/docs-internal/chat_protocol.md+++ b/docs-internal/chat_protocol.md@@ -10,7 +10,12 @@ AG-UI JSON.Status: being built on `feature/chat-genui`; the design and the owner's decisions are in the plan recorded at`~/.claude/plans/analyze-the-new-generative-zazzy-cosmos.md`. Change this document in the same commit as any change-to the shapes below, and bump `protocol_version` when an existing field changes meaning.+to the shapes below, and bump `protocol_version` (returned by `get_chat_messages`) when an existing field changes+meaning.++Built so far: runs and their storage, the lifecycle events (`started`, `finished`, `failed`), subscriptions,+`get_chat_messages` v2, `get_chat_run`, the watchdog and the MachinaWorkflow patch (see [Runs](#runs)). Every other+event, handler and part below is the contract the later phases build to.## Concepts@@ -42,8 +47,15 @@ internal socket, load the workflow, and compare its owner with the socket's execattributes.- `seq` increases by one per run. The client drops duplicates `(run_id, seq)` and asks for a snapshot on a gap.- `hub_epoch` changes when the server process restarts. A changed epoch means the client must resync.-- Each subscriber has a bounded queue. Under pressure the hub merges consecutive text deltas of one segment, never- drops lifecycle events, and finally sends `custom` `opencompany.resync`.+- `chat_subscribe` answers with the session's live runs as snapshots, each with the `seq` it reflects. The hub is+ read before the database, so an event published meanwhile carries a higher `seq` than its snapshot: drop events+ with `seq` at or below the snapshot's, and buffer events that arrive before the response.+- Each subscriber has a bounded queue (`hub.subscriber_queue_size` in `server/config/chat_defaults.json`). A socket+ that falls that far behind has its queue replaced by one `custom` `opencompany.resync` per session it follows+ (`data: {session_id, hub_epoch, name, value}`, no `run_id`); the client then takes fresh snapshots. Merging+ consecutive text deltas of one segment comes with streaming.+- Delivery is in-process: events published by an activity on a worker in another process do not reach sockets here+ yet (a relay is planned).- `chat.updated` remains an identity-only broadcast (`{workflow_id, session_id, role}`) that tells every open threadto refetch `get_chat_messages`.@@ -89,7 +101,10 @@ streamed.| `interrupt` | Drafts wait for the owner. `outcome.interrupts: [{id, reason: "tool_call", message, tool_call_id, response_schema, expires_at}]`; `id` is the approval id. || `stopped` | The owner pressed Stop. OpenCompany extension; AG-UI defines only `success` and `interrupt`. |-Never-picked-up runs end with `run.failed` code `not_delivered`; a Reset ends live runs with code `reset`.+Runs nothing will finish end with `run.failed`: code `not_delivered` (never picked up, or the employee stopped+first), `timed_out` (running longer than `runs.max_running_s`), `interrupted` (its workflow closed without finishing+it and no reply was saved; with a saved reply it finishes instead). A Reset ends live runs with code `reset`, the+owner's Clear with `cleared`, and both delete them.## Handlers@@ -100,10 +115,11 @@ All are WebSocket request/response handlers with snake_case payloads. Failures a| Handler | Request | Response ||---|---|---|-| `send_chat_message` | `{session_id, message, role: "user", timestamp?, client_message_id?, attachments?: FileRef[], options?: {web?}, ui_event?: {part_id, element_id, action, params}}` | `{success, message_id, run_id, delivery: "now" \| "queued", timestamp}` |-| `get_chat_messages` | `{session_id, limit?, all_generations?}` | `{success, messages, thread: {active_leaf_id, revision}, active_runs}` |-| `chat_subscribe` / `chat_unsubscribe` | `{session_id}` | `{success, hub_epoch, active_runs: [RunSnapshot]}` |-| `get_chat_run` | `{run_id}` | `{success, run: RunSnapshot}` |+| `send_chat_message` | `{session_id, message, role: "user", timestamp?, client_message_id?, attachments?: FileRef[], options?: {web?}, ui_event?: {part_id, element_id, action, params}}` | `{success, message_id, run_id, delivery: "now" \| "queued", timestamp}`; `run_id` is null when the message starts no run |+| `get_chat_messages` | `{session_id, limit?, all_generations?}` | `{success, protocol_version, session_id, messages, thread: {active_leaf_id, revision}, active_runs: [RunSnapshot]}` |+| `chat_subscribe` | `{session_id}` | `{success, session_id, hub_epoch, active_runs: [RunSnapshot]}` |+| `chat_unsubscribe` | `{session_id}` | `{success, session_id, hub_epoch}` |+| `get_chat_run` | `{run_id}` | `{success, run: RunSnapshot}`, or `not_found` || `stop_chat_run` | `{run_id}` or `{session_id}` | `{success, run_id, state}` || `edit_chat_message` | `{message_id, text, attachments?, expected_revision}` | `{success, message_id, run_id}` || `regenerate_chat_reply` | `{message_id, expected_revision}` | `{success, run_id}` |@@ -112,10 +128,15 @@ All are WebSocket request/response handlers with snake_case payloads. Failures a| `get_chat_context` | `{session_id}` | `{success, commands, suggestions, capabilities, genui_catalog, limits, ask_first: {value, editable, replies_gated}}` || `clear_chat_messages` | `{session_id}` | `{success}`; also clears runs, parts, snapshots, notes and feedback, and makes the employee forget the conversation |-`send_chat_message` with the same `client_message_id` returns the run the first call created.+`send_chat_message` with the same `client_message_id` (1 to 100 letters, digits or `_.:-`) returns the message+and run the first call created, and dispatches nothing again.-**RunSnapshot** is the state the client reducer would hold after replaying the run's events:-`{run_id, kind, state, seq, steps, segments: [{message_id, text, final}], activities, interrupts, started_at}`.+**RunSnapshot** is the state the client reducer would hold after replaying the run's events+(`server/services/chat/reducer.py` folds them the same way):+`{run_id, session_id, workflow_id, kind, state, seq, hub_epoch, user_message_id, reply_message_id, parent_run_id,+created_at, started_at, finished_at, steps, segments: [{message_id, text, final}], activities, interrupts, outcome,+result, error, error_code}`. `state` is `queued`, `pending`, `running`, `stopping`, `finished`, `error` or `stopped`.+A snapshot read after a server restart has its steps but no text segments (text is stored with the reply).### Approvals (`server/services/approvals/handlers.py`)@@ -147,10 +168,15 @@ summary adds `kind` (`gate` or `tool_call`), `node_type`, `tool_name`, `tool_cal```{id, legacy_id, role, kind, text, message, timestamp, run_key, run_id, parent_id, status, attachments,- parts, feedback, siblings: {index, count, ids}, editable}+ parts, feedback, siblings: {index, count, ids}, editable, client_message_id?}```+- `id` is the message's stable id (`m_…` for the owner's, `a_<run id>` for a run's reply; rows saved before chat+ runs are `m<number>`). `legacy_id` is the database row number.- `message` equals `text`; it is kept for older readers.+- `client_message_id` is echoed on the owner's messages that carried one, so an optimistic row and its saved row+ keep one key.+- Until branches land, `siblings` always holds the message alone and `editable` is false.- `kind`: `text`, `report` (Post to Talk: no parent, not editable), `action` (a button press), `notice`.- `status`: `complete`, `stopped`, `error`.- `siblings` are branches for owner messages and versions for replies.@@ -197,6 +223,33 @@ Approvals are joined live from the approvals store; the part only names them.json-render renders each element inside its own error boundary, so one failing element renders nothing instead ofbreaking the reply.+## Runs++`server/services/chat/` (never imports `nodes/`):++- **Admission** (`ledger.admit_message`). `send_chat_message` writes the owner's message and its run in one write+ transaction reserved before its first read (`Database.reserved_session`, SQLite `BEGIN IMMEDIATE`), after checking+ the lane; a partial unique index on `chat_runs` enforces it too. A message starts a run only when a chat trigger in+ the deployed graph (the control generation's `graph_snapshot`) accepts its session, judged by that trigger's own+ filter (`event_waiter.build_filter`). Otherwise it is saved and dispatched without a run, like the editor's+ `"default"` session, which keeps its unscoped delivery.+- **Dispatch.** The `chat_message_received` event (`services/chat/events.py`, source `opencompany://services/chat`)+ has the run id as its CloudEvent id and carries `message_id` and `run_id` in `data`, so the listener's child run+ id is `<slug>-<trigger label>-<run id>`. It is never broadcast.+- **Start and finish.** MachinaWorkflow, behind the `machina-chat-run-v1` patch, reads the run id only from an event+ with that source and type, claims the run (`chat_run.start`: `pending` or `queued` to `running`, recording the+ Temporal workflow and run ids) once the firing trigger's output is stored, passes `run_scope {run_id, session_id}`+ to every node context, and finishes it at its single exit (`chat_run.finish`). With several chat triggers in one+ graph, the first to claim tracks the run and the others run untracked. Only the claimant may finish.+- **The reply.** `chatReply` with a `run_scope` saves the answer through `ledger.post_reply`: the first reply takes+ the run's reply id `a_<run id>`, another reply node in the same run `a_<run id>.<n>`, and a retry from the same node+ saves nothing new. A run whose conversation was reset or cleared posts nothing.+- **One chain.** Every message is appended after the session's active leaf (`chat_threads`) inside the same reserved+ transaction, so concurrent writes never fork the thread.+- **The watchdog** (`services/chat/watchdog.py`, started by `main.py`) sweeps every `runs.watchdog_interval_s` and+ ends the runs nothing will finish (codes above). A pending run's wait (`runs.pickup_timeout_s`) counts from the+ later of its creation and the server's start.+## Notes to the employeeThings the employee should learn on its next turn are queued in `chat_notes` and prepended to that turn's user@@ -223,9 +276,12 @@ and server tests alike:| `run_in_progress` | send, edit, regenerate, switch | A run is live in this session; `run_id` names it. || `save_failed` | send | The message could not be saved; nothing was dispatched. || `not_running` | send | The employee is not running and cannot queue messages. |+| `invalid_request` | send, save | A malformed field (`detail` says which): an empty message, a role other than the owner's, or a bad `client_message_id`. |+| `read_failed` | get_chat_messages, chat_subscribe | The thread could not be read. Never answered as an empty thread. |+| `not_found` | get_chat_run | No such run. || `conflict` | edit, regenerate, switch, set_ask_first | `expected_revision` is stale. || `not_editable` | edit, regenerate, decide | The message belongs to an older generation, its prefix was compacted, or an edited argument is not editable. || `ui_event_rejected` | send | The element or action does not match the saved spec. |-| `access_denied` | all | The socket's principal does not own the workflow. |+| `access_denied` | all | The socket's principal does not own the workflow, or it is the internal worker socket. || `too_late` | decide | The Undo or Restore window has passed. || `speech_unavailable` | transcribe_audio | No dictation provider has a stored key. |diff --git a/docs-internal/event_framework.md b/docs-internal/event_framework.mdindex 5aed1455..1adef0ad 100644--- a/docs-internal/event_framework.md+++ b/docs-internal/event_framework.md@@ -98,9 +98,16 @@ broadcast semantics) are unchanged. First producer: chat — the panel isbound to one workflow (`session_id` IS its workflow id), and without thescope one workflow's chat message fired every deployed workflow'schatTrigger. The scoping DECISION lives in the core call site-(`routers/websocket.py:handle_send_chat_message`; the legacy `"default"`-session stays unscoped) and the narrowing in core dispatch — plugin-`_events.py` factories only plumb the field of their own wire shape.+(`services/chat/handlers.py:handle_send_chat_message`; the legacy+`"default"` session stays unscoped) and the narrowing in core dispatch —+event factories (the chat one is `services/chat/events.py`, source+`opencompany://services/chat`) only plumb the field of their own wire+shape. The chat message is emitted with `broadcast=False` (it carries the+owner's text), and its CloudEvent id is the chat run it started, so the+listener's child run id is predictable; chat run events (`chat_run_event`)+never ride `emit` or the broadcast at all: `services/chat/hub.py` delivers+them only to sockets subscribed to the session+([chat_protocol.md](./chat_protocol.md)).Worker is embedded in the FastAPI process (the `main.py` lifespan schedules`run_temporal_lifecycle` from `services/temporal/lifecycle.py` as onediff --git a/docs-internal/node-logic-flows/chat_utility/chatReply.md b/docs-internal/node-logic-flows/chat_utility/chatReply.mdindex 6b5d871e..8906637b 100644--- a/docs-internal/node-logic-flows/chat_utility/chatReply.md+++ b/docs-internal/node-logic-flows/chat_utility/chatReply.md@@ -48,8 +48,10 @@ like `console`. It does not carry console's `isConsoleSink` hint.```ts// ChatReplyOutput (model_config extra="allow"){- posted: boolean; // false when there was nothing to post- message?: string; // the trimmed text, when posted+ posted: boolean; // false when there was nothing to post+ message?: string; // the trimmed text, when posted+ message_id?: string; // the saved message's id+ run_id?: string; // the chat run it answered, in a run the owner's message started}```@@ -62,7 +64,12 @@ flowchart TDC -- yes --> D[Return posted=false: nothing written or broadcast]C -- no --> E{ctx.workflow_id set?}E -- no --> F[Raise NodeUserError: save the workflow first]- E -- yes --> G[services.chat_thread.record_chat_message<br/>session = workflow id, role = assistant]+ E -- yes --> R{run_scope in the context?}+ R -- yes --> S{the chat run still exists?}+ S -- no --> T[Return posted=false: the conversation was reset or cleared]+ S -- yes --> U[services.chat.ledger.post_reply<br/>id a_run id, linked to the run]+ U --> J+ R -- no --> G[services.chat_thread.record_chat_message<br/>session = workflow id, role = assistant]G --> H{row saved?}H -- no --> I[Raise RuntimeError: the reply could not be saved]H -- yes --> J[broadcast chat.updated]@@ -79,6 +86,15 @@ flowchart TDnothing to send) posts nothing and succeeds with `posted: false`. The edgesthe builders wire into the node carry the same `result.response neq NO_REPLY`condition, so normally the node does not run then at all.+- **Chat runs**: in a run the owner's chat message started, MachinaWorkflow+ passes `run_scope {run_id, session_id}` in the context+ ([chat_protocol.md → Runs](../../chat_protocol.md#runs)). The reply is then+ saved through `services.chat.ledger.post_reply` as that run's answer: the+ first reply takes the run's reply id `a_<run id>`, another reply node in the+ same run `a_<run id>.<n>`, and a retried step returns the row it saved. When+ the run no longer exists (a Reset or the owner's Clear removed the+ conversation it answered) nothing is posted and the node succeeds with+ `posted: false`.- **Generation**: `record_chat_message` stamps the row with the latestcontrol's `root_execution_id` (`chat_thread.chat_execution_id`; none whennothing was started since the last Reset). The editor's chat pane reads the@@ -107,8 +123,9 @@ flowchart TD- **Database writes**: one `chat_messages` row: `session_id` = the workflow id,`role` = `assistant`, `message` = the trimmed text, `execution_id` = the- live generation. On a Reset or a workflow delete, every row of the- workflow's thread is deleted.+ live generation, appended after the thread's last message; in a chat run,+ `run_id` and `uid` name the run and its reply. On a Reset or a workflow+ delete, every row of the workflow's thread is deleted, with its chat runs.- **Broadcasts**: `chat.updated` (CloudEvents `com.opencompany.chat.updated`,data `{workflow_id, session_id, role: "assistant"}`), sent directly throughthe status broadcaster, not `services.events.dispatch.emit`. Home's thread@@ -120,7 +137,7 @@ flowchart TD## External Dependencies- **Credentials**: none.-- **Services**: `services.chat_thread` (`record_chat_message`), the database+- **Services**: `services.chat_thread` (`record_chat_message`), `services.chat.ledger` (`get_run`, `post_reply`), the database(`services.plugin.deps.get_database`), `StatusBroadcaster`.- **Python packages**: stdlib only (`json`).- **Environment variables**: none.diff --git a/docs-internal/node-logic-flows/workflow_triggers/chatTrigger.md b/docs-internal/node-logic-flows/workflow_triggers/chatTrigger.mdindex d29f9ea1..661900a0 100644--- a/docs-internal/node-logic-flows/workflow_triggers/chatTrigger.md+++ b/docs-internal/node-logic-flows/workflow_triggers/chatTrigger.md@@ -12,9 +12,10 @@Fires when the user sends a chat message: from the editor's chat pane(Console Panel chat tab), or from Talk on an employee's Home page. The-producer [`server/nodes/trigger/chat_trigger/_events.py`](../../../server/nodes/trigger/chat_trigger/_events.py)-emits a CloudEvents `WorkflowEvent` (`type: com.opencompany.chat.message.received`)-via `dispatch.emit`. `chatTrigger` is canary-registered+producer [`server/services/chat/events.py`](../../../server/services/chat/events.py)+emits a CloudEvents `WorkflowEvent` (`type: com.opencompany.chat.message.received`,+source `opencompany://services/chat`) via `dispatch.emit`, never broadcast to+sockets. `chatTrigger` is canary-registered(`register_canary_trigger_type`), so `DeploymentManager` starts a`TriggerListenerWorkflow` for it; the listener receives the event via TemporalSignal and spawns a child `MachinaWorkflow` per matching event. Any `chatTrigger`@@ -53,17 +54,23 @@ trigger that Hire and Turn on Talk add beside another worker) has### Output payload-`routers/websocket.py::handle_send_chat_message` builds the event as:+`services/chat/handlers.py::handle_send_chat_message` builds the event as:```ts{message: string;timestamp: string; // ISO 8601; the client's, else the server's time (UTC)session_id: string;+ message_id: string; // the saved message's id+ run_id?: string; // the chat run the message started, when one did}```-Wrapped in the standard envelope.+Wrapped in the standard envelope, whose id is the run id when there is one.+A run spawned from it claims that chat run (MachinaWorkflow,+`machina-chat-run-v1`) and passes `run_scope` to every node, so+[`chatReply`](../chat_utility/chatReply.md) saves its answer as the run's reply.+See [chat_protocol.md](../../chat_protocol.md#runs).## Logic Flow@@ -71,8 +78,8 @@ Wrapped in the standard envelope.flowchart TDP[chat pane or Home Talk sends a message] --> V{workflow session with no<br/>controller to read it?}V -- yes --> X[not_running: nothing saved or sent]- V -- no --> W[chat_thread.record_chat_message]- W --> Q[_events.py dispatch.emit<br/>WorkflowEvent com.opencompany.chat.message.received]+ V -- no --> W[chat ledger: save the message,<br/>and its run when a chat trigger answers the session]+ W --> Q[services/chat/events.py dispatch.emit<br/>WorkflowEvent com.opencompany.chat.message.received]Q --> R[TriggerListenerWorkflow receives via Temporal Signal]R --> S[ChatTriggerNode.build_filter:<br/>if session_id != 'default' require exact match]S -- match --> T[spawn child MachinaWorkflow<br/>trigger pre-executed with event payload]@@ -95,16 +102,17 @@ flowchart TD## Side Effects- **Database writes**: none while it fires. (`send_chat_message` keeps the- message in the session's thread first, through- `services/chat_thread.record_chat_message`, which stamps the live generation- and broadcasts `chat.updated`.) On a workflow Reset its+ message in the session's thread first, with its chat run, through+ `services/chat/ledger.admit_message`, stamped with the live generation, and+ announces `chat.updated`.) On a workflow Reset its`reset_execution_state` clears the workflow's own thread (session = theworkflow id, never a custom `session_id` another workflow may share),as `chatReply`'s does: the conversation ended with the generation. Itmatters for a graph with a trigger and no reply yet, such as the oneTurn on Talk resets before its new reply node runs.- **Broadcasts**: the producer emits a CloudEvents `WorkflowEvent` via- `dispatch.emit` (Temporal Signal fan-out + in-process WS broadcast). The+ `dispatch.emit` (Temporal Signal fan-out only: `broadcast=False`, since the+ envelope carries the owner's text). The`TriggerListenerWorkflow` emits firing-pulse status via`broadcast_trigger_status_activity` before/after each child spawn.- **External API calls**: none.@@ -124,8 +132,10 @@ flowchart TDper trigger is the only way to scope messages to a specific node.- When multiple `chatTrigger` nodes exist with the same session ID, all ofthem fire for a matching message, so one message starts one run per- trigger. Hire and Turn on Talk never add a second trigger on a workflow's- own session: a chat hire's "Chat" trigger is already its talk line.+ trigger; the first run to start claims the chat run and the others run+ untracked. Hire and Turn on Talk never add a second trigger on a+ workflow's own session: a chat hire's "Chat" trigger is already its talk+ line.- The handler has no timeout; it waits forever until an event arrives orthe run is cancelled.- For a workflow's session, `send_chat_message` first reads the latest@@ -134,7 +144,9 @@ flowchart TDand starts a run on Resume (`delivery: "queued"`). In any other state thehandler answers `not_running` and neither saves nor dispatches the message.Session `'default'` is saved and dispatched as before, so a message there is- stored even when no `chatTrigger` is waiting.+ stored even when no `chatTrigger` is waiting. A workflow's message starts a+ chat run only when a deployed `chatTrigger` accepts its session; while a run+ is live the next message is refused with `run_in_progress`.## Relateddiff --git a/docs-internal/normal_mode.md b/docs-internal/normal_mode.mdindex 27b845c3..e7682422 100644--- a/docs-internal/normal_mode.md+++ b/docs-internal/normal_mode.md@@ -19,7 +19,7 @@ up in Home too, with details derived from their graph.| Setup-screen pipeline (parse, normalise, render) | [client/src/features/home/genui/](../client/src/features/home/genui/) || Employees service (list, setup, hire, start, run records) | [server/services/employees/](../server/services/employees/) || Approvals (the "ask me first" step) | [server/services/approvals/](../server/services/approvals/), node [approvalGate](./node-logic-flows/workflow_triggers/approvalGate.md) |-| Talk (the chat thread and the talk line) | [server/services/chat_thread.py](../server/services/chat_thread.py), [services/employees/talk.py](../server/services/employees/talk.py), node [chatReply](./node-logic-flows/chat_utility/chatReply.md) |+| Talk (the chat thread and the talk line) | [server/services/chat_thread.py](../server/services/chat_thread.py), chat runs in [server/services/chat/](../server/services/chat/) ([chat_protocol.md](./chat_protocol.md)), [services/employees/talk.py](../server/services/employees/talk.py), node [chatReply](./node-logic-flows/chat_utility/chatReply.md) || Growing a saved employee (Turn on Talk, the Agent Builder, Apply) | [server/services/graph_build.py](../server/services/graph_build.py), [services/employees/policy.py](../server/services/employees/policy.py), [services/workflow_storage/mutate.py](../server/services/workflow_storage/mutate.py), [services/deployment/restart.py](../server/services/deployment/restart.py), node [agentBuilder](./node-logic-flows/ai_tools/agentBuilder.md) || Built-in skills offered to new hires (Settings > Skills > Discover) | [server/skills/employee/](../server/skills/employee/) |@@ -390,17 +390,24 @@ Connect an AI model action.The owner talks to an employee on its page. The conversation is the chatsession whose id is the employee's workflow id, the thread the editor's chat-pane also shows for that workflow (there, only the live generation). Every-row goes through-[services/chat_thread.py](../server/services/chat_thread.py): the owner's-messages from `send_chat_message`, answers and reports from the-[chatReply](./node-logic-flows/chat_utility/chatReply.md) ("Reply in Chat")-node. Each row is stamped with the live generation, and each insert or clear-is announced as `chat.updated`. Clearing the chat in the editor's chat pane+pane also shows for that workflow (there, only the live generation). The+owner's messages come in through `send_chat_message`+([services/chat/handlers.py](../server/services/chat/handlers.py)), answers+and reports from the [chatReply](./node-logic-flows/chat_utility/chatReply.md)+("Reply in Chat") node. Each row is stamped with the live generation and+appended to one chain, and each insert or clear is announced as+`chat.updated`. Clearing the chat in the editor's chat pane(`clear_chat_messages`, `chat_thread.clear_chat_session`) deletes the threadand, through the Context plugin's chat-cleared listener, every conversationof the workflow, so the employee starts over with the chat.+A message the talk line will answer starts a **chat run**, saved with it: one+run at a time per conversation (a second send while one is live answers+`run_in_progress`), claimed and finished by the workflow run that answers it,+with the reply saved as the run's (`a_<run id>`). Sockets that subscribe+(`chat_subscribe`) receive the run's events; a watchdog ends runs nothing+will finish. The wire contract is [chat_protocol.md](./chat_protocol.md).+### On the employee's page[employee/EmployeeTalk.tsx](../client/src/features/home/employee/EmployeeTalk.tsx)@@ -710,8 +717,8 @@ WebSocket requests (snake_case; failures come back as `success: false` with an| `start_employee` | `{workflow_id, expected_revision, idempotency_key}` | as `start_workflow` || `enable_employee_talk` | `{workflow_id, idempotency_key}` | `{employee}`. Errors: `invalid_request`, `not_found`, `unsupported`, `conflict` (a start, pause, resume or reset is under way, or the graph changed meanwhile), `restart_failed`; the last three carry `employee` too || `apply_employee_changes` | `{workflow_id, idempotency_key}` | `{employee}`: running ends running, paused or failed ends ready, ready is left alone. Errors: `invalid_request`, `not_found`, `conflict`, `restart_failed` (the last two with `employee`) |-| `send_chat_message` | `{message, role: "user", session_id: <workflow_id>, timestamp}` | `{timestamp, delivery}`: `"now"` while running, starting or resuming; `"queued"` while paused or pausing (it runs on Resume). In any other state `not_running`, and nothing is saved or sent. Session `"default"` works as before, with no `delivery` |-| `get_chat_messages` | `{session_id, limit?, all_generations?}` | `{messages}`, oldest first, each `{id, role, message, timestamp, run_key}`. Timestamps carry their UTC offset; `run_key` is the generation the row was written in. Without `all_generations`, only the latest generation's rows (none after a Reset; every row when the workflow was never started) |+| `send_chat_message` | `{message, role: "user", session_id: <workflow_id>, timestamp, client_message_id?}` | `{timestamp, delivery, message_id, run_id}`: `"now"` while running, starting or resuming; `"queued"` while paused or pausing (it runs on Resume). In any other state `not_running`, and nothing is saved or sent; `run_in_progress` (with the live `run_id`) while a run is live. `run_id` is null when no deployed chat trigger answers the session. Session `"default"` works as before, with no `delivery` |+| `get_chat_messages` | `{session_id, limit?, all_generations?}` | `{messages, thread, active_runs}`, messages oldest first, each `{id, role, message, timestamp, run_key, ...}` (the full shape is in [chat_protocol.md](./chat_protocol.md#messages)). Timestamps carry their UTC offset; `run_key` is the generation the row was written in. Without `all_generations`, only the latest generation's rows (none after a Reset; every row when the workflow was never started). A failed read answers `read_failed`, never an empty thread || `list_approvals` | `{workflow_id?, status?, limit <= 100}` | `{approvals, counts, server_time}` || `decide_approval` | `{approval_id, decision, text?, subject?, decision_key}` | `{approval, will_send_on_resume}` || `delete_workflow` | `{workflow_id}` | `{workflow_id, contexts_archived, context_archives_pending}`; `DELETE /api/database/workflows/{id}` is the same handler. Error `workflow_shutdown_failed` (with `detail`) when stopping the employee failed: nothing was deleted |@@ -743,7 +750,8 @@ history). Talk and growing a saved employee: `tests/services/employees/`(`test_talk.py`, `test_enable_talk.py`, `test_apply_changes.py`,`test_policy.py`, and `test_builder_snapshot.py` against`tests/fixtures/employee_builder_snapshot.json`),-`tests/services/test_chat_thread.py`, `tests/nodes/test_chat_reply.py`,+`tests/services/test_chat_thread.py`, `tests/services/chat/`,+`tests/temporal/test_machina_chat_run.py`, `tests/nodes/test_chat_reply.py`,`tests/services/test_graph_build.py`, `tests/services/test_graph_additions.py`,`tests/services/test_graph_listeners.py`,`tests/services/test_deployment_restart.py`,diff --git a/docs-internal/status_broadcaster.md b/docs-internal/status_broadcaster.mdindex 7f758dd0..35e7c7fd 100644--- a/docs-internal/status_broadcaster.md+++ b/docs-internal/status_broadcaster.md@@ -81,9 +81,14 @@ Message loop:Frontend unmounts or logs out|v-StatusBroadcaster.disconnect(ws) -> remove from _connections+StatusBroadcaster.disconnect(ws) -> remove from _connections, then call each+ register_disconnect_listener(fn) listener```+`register_disconnect_listener(fn)` (module-level, idempotent) is for services that keep per-socket state:+`fn(websocket)` runs after the socket leaves `_connections`, synchronously, and must not block. Chat registers one so a+closed socket's run subscriptions are dropped (`services/chat/__init__.py`).+Auto-reconnect is handled by `WebSocketContext.tsx` through PartySocket's `ReconnectingWebSocket` (`partysocket/ws`), with jittered exponential backoff (`MIN_DELAY_MS` / `MAX_DELAY_MS` / `GROW_FACTOR` in `client/src/lib/connectionConfig.ts`). Requests queued while disconnected are sent after the next open; requests already in flight are rejected on close and never replayed. A remote close, even with code 1000, keeps the loop retrying; only the client's own `disposeConnection` (logout or unmount) closes with 1000 and stops it. The full client-side rules are in [frontend_architecture.md → Real-time](./frontend_architecture.md#real-time). A 100ms mount delay avoids React Strict Mode double-connect in dev.## Handler Registry@@ -120,7 +125,7 @@ Live total = `len(MESSAGE_HANDLERS) + len(get_ws_handlers())` -- the `MESSAGE_HA| WhatsApp | `whatsapp_status`, `whatsapp_qr`, `whatsapp_send`, `whatsapp_chat_history`, `whatsapp_newsletters`, `whatsapp_diagnostics`, ... || Telegram | `telegram_connect`, `telegram_disconnect`, `telegram_status`, `telegram_send`, `telegram_get_me`, `telegram_get_chat` || Workflow storage | `save_workflow`, `get_workflow`, `get_all_workflows`, `delete_workflow` |-| Chat messages | `send_chat_message`, `get_chat_messages`, `clear_chat_messages`, `get_chat_sessions` |+| Chat messages | `send_chat_message`, `get_chat_messages`, `chat_subscribe`, `chat_unsubscribe`, `get_chat_run`, `clear_chat_messages`, `save_chat_message` (`services/chat/handlers.py`; run events reach only subscribed sockets, see [chat_protocol.md](./chat_protocol.md)) || Console / terminal | `get_console_logs`, `clear_console_logs`, `get_terminal_logs`, `clear_terminal_logs` || Skills | `get_skill_content`, `save_skill_content`, `get_user_skills`, `create_user_skill`, `scan_skill_folder` || Memory | `clear_memory`, `reset_skill`, `configure_compaction`, `get_compaction_stats` |diff --git a/server/config/chat_defaults.json b/server/config/chat_defaults.jsonnew file mode 100644index 00000000..dc364f71--- /dev/null+++ b/server/config/chat_defaults.json@@ -0,0 +1,10 @@+{+ "runs": {+ "pickup_timeout_s": 120,+ "max_running_s": 86400,+ "watchdog_interval_s": 30+ },+ "hub": {+ "subscriber_queue_size": 512+ }+}diff --git a/server/core/database.py b/server/core/database.pyindex ba954afb..95968491 100644--- a/server/core/database.py+++ b/server/core/database.py@@ -58,6 +58,11 @@ from models.employees import ( # noqa: F401 - registers SQLModel tablesfrom models.approvals import ( # noqa: F401 - registers SQLModel tablesApprovalRequest,)+from models.chat import ( # noqa: F401 - registers SQLModel tables+ ChatRun,+ ChatRunPart,+ ChatThread,+)from models.cache import CacheEntry # SQLite-backed cache for Redis alternativefrom core.logging import get_logger@@ -132,6 +137,7 @@ class Database:await self._migrate_agent_teams()await self._migrate_workflow_controls()await self._migrate_generation_scoped_runtime_data()+ await self._migrate_chat_messages()logger.info("Database initialized successfully")@@ -436,6 +442,21 @@ class Database:else:await asyncio.shield(_teardown_session(session, rollback=False))+ @asynccontextmanager+ async def reserved_session(self):+ """A session whose write transaction is reserved before its first read.++ SQLite's deferred transactions let two writers read the same rows+ before either writes; ``BEGIN IMMEDIATE`` takes the write reservation+ first, so a whole read/modify/write runs alone (as in+ :meth:`run_runtime_mutation`). The caller commits; leaving the block+ without committing rolls back.+ """+ async with self.get_session() as session:+ if self.engine is not None and self.engine.dialect.name == "sqlite":+ await session.execute(text("BEGIN IMMEDIATE"))+ yield session+async def _migrate_workflow_controls(self):"""Backfill control-plane columns when upgrading an early preview DB."""try:@@ -503,6 +524,50 @@ class Database:except Exception as exc:logger.warning(f"Generation runtime-data migration check failed: {exc}")+ async def _migrate_chat_messages(self):+ """Give older chat rows the columns chat runs need.++ Rows written before this migration get a ``uid`` (``m<id>``) and a+ ``parent_uid`` chaining each session's rows in order, the shape every+ new row is written in. The backfill runs once: only in the startup+ that adds the ``uid`` column, so a session's first message or a+ report (both legitimately parentless) never gains a parent later.+ """+ try:+ async with self.engine.begin() as conn:+ result = await conn.execute(text("PRAGMA table_info(chat_messages)"))+ columns = {row[1] for row in result.fetchall()}+ if not columns:+ return+ additions = {+ "uid": "VARCHAR(64)",+ "parent_uid": "VARCHAR(64)",+ "run_id": "VARCHAR(64)",+ "kind": "VARCHAR(20) DEFAULT 'text'",+ "status": "VARCHAR(20) DEFAULT 'complete'",+ "parts": "JSON DEFAULT '{}'",+ "attachments": "JSON DEFAULT '[]'",+ "meta": "JSON DEFAULT '{}'",+ }+ backfill = "uid" not in columns+ for column, definition in additions.items():+ if column not in columns:+ await conn.execute(text(f"ALTER TABLE chat_messages ADD COLUMN {column} {definition}"))+ if backfill:+ await conn.execute(text("UPDATE chat_messages SET uid = 'm' || id WHERE uid IS NULL"))+ await conn.execute(text(+ "UPDATE chat_messages SET parent_uid = ("+ " SELECT prev.uid FROM chat_messages AS prev"+ " WHERE prev.session_id = chat_messages.session_id"+ " AND (prev.created_at < chat_messages.created_at"+ " OR (prev.created_at = chat_messages.created_at AND prev.id < chat_messages.id))"+ " ORDER BY prev.created_at DESC, prev.id DESC LIMIT 1)"+ ))+ await conn.execute(text("CREATE UNIQUE INDEX IF NOT EXISTS ux_chat_messages_uid ON chat_messages(uid)"))+ await conn.execute(text("CREATE INDEX IF NOT EXISTS ix_chat_messages_run_id ON chat_messages(run_id)"))+ except Exception as exc:+ logger.warning(f"Chat message migration check failed: {exc}")+async def run_runtime_mutation(self,*,@@ -1356,62 +1421,164 @@ class Database:# Chat Messages (Console Panel persistence)# ============================================================================+ async def append_chat_row(+ self,+ session: AsyncSession,+ *,+ session_id: str,+ role: str,+ message: str,+ execution_id: Optional[str] = None,+ uid: Optional[str] = None,+ run_id: Optional[str] = None,+ kind: str = "text",+ status: str = "complete",+ parts: Optional[Dict[str, Any]] = None,+ attachments: Optional[List[Dict[str, Any]]] = None,+ meta: Optional[Dict[str, Any]] = None,+ ) -> ChatMessage:+ """Append a message to the end of a session's shown path, inside the+ caller's :meth:`reserved_session` (the caller commits).++ The new row follows the thread's active leaf and becomes the leaf, so+ every message joins one chain: two writers at once would otherwise+ both follow the same leaf and fork it, and a fork hides one branch. A+ session written before threads existed follows its newest message.+ """+ from uuid import uuid4++ thread = await session.get(ChatThread, session_id)+ if thread is None:+ newest = await session.execute(+ select(ChatMessage.uid)+ .where(ChatMessage.session_id == session_id)+ .order_by(ChatMessage.created_at.desc(), ChatMessage.id.desc())+ .limit(1)+ )+ parent_uid = newest.scalar_one_or_none()+ thread = ChatThread(session_id=session_id)+ session.add(thread)+ else:+ parent_uid = thread.active_leaf_uid+ row = ChatMessage(+ session_id=session_id, role=role, message=message,+ execution_id=execution_id,+ uid=uid or f"m_{uuid4().hex}",+ parent_uid=parent_uid,+ run_id=run_id,+ kind=kind,+ status=status,+ parts=parts or {},+ attachments=attachments or [],+ meta=meta or {},+ )+ session.add(row)+ thread.active_leaf_uid = row.uid+ thread.revision = (thread.revision or 0) + 1+ thread.updated_at = datetime.now(timezone.utc)+ await session.flush()+ return row+async def add_chat_message(self, session_id: str, role: str, message: str,execution_id: Optional[str] = None,- ) -> bool:- """Add a chat message to the console panel history."""+ *,+ uid: Optional[str] = None,+ run_id: Optional[str] = None,+ kind: str = "text",+ status: str = "complete",+ parts: Optional[Dict[str, Any]] = None,+ attachments: Optional[List[Dict[str, Any]]] = None,+ meta: Optional[Dict[str, Any]] = None,+ ) -> Optional[Dict[str, Any]]:+ """Append a chat message (:meth:`append_chat_row`) and return it as+ :meth:`chat_row` shapes it, or None when it could not be saved.++ A ``uid`` already in the session returns the stored row unchanged, so+ a retried write (an activity attempt, a resent message) saves once.+ """try:- async with self.get_session() as session:- chat_msg = ChatMessage(- session_id=session_id, role=role, message=message,- execution_id=execution_id,+ async with self.reserved_session() as session:+ if uid is not None:+ found = await session.execute(select(ChatMessage).where(ChatMessage.uid == uid))+ existing = found.scalar_one_or_none()+ if existing is not None:+ if existing.session_id != session_id:+ raise ValueError("message id belongs to another chat")+ return self.chat_row(existing)+ row = await self.append_chat_row(+ session,+ session_id=session_id, role=role, message=message, execution_id=execution_id,+ uid=uid, run_id=run_id, kind=kind, status=status,+ parts=parts, attachments=attachments, meta=meta,)- session.add(chat_msg)await session.commit()logger.debug(f"[Chat] Added {role} message to session '{session_id}'")- return True+ return self.chat_row(row)except Exception as e:logger.error("Failed to add chat message", session_id=session_id, error=str(e))- return False+ return None++ @staticmethod+ def chat_row(message: ChatMessage) -> Dict[str, Any]:+ """A chat row as the handlers read it. The timestamp carries its UTC+ offset (SQLite returns the stored UTC time naive)."""+ moment = message.created_at+ if moment is not None and moment.tzinfo is None:+ moment = moment.replace(tzinfo=timezone.utc)+ return {+ "id": message.id,+ "uid": message.uid,+ "parent_uid": message.parent_uid,+ "run_id": message.run_id,+ "role": message.role,+ "message": message.message,+ "kind": message.kind or "text",+ "status": message.status or "complete",+ "parts": message.parts or {},+ "attachments": message.attachments or [],+ "meta": message.meta or {},+ "timestamp": moment.isoformat() if moment is not None else None,+ "execution_id": message.execution_id,+ }async def get_chat_messages(self, session_id: str, limit: Optional[int] = None,execution_id: Optional[str] = None,) -> List[Dict[str, Any]]:"""Chat messages for a session, oldest first: the newest ``limit``- when given, of one generation when ``execution_id`` is. Each is- ``{id, role, message, timestamp, execution_id}``; the timestamp- carries its UTC offset (SQLite returns the stored UTC time naive)."""+ when given, of one generation when ``execution_id`` is. Rows are+ shaped by :meth:`chat_row`. A read that fails returns ``[]``; use+ :meth:`read_chat_messages` where a failure must be told apart from an+ empty thread."""try:- async with self.get_session() as session:- stmt = select(ChatMessage).where(ChatMessage.session_id == session_id)- if execution_id is not None:- stmt = stmt.where(ChatMessage.execution_id == execution_id)- stmt = stmt.order_by(ChatMessage.created_at.desc(), ChatMessage.id.desc())- if limit and limit > 0:- stmt = stmt.limit(limit)-- result = await session.execute(stmt)- messages = list(reversed(result.scalars().all()))-- def utc(moment: Optional[datetime]) -> Optional[str]:- if moment is None:- return None- return (moment if moment.tzinfo else moment.replace(tzinfo=timezone.utc)).isoformat()-- return [- {"id": m.id, "role": m.role, "message": m.message, "timestamp": utc(m.created_at), "execution_id": m.execution_id}- for m in messages- ]-+ return await self.read_chat_messages(session_id, limit, execution_id=execution_id)except Exception as e:logger.error("Failed to get chat messages", session_id=session_id, error=str(e))return []+ async def read_chat_messages(+ self, session_id: str, limit: Optional[int] = None,+ execution_id: Optional[str] = None,+ ) -> List[Dict[str, Any]]:+ """:meth:`get_chat_messages`, raising when the read fails."""+ async with self.get_session() as session:+ stmt = select(ChatMessage).where(ChatMessage.session_id == session_id)+ if execution_id is not None:+ stmt = stmt.where(ChatMessage.execution_id == execution_id)+ stmt = stmt.order_by(ChatMessage.created_at.desc(), ChatMessage.id.desc())+ if limit and limit > 0:+ stmt = stmt.limit(limit)++ result = await session.execute(stmt)+ return [self.chat_row(m) for m in reversed(result.scalars().all())]+async def clear_chat_messages(self, session_id: str) -> int:- """Clear all chat messages for a session. Returns count deleted."""+ """Clear a session's chat: its messages, and the runs, run parts and+ thread state that belong to them. Returns the messages deleted."""+ from sqlalchemy import delete as sa_delete+try:async with self.get_session() as session:stmt = select(ChatMessage).where(ChatMessage.session_id == session_id)@@ -1422,6 +1589,11 @@ class Database:for message in messages:await session.delete(message)+ run_ids = select(ChatRun.run_id).where(ChatRun.session_id == session_id)+ await session.execute(sa_delete(ChatRunPart).where(ChatRunPart.run_id.in_(run_ids)))+ await session.execute(sa_delete(ChatRun).where(ChatRun.session_id == session_id))+ await session.execute(sa_delete(ChatThread).where(ChatThread.session_id == session_id))+await session.commit()logger.info(f"[Chat] Cleared {count} messages from session '{session_id}'")return countdiff --git a/server/main.py b/server/main.pyindex 651df1df..bd4492e0 100644--- a/server/main.py+++ b/server/main.py@@ -166,6 +166,11 @@ async def lifespan(app: FastAPI):# runs on workflow delete, and the employee_lifecycle summary builder.import services.employees # noqa: F401+ # services/chat/__init__.py self-registers the chat commands+ # (send_chat_message / get_chat_messages / chat_subscribe / ...) and the+ # disconnect listener that drops a closed socket's run subscriptions.+ import services.chat # noqa: F401+# Wave 13.8: services/pricing_handlers.py self-registers the 3# pricing handlers (get_pricing_config / save_pricing_config /# get_api_usage_summary). Flat module (sibling to services/pricing.py)@@ -188,6 +193,13 @@ async def lifespan(app: FastAPI):await container.cache().startup()_startup_log("Database + cache started")+ # Ends chat runs nothing will finish (never picked up, or their run+ # closed without finishing them). Its first sweep runs now.+ from services.chat.watchdog import ChatRunWatchdog++ chat_watchdog = ChatRunWatchdog(container.database())+ chat_watchdog.start()+# Initialize credentials database (creates tables if not exist)credentials_db = container.credentials_database()salt = await credentials_db.initialize()@@ -503,6 +515,9 @@ async def lifespan(app: FastAPI):logger.debug("[CLI MCP] lifespan shutdown: %s", exc)_startup_log("Lifespan shutdown: CLI MCP lifespan exited")+ await chat_watchdog.stop()+ _startup_log("Lifespan shutdown: chat run watchdog stopped")+# Drain cached native SDK clients before credentials/database teardown.await container.chat_unifier().aclose()_startup_log("Lifespan shutdown: LLM clients closed")diff --git a/server/models/chat.py b/server/models/chat.pynew file mode 100644index 00000000..efb6b9ef--- /dev/null+++ b/server/models/chat.py@@ -0,0 +1,98 @@+"""Chat runs: the answer an employee works on after each owner message.++A run is admitted in the same transaction as the owner's message+(``services/chat/ledger.py``). It moves ``queued`` (the employee is paused)+or ``pending`` -> ``running`` -> ``finished`` | ``error`` | ``stopped``, each+step a compare-and-swap on ``state``. ``kind`` says what started it:+``message`` (the owner wrote), ``edit``, ``regenerate``, ``action`` (a button+in generated UI) or ``resume`` (an approved send executing later).++**The lane.** At most one non-terminal run per session, resume runs aside,+enforced by a partial unique index: two runs that loaded the same stored+conversation would each save over the other's (docs-internal/+agent_context_flow.md). A second message while one is live is refused with+``run_in_progress``; the composer shows Stop instead of Send meanwhile.++``ChatRunPart`` collects what a run's tools produce for its reply (generated+UI, sources, artifacts, approvals), keyed so an activity retry writes the+same row again. ``ChatThread`` holds a session's active leaf: the message the+shown path ends at, which branches will move.+"""++from __future__ import annotations++from datetime import datetime, timezone+from typing import Any, Dict, List, Optional++from sqlalchemy import JSON, Column, DateTime, Index, UniqueConstraint, text+from sqlmodel import Field, SQLModel+++def _utcnow() -> datetime:+ return datetime.now(timezone.utc)+++#: States that hold the lane.+LIVE_STATES = ("queued", "pending", "running", "stopping")+#: States a run ends in.+TERMINAL_STATES = ("finished", "error", "stopped")++_LANE_WHERE = "state IN ('queued', 'pending', 'running', 'stopping') AND kind != 'resume'"+++class ChatThread(SQLModel, table=True):+ __tablename__ = "chat_threads"++ session_id: str = Field(primary_key=True, max_length=255)+ active_leaf_uid: Optional[str] = Field(default=None, max_length=64)+ revision: int = Field(default=0)+ updated_at: datetime = Field(default_factory=_utcnow, sa_column=Column(DateTime(timezone=True), nullable=False))+++class ChatRun(SQLModel, table=True):+ __tablename__ = "chat_runs"+ __table_args__ = (Index("ux_chat_runs_lane", "session_id", unique=True, sqlite_where=text(_LANE_WHERE)),)++ run_id: str = Field(primary_key=True, max_length=64)+ session_id: str = Field(index=True, max_length=255)+ workflow_id: Optional[str] = Field(default=None, index=True, max_length=255)+ #: The generation's root execution id, like ``chat_messages.execution_id``.+ run_key: Optional[str] = Field(default=None, max_length=255)+ kind: str = Field(default="message", max_length=20)+ state: str = Field(default="pending", index=True, max_length=20)+ user_message_uid: Optional[str] = Field(default=None, max_length=64)+ reply_message_uid: Optional[str] = Field(default=None, max_length=64)+ parent_run_id: Optional[str] = Field(default=None, max_length=64)+ #: Per-message choices the owner made (``web``).+ options: Dict[str, Any] = Field(default_factory=dict, sa_column=Column(JSON))+ #: Working steps as they finished, for "Worked for Ns · N steps" on reload.+ steps: List[Dict[str, Any]] = Field(default_factory=list, sa_column=Column(JSON))+ #: How a finished run ended: ``success``, ``stopped`` or ``interrupt``+ #: (drafts wait for the owner). None until it ends, and for errors.+ outcome: Optional[str] = Field(default=None, max_length=20)+ #: The terminal event's ``result`` (``reply_message_id`` or ``no_reply``).+ result: Dict[str, Any] = Field(default_factory=dict, sa_column=Column(JSON))+ error: Optional[str] = Field(default=None, max_length=500)+ error_code: Optional[str] = Field(default=None, max_length=60)+ temporal_workflow_id: Optional[str] = Field(default=None, max_length=500)+ temporal_run_id: Optional[str] = Field(default=None, max_length=255)+ created_at: datetime = Field(default_factory=_utcnow, sa_column=Column(DateTime(timezone=True), nullable=False))+ started_at: Optional[datetime] = Field(default=None, sa_column=Column(DateTime(timezone=True), nullable=True))+ finished_at: Optional[datetime] = Field(default=None, sa_column=Column(DateTime(timezone=True), nullable=True))+++class ChatRunPart(SQLModel, table=True):+ __tablename__ = "chat_run_parts"+ __table_args__ = (UniqueConstraint("run_id", "key", name="uq_chat_run_parts_key"),)++ id: Optional[int] = Field(default=None, primary_key=True)+ run_id: str = Field(index=True, max_length=64)+ #: ``ui``, ``sources``, ``artifacts``, ``approvals``, ``followups``.+ kind: str = Field(max_length=20)+ #: Unique within the run; a retried activity writes the same key again.+ key: str = Field(max_length=255)+ payload: Dict[str, Any] = Field(default_factory=dict, sa_column=Column(JSON))+ created_at: datetime = Field(default_factory=_utcnow, sa_column=Column(DateTime(timezone=True), nullable=False))+++__all__ = ["ChatRun", "ChatRunPart", "ChatThread", "LIVE_STATES", "TERMINAL_STATES"]diff --git a/server/models/database.py b/server/models/database.pyindex 95adc602..ad92ccc4 100644--- a/server/models/database.py+++ b/server/models/database.py@@ -3,7 +3,7 @@from datetime import datetime, timezonefrom typing import Optional, Dict, Anyfrom sqlmodel import SQLModel, Field, Column, DateTime, JSON-from sqlalchemy import UniqueConstraint, func+from sqlalchemy import Index, UniqueConstraint, funcclass NodeParameter(SQLModel, table=True):@@ -215,15 +215,33 @@ class ChatMessage(SQLModel, table=True):Stores user and assistant messages from the chat panel (Console Panel chat section).These are separate from ConversationMessage which stores AI agent memory.++ ``uid`` is the message's stable id on the wire (``m_…`` for the owner's,+ ``a_<run id>`` for a run's reply). ``parent_uid`` links a message to the+ one it follows, so edits can branch (models/chat.py holds the runs and+ each session's active leaf). ``parts`` is the structured content a reply+ gathered (steps, generated UI, sources, ...); ``message`` stays the text."""__tablename__ = "chat_messages"+ # Named like core/database.py's migration creates it on older databases.+ __table_args__ = (Index("ux_chat_messages_uid", "uid", unique=True),)id: Optional[int] = Field(default=None, primary_key=True)session_id: str = Field(default="default", index=True, max_length=255)execution_id: Optional[str] = Field(default=None, index=True, max_length=255)role: str = Field(max_length=20) # 'user' or 'assistant'message: str = Field(max_length=50000) # Large content support+ uid: Optional[str] = Field(default=None, max_length=64)+ parent_uid: Optional[str] = Field(default=None, max_length=64)+ run_id: Optional[str] = Field(default=None, index=True, max_length=64)+ #: ``text``; ``report`` (Post to Talk); ``action`` (a button press).+ kind: str = Field(default="text", max_length=20)+ #: ``complete``, ``stopped`` (the owner pressed Stop), ``error``.+ status: str = Field(default="complete", max_length=20)+ parts: Dict[str, Any] = Field(default_factory=dict, sa_column=Column(JSON))+ attachments: list = Field(default_factory=list, sa_column=Column(JSON))+ meta: Dict[str, Any] = Field(default_factory=dict, sa_column=Column(JSON))created_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc), sa_column=Column(DateTime(timezone=True), server_default=func.now()))diff --git a/server/nodes/chat/chat_reply/__init__.py b/server/nodes/chat/chat_reply/__init__.pyindex ec37495f..4e2063ab 100644--- a/server/nodes/chat/chat_reply/__init__.py+++ b/server/nodes/chat/chat_reply/__init__.py@@ -9,6 +9,13 @@ which stamps the live generation and announces ``chat.updated``.A message that is empty, or exactly NO_REPLY (the agent had nothing tosay), posts nothing.+In a run the owner's chat message started, the reply is that chat run's+answer (``run_scope``, set by MachinaWorkflow): it is saved under the run's+reply id through ``services/chat/ledger.py``, so a retried step saves it+once, and the run reports it when it finishes. A run whose conversation was+reset or cleared meanwhile posts nothing: its answer belongs to a thread+that no longer exists.+The thread ends with the workflow's generation: a Reset (every restart)clears it through ``reset_execution_state``, as the Context node forgetsthe conversation in the same Reset, and deleting the workflow deletes it@@ -52,6 +59,9 @@ class ChatReplyOutput(BaseModel):#: False when there was nothing to post.posted: bool = Falsemessage: Optional[str] = None+ #: The saved message's id, and the chat run it answers (if any).+ message_id: Optional[str] = None+ run_id: Optional[str] = Nonemodel_config = ConfigDict(extra="allow")@@ -81,9 +91,30 @@ class ChatReplyNode(ActionNode):return ChatReplyOutput(posted=False)if not ctx.workflow_id:raise NodeUserError("Reply in Chat posts to the workflow's chat: save the workflow first.")- if not await record_chat_message(get_database(), ctx.workflow_id, "assistant", text):+ database = get_database()+ scope = ctx.raw.get("run_scope") if isinstance(ctx.raw, dict) else None+ run_id = scope.get("run_id") if isinstance(scope, dict) else None+ if isinstance(run_id, str) and run_id:+ return await self._reply_to_run(database, ctx, run_id, text)+ saved = await record_chat_message(database, ctx.workflow_id, "assistant", text)+ if not saved:raise RuntimeError("The reply could not be saved to the chat")- return ChatReplyOutput(posted=True, message=text)+ return ChatReplyOutput(posted=True, message=text, message_id=_message_id(saved))++ async def _reply_to_run(self, database, ctx: NodeContext, run_id: str, text: str) -> ChatReplyOutput:+ from services.chat import ledger+ from services.chat_thread import announce_chat_updated, chat_execution_id++ run = await ledger.get_run(database, run_id)+ if run is None or run.session_id != ctx.workflow_id:+ # Reset or Clear removed the conversation this run answered.+ return ChatReplyOutput(posted=False, run_id=run_id)+ control = await database.get_latest_workflow_control(run.session_id)+ saved = await ledger.post_reply(+ database, run=run, node_id=ctx.node_id, text=text, execution_id=chat_execution_id(control)+ )+ await announce_chat_updated(run.session_id, "assistant")+ return ChatReplyOutput(posted=True, message=text, message_id=saved.get("uid"), run_id=run_id)@classmethodasync def reset_execution_state(@@ -106,6 +137,10 @@ class ChatReplyNode(ActionNode):return {"reset": bool(cleared), "cleared_chat_messages": cleared}+def _message_id(saved: Any) -> Optional[str]:+ return saved.get("uid") if isinstance(saved, dict) else None++# The thread belongs to its workflow: deleting the workflow deletes it.from services.chat_thread import clear_chat_thread # noqa: E402from services.workflow_storage.hooks import register_workflow_deleted_hook # noqa: E402diff --git a/server/nodes/trigger/chat_trigger/__init__.py b/server/nodes/trigger/chat_trigger/__init__.pyindex 313d682f..5e78417c 100644--- a/server/nodes/trigger/chat_trigger/__init__.py+++ b/server/nodes/trigger/chat_trigger/__init__.py@@ -9,6 +9,12 @@ A Reset clears the workflow's chat thread (``reset_execution_state``), asTalk resets on that graph before adding the reply) must not carry its oldmessages into the next generation.+The owner's message reaches it from ``services/chat/handlers.py``+(``send_chat_message``), which saves it and signals it through+``services/chat/events.py``. Its output names the saved message+(``message_id``) and, when the message started a chat run, the run+(``run_id``): the run that answers it is that chat run.+Replaces:- ``nodes/triggers.py:chatTrigger`` metadata-only registration.- ``event_waiter.build_chat_filter`` stays wired until the generic@@ -37,6 +43,8 @@ class ChatTriggerOutput(BaseModel):message: Optional[str] = Nonetimestamp: Optional[str] = Nonesession_id: Optional[str] = None+ message_id: Optional[str] = None+ run_id: Optional[str] = Nonemodel_config = ConfigDict(extra="allow")@@ -94,7 +102,7 @@ class ChatTriggerNode(TriggerNode):# Wave 12 C1 rollout #1: opt this trigger into the TriggerListenerWorkflow# consumer path. Producer side: dispatch_chat_message_received in-# ./_events.py calls dispatch.emit unconditionally; the canary registry-# tells DeploymentManager to start a listener for this type. See+# services/chat/events.py calls dispatch.emit; the canary registry tells+# DeploymentManager to start a listener for this type. See# services/deployment/canary_registry.py.register_canary_trigger_type(ChatTriggerNode.type, "com.opencompany.chat.message.received")diff --git a/server/nodes/trigger/chat_trigger/_events.py b/server/nodes/trigger/chat_trigger/_events.pydeleted file mode 100644index fd358d3e..00000000--- a/server/nodes/trigger/chat_trigger/_events.py+++ /dev/null@@ -1,73 +0,0 @@-"""CloudEvents factory + dispatch wrapper for chat_trigger.--Per RFC plugin_authoring_rfc.md §6.4: plugin-specific factories live in-the plugin folder.--Delivery: single ``dispatch.emit`` call routes events to running-:class:`TriggerListenerWorkflow` consumers via Temporal Visibility.-chatTrigger is canary-registered (see ``nodes/trigger/chat_trigger/__init__.py``)-so the deployment manager skips ``setup_event_trigger`` and the legacy-``event_waiter.dispatch`` path has zero consumers — removed.--It is never broadcast. The envelope carries the owner's message text, and-the WebSocket broadcast reaches every connected socket; no client handled-it anyway. Clients learn about a new message from the identity-only-``chat.updated`` that ``services/chat_thread.py`` sends when it is saved.-"""--from __future__ import annotations--from typing import Any, Mapping, Optional--from services.events.envelope import WorkflowEvent---# Outer wire-routing key. Matches ``ChatTriggerNode.event_type`` and the-# FE WS channel; the inner envelope carries-# ``com.opencompany.chat.message.received``.-_WIRE_ROUTING_KEY = "chat_message_received"---def chat_message_received(- event_data: Mapping[str, Any],- *,- workflow_id: Optional[str] = None,-) -> WorkflowEvent:- """Incoming chat message envelope. ``subject`` is the session_id so- consumers can route per-conversation.-- ``workflow_id`` is carried verbatim onto the envelope — the scoping- DECISION belongs to the core call site (routers/websocket.py) and- the scoped-delivery rule to core dispatch; this factory only plumbs- the field of its own wire shape.- """- payload = dict(event_data)- session_id = payload.get("session_id")- return WorkflowEvent(- source="opencompany://nodes/chat_trigger",- type="com.opencompany.chat.message.received",- subject=str(session_id) if session_id else None,- workflow_id=workflow_id,- data=payload,- )---async def dispatch_chat_message_received(- event_data: Mapping[str, Any],- *,- workflow_id: Optional[str] = None,-) -> None:- """Dispatch an incoming chat message via the canary CloudEvents path."""- from services.events.dispatch import emit-- await emit(- chat_message_received(dict(event_data), workflow_id=workflow_id),- wire_routing_key=_WIRE_ROUTING_KEY,- broadcast=False,- )---__all__ = [- "chat_message_received",- "dispatch_chat_message_received",-]diff --git a/server/routers/websocket.py b/server/routers/websocket.pyindex 8ef7cadf..f25cf1eb 100644--- a/server/routers/websocket.py+++ b/server/routers/websocket.py@@ -15,7 +15,6 @@ import asyncioimport uuidimport weakreffrom typing import Dict, Any, Callable, Awaitable, Optional, Set-from datetime import datetime, timezonefrom fastapi import APIRouter, WebSocket, WebSocketDisconnectfrom services.status_broadcaster import get_status_broadcaster@@ -455,6 +454,11 @@ async def handle_execute_node(data: Dict[str, Any], websocket: WebSocket) -> Dic)if key in data}+ # A chat run's scope links what a node produces to the owner's chat, so+ # only the worker relaying MachinaWorkflow's context may set it.+ socket_scope = getattr(websocket, "scope", None) or {}+ if "run_scope" in data and isinstance(socket_scope, dict) and socket_scope.get("path") == "/ws/internal":+ invocation_extras["run_scope"] = data["run_scope"]user_id = execution_principal(data, websocket)await broadcaster.update_node_status(@@ -1071,113 +1075,11 @@ async def handle_get_ai_models(data: Dict[str, Any], websocket: WebSocket) -> Di# ============================================================================-# Chat Message Handler (for chatTrigger nodes)+# Chat — moved to services/chat/handlers.py with the chat run runtime# ============================================================================---@ws_handler("message")-async def handle_send_chat_message(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:- """Send a chat message to a workflow's chatTrigger nodes, and keep it in- the workflow's thread (services/chat_thread.py).-- A workflow's session (``session_id`` = its id) takes a message only- while a deployment would read it: the response's ``delivery`` is- ``"now"`` while it runs, ``"queued"`` while it is paused (it runs on- Resume). Otherwise the answer is ``not_running`` and nothing is saved- or sent. Session ``"default"`` (no workflow open) is saved and- broadcast to every deployment, as it always was.- """- # Wave 12 B7: route through chat_trigger plugin _events.py wrapper —- # the chat_message_received event wire shape lives with the plugin,- # not in this central WS router.- from nodes.trigger.chat_trigger._events import dispatch_chat_message_received- from services.chat_thread import delivery_for, record_chat_message-- message = data["message"]- role = data.get("role", "user")- session_id = data.get("session_id", "default")- timestamp = data.get("timestamp") or datetime.now(timezone.utc).isoformat()-- # Chat is inherently workflow-scoped: the panel is bound to ONE- # workflow and its ``session_id`` IS that workflow's id ("default"- # only when no workflow is open — that legacy session keeps broadcast- # semantics). The scope rides the envelope's ``workflow_id`` so- # ``dispatch.emit`` signals only this workflow's consumers via the- # EventWorkflowId Search Attribute; without it, one workflow's chat- # message fired EVERY deployed workflow's chatTrigger. The scoping- # decision lives here (core router) and the narrowing in core- # dispatch — the plugin factory only carries the field.- workflow_scope = session_id if session_id and session_id != "default" else None-- database = container.database()- delivery = None- if workflow_scope is not None:- delivery = delivery_for(await database.get_latest_workflow_control(session_id))- if delivery is None:- return {"success": False, "error": "not_running"}-- # Kept in the thread, stamped with the live generation.- await record_chat_message(database, session_id, role, message)-- # Build event data matching chatTrigger output schema- event_data = {"message": message, "timestamp": timestamp, "session_id": session_id}-- # Dispatch via canary CloudEvents path — Visibility-query Signal- # fan-out to running TriggerListenerWorkflow consumers + in-process- # WS broadcast on ``chat_message_received``.- await dispatch_chat_message_received(event_data, workflow_id=workflow_scope)-- logger.info(f"[ChatMessage] Dispatched canary event for session={session_id}")-- response = {"success": True, "message": "Chat message sent", "timestamp": timestamp}- if delivery is not None:- response["delivery"] = delivery- return response---@ws_handler()-async def handle_get_chat_messages(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:- """A session's messages, oldest first, the newest ``limit`` of them:- ``{id, role, message, timestamp, run_key}``. ``run_key`` is the- generation the message was written in. The live generation's only,- unless ``all_generations`` (Home's thread, which also shows a message- from before a Start). A Reset clears a workflow's thread (the chat- nodes' Reset hook), so generations rarely mix."""- session_id = data.get("session_id", "default")- limit = data.get("limit") # Optional limit-- database = container.database()- if data.get("all_generations"):- rows = await database.get_chat_messages(session_id, limit)- else:- control = await database.get_latest_workflow_control(session_id)- if control is not None and control.status == "reset":- rows = []- else:- rows = await database.get_chat_messages(- session_id, limit,- execution_id=control.root_execution_id if control is not None else None,- )- messages = [- {"id": row["id"], "role": row["role"], "message": row["message"], "timestamp": row["timestamp"], "run_key": row["execution_id"]}- for row in rows- ]-- return {"success": True, "messages": messages, "session_id": session_id}---@ws_handler()-async def handle_clear_chat_messages(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:- """Clear a session's chat, every generation of it. For a workflow's- session the agent forgets the conversation too- (``services.chat_thread.clear_chat_session``: the Context plugin clears- the workflow's conversations), so it starts over with the chat."""- from services.chat_thread import clear_chat_session-- session_id = data.get("session_id", "default")- count = await clear_chat_session(container.database(), session_id)-- return {"success": True, "message": f"Cleared {count} chat messages", "cleared_count": count}+# send_chat_message / get_chat_messages / chat_subscribe / chat_unsubscribe /+# get_chat_run / clear_chat_messages / save_chat_message self-register via+# services/chat/__init__.py.@ws_handler()@@ -1244,26 +1146,6 @@ async def handle_clear_console_logs(data: Dict[str, Any], websocket: WebSocket)}-@ws_handler("message", "role")-async def handle_save_chat_message(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:- """Save a single chat message (used for assistant responses)."""- from services.chat_thread import record_chat_message-- session_id = data.get("session_id", "default")- success = await record_chat_message(container.database(), session_id, data["role"], data["message"])-- return {"success": success, "message": "Chat message saved" if success else "Failed to save chat message"}---@ws_handler()-async def handle_get_chat_sessions(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:- """Get list of all chat sessions."""- database = container.database()- sessions = await database.get_chat_sessions()-- return {"success": True, "sessions": sessions}--# ============================================================================# Terminal Logs Handlers# ============================================================================@@ -1533,11 +1415,7 @@ MESSAGE_HANDLERS: Dict[str, MessageHandler] = {# via _resolve_handler() defined above.# Workflow storage operations — extracted to services/workflow_storage/handlers.py# (Wave 13.7); registered via ws_handler_registry on package import.- # Chat message (for chatTrigger nodes)- "send_chat_message": handle_send_chat_message,- "get_chat_messages": handle_get_chat_messages,- "clear_chat_messages": handle_clear_chat_messages,- "save_chat_message": handle_save_chat_message,+ # Chat commands live in services/chat/handlers.py (registered on import).# Console logs (for Console nodes)"get_console_logs": handle_get_console_logs,"clear_console_logs": handle_clear_console_logs,diff --git a/server/services/chat/__init__.py b/server/services/chat/__init__.pynew file mode 100644index 00000000..4480e551--- /dev/null+++ b/server/services/chat/__init__.py@@ -0,0 +1,40 @@+"""Chat runs: the conversation between the owner and a workflow, as runs.++The wire contract is docs-internal/chat_protocol.md. This package never+imports ``nodes/``.++- ``ledger``: admits a run with the owner's message, starts and finishes it,+ and the watchdog's sweep ends runs nothing will.+- ``hub``: delivers run events to the sockets subscribed to a session.+- ``reducer``: a run's snapshot, folded from its events.+- ``events``: the CloudEvents (the owner's message to the listeners, run+ events).+- ``access``: who may use a session.+- ``handlers``: the WebSocket commands.+- ``activities``: the Temporal activities MachinaWorkflow starts and+ finishes runs through.+- ``watchdog``: the periodic sweep main.py runs.++Thread rows (and ``chat.updated``) stay with ``services/chat_thread.py``.++Side-effect import (from main.py) registers the WebSocket commands and the+disconnect listener that drops a closed socket's subscriptions.+"""++from __future__ import annotations++from services.status_broadcaster import register_disconnect_listener as _register_disconnect_listener+from services.ws_handler_registry import register_ws_handlers as _register_ws_handlers++from .handlers import WS_HANDLERS as _CHAT_WS_HANDLERS+from .hub import get_chat_hub+++def _drop_socket(websocket) -> None:+ get_chat_hub().drop_socket(websocket)+++_register_ws_handlers(_CHAT_WS_HANDLERS)+_register_disconnect_listener(_drop_socket)++__all__ = ["get_chat_hub"]diff --git a/server/services/chat/access.py b/server/services/chat/access.pynew file mode 100644index 00000000..68113ebd--- /dev/null+++ b/server/services/chat/access.py@@ -0,0 +1,61 @@+"""Who may use a chat session.++A workflow's session (its id) answers only the workflow's owner, compared+with the socket's principal as the canvas and Memory panels do, and the+internal worker socket may use none of chat's commands. The editor's+``"default"`` session belongs to no workflow. A session that names no saved+workflow is still scoped to that id: its messages reach no other workflow.+"""++from __future__ import annotations++from dataclasses import dataclass+from typing import Any, Optional++from services.authz.ws_surface import execution_principal+from services.chat_thread import DEFAULT_SESSION+++class ChatAccessDenied(Exception):+ """The socket may not use this session."""++ def __init__(self) -> None:+ super().__init__("access_denied")+++@dataclass(frozen=True)+class ChatScope:+ session_id: str+ #: The workflow the session belongs to; None for ``"default"``.+ workflow_id: Optional[str]+++def _owner_of(saved: Any) -> str:+ data = getattr(saved, "data", None)+ if data is None and isinstance(saved, dict):+ data = saved.get("data", saved)+ return str(data.get("owner_id") or "") if isinstance(data, dict) else ""+++def session_id_of(data: Any) -> str:+ """The request's session, ``"default"`` when it names none."""+ value = data.get("session_id") if isinstance(data, dict) else None+ text = str(value).strip() if value is not None else ""+ return text or DEFAULT_SESSION+++async def authorize_session(database: Any, websocket: Any, session_id: str) -> ChatScope:+ """The scope a socket may use, or :class:`ChatAccessDenied`."""+ scope = getattr(websocket, "scope", None)+ if isinstance(scope, dict) and scope.get("path") == "/ws/internal":+ raise ChatAccessDenied()+ if session_id == DEFAULT_SESSION:+ return ChatScope(session_id=session_id, workflow_id=None)+ saved = await database.get_workflow(session_id)+ owner = _owner_of(saved) if saved is not None else ""+ if owner and owner != execution_principal({}, websocket):+ raise ChatAccessDenied()+ return ChatScope(session_id=session_id, workflow_id=session_id)+++__all__ = ["ChatAccessDenied", "ChatScope", "authorize_session", "session_id_of"]diff --git a/server/services/chat/activities.py b/server/services/chat/activities.pynew file mode 100644index 00000000..de78bb66--- /dev/null+++ b/server/services/chat/activities.py@@ -0,0 +1,66 @@+"""The Temporal activities a chat run starts and finishes through.++MachinaWorkflow schedules them, behind the ``machina-chat-run-v1`` patch,+for a run that the owner's chat message spawned (services/temporal/+workflow.py). Both are safe to retry: the ledger accepts a repeat from the+workflow that claimed the run and publishes nothing twice. A failure raises,+so Temporal retries; when the retries run out the workflow carries on and+the watchdog ends the run.+"""++from __future__ import annotations++from typing import Any, Dict++from temporalio import activity++[email protected](name="chat_run.start")+async def start_chat_run_activity(payload: Dict[str, Any]) -> Dict[str, Any]:+ """Claim the run for the calling workflow.++ Payload ``{run_id, temporal_workflow_id, temporal_run_id}``. Returns+ ``{claimed}``: False when another workflow claimed it first (several+ chat triggers in one graph) or it can no longer start (it was stopped,+ or the watchdog ended it); that workflow then runs without tracking.+ """+ from core.container import container+ from services.chat import ledger++ run = await ledger.start_run(+ container.database(),+ run_id=str(payload.get("run_id") or ""),+ temporal_workflow_id=str(payload.get("temporal_workflow_id") or ""),+ temporal_run_id=payload.get("temporal_run_id"),+ )+ return {"claimed": run is not None}++[email protected](name="chat_run.finish")+async def finish_chat_run_activity(payload: Dict[str, Any]) -> Dict[str, Any]:+ """End the run the calling workflow claimed.++ Payload ``{run_id, temporal_workflow_id, temporal_run_id, success,+ error?, code?, hint?, requires_user_action?}``. Returns ``{state}``, the+ run's state afterwards, or None when the caller does not hold it.+ """+ from core.container import container+ from services.chat import ledger++ run = await ledger.finish_run(+ container.database(),+ run_id=str(payload.get("run_id") or ""),+ temporal_workflow_id=str(payload.get("temporal_workflow_id") or ""),+ temporal_run_id=payload.get("temporal_run_id"),+ success=bool(payload.get("success")),+ error=payload.get("error"),+ code=payload.get("code"),+ hint=payload.get("hint"),+ requires_user_action=payload.get("requires_user_action"),+ )+ return {"state": run.state if run is not None else None}+++CHAT_RUN_ACTIVITIES = (start_chat_run_activity, finish_chat_run_activity)++__all__ = ["CHAT_RUN_ACTIVITIES", "finish_chat_run_activity", "start_chat_run_activity"]diff --git a/server/services/chat/config.py b/server/services/chat/config.pynew file mode 100644index 00000000..2a4c45b4--- /dev/null+++ b/server/services/chat/config.py@@ -0,0 +1,27 @@+"""Chat runtime settings (``config/chat_defaults.json``), loaded once."""++from __future__ import annotations++import json+from functools import lru_cache+from pathlib import Path+from typing import Any, Mapping++CONFIG_PATH = Path(__file__).resolve().parents[2] / "config" / "chat_defaults.json"+++@lru_cache(maxsize=1)+def load_chat_config() -> Mapping[str, Any]:+ return json.loads(CONFIG_PATH.read_text(encoding="utf-8"))+++def runs_setting(name: str) -> float:+ """``runs.<name>``: seconds, from the config file."""+ return float(load_chat_config()["runs"][name])+++def hub_setting(name: str) -> int:+ return int(load_chat_config()["hub"][name])+++__all__ = ["CONFIG_PATH", "hub_setting", "load_chat_config", "runs_setting"]diff --git a/server/services/chat/events.py b/server/services/chat/events.pynew file mode 100644index 00000000..6a0f4caf--- /dev/null+++ b/server/services/chat/events.py@@ -0,0 +1,156 @@+"""Chat's CloudEvents: the owner's message to a workflow, and run events.++**The owner's message.** ``chat_message_received`` carries a message to the+workflow's chatTrigger listeners through ``dispatch.emit`` (Temporal Signals,+narrowed to the workflow by the envelope's ``workflow_id``). It is never+broadcast: it carries the owner's text, and a broadcast reaches every socket;+clients learn of new messages from the identity-only ``chat.updated``. When+the message started a run, the event id is the run id, so the run a listener+spawns has a predictable id (``<slug>-<trigger label>-<run id>``) and a+re-sent event starts nothing twice. ``data.run_id`` names the run;+MachinaWorkflow trusts it only from this module's source and type+(``services/temporal/workflow.py``).++**Run events.** ``chat_run_event`` frames tell a session's subscribers how a+run moves (``services/chat/hub.py``). AG-UI's event model: the type is+``com.opencompany.chat.run.<suffix>``, the id ``<run id>:<seq>``, the subject+the run id, and the scope (``workflow_id``, ``session_id``, ``run_id``, ``seq``,+``hub_epoch``) sits in ``data``, never in top-level extension attributes.+docs-internal/chat_protocol.md lists every suffix and its fields.+"""++from __future__ import annotations++from typing import Any, Dict, Mapping, Optional++from services.events.envelope import WorkflowEvent++SOURCE = "opencompany://services/chat"+MESSAGE_RECEIVED_TYPE = "com.opencompany.chat.message.received"+RUN_TYPE_PREFIX = "com.opencompany.chat.run."+#: WebSocket ``type`` of a run event frame.+RUN_WIRE_KEY = "chat_run_event"+#: ``dispatch.emit``'s wire routing key for the owner's message; matches+#: ``ChatTriggerNode.event_type``.+MESSAGE_WIRE_ROUTING_KEY = "chat_message_received"++RUN_SUFFIXES = frozenset(+ {+ "started",+ "finished",+ "failed",+ "step.started",+ "step.finished",+ "text.started",+ "text.content",+ "text.ended",+ "tool_call.started",+ "tool_call.args",+ "tool_call.ended",+ "tool_call.result",+ "activity.snapshot",+ "activity.delta",+ "reasoning.started",+ "reasoning.ended",+ "custom",+ }+)+#: Suffixes after which a run publishes nothing more.+TERMINAL_SUFFIXES = frozenset({"finished", "failed"})+++def chat_message_received(+ event_data: Mapping[str, Any],+ *,+ workflow_id: Optional[str] = None,+ event_id: Optional[str] = None,+) -> WorkflowEvent:+ """The owner's message as the chatTrigger listeners receive it.+ ``subject`` is the session. ``workflow_id`` scopes delivery to one+ workflow's listeners; the caller decides it."""+ payload = dict(event_data)+ session_id = payload.get("session_id")+ fields: Dict[str, Any] = {}+ if event_id:+ fields["id"] = event_id+ return WorkflowEvent(+ source=SOURCE,+ type=MESSAGE_RECEIVED_TYPE,+ subject=str(session_id) if session_id else None,+ workflow_id=workflow_id,+ data=payload,+ **fields,+ )+++async def dispatch_chat_message_received(+ event_data: Mapping[str, Any],+ *,+ workflow_id: Optional[str] = None,+ event_id: Optional[str] = None,+) -> None:+ """Signal the message to the listeners, without a broadcast."""+ from services.events.dispatch import emit++ await emit(+ chat_message_received(event_data, workflow_id=workflow_id, event_id=event_id),+ wire_routing_key=MESSAGE_WIRE_ROUTING_KEY,+ broadcast=False,+ )+++def chat_run_event(+ *,+ suffix: str,+ run_id: str,+ seq: int,+ hub_epoch: str,+ workflow_id: Optional[str],+ session_id: str,+ fields: Optional[Mapping[str, Any]] = None,+) -> WorkflowEvent:+ """One run event. ``fields`` are the suffix's own fields; the scope is+ added here and cannot be overridden by them."""+ if suffix not in RUN_SUFFIXES:+ raise ValueError(f"not a chat run event: {suffix!r}")+ data: Dict[str, Any] = {**dict(fields or {})}+ data.update(+ {+ "workflow_id": workflow_id,+ "session_id": session_id,+ "run_id": run_id,+ "seq": seq,+ "hub_epoch": hub_epoch,+ }+ )+ return WorkflowEvent(+ id=f"{run_id}:{seq}",+ source=SOURCE,+ type=f"{RUN_TYPE_PREFIX}{suffix}",+ subject=run_id,+ data=data,+ )+++def run_event_suffix(event_type: str) -> Optional[str]:+ """``started`` for ``com.opencompany.chat.run.started``; None for any+ other type."""+ if not event_type.startswith(RUN_TYPE_PREFIX):+ return None+ suffix = event_type[len(RUN_TYPE_PREFIX):]+ return suffix if suffix in RUN_SUFFIXES else None+++__all__ = [+ "MESSAGE_RECEIVED_TYPE",+ "MESSAGE_WIRE_ROUTING_KEY",+ "RUN_SUFFIXES",+ "RUN_TYPE_PREFIX",+ "RUN_WIRE_KEY",+ "SOURCE",+ "TERMINAL_SUFFIXES",+ "chat_message_received",+ "chat_run_event",+ "dispatch_chat_message_received",+ "run_event_suffix",+]diff --git a/server/services/chat/handlers.py b/server/services/chat/handlers.pynew file mode 100644index 00000000..264f5715--- /dev/null+++ b/server/services/chat/handlers.py@@ -0,0 +1,330 @@+"""Chat's WebSocket commands (docs-internal/chat_protocol.md).++Moved here from ``routers/websocket.py`` with the run runtime. Every command+that names a session checks the socket first (``services/chat/access.py``).++``send_chat_message`` saves the owner's message and dispatches it to the+workflow's chat triggers. A workflow's session takes a message only while+its deployment would read it (``delivery``: ``"now"`` while it runs,+``"queued"`` while it is paused); otherwise nothing is saved and the answer+is ``not_running``. When a deployed chat trigger will answer the session,+the message starts a run, admitted in the same transaction+(``services/chat/ledger.py``), and the response names it. The editor's+``"default"`` session is saved and dispatched to every deployment, without a+run, as it always was.+"""++from __future__ import annotations++from datetime import datetime, timezone+from typing import Any, Dict, List, Optional++from fastapi import WebSocket++from core.container import container+from core.logging import get_logger+from services.chat import ledger, reducer+from services.chat.access import ChatAccessDenied, authorize_session, session_id_of+from services.chat.events import MESSAGE_WIRE_ROUTING_KEY, dispatch_chat_message_received+from services.chat.hub import get_chat_hub+from services.chat_thread import (+ DEFAULT_SESSION,+ announce_chat_updated,+ chat_execution_id,+ clear_chat_session,+ delivery_for,+ record_chat_message,+)+from services.ws_handler_registry import ws_handler++logger = get_logger(__name__)++PROTOCOL_VERSION = 1+_DENIED = {"success": False, "error": "access_denied"}+++def _wire_message(row: Dict[str, Any]) -> Dict[str, Any]:+ """A stored message as ``get_chat_messages`` sends it."""+ message_id = row.get("uid") or f"m{row.get('id')}"+ wire: Dict[str, Any] = {+ "id": message_id,+ "legacy_id": row.get("id"),+ "role": row.get("role"),+ "kind": row.get("kind") or "text",+ "text": row.get("message"),+ "message": row.get("message"),+ "timestamp": row.get("timestamp"),+ "run_key": row.get("execution_id"),+ "run_id": row.get("run_id"),+ "parent_id": row.get("parent_uid"),+ "status": row.get("status") or "complete",+ "attachments": row.get("attachments") or [],+ "parts": row.get("parts") or {},+ "feedback": None,+ "siblings": {"index": 0, "count": 1, "ids": [message_id]},+ # Editing arrives with branches; until then no message is editable.+ "editable": False,+ }+ client_message_id = (row.get("meta") or {}).get("client_message_id")+ if client_message_id:+ wire["client_message_id"] = client_message_id+ return wire+++async def run_snapshots(database: Any, session_id: str) -> List[Dict[str, Any]]:+ """Snapshots of a session's live runs. The hub is read before the rows,+ so an event published in between carries a newer ``seq`` than the+ snapshot and still reaches a subscriber."""+ hub = get_chat_hub()+ live = hub.session_live(session_id)+ rows = await ledger.live_runs(database, session_id)+ return [{**reducer.merge_snapshot(reducer.snapshot_from_row(run), live.get(run.run_id)), "hub_epoch": hub.epoch} for run in rows]+++async def _thread_state(database: Any, session_id: str) -> Dict[str, Any]:+ from models.chat import ChatThread++ async with database.get_session() as session:+ thread = await session.get(ChatThread, session_id)+ return {+ "active_leaf_id": thread.active_leaf_uid if thread is not None else None,+ "revision": thread.revision if thread is not None else 0,+ }+++async def _answers_session(database: Any, control: Any, session_id: str) -> bool:+ """Whether a chat trigger in the deployed graph accepts this session, so+ a run started for the message will be picked up. Decided from the+ trigger's own filter (``event_waiter.build_filter``), never from a node+ type name."""+ from services import event_waiter++ graph = getattr(control, "graph_snapshot", None) or {}+ for node in graph.get("nodes") or []:+ node_type = str(node.get("type") or "")+ config = event_waiter.get_trigger_config(node_type)+ if config is None or config.event_type != MESSAGE_WIRE_ROUTING_KEY:+ continue+ if (node.get("data") or {}).get("disabled"):+ continue+ params = await database.get_node_parameters(str(node.get("id") or "")) or {}+ try:+ if event_waiter.build_filter(node_type, params)({"session_id": session_id, "message": ""}):+ return True+ except Exception: # noqa: BLE001 - a broken filter admits, as the listener's does+ return True+ return False+++@ws_handler("message")+async def handle_send_chat_message(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:+ """Save the owner's message, start its run, and dispatch it. Answers+ ``{message_id, run_id, delivery, timestamp}``; ``run_in_progress`` (with+ the live ``run_id``), ``not_running``, ``save_failed``, ``access_denied``+ or ``invalid_request`` otherwise."""+ message = data.get("message")+ if not isinstance(message, str) or not message.strip():+ return {"success": False, "error": "invalid_request", "detail": "message must be text"}+ if data.get("role", "user") != "user":+ return {"success": False, "error": "invalid_request", "detail": "send_chat_message sends the owner's messages"}+ session_id = session_id_of(data)+ timestamp = data.get("timestamp") or datetime.now(timezone.utc).isoformat()+ database = container.database()+ try:+ scope = await authorize_session(database, websocket, session_id)+ except ChatAccessDenied:+ return dict(_DENIED)++ # The scope rides the envelope's ``workflow_id``, so ``dispatch.emit``+ # signals only this workflow's listeners (without it one workflow's chat+ # fired every deployed chat trigger). The editor's "default" session+ # keeps its unscoped delivery.+ control = None+ delivery: Optional[str] = None+ if scope.workflow_id is not None:+ control = await database.get_latest_workflow_control(session_id)+ delivery = delivery_for(control)+ if delivery is None:+ return {"success": False, "error": "not_running"}+ track = scope.workflow_id is not None and await _answers_session(database, control, session_id)++ try:+ admission = await ledger.admit_message(+ database,+ session_id=session_id,+ workflow_id=scope.workflow_id,+ execution_id=chat_execution_id(control) if scope.workflow_id is not None else None,+ text=message,+ track=track,+ state="queued" if delivery == "queued" else "pending",+ client_message_id=data.get("client_message_id"),+ )+ except ledger.RunInProgress as exc:+ return {"success": False, "error": "run_in_progress", "run_id": exc.run.run_id}+ except ValueError as exc:+ return {"success": False, "error": "invalid_request", "detail": str(exc)}+ except Exception:+ logger.warning("Chat message could not be saved", session_id=session_id, exc_info=True)+ return {"success": False, "error": "save_failed"}++ row, run = admission.message, admission.run+ if admission.created:+ await announce_chat_updated(session_id, "user")+ event_data: Dict[str, Any] = {+ "message": message,+ "timestamp": timestamp,+ "session_id": session_id,+ "message_id": row["uid"],+ }+ if run is not None:+ event_data["run_id"] = run.run_id+ await dispatch_chat_message_received(+ event_data,+ workflow_id=scope.workflow_id,+ event_id=run.run_id if run is not None else None,+ )+ logger.info("Chat message dispatched", session_id=session_id, run_id=run.run_id if run is not None else None)++ response: Dict[str, Any] = {+ "success": True,+ "message": "Chat message sent",+ "timestamp": timestamp,+ "message_id": row["uid"],+ "run_id": run.run_id if run is not None else None,+ }+ if delivery is not None:+ response["delivery"] = delivery+ return response+++@ws_handler()+async def handle_get_chat_messages(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:+ """A session's messages, oldest first, the newest ``limit`` of them, with+ the thread's state and its live runs. The live generation's messages+ only, unless ``all_generations`` (Home's thread, which also shows a+ message from before a Start). A read that fails answers ``read_failed``,+ never an empty thread."""+ session_id = session_id_of(data)+ limit = data.get("limit")+ database = container.database()+ try:+ await authorize_session(database, websocket, session_id)+ except ChatAccessDenied:+ return dict(_DENIED)+ try:+ if data.get("all_generations"):+ rows = await database.read_chat_messages(session_id, limit)+ else:+ control = await database.get_latest_workflow_control(session_id)+ if control is not None and control.status == "reset":+ rows = []+ else:+ rows = await database.read_chat_messages(+ session_id, limit,+ execution_id=control.root_execution_id if control is not None else None,+ )+ thread = await _thread_state(database, session_id)+ active_runs = await run_snapshots(database, session_id)+ except Exception:+ logger.warning("Chat messages could not be read", session_id=session_id, exc_info=True)+ return {"success": False, "error": "read_failed", "session_id": session_id}+ return {+ "success": True,+ "protocol_version": PROTOCOL_VERSION,+ "session_id": session_id,+ "messages": [_wire_message(row) for row in rows],+ "thread": thread,+ "active_runs": active_runs,+ }+++@ws_handler()+async def handle_chat_subscribe(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:+ """Send the session's run events to this socket from now on. Answers+ the hub's epoch and the live runs' snapshots; events after a snapshot+ carry a higher ``seq`` than it."""+ session_id = session_id_of(data)+ database = container.database()+ try:+ await authorize_session(database, websocket, session_id)+ except ChatAccessDenied:+ return dict(_DENIED)+ hub = get_chat_hub()+ hub.subscribe(websocket, session_id)+ try:+ active_runs = await run_snapshots(database, session_id)+ except Exception:+ logger.warning("Chat runs could not be read", session_id=session_id, exc_info=True)+ return {"success": False, "error": "read_failed", "session_id": session_id, "hub_epoch": hub.epoch}+ return {"success": True, "session_id": session_id, "hub_epoch": hub.epoch, "active_runs": active_runs}+++@ws_handler()+async def handle_chat_unsubscribe(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:+ """Stop sending the session's run events to this socket."""+ session_id = session_id_of(data)+ hub = get_chat_hub()+ hub.unsubscribe(websocket, session_id)+ return {"success": True, "session_id": session_id, "hub_epoch": hub.epoch}+++@ws_handler("run_id")+async def handle_get_chat_run(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:+ """One run's snapshot, live or ended."""+ database = container.database()+ run = await ledger.get_run(database, str(data["run_id"]))+ if run is None:+ return {"success": False, "error": "not_found"}+ try:+ await authorize_session(database, websocket, run.session_id)+ except ChatAccessDenied:+ return dict(_DENIED)+ hub = get_chat_hub()+ live = hub.live_snapshot(run.run_id)+ run = await ledger.get_run(database, run.run_id) or run+ snapshot = reducer.merge_snapshot(reducer.snapshot_from_row(run), live)+ return {"success": True, "run": {**snapshot, "hub_epoch": hub.epoch}}+++@ws_handler()+async def handle_clear_chat_messages(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:+ """Clear a session's chat, every generation of it, with its runs. For a+ workflow's session the agent forgets the conversation too+ (``services.chat_thread.clear_chat_session``)."""+ session_id = session_id_of(data)+ database = container.database()+ try:+ await authorize_session(database, websocket, session_id)+ except ChatAccessDenied:+ return dict(_DENIED)+ count = await clear_chat_session(database, session_id)+ return {"success": True, "message": f"Cleared {count} chat messages", "cleared_count": count}+++@ws_handler("message", "role")+async def handle_save_chat_message(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:+ """Save one message with the given role, without dispatching it."""+ role = data["role"]+ if role not in ("user", "assistant"):+ return {"success": False, "error": "invalid_request", "detail": "role is user or assistant"}+ session_id = session_id_of(data)+ database = container.database()+ try:+ await authorize_session(database, websocket, session_id)+ except ChatAccessDenied:+ return dict(_DENIED)+ saved = await record_chat_message(database, session_id, role, data["message"])+ return {"success": bool(saved), "message": "Chat message saved" if saved else "Failed to save chat message"}+++WS_HANDLERS = {+ "send_chat_message": handle_send_chat_message,+ "get_chat_messages": handle_get_chat_messages,+ "chat_subscribe": handle_chat_subscribe,+ "chat_unsubscribe": handle_chat_unsubscribe,+ "get_chat_run": handle_get_chat_run,+ "clear_chat_messages": handle_clear_chat_messages,+ "save_chat_message": handle_save_chat_message,+}++__all__ = ["DEFAULT_SESSION", "PROTOCOL_VERSION", "WS_HANDLERS", "run_snapshots"]diff --git a/server/services/chat/hub.py b/server/services/chat/hub.pynew file mode 100644index 00000000..9ba8aa8f--- /dev/null+++ b/server/services/chat/hub.py@@ -0,0 +1,266 @@+"""Live run events for the sockets that asked for them.++A socket subscribes to a chat session (``chat_subscribe``, once the handler+has authorized it) and from then on receives that session's+``chat_run_event`` frames, in order, from its own writer task, so a slow+socket never holds up the publisher or any other socket. Run events never+go through ``StatusBroadcaster.broadcast``, which reaches every socket.++- ``seq`` counts a run's events from 1 within this process. ``epoch`` names+ the process: after a restart the counters start over and the epoch+ changes, which tells a client to take a fresh snapshot.+- An event whose ``event_key`` the run already published is dropped, so a+ retried activity publishes once. Nothing is published for a run after its+ terminal event.+- The hub keeps each live run's snapshot (``services/chat/reducer.py``), so a+ client that subscribes mid-run gets what the earlier events built.+- Each socket queues at most ``hub.subscriber_queue_size`` frames+ (``config/chat_defaults.json``). A socket that falls that far behind has+ its queue replaced by one ``opencompany.resync`` frame per session it+ follows: the client takes a fresh snapshot instead of reading a stream+ with a hole in it.++Delivery is in-process. An activity on a worker in another process+publishes into that process's hub, which no socket here reads.+"""++from __future__ import annotations++import asyncio+from collections import OrderedDict, deque+from typing import Any, Deque, Dict, Mapping, Optional, Set+from uuid import uuid4++import orjson+from starlette.websockets import WebSocketState++from core.logging import get_logger+from services.chat import reducer+from services.chat.events import RUN_WIRE_KEY, SOURCE, TERMINAL_SUFFIXES, chat_run_event+from services.events.envelope import WorkflowEvent++logger = get_logger(__name__)++#: Ended runs remembered so a late publish for one is dropped.+_ENDED_RUNS_KEPT = 2048+++def _open(websocket: Any) -> bool:+ return (+ getattr(websocket, "client_state", WebSocketState.CONNECTED) is WebSocketState.CONNECTED+ and getattr(websocket, "application_state", WebSocketState.CONNECTED) is WebSocketState.CONNECTED+ )+++class _Subscriber:+ """One socket's subscriptions and its writer."""++ def __init__(self, hub: "ChatRunHub", websocket: Any) -> None:+ self.hub = hub+ self.websocket = websocket+ self.sessions: Set[str] = set()+ self.frames: Deque[str] = deque()+ self.wake = asyncio.Event()+ self.task: Optional[asyncio.Task] = None++ def offer(self, frame: str) -> None:+ if len(self.frames) >= self.hub.queue_size:+ self.frames.clear()+ for session_id in sorted(self.sessions):+ self.frames.append(self.hub.resync_frame(session_id))+ logger.info("Chat subscriber fell behind; sent resync", sessions=len(self.sessions))+ self.frames.append(frame)+ self.wake.set()+ if self.task is None or self.task.done():+ self.task = asyncio.get_running_loop().create_task(self._write(), name="chat-hub-writer")++ async def _write(self) -> None:+ try:+ while True:+ await self.wake.wait()+ self.wake.clear()+ while self.frames:+ frame = self.frames.popleft()+ if not _open(self.websocket):+ self.hub.drop_socket(self.websocket)+ return+ await self.websocket.send_text(frame)+ except asyncio.CancelledError:+ raise+ except Exception as exc: # noqa: BLE001 - the socket is gone; forget it+ logger.debug("Chat hub send failed; dropping the socket", error=str(exc))+ self.hub.drop_socket(self.websocket)++ def close(self) -> None:+ self.frames.clear()+ if self.task is not None and not self.task.done() and self.task is not asyncio.current_task():+ self.task.cancel()+++class ChatRunHub:+ def __init__(self, *, queue_size: Optional[int] = None) -> None:+ if queue_size is None:+ from services.chat.config import hub_setting++ queue_size = hub_setting("subscriber_queue_size")+ self.queue_size = max(1, int(queue_size))+ self.epoch = f"e_{uuid4().hex[:12]}"+ # Keyed by the socket object itself, never ``id()``: a reused id+ # would hand a new socket an old one's sessions.+ self._subscribers: Dict[Any, _Subscriber] = {}+ self._sessions: Dict[str, Set[Any]] = {}+ self._seq: Dict[str, int] = {}+ self._keys: Dict[str, Set[str]] = {}+ self._live: Dict[str, Dict[str, Any]] = {}+ self._ended: "OrderedDict[str, None]" = OrderedDict()++ # ---- subscriptions -------------------------------------------------++ def subscribe(self, websocket: Any, session_id: str) -> None:+ subscriber = self._subscribers.get(websocket)+ if subscriber is None:+ subscriber = _Subscriber(self, websocket)+ self._subscribers[websocket] = subscriber+ subscriber.sessions.add(session_id)+ self._sessions.setdefault(session_id, set()).add(websocket)++ def unsubscribe(self, websocket: Any, session_id: Optional[str] = None) -> None:+ """Stop sending one session (or, without ``session_id``, any) to+ the socket."""+ subscriber = self._subscribers.get(websocket)+ if subscriber is None:+ return+ sessions = [session_id] if session_id is not None else list(subscriber.sessions)+ for session in sessions:+ subscriber.sessions.discard(session)+ members = self._sessions.get(session)+ if members is not None:+ members.discard(websocket)+ if not members:+ del self._sessions[session]+ if not subscriber.sessions:+ subscriber.close()+ del self._subscribers[websocket]++ def drop_socket(self, websocket: Any) -> None:+ """Forget a closed socket (the status broadcaster calls this on+ disconnect)."""+ self.unsubscribe(websocket)++ def subscribed(self, websocket: Any, session_id: str) -> bool:+ subscriber = self._subscribers.get(websocket)+ return subscriber is not None and session_id in subscriber.sessions++ # ---- runs ----------------------------------------------------------++ def current_seq(self, run_id: str) -> int:+ return self._seq.get(run_id, 0)++ def live_snapshot(self, run_id: str) -> Optional[Dict[str, Any]]:+ snapshot = self._live.get(run_id)+ return dict(snapshot) if snapshot is not None else None++ def session_live(self, session_id: str) -> Dict[str, Dict[str, Any]]:+ """The live snapshots of a session's runs, by run id. Read it before+ reading the runs' rows: an event published in between then reaches a+ subscriber with a ``seq`` above its snapshot's, never below."""+ return {run_id: dict(snapshot) for run_id, snapshot in self._live.items() if snapshot.get("session_id") == session_id}++ def ended(self, run_id: str) -> bool:+ return run_id in self._ended++ def publish(+ self,+ *,+ run_id: str,+ session_id: str,+ workflow_id: Optional[str],+ suffix: str,+ fields: Optional[Mapping[str, Any]] = None,+ event_key: Optional[str] = None,+ ) -> Optional[WorkflowEvent]:+ """Publish one run event to the session's subscribers. Returns the+ event, or None when it was dropped (a repeated ``event_key``, or the+ run already ended). Never raises for delivery problems: a socket that+ cannot be written to is dropped."""+ if run_id in self._ended:+ return None+ keys = self._keys.setdefault(run_id, set())+ if event_key is not None:+ if event_key in keys:+ return None+ keys.add(event_key)+ seq = self._seq.get(run_id, 0) + 1+ self._seq[run_id] = seq+ event = chat_run_event(+ suffix=suffix,+ run_id=run_id,+ seq=seq,+ hub_epoch=self.epoch,+ workflow_id=workflow_id,+ session_id=session_id,+ fields=fields,+ )+ payload = event.model_dump(mode="json", exclude_none=True)+ self._live[run_id] = reducer.apply_event(self._live.get(run_id) or reducer.empty_snapshot(run_id), payload)+ if suffix in TERMINAL_SUFFIXES:+ self._end(run_id)+ self._fan_out(session_id, orjson.dumps({"type": RUN_WIRE_KEY, "data": payload}).decode())+ return event++ def _end(self, run_id: str) -> None:+ self._seq.pop(run_id, None)+ self._keys.pop(run_id, None)+ self._live.pop(run_id, None)+ self._ended[run_id] = None+ while len(self._ended) > _ENDED_RUNS_KEPT:+ self._ended.popitem(last=False)++ def _fan_out(self, session_id: str, frame: str) -> None:+ members = self._sessions.get(session_id)+ if not members:+ return+ try:+ asyncio.get_running_loop()+ except RuntimeError:+ return+ for websocket in list(members):+ subscriber = self._subscribers.get(websocket)+ if subscriber is not None and session_id in subscriber.sessions:+ subscriber.offer(frame)++ def resync_frame(self, session_id: str) -> str:+ event = WorkflowEvent(+ id=f"resync:{uuid4().hex}",+ source=SOURCE,+ type="com.opencompany.chat.run.custom",+ subject=session_id,+ data={+ "session_id": session_id,+ "hub_epoch": self.epoch,+ "name": "opencompany.resync",+ "value": {},+ },+ )+ return orjson.dumps({"type": RUN_WIRE_KEY, "data": event.model_dump(mode="json", exclude_none=True)}).decode()+++_hub: Optional[ChatRunHub] = None+++def get_chat_hub() -> ChatRunHub:+ """The process's hub."""+ global _hub+ if _hub is None:+ _hub = ChatRunHub()+ return _hub+++def reset_chat_hub_for_tests(hub: Optional[ChatRunHub] = None) -> Optional[ChatRunHub]:+ """Swap the process hub (tests). Returns the previous one."""+ global _hub+ previous, _hub = _hub, hub+ return previous+++__all__ = ["ChatRunHub", "get_chat_hub", "reset_chat_hub_for_tests"]diff --git a/server/services/chat/ledger.py b/server/services/chat/ledger.pynew file mode 100644index 00000000..ac044935--- /dev/null+++ b/server/services/chat/ledger.py@@ -0,0 +1,532 @@+"""Chat runs: admitted with the owner's message, started and finished by the+run that answers it, and ended by the watchdog when nothing will.++**Admission** (``admit_message``) writes the owner's message and its run in+one reserved transaction (``Database.reserved_session``), after checking the+lane: one live run per session, resume runs aside, also enforced by the+partial unique index on ``chat_runs``. A message re-sent with the same+``client_message_id`` returns the first send's message and run instead of+writing a second one; the id maps to a message id that only this session can+produce (``client_message_uid``).++**Start** (``start_run``) moves ``pending``/``queued`` to ``running`` and+records the Temporal workflow that claimed it. Several chatTrigger nodes in+one graph each spawn a run for the same message: the first claims the chat+run, the others run untracked. **Finish** (``finish_run``) is accepted only+from the claimant. A retried start or finish publishes nothing new: events+carry an ``event_key`` the hub publishes once.++**The watchdog** (``sweep``) ends runs nothing will end: a pending run never+picked up (``not_delivered``, counted from the later of its creation and+this process's start, so a restart does not expire runs it held up), a+queued run whose deployment stopped or did not pick it up after resuming, a+running run past ``runs.max_running_s`` (``timed_out``), and a running run+whose Temporal workflow closed without finishing it: finished when its reply+was saved, ``interrupted`` otherwise.++Every transition commits before its event is published, so a client that+reads the run after an event never sees an older state than the event.+"""++from __future__ import annotations++import hashlib+import re+from dataclasses import dataclass+from datetime import datetime, timedelta, timezone+from typing import Any, Awaitable, Callable, Dict, List, Optional++from sqlalchemy import update+from sqlalchemy.exc import IntegrityError+from sqlmodel import select++from core.logging import get_logger+from models.chat import LIVE_STATES, TERMINAL_STATES, ChatRun+from models.database import ChatMessage+from services.chat.config import runs_setting+from services.chat.hub import get_chat_hub+from services.chat_thread import delivery_for++logger = get_logger(__name__)++PROCESS_STARTED = datetime.now(timezone.utc)++_CLIENT_MESSAGE_ID = re.compile(r"^[A-Za-z0-9_.:-]{1,100}$")+#: How long a closed Temporal workflow may leave its run open before the+#: watchdog ends it (its finish activity may still be on its way).+_CLOSED_GRACE = timedelta(seconds=60)++#: ``await status(temporal_workflow_id, temporal_run_id)`` -> True when the+#: workflow has closed, False while it runs, None when unknown.+TemporalStatus = Callable[[str, Optional[str]], Awaitable[Optional[bool]]]+++class RunInProgress(Exception):+ """The session's lane is held by ``run``."""++ def __init__(self, run: ChatRun) -> None:+ super().__init__("run_in_progress")+ self.run = run+++@dataclass(frozen=True)+class Admission:+ #: The owner's message, as ``Database.chat_row`` shapes it.+ message: Dict[str, Any]+ #: Its run, or None for a message that starts none.+ run: Optional[ChatRun]+ #: False when ``client_message_id`` matched an earlier send.+ created: bool+++def _utcnow() -> datetime:+ return datetime.now(timezone.utc)+++def _aware(moment: Optional[datetime]) -> Optional[datetime]:+ if moment is None:+ return None+ return moment if moment.tzinfo else moment.replace(tzinfo=timezone.utc)+++def new_run_id() -> str:+ from uuid import uuid4++ return f"r_{uuid4().hex}"+++def reply_uid(run_id: str) -> str:+ """The id a run's reply is saved under."""+ return f"a_{run_id}"+++def client_message_uid(session_id: str, client_message_id: Any) -> str:+ """The message id for a client's ``client_message_id`` in a session.+ Hashed with the session, so a client cannot name another chat's+ message."""+ if not isinstance(client_message_id, str) or not _CLIENT_MESSAGE_ID.match(client_message_id):+ raise ValueError("client_message_id must be 1 to 100 letters, digits or the characters _.:-")+ digest = hashlib.sha256(f"{session_id}\x00{client_message_id}".encode()).hexdigest()[:40]+ return f"m_{digest}"+++# ---- reads -------------------------------------------------------------+++async def get_run(database: Any, run_id: str) -> Optional[ChatRun]:+ async with database.get_session() as session:+ return await session.get(ChatRun, run_id)+++async def lane_run(database: Any, session_id: str) -> Optional[ChatRun]:+ """The run holding the session's lane, if any."""+ async with database.get_session() as session:+ result = await session.execute(+ select(ChatRun)+ .where(ChatRun.session_id == session_id, ChatRun.state.in_(LIVE_STATES), ChatRun.kind != "resume")+ .limit(1)+ )+ return result.scalar_one_or_none()+++async def live_runs(database: Any, session_id: Optional[str] = None) -> List[ChatRun]:+ """Live runs, oldest first: one session's, or every session's."""+ async with database.get_session() as session:+ query = select(ChatRun).where(ChatRun.state.in_(LIVE_STATES))+ if session_id is not None:+ query = query.where(ChatRun.session_id == session_id)+ result = await session.execute(query.order_by(ChatRun.created_at, ChatRun.run_id))+ return list(result.scalars().all())+++async def _message_exists(database: Any, uid: Optional[str]) -> bool:+ if not uid:+ return False+ async with database.get_session() as session:+ result = await session.execute(select(ChatMessage.id).where(ChatMessage.uid == uid))+ return result.scalar_one_or_none() is not None+++# ---- admission ---------------------------------------------------------+++async def admit_message(+ database: Any,+ *,+ session_id: str,+ workflow_id: Optional[str],+ execution_id: Optional[str],+ text: str,+ track: bool,+ state: str = "pending",+ kind: str = "message",+ message_kind: str = "text",+ client_message_id: Optional[str] = None,+ options: Optional[Dict[str, Any]] = None,+ attachments: Optional[List[Dict[str, Any]]] = None,+) -> Admission:+ """Save the owner's message and, when ``track``, the run that will answer+ it. Raises :class:`RunInProgress` when the lane is held, ``ValueError``+ for a malformed ``client_message_id``."""+ if state not in ("pending", "queued"):+ raise ValueError(f"a run is admitted pending or queued, not {state!r}")+ uid = client_message_uid(session_id, client_message_id) if client_message_id is not None else None+ meta = {"client_message_id": client_message_id} if client_message_id is not None else {}+ async with database.reserved_session() as session:+ if uid is not None:+ found = (await session.execute(select(ChatMessage).where(ChatMessage.uid == uid))).scalar_one_or_none()+ if found is not None:+ run = await session.get(ChatRun, found.run_id) if found.run_id else None+ return Admission(message=database.chat_row(found), run=run, created=False)+ run_id: Optional[str] = None+ if track:+ held = await session.execute(+ select(ChatRun)+ .where(ChatRun.session_id == session_id, ChatRun.state.in_(LIVE_STATES), ChatRun.kind != "resume")+ .limit(1)+ )+ holder = held.scalar_one_or_none()+ if holder is not None:+ # Detached first: the rollback on the way out would expire it.+ session.expunge(holder)+ raise RunInProgress(holder)+ run_id = new_run_id()+ row = await database.append_chat_row(+ session,+ session_id=session_id,+ role="user",+ message=text,+ execution_id=execution_id,+ uid=uid,+ run_id=run_id,+ kind=message_kind,+ attachments=attachments,+ meta=meta,+ )+ run: Optional[ChatRun] = None+ if run_id is not None:+ run = ChatRun(+ run_id=run_id,+ session_id=session_id,+ workflow_id=workflow_id,+ run_key=execution_id,+ kind=kind,+ state=state,+ user_message_uid=row.uid,+ reply_message_uid=reply_uid(run_id),+ options=dict(options or {}),+ )+ session.add(run)+ try:+ await session.commit()+ except IntegrityError:+ # Without a write reservation (another database) the lane index+ # is what refuses the second run.+ await session.rollback()+ holder = await lane_run(database, session_id)+ if holder is not None:+ raise RunInProgress(holder)+ raise+ return Admission(message=database.chat_row(row), run=run, created=True)+++async def post_reply(+ database: Any,+ *,+ run: ChatRun,+ node_id: str,+ text: str,+ execution_id: Optional[str],+) -> Dict[str, Any]:+ """Save a reply to the run's thread and return the row.++ The run's first reply takes the run's reply id (``a_<run id>``), the id+ its events name, so a client can match the answer it watched to the one+ saved. Another reply node in the same run gets ``a_<run id>.<n>``. A+ retry from the same node returns the row it saved. Serialized by the+ write reservation, so two reply nodes at once cannot take the same id.+ """+ async with database.reserved_session() as session:+ result = await session.execute(+ select(ChatMessage)+ .where(ChatMessage.run_id == run.run_id, ChatMessage.role == "assistant")+ .order_by(ChatMessage.id)+ )+ replies = list(result.scalars().all())+ for reply in replies:+ if (reply.meta or {}).get("node_id") == node_id:+ return database.chat_row(reply)+ uid = run.reply_message_uid or reply_uid(run.run_id)+ if replies:+ uid = f"{uid}.{len(replies) + 1}"+ row = await database.append_chat_row(+ session,+ session_id=run.session_id,+ role="assistant",+ message=text,+ execution_id=execution_id,+ uid=uid,+ run_id=run.run_id,+ meta={"node_id": node_id},+ )+ await session.commit()+ return database.chat_row(row)+++# ---- transitions -------------------------------------------------------+++def _publish(run: ChatRun, suffix: str, fields: Dict[str, Any], event_key: str) -> None:+ try:+ get_chat_hub().publish(+ run_id=run.run_id,+ session_id=run.session_id,+ workflow_id=run.workflow_id,+ suffix=suffix,+ fields=fields,+ event_key=event_key,+ )+ except Exception: # noqa: BLE001 - the row is the record; an event is a courtesy+ logger.warning("Chat run event could not be published", run_id=run.run_id, suffix=suffix, exc_info=True)+++def _iso(moment: Optional[datetime]) -> Optional[str]:+ moment = _aware(moment)+ return moment.isoformat() if moment is not None else None+++def publish_started(run: ChatRun) -> None:+ fields: Dict[str, Any] = {+ "kind": run.kind,+ "reply_message_id": run.reply_message_uid,+ "started_at": _iso(run.started_at),+ }+ if run.user_message_uid:+ fields["user_message_id"] = run.user_message_uid+ if run.parent_run_id:+ fields["parent_run_id"] = run.parent_run_id+ _publish(run, "started", fields, "started")+++def publish_terminal(run: ChatRun) -> None:+ started, finished = _aware(run.started_at), _aware(run.finished_at)+ duration_ms = int((finished - started).total_seconds() * 1000) if started and finished else 0+ result = dict(run.result or {})+ if run.state == "error":+ fields: Dict[str, Any] = {"message": run.error or "The run failed", "code": run.error_code or "run_failed"}+ for key in ("hint", "requires_user_action"):+ if result.get(key) is not None:+ fields[key] = result[key]+ _publish(run, "failed", fields, "terminal")+ return+ outcome: Dict[str, Any] = {"type": run.outcome or "success"}+ interrupts = result.pop("interrupts", None)+ if run.outcome == "interrupt":+ outcome["interrupts"] = list(interrupts or [])+ _publish(+ run,+ "finished",+ {"outcome": outcome, "result": result, "duration_ms": duration_ms, "step_count": len(run.steps or [])},+ "terminal",+ )+++async def start_run(+ database: Any,+ *,+ run_id: str,+ temporal_workflow_id: str,+ temporal_run_id: Optional[str],+) -> Optional[ChatRun]:+ """Claim a run for the Temporal workflow that answers it. Returns the+ run when this workflow holds it (also on a retry), None when another+ workflow claimed it first or it can no longer start."""+ async with database.get_session() as session:+ await session.execute(+ update(ChatRun)+ .where(ChatRun.run_id == run_id, ChatRun.state.in_(("pending", "queued")))+ .values(+ state="running",+ started_at=_utcnow(),+ temporal_workflow_id=temporal_workflow_id,+ temporal_run_id=temporal_run_id,+ )+ )+ await session.commit()+ run = await get_run(database, run_id)+ if (+ run is None+ or run.state not in ("running", "stopping")+ or run.temporal_workflow_id != temporal_workflow_id+ or run.temporal_run_id != temporal_run_id+ ):+ return None+ publish_started(run)+ return run+++async def _settle(database: Any, run: ChatRun, values: Dict[str, Any]) -> Optional[ChatRun]:+ """Move ``run`` from the state it was read in to a terminal one, then+ publish it. None when the run moved meanwhile."""+ async with database.get_session() as session:+ result = await session.execute(+ update(ChatRun).where(ChatRun.run_id == run.run_id, ChatRun.state == run.state).values(finished_at=_utcnow(), **values)+ )+ await session.commit()+ if not result.rowcount:+ return None+ settled = await get_run(database, run.run_id)+ if settled is not None and settled.state in TERMINAL_STATES:+ publish_terminal(settled)+ return settled+++async def _success_values(database: Any, run: ChatRun, outcome: str) -> Dict[str, Any]:+ if await _message_exists(database, run.reply_message_uid):+ result: Dict[str, Any] = {"reply_message_id": run.reply_message_uid}+ else:+ result = {"no_reply": True}+ return {"state": "stopped" if outcome == "stopped" else "finished", "outcome": outcome, "result": result}+++async def finish_run(+ database: Any,+ *,+ run_id: str,+ temporal_workflow_id: str,+ temporal_run_id: Optional[str],+ success: bool,+ error: Optional[str] = None,+ code: Optional[str] = None,+ hint: Optional[str] = None,+ requires_user_action: Optional[bool] = None,+) -> Optional[ChatRun]:+ """End a run its claimant answered. A run stopped meanwhile ends+ ``stopped``. Returns the ended run, or None when the caller does not+ hold it."""+ run = await get_run(database, run_id)+ if run is None or run.temporal_workflow_id != temporal_workflow_id or run.temporal_run_id != temporal_run_id:+ return None+ if run.state in TERMINAL_STATES:+ publish_terminal(run)+ return run+ if run.state not in ("running", "stopping"):+ return None+ if run.state == "stopping":+ values = await _success_values(database, run, "stopped")+ elif success:+ values = await _success_values(database, run, "success")+ else:+ details = {key: value for key, value in (("hint", hint), ("requires_user_action", requires_user_action)) if value is not None}+ values = {+ "state": "error",+ "error": (error or "The run failed")[:500],+ "error_code": (code or "run_failed")[:60],+ "result": details,+ }+ return await _settle(database, run, values)+++async def fail_run(database: Any, run: ChatRun, *, code: str, message: str) -> Optional[ChatRun]:+ """End a live run with an error. None when it moved meanwhile."""+ if run.state not in LIVE_STATES:+ return None+ return await _settle(database, run, {"state": "error", "error": message[:500], "error_code": code[:60], "result": {}})+++def publish_cleared(runs: List[ChatRun], *, code: str) -> None:+ """Tell subscribers that the runs ended with their conversation (a Reset+ or the owner's Clear deleted them)."""+ message = "The conversation was reset." if code == "reset" else "The conversation was cleared."+ for run in runs:+ _publish(run, "failed", {"message": message, "code": code}, "terminal")+++# ---- the watchdog ------------------------------------------------------+++async def sweep(+ database: Any,+ *,+ now: Optional[datetime] = None,+ process_started: datetime = PROCESS_STARTED,+ temporal_status: Optional[TemporalStatus] = None,+) -> List[str]:+ """End the live runs nothing will end. Returns their ids."""+ now = now or _utcnow()+ pickup = timedelta(seconds=runs_setting("pickup_timeout_s"))+ longest = timedelta(seconds=runs_setting("max_running_s"))+ ended: List[str] = []+ for run in await live_runs(database):+ try:+ closed = await _sweep_one(database, run, now, process_started, pickup, longest, temporal_status)+ except Exception: # noqa: BLE001 - one bad row must not stop the sweep+ logger.warning("Chat run sweep failed for a run", run_id=run.run_id, exc_info=True)+ continue+ if closed is not None:+ ended.append(closed.run_id)+ if ended:+ logger.info("Ended chat runs nothing would finish", count=len(ended))+ return ended+++async def _sweep_one(+ database: Any,+ run: ChatRun,+ now: datetime,+ process_started: datetime,+ pickup: timedelta,+ longest: timedelta,+ temporal_status: Optional[TemporalStatus],+) -> Optional[ChatRun]:+ created = max(_aware(run.created_at) or now, process_started)+ if run.state == "pending":+ if now - created > pickup:+ return await fail_run(database, run, code="not_delivered", message="The employee did not pick up this message.")+ return None+ if run.state == "queued":+ control = await database.get_latest_workflow_control(run.workflow_id) if run.workflow_id else None+ delivery = delivery_for(control)+ if delivery == "queued":+ return None+ if delivery == "now":+ resumed = max(created, _aware(getattr(control, "updated_at", None)) or created)+ if now - resumed > pickup:+ return await fail_run(database, run, code="not_delivered", message="The employee did not pick up this message.")+ return None+ return await fail_run(database, run, code="not_delivered", message="The employee stopped before reading this message.")+ started = _aware(run.started_at) or created+ if now - started > longest:+ return await fail_run(database, run, code="timed_out", message="The employee took too long to answer.")+ if temporal_status is None or not run.temporal_workflow_id or now - started < _CLOSED_GRACE:+ return None+ if await temporal_status(run.temporal_workflow_id, run.temporal_run_id) is not True:+ return None+ current = await get_run(database, run.run_id)+ if current is None or current.state not in ("running", "stopping"):+ return None+ if await _message_exists(database, current.reply_message_uid):+ return await _settle(database, current, await _success_values(database, current, "stopped" if current.state == "stopping" else "success"))+ return await fail_run(database, current, code="interrupted", message="The run ended before it answered.")+++__all__ = [+ "Admission",+ "PROCESS_STARTED",+ "RunInProgress",+ "admit_message",+ "client_message_uid",+ "fail_run",+ "finish_run",+ "get_run",+ "lane_run",+ "live_runs",+ "new_run_id",+ "post_reply",+ "publish_cleared",+ "publish_started",+ "publish_terminal",+ "reply_uid",+ "start_run",+ "sweep",+]diff --git a/server/services/chat/reducer.py b/server/services/chat/reducer.pynew file mode 100644index 00000000..a431907e--- /dev/null+++ b/server/services/chat/reducer.py@@ -0,0 +1,190 @@+"""A run's snapshot: what a client needs to draw a run without its events.++``snapshot_from_row`` shapes a ``ChatRun`` row; ``apply_event`` folds one+run event into a snapshot the way the client folds the stream, so a client+that subscribes mid-run starts from the same state the events before would+have built. The hub keeps one live snapshot per run it publishes for+(``services/chat/hub.py``); ``merge_snapshot`` lays it over the stored row.++Lifecycle, step and text events are folded. Activity events (generated UI,+approvals, sources) are not yet: nothing publishes them before generated UI+lands, and folding them needs the RFC 6902 patch code that comes with it.+"""++from __future__ import annotations++from copy import deepcopy+from datetime import datetime, timezone+from typing import Any, Dict, List, Mapping, Optional++from services.chat.events import run_event_suffix++#: Run states a client draws as working.+LIVE_STATES = frozenset({"queued", "pending", "running", "stopping"})+++def _iso(moment: Optional[datetime]) -> Optional[str]:+ if moment is None:+ return None+ if moment.tzinfo is None:+ moment = moment.replace(tzinfo=timezone.utc)+ return moment.isoformat()+++def empty_snapshot(run_id: str) -> Dict[str, Any]:+ return {+ "run_id": run_id,+ "session_id": None,+ "workflow_id": None,+ "kind": "message",+ "state": "pending",+ "seq": 0,+ "user_message_id": None,+ "reply_message_id": None,+ "parent_run_id": None,+ "created_at": None,+ "started_at": None,+ "finished_at": None,+ "steps": [],+ "segments": [],+ "activities": [],+ "interrupts": [],+ "outcome": None,+ "result": {},+ "error": None,+ "error_code": None,+ }+++def snapshot_from_row(run: Any) -> Dict[str, Any]:+ """A stored run as a snapshot. Steps come from the row (saved as they+ finish); text is not stored until the reply is, so a run read after a+ restart has no segments."""+ snapshot = empty_snapshot(run.run_id)+ outcome = {"type": run.outcome} if run.outcome else None+ if outcome is not None and run.outcome == "interrupt":+ outcome["interrupts"] = list((run.result or {}).get("interrupts") or [])+ snapshot.update(+ {+ "session_id": run.session_id,+ "workflow_id": run.workflow_id,+ "kind": run.kind,+ "state": run.state,+ "user_message_id": run.user_message_uid,+ "reply_message_id": run.reply_message_uid,+ "parent_run_id": run.parent_run_id,+ "created_at": _iso(run.created_at),+ "started_at": _iso(run.started_at),+ "finished_at": _iso(run.finished_at),+ "steps": [dict(step) for step in (run.steps or [])],+ "outcome": outcome,+ "result": {key: value for key, value in (run.result or {}).items() if key != "interrupts"},+ "error": run.error,+ "error_code": run.error_code,+ }+ )+ return snapshot+++def merge_snapshot(stored: Mapping[str, Any], live: Optional[Mapping[str, Any]]) -> Dict[str, Any]:+ """The stored run with the hub's live view laid over it. The stored state+ wins once it has ended: the row commits before its terminal event goes+ out, so the row can be ahead of the hub, never behind it."""+ merged = deepcopy(dict(stored))+ if not live:+ return merged+ for key in ("seq", "steps", "segments", "activities", "interrupts"):+ merged[key] = deepcopy(live.get(key, merged.get(key)))+ if stored.get("state") in LIVE_STATES and live.get("state") not in (None, "pending"):+ merged["state"] = live["state"]+ return merged+++def _step(snapshot: Dict[str, Any], step_id: str) -> Dict[str, Any]:+ for step in snapshot["steps"]:+ if step.get("step_id") == step_id:+ return step+ step: Dict[str, Any] = {"step_id": step_id}+ snapshot["steps"].append(step)+ return step+++def _segment(snapshot: Dict[str, Any], message_id: str) -> Dict[str, Any]:+ for segment in snapshot["segments"]:+ if segment.get("message_id") == message_id:+ return segment+ segment: Dict[str, Any] = {"message_id": message_id, "text": "", "final": None}+ snapshot["segments"].append(segment)+ return segment+++def apply_event(snapshot: Mapping[str, Any], event: Mapping[str, Any]) -> Dict[str, Any]:+ """Fold one ``chat_run_event`` envelope (as a dict) into ``snapshot``.++ Returns a new snapshot; the input is not modified. An event whose ``seq``+ is not newer than the snapshot's is a duplicate and changes nothing."""+ data = event.get("data") or {}+ suffix = run_event_suffix(str(event.get("type") or ""))+ seq = int(data.get("seq") or 0)+ if suffix is None or seq <= int(snapshot.get("seq") or 0):+ return dict(snapshot)+ out: Dict[str, Any] = deepcopy(dict(snapshot))+ out["seq"] = seq+ for key in ("session_id", "workflow_id"):+ if data.get(key) is not None:+ out[key] = data[key]++ if suffix == "started":+ out["state"] = "running"+ for key in ("kind", "parent_run_id", "user_message_id", "reply_message_id", "started_at"):+ if data.get(key) is not None:+ out[key] = data[key]+ elif suffix == "finished":+ outcome = dict(data.get("outcome") or {"type": "success"})+ out["outcome"] = outcome+ out["result"] = dict(data.get("result") or {})+ out["interrupts"] = list(outcome.get("interrupts") or [])+ out["state"] = "stopped" if outcome.get("type") == "stopped" else "finished"+ elif suffix == "failed":+ out["state"] = "error"+ out["error"] = data.get("message")+ out["error_code"] = data.get("code")+ elif suffix == "step.started":+ step = _step(out, str(data.get("step_id")))+ step.update({"name": data.get("step_name"), "state": "running"})+ if data.get("icon"):+ step["icon"] = data["icon"]+ elif suffix == "step.finished":+ step = _step(out, str(data.get("step_id")))+ step.update({"name": data.get("step_name") or step.get("name"), "state": data.get("state") or "done"})+ for key in ("detail", "duration_ms", "narration"):+ if data.get(key) is not None:+ step[key] = data[key]+ elif suffix == "text.started":+ _segment(out, str(data.get("message_id")))+ elif suffix == "text.content":+ segment = _segment(out, str(data.get("message_id")))+ segment["text"] += str(data.get("delta") or "")+ elif suffix == "text.ended":+ segment = _segment(out, str(data.get("message_id")))+ segment["final"] = bool(data.get("final"))+ elif suffix == "custom":+ name = data.get("name")+ value = data.get("value") or {}+ if name == "opencompany.segment_discarded":+ discarded = value.get("message_id")+ out["segments"] = [segment for segment in out["segments"] if segment.get("message_id") != discarded]+ elif name == "opencompany.stopping":+ out["state"] = "stopping"+ return out+++def replay(run_id: str, events: List[Mapping[str, Any]]) -> Dict[str, Any]:+ """Fold a run's events in order, from an empty snapshot."""+ snapshot = empty_snapshot(run_id)+ for event in events:+ snapshot = apply_event(snapshot, event)+ return snapshot+++__all__ = ["LIVE_STATES", "apply_event", "empty_snapshot", "merge_snapshot", "replay", "snapshot_from_row"]diff --git a/server/services/chat/watchdog.py b/server/services/chat/watchdog.pynew file mode 100644index 00000000..ebc98d10--- /dev/null+++ b/server/services/chat/watchdog.py@@ -0,0 +1,78 @@+"""The chat run watchdog: ``ledger.sweep`` every ``runs.watchdog_interval_s``+(config/chat_defaults.json).++main.py starts it once the database is up and stops it at shutdown. The+first sweep runs at once, so a run a crash left open ends once the server+is back; a pending run's wait counts from this process's start, so the+restart itself does not expire it.+"""++from __future__ import annotations++import asyncio+from typing import Any, Optional++from core.logging import get_logger+from services.chat import ledger+from services.chat.config import runs_setting++logger = get_logger(__name__)+++async def temporal_workflow_closed(workflow_id: str, run_id: Optional[str]) -> Optional[bool]:+ """Whether the Temporal workflow has closed: True once it is no longer+ running (or Temporal no longer knows it), False while it runs, None when+ Temporal cannot be asked."""+ try:+ from core.container import container++ wrapper = container.temporal_client()+ except Exception: # noqa: BLE001 - Temporal is optional+ return None+ client = getattr(wrapper, "client", None) if wrapper is not None else None+ if client is None:+ return None+ from temporalio.client import WorkflowExecutionStatus+ from temporalio.service import RPCError, RPCStatusCode++ try:+ description = await client.get_workflow_handle(workflow_id, run_id=run_id).describe()+ except RPCError as exc:+ return True if exc.status == RPCStatusCode.NOT_FOUND else None+ except Exception: # noqa: BLE001 - unknown is not closed+ return None+ return description.status != WorkflowExecutionStatus.RUNNING+++class ChatRunWatchdog:+ def __init__(self, database: Any, *, interval: Optional[float] = None) -> None:+ self.database = database+ self.interval = float(interval if interval is not None else runs_setting("watchdog_interval_s"))+ self._task: Optional[asyncio.Task] = None++ def start(self) -> None:+ if self._task is None or self._task.done():+ self._task = asyncio.get_running_loop().create_task(self._loop(), name="chat-run-watchdog")++ async def stop(self) -> None:+ task, self._task = self._task, None+ if task is None or task.done():+ return+ task.cancel()+ try:+ await task+ except asyncio.CancelledError:+ pass++ async def _loop(self) -> None:+ while True:+ try:+ await ledger.sweep(self.database, temporal_status=temporal_workflow_closed)+ except asyncio.CancelledError:+ raise+ except Exception: # noqa: BLE001 - the next sweep tries again+ logger.warning("Chat run sweep failed", exc_info=True)+ await asyncio.sleep(self.interval)+++__all__ = ["ChatRunWatchdog", "temporal_workflow_closed"]diff --git a/server/services/chat_thread.py b/server/services/chat_thread.pyindex a4e58c19..9ded1787 100644--- a/server/services/chat_thread.py+++ b/server/services/chat_thread.py@@ -14,13 +14,18 @@ Talk, and the editor shows it in its chat pane.- ``record_chat_message``: add a row stamped with the session's livegeneration (``root_execution_id``, as ``chat_execution_id`` says; theeditor's chat reads one generation, so a row without it would never show- there), then announce it.-- ``clear_chat_thread``: delete the session's rows, every generation, then- announce it when there were any.-- ``clear_chat_session``: the owner's Clear. The thread goes, and the- listeners registered with ``register_chat_cleared_listener`` forget what- it held: the Context plugin clears the workflow's conversations, so the- agent starts over with the chat.+ there), then announce it. A reply names its chat run (``run_id``) and is+ saved under the run's reply id, so a retried save writes it once. The+ owner's messages are written with their runs by ``services/chat/ledger.py``,+ which announces them the same way.+- ``clear_chat_thread``: delete the session's rows, every generation, and+ its chat runs, then announce it when there were any. Subscribers watching+ a run that was still live are told it ended (``reset``).+- ``clear_chat_session``: the owner's Clear. The thread goes (live runs end+ with ``cleared``), and the listeners registered with+ ``register_chat_cleared_listener`` forget what it held: the Context plugin+ clears the workflow's conversations, so the agent starts over with the+ chat.A workflow's thread lives as long as its generation and its workflow. AReset (every restart, Home's Apply and Turn on Talk included) clears it@@ -43,7 +48,7 @@ query that matches nothing on every message (the same reason asfrom __future__ import annotations-from typing import Any, Awaitable, Callable, List, Optional+from typing import Any, Awaitable, Callable, Dict, List, Optionalfrom core.logging import get_loggerfrom services.events.envelope import WorkflowEvent@@ -100,7 +105,9 @@ def chat_updated(*, session_id: str, role: Optional[str]) -> WorkflowEvent:)-async def _announce(session_id: str, role: Optional[str]) -> None:+async def announce_chat_updated(session_id: str, role: Optional[str]) -> None:+ """Send ``chat.updated`` for a session: a message was added (``role``)+ or the thread was cleared (``role`` None). Never raises."""from services.status_broadcaster import get_status_broadcasterevent = chat_updated(session_id=session_id, role=role)@@ -110,24 +117,48 @@ async def _announce(session_id: str, role: Optional[str]) -> None:logger.warning("chat.updated broadcast failed", session_id=session_id, exc_info=True)-async def record_chat_message(database: Any, session_id: str, role: str, message: str) -> bool:+async def record_chat_message(+ database: Any,+ session_id: str,+ role: str,+ message: str,+ *,+ uid: Optional[str] = None,+ run_id: Optional[str] = None,+ kind: str = "text",+ status: str = "complete",+ parts: Optional[Dict[str, Any]] = None,+) -> Optional[Dict[str, Any]]:"""Add ``message`` to the thread, in the session's live generation, and- announce it. False when the row could not be saved (nothing is- announced)."""+ announce it. Returns the saved row, or None when it could not be saved+ (nothing is announced). A ``uid`` already saved returns that row; the+ repeated announcement only makes clients read the thread again."""control = await database.get_latest_workflow_control(session_id)- saved = await database.add_chat_message(session_id, role, message, execution_id=chat_execution_id(control))+ saved = await database.add_chat_message(+ session_id, role, message, execution_id=chat_execution_id(control),+ uid=uid, run_id=run_id, kind=kind, status=status, parts=parts,+ )if saved:- await _announce(session_id, role)- return bool(saved)+ await announce_chat_updated(session_id, role)+ return saved or None++async def clear_chat_thread(database: Any, session_id: str, *, reason: str = "reset") -> int:+ """Delete the thread, every generation of it, with its chat runs, and+ announce it when there was anything to delete (a Reset runs one clear+ per chat node). Runs still live end for their subscribers with+ ``reason`` as the error code. Returns the messages deleted."""+ from services.chat import ledger-async def clear_chat_thread(database: Any, session_id: str) -> int:- """Delete the thread, every generation of it, and announce it when there- was anything to delete (a Reset runs one clear per chat node). Returns- the rows deleted."""+ try:+ live = await ledger.live_runs(database, session_id)+ except Exception:+ logger.warning("Could not read live chat runs before a clear", session_id=session_id, exc_info=True)+ live = []count = await database.clear_chat_messages(session_id)+ ledger.publish_cleared(live, code=reason)if count:- await _announce(session_id, None)+ await announce_chat_updated(session_id, None)return count@@ -145,7 +176,7 @@ async def clear_chat_session(database: Any, session_id: str) -> int:session has listeners to tell (``"default"`` is no workflow's). Afailing listener is logged and never fails the clear. Returns the rowsdeleted."""- count = await clear_chat_thread(database, session_id)+ count = await clear_chat_thread(database, session_id, reason="cleared")if session_id == DEFAULT_SESSION:return countfor listener in list(_CLEARED_LISTENERS):@@ -165,6 +196,7 @@ __all__ = ["DEFAULT_SESSION","WIRE_KEY","ChatClearedListener",+ "announce_chat_updated","chat_execution_id","chat_updated","clear_chat_session",diff --git a/server/services/plugin/base.py b/server/services/plugin/base.pyindex d1be9cf4..b083af18 100644--- a/server/services/plugin/base.py+++ b/server/services/plugin/base.py@@ -1150,6 +1150,9 @@ class BaseNode:"context_execution_id","context_session_id","data_scope_id",+ # The chat run this run answers (MachinaWorkflow sets it+ # for a run the owner's chat message started).+ "run_scope",):if key in context:extras[key] = context[key]diff --git a/server/services/status_broadcaster.py b/server/services/status_broadcaster.pyindex feb38ca1..2a4de622 100644--- a/server/services/status_broadcaster.py+++ b/server/services/status_broadcaster.py@@ -184,10 +184,16 @@ class StatusBroadcaster:)async def disconnect(self, websocket: WebSocket):- """Remove a WebSocket connection."""+ """Remove a WebSocket connection, then tell the disconnect listeners+ (``register_disconnect_listener``)."""async with self._lock:self._connections.discard(websocket)logger.info(f"[StatusBroadcaster] Client disconnected. Total: {len(self._connections)}")+ for listener in list(_DISCONNECT_LISTENERS):+ try:+ listener(websocket)+ except Exception:+ logger.warning("[StatusBroadcaster] Disconnect listener failed", exc_info=True)async def _refresh_all_services(self):"""Fan out plugin-registered refresh callbacks.@@ -1344,6 +1350,18 @@ def register_service_refresh(callback: _ServiceRefreshCallback) -> None:_SERVICE_REFRESH_FANOUT.register(callback)+_DisconnectListener = _typing.Callable[[WebSocket], None]+_DISCONNECT_LISTENERS: _typing.List[_DisconnectListener] = []+_DISCONNECT_FANOUT: _IdempotentList[_DisconnectListener] = _IdempotentList("disconnect", items=_DISCONNECT_LISTENERS)+++def register_disconnect_listener(listener: _DisconnectListener) -> None:+ """Call ``listener(websocket)`` after a status socket disconnects, so a+ service that keeps per-socket state (chat's run subscriptions) can drop+ it. Synchronous and must not block. Idempotent on re-import."""+ _DISCONNECT_FANOUT.register(listener)++# Global singleton instance_broadcaster: Optional[StatusBroadcaster] = Nonediff --git a/server/services/temporal/activities.py b/server/services/temporal/activities.pyindex 64339bd1..d331775a 100644--- a/server/services/temporal/activities.py+++ b/server/services/temporal/activities.py@@ -233,6 +233,7 @@ class NodeExecutionActivities:"agent_iteration","tool_call_index","tool_call_id",+ "run_scope",):if key in context:message[key] = context[key]diff --git a/server/services/temporal/agent_workflow.py b/server/services/temporal/agent_workflow.pyindex cc6dee8d..6156ab83 100644--- a/server/services/temporal/agent_workflow.py+++ b/server/services/temporal/agent_workflow.py@@ -261,6 +261,9 @@ _INHERITED_SCOPE_KEYS = ("user_id","workspace_dir","temporal_worker_pool_enabled",+ # The chat run the root run answers (services/chat/), so a tool call+ # or a delegated agent can attach what it produces to that answer.+ "run_scope",)diff --git a/server/services/temporal/worker.py b/server/services/temporal/worker.pyindex 2d97c778..3ef70d79 100644--- a/server/services/temporal/worker.py+++ b/server/services/temporal/worker.py@@ -226,6 +226,7 @@ class TemporalWorkerManager:)from services.temporal.agent_activities import collect_agent_activitiesfrom services.temporal.workspace_task_activities import reset_workspace_task_runtime+ from services.chat.activities import CHAT_RUN_ACTIVITIESfrom services.temporal.activities import (broadcast_trigger_status_activity,evaluate_trigger_filter_activity,@@ -282,6 +283,7 @@ class TemporalWorkerManager:pause_workflow_on_failure_activity,record_run_completion_activity,reset_workspace_task_runtime,+ *CHAT_RUN_ACTIVITIES,store_node_output_activity,*per_type,*agent_activities,@@ -741,6 +743,7 @@ async def run_standalone_worker()from services.temporal.agent_activities import collect_agent_activitiesfrom services.temporal.workspace_task_activities import reset_workspace_task_runtime+ from services.chat.activities import CHAT_RUN_ACTIVITIESfrom services.temporal.agent_workflow import (AgentWorkflow,DelegatedTaskWorkflow,@@ -767,6 +770,7 @@ async def run_standalone_worker(pause_workflow_on_failure_activity,record_run_completion_activity,reset_workspace_task_runtime,+ *CHAT_RUN_ACTIVITIES,store_node_output_activity,*registered_plugin_activities,*registered_agent_activities,@@ -824,6 +828,7 @@ async def create_worker()from services.temporal.agent_activities import collect_agent_activitiesfrom services.temporal.workspace_task_activities import reset_workspace_task_runtime+ from services.chat.activities import CHAT_RUN_ACTIVITIESfrom services.temporal.agent_workflow import (AgentWorkflow,DelegatedTaskWorkflow,@@ -849,6 +854,7 @@ async def create_worker(pause_workflow_on_failure_activity,record_run_completion_activity,reset_workspace_task_runtime,+ *CHAT_RUN_ACTIVITIES,store_node_output_activity,*registered_plugin_activities,*registered_agent_activities,diff --git a/server/services/temporal/workflow.py b/server/services/temporal/workflow.pyindex fbf6bce9..5166d54a 100644--- a/server/services/temporal/workflow.py+++ b/server/services/temporal/workflow.py@@ -42,6 +42,17 @@ CONDITIONAL_EDGES_PATCH = "machina-conditional-edges-v1"# it (which end without that activity) still replay.RUN_RECORD_PATCH = "machina-run-record-v1"+# A run that the owner's chat message spawned claims its chat run when it+# starts and finishes it at its end (services/chat/ledger.py), so Talk and+# the editor's chat follow the answer while it is worked on. Gated: the two+# activities are new commands.+CHAT_RUN_PATCH = "machina-chat-run-v1"+# The source and type services/chat/events.py gives the owner's message. A+# chat run id is trusted only from an event with both, never from what+# another producer put in its event data.+CHAT_MESSAGE_SOURCE = "opencompany://services/chat"+CHAT_MESSAGE_TYPE = "com.opencompany.chat.message.received"+# Config handles - nodes connecting via these are config nodes (not executed)# AI Agent handles: input-context, input-tools, input-model, input-task, input-teammates# Zeenie handles: input-skill, input-tools@@ -304,6 +315,75 @@ class MachinaWorkflow:except Exception as exc: # noqa: BLE001 — cosmetic relative to the run resultworkflow.logger.warning(f"run-record activity failed (non-fatal): {exc}")+ @staticmethod+ def _chat_run_id(nodes: List[Dict[str, Any]]) -> Optional[str]:+ """The chat run the firing trigger's event started, or None.++ Read from the event envelope the trigger listener attached to the+ trigger's output, and only when the chat service produced it. Depends+ on the workflow input alone, so it is deterministic."""+ for node in nodes:+ if not node.get("_pre_executed"):+ continue+ output = node.get("_trigger_output")+ if not isinstance(output, dict) or output.get("not_triggered"):+ continue+ envelope = output.get("_event_envelope")+ if not isinstance(envelope, dict):+ continue+ if envelope.get("source") != CHAT_MESSAGE_SOURCE or envelope.get("type") != CHAT_MESSAGE_TYPE:+ continue+ data = envelope.get("data")+ run_id = data.get("run_id") if isinstance(data, dict) else None+ if isinstance(run_id, str) and run_id:+ return run_id+ return None++ async def _start_chat_run(self, run_id: str, session_id: str) -> Optional[Dict[str, Any]]:+ """Claim the chat run; returns the run scope node contexts carry, or+ None when this run does not hold it (another chat trigger in the+ graph claimed it, or it ended) and runs untracked."""+ info = workflow.info()+ try:+ claimed = await workflow.execute_activity(+ "chat_run.start",+ {"run_id": run_id, "temporal_workflow_id": info.workflow_id, "temporal_run_id": info.run_id},+ start_to_close_timeout=timedelta(seconds=30),+ retry_policy=DEFAULT_ACTIVITY_RETRY,+ )+ except Exception as exc: # noqa: BLE001 — the answer matters more than its tracking+ workflow.logger.warning(f"chat run start failed (running untracked): {exc}")+ return None+ if not (isinstance(claimed, dict) and claimed.get("claimed")):+ return None+ return {"run_id": run_id, "session_id": session_id}++ async def _finish_chat_run(self, run_scope: Dict[str, Any], success: bool, errors: List[Dict]) -> None:+ """End the claimed chat run with the run's outcome. Non-fatal: the+ watchdog ends a run whose finish never arrived."""+ info = workflow.info()+ payload: Dict[str, Any] = {+ "run_id": run_scope["run_id"],+ "temporal_workflow_id": info.workflow_id,+ "temporal_run_id": info.run_id,+ "success": success,+ }+ if errors:+ failure = errors[0] or {}+ payload["error"] = str(failure.get("error") or "run_failed")[:500]+ for key in ("hint", "requires_user_action"):+ if failure.get(key) is not None:+ payload[key] = failure[key]+ try:+ await workflow.execute_activity(+ "chat_run.finish",+ payload,+ start_to_close_timeout=timedelta(seconds=30),+ retry_policy=DEFAULT_ACTIVITY_RETRY,+ )+ except Exception as exc: # noqa: BLE001 — cosmetic relative to the run result+ workflow.logger.warning(f"chat run finish failed (non-fatal): {exc}")+@workflow.signalasync def on_event(self, event_payload: Dict[str, Any]) -> None:"""Wave 12 A7: receive an event from ``services.events.dispatch.emit``.@@ -490,6 +570,15 @@ class MachinaWorkflow:workflow.logger.info(f"Pre-executed: {pre_executed_count}, To execute: {len(node_map) - pre_executed_count}")+ # The owner's chat message started a chat run: claim it. The patch+ # check comes after the eligibility test, which reads only the+ # workflow input, so it is deterministic.+ run_scope: Optional[Dict[str, Any]] = None+ chat_run_id = self._chat_run_id(nodes)+ if chat_run_id is not None and workflow.patched(CHAT_RUN_PATCH):+ await self._wait_until_resumed()+ run_scope = await self._start_chat_run(chat_run_id, session_id)+# 5. Retry policy for node activities. Wave 12 D1: imported# shared constant declares ``non_retryable_error_types`` so# NodeUserError fails fast instead of burning 3 retries on@@ -596,6 +685,8 @@ class MachinaWorkflow:)if "user_id" in workflow_data:context["user_id"] = workflow_data.get("user_id")+ if run_scope is not None:+ context["run_scope"] = dict(run_scope)context["temporal_worker_pool_enabled"] = bool(frozen_routing.get("worker_pool_enabled"))@@ -722,6 +813,8 @@ class MachinaWorkflow:# Build final resultsuccess = len(errors) == 0 and len(completed) == len(node_map)+ if run_scope is not None:+ await self._finish_chat_run(run_scope, success, errors)if errors:await self._pause_deployment_on_failure(workflow_data, nodes, errors)await self._record_run_completion(workflow_data, nodes, success)diff --git a/server/tests/nodes/test_chat_reply.py b/server/tests/nodes/test_chat_reply.pyindex fd6fd6d4..70653a10 100644--- a/server/tests/nodes/test_chat_reply.py+++ b/server/tests/nodes/test_chat_reply.py@@ -1,6 +1,8 @@"""Reply in Chat (``chatReply``): posts an answer to the thread of theworkflow it runs in, as the assistant, in the live generation, and says so-on ``chat.updated``; posts nothing for an empty answer or NO_REPLY."""+on ``chat.updated``; posts nothing for an empty answer or NO_REPLY. In a run+the owner's chat message started (``run_scope``), the answer is that chat+run's reply (tests/services/chat/ cover the ledger itself)."""from __future__ import annotations@@ -22,20 +24,26 @@ def thread(harness):frames.append(message)harness.database.get_latest_workflow_control = AsyncMock(return_value=SimpleNamespace(status="running", root_execution_id="gen-2"))- harness.database.add_chat_message = AsyncMock(return_value=True)+ harness.database.add_chat_message = AsyncMock(return_value={"uid": "m_1", "message": "saved"})with patch("services.status_broadcaster.get_status_broadcaster", return_value=Broadcaster()):yield SimpleNamespace(database=harness.database, frames=frames)-async def _reply(harness, message, *, workflow_id="wf-1"):- return await harness.execute("chatReply", {"message": message}, context=harness.build_context(workflow_id=workflow_id))+async def _reply(harness, message, *, workflow_id="wf-1", run_scope=None):+ context = harness.build_context(workflow_id=workflow_id)+ if run_scope is not None:+ context["run_scope"] = run_scope+ return await harness.execute("chatReply", {"message": message}, context=context)async def test_the_answer_lands_in_the_workflows_thread(harness, thread):result = await _reply(harness, " Booked you for 3pm. ")harness.assert_envelope(result, success=True)- assert result["result"] == {"posted": True, "message": "Booked you for 3pm."}- thread.database.add_chat_message.assert_awaited_once_with("wf-1", "assistant", "Booked you for 3pm.", execution_id="gen-2")+ assert result["result"] == {"posted": True, "message": "Booked you for 3pm.", "message_id": "m_1", "run_id": None}+ thread.database.add_chat_message.assert_awaited_once_with(+ "wf-1", "assistant", "Booked you for 3pm.", execution_id="gen-2",+ uid=None, run_id=None, kind="text", status="complete", parts=None,+ )[frame] = thread.framesassert frame["type"] == "chat.updated"assert frame["data"]["type"] == "com.opencompany.chat.updated"@@ -66,12 +74,42 @@ async def test_it_needs_a_saved_workflow(harness, thread):async def test_a_failed_save_is_an_error_and_announces_nothing(harness, thread):- thread.database.add_chat_message = AsyncMock(return_value=False)+ thread.database.add_chat_message = AsyncMock(return_value=None)result = await _reply(harness, "hello")harness.assert_envelope(result, success=False)assert thread.frames == []+async def test_in_a_chat_run_the_answer_is_the_runs_reply(harness, thread):+ run = SimpleNamespace(run_id="r_1", session_id="wf-1", reply_message_uid="a_r_1")+ with (+ patch("services.chat.ledger.get_run", AsyncMock(return_value=run)) as get_run,+ patch("services.chat.ledger.post_reply", AsyncMock(return_value={"uid": "a_r_1"})) as post_reply,+ ):+ result = await _reply(harness, "Booked.", run_scope={"run_id": "r_1", "session_id": "wf-1"})+ harness.assert_envelope(result, success=True)+ assert result["result"] == {"posted": True, "message": "Booked.", "message_id": "a_r_1", "run_id": "r_1"}+ assert get_run.await_args.args[1] == "r_1"+ kwargs = post_reply.await_args.kwargs+ assert (kwargs["run"], kwargs["text"], kwargs["execution_id"]) == (run, "Booked.", "gen-2")+ assert kwargs["node_id"]+ thread.database.add_chat_message.assert_not_awaited()+ assert [frame["data"]["data"]["role"] for frame in thread.frames] == ["assistant"]+++async def test_a_run_whose_conversation_was_cleared_posts_nothing(harness, thread):+ with (+ patch("services.chat.ledger.get_run", AsyncMock(return_value=None)),+ patch("services.chat.ledger.post_reply", AsyncMock()) as post_reply,+ ):+ result = await _reply(harness, "Too late.", run_scope={"run_id": "r_gone", "session_id": "wf-1"})+ harness.assert_envelope(result, success=True)+ assert result["result"] == {"posted": False, "message": None, "message_id": None, "run_id": "r_gone"}+ post_reply.assert_not_awaited()+ thread.database.add_chat_message.assert_not_awaited()+ assert thread.frames == []++def test_it_is_a_sink_the_hire_builder_may_use():import nodes # noqa: F401from services.node_allowlist import CONFIG_PATH, NodeAllowlistServicediff --git a/server/tests/services/chat/__init__.py b/server/tests/services/chat/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/server/tests/services/chat/_helpers.py b/server/tests/services/chat/_helpers.pynew file mode 100644index 00000000..719a7acb--- /dev/null+++ b/server/tests/services/chat/_helpers.py@@ -0,0 +1,68 @@+"""Helpers for the chat run tests."""++from __future__ import annotations++from types import SimpleNamespace+from typing import Any, Dict, List, Optional++from starlette.websockets import WebSocketState++from models.database import WorkflowControlExecution+++class FakeSocket:+ """A status socket: records what the hub writes to it."""++ def __init__(self, *, path: str = "/ws/status", user_id: Optional[str] = None) -> None:+ self.scope = {"path": path}+ self.state = SimpleNamespace(user_id=user_id)+ self.client_state = WebSocketState.CONNECTED+ self.application_state = WebSocketState.CONNECTED+ self.sent: List[str] = []++ async def send_text(self, text: str) -> None:+ self.sent.append(text)++ def events(self) -> List[Dict[str, Any]]:+ import json++ return [json.loads(text)["data"] for text in self.sent]+++async def add_control(+ database,+ workflow_id: str,+ status: str,+ *,+ generation: int = 1,+ nodes: Optional[List[Dict[str, Any]]] = None,+):+ """A control generation, its graph holding ``nodes``."""+ async with database.get_session() as session:+ session.add(+ WorkflowControlExecution(+ id=f"{workflow_id}-{generation}",+ workflow_id=workflow_id,+ generation=generation,+ execution_id=f"gen-{generation}",+ root_execution_id=f"gen-{generation}",+ graph_hash="0" * 64,+ graph_snapshot={"nodes": list(nodes or []), "edges": []},+ status=status,+ idempotency_key=f"k-{generation}",+ )+ )+ await session.commit()+++def chat_trigger(workflow_id: str, **data: Any) -> Dict[str, Any]:+ return {"id": f"{workflow_id}:chatTrigger:1", "type": "chatTrigger", "data": {"label": "Chat", **data}}+++async def talking(database, workflow_id: str = "wf", status: str = "running", generation: int = 1) -> None:+ """A deployed workflow whose chat trigger answers its own session."""+ await add_control(database, workflow_id, status, generation=generation, nodes=[chat_trigger(workflow_id)])+++def chat_updates(frames) -> List[Dict[str, Any]]:+ return [frame["data"]["data"] for frame in frames if frame["type"] == "chat.updated"]diff --git a/server/tests/services/chat/conftest.py b/server/tests/services/chat/conftest.pynew file mode 100644index 00000000..c6b61487--- /dev/null+++ b/server/tests/services/chat/conftest.py@@ -0,0 +1,85 @@+"""Fixtures for the chat run tests: a real database on a throwaway file, the+chat handlers' container pointed at it, a fresh hub, and the broadcaster,+dispatch and Clear listeners captured."""++from __future__ import annotations++import importlib.util+import sys+import uuid+from pathlib import Path+from types import SimpleNamespace+from typing import Any, Dict, List++import pytest+++async def database(tmp_path: Path):+ """A real ``core.database.Database`` (the root conftest stubs the module,+ so the real one is loaded privately)."""+ module_name = f"tests._chat_database_{uuid.uuid4().hex}"+ spec = importlib.util.spec_from_file_location(module_name, Path(__file__).resolve().parents[3] / "core" / "database.py")+ assert spec is not None and spec.loader is not None+ module = importlib.util.module_from_spec(spec)+ sys.modules[module_name] = module+ spec.loader.exec_module(module)+ db_path = tmp_path / f"chat-{uuid.uuid4().hex}.db"+ db = module.Database(+ SimpleNamespace(+ database_url=f"sqlite+aiosqlite:///{db_path.as_posix()}",+ database_echo=False,+ database_pool_size=5,+ database_max_overflow=5,+ )+ )+ await db.startup()+ try:+ yield db+ finally:+ await db.shutdown()+ sys.modules.pop(module_name, None)+++def hub():+ from services.chat.hub import ChatRunHub, reset_chat_hub_for_tests++ fresh = ChatRunHub(queue_size=8)+ previous = reset_chat_hub_for_tests(fresh)+ try:+ yield fresh+ finally:+ reset_chat_hub_for_tests(previous)+++def frames(monkeypatch):+ """Everything the status broadcaster would have sent to every socket."""+ import services.status_broadcaster as status_broadcaster++ sent: List[Dict[str, Any]] = []++ class Broadcaster:+ async def broadcast(self, message):+ sent.append(message)++ monkeypatch.setattr(status_broadcaster, "get_status_broadcaster", lambda: Broadcaster())+ return sent+++def chat(monkeypatch, database, hub, frames):+ """The chat handlers on the test database, with the dispatch captured."""+ import services.chat.handlers as handlers+ from services import chat_thread++ dispatched: List[Dict[str, Any]] = []++ async def dispatch(event_data, *, workflow_id=None, event_id=None):+ dispatched.append({"data": dict(event_data), "workflow_id": workflow_id, "event_id": event_id})++ monkeypatch.setattr(handlers, "container", SimpleNamespace(database=lambda: database))+ monkeypatch.setattr(handlers, "dispatch_chat_message_received", dispatch)+ monkeypatch.setattr(chat_thread, "_CLEARED_LISTENERS", [])+ return SimpleNamespace(database=database, hub=hub, frames=frames, dispatched=dispatched, handlers=handlers)diff --git a/server/tests/services/chat/test_handlers.py b/server/tests/services/chat/test_handlers.pynew file mode 100644index 00000000..f0a53a83--- /dev/null+++ b/server/tests/services/chat/test_handlers.py@@ -0,0 +1,160 @@+"""Chat's WebSocket commands (services/chat/handlers.py) with runs: a message+a deployed chat trigger will answer starts a run named in the response and+dispatched as the event id; a second send while it is live is refused; a+re-sent message returns the first send; reads return the v2 shape with the+thread and live runs; subscribing returns snapshots and delivers events; and+only the workflow's owner may use its session."""++from __future__ import annotations++import asyncio++from services.chat import ledger+from tests.services.chat._helpers import FakeSocket, add_control, chat_trigger, chat_updates, talking+++async def send(chat, message="Book Saturday", session_id="wf", socket=None, **extra):+ return await chat.handlers.handle_send_chat_message({"message": message, "session_id": session_id, "timestamp": "t", **extra}, socket)+++async def test_a_message_the_employee_will_answer_starts_a_run(chat):+ await talking(chat.database)+ result = await send(chat)+ assert result["success"] is True and result["delivery"] == "now"+ run = await ledger.get_run(chat.database, result["run_id"])+ assert run.state == "pending" and run.user_message_uid == result["message_id"]+ [dispatched] = chat.dispatched+ # The run id is the event id, so the run a listener spawns has a predictable id.+ assert dispatched["event_id"] == result["run_id"]+ assert dispatched["workflow_id"] == "wf"+ assert dispatched["data"] == {+ "message": "Book Saturday",+ "timestamp": "t",+ "session_id": "wf",+ "message_id": result["message_id"],+ "run_id": result["run_id"],+ }+ assert chat_updates(chat.frames) == [{"workflow_id": "wf", "session_id": "wf", "role": "user"}]+++async def test_a_paused_employee_queues_the_run(chat):+ await talking(chat.database, status="paused")+ result = await send(chat)+ assert result["delivery"] == "queued"+ assert (await ledger.get_run(chat.database, result["run_id"])).state == "queued"+++async def test_a_second_message_while_a_run_is_live_is_refused(chat):+ await talking(chat.database)+ first = await send(chat)+ second = await send(chat, "And Sunday?")+ assert second == {"success": False, "error": "run_in_progress", "run_id": first["run_id"]}+ assert len(chat.dispatched) == 1+ assert [row["message"] for row in await chat.database.get_chat_messages("wf")] == ["Book Saturday"]+++async def test_a_resent_message_returns_the_first_send_and_dispatches_once(chat):+ await talking(chat.database)+ first = await send(chat, client_message_id="c-1")+ again = await send(chat, client_message_id="c-1")+ assert (again["message_id"], again["run_id"]) == (first["message_id"], first["run_id"])+ assert len(chat.dispatched) == 1+ bad = await send(chat, client_message_id="not ok")+ assert bad["error"] == "invalid_request"+++async def test_no_run_when_no_deployed_chat_trigger_answers_the_session(chat):+ # A graph without a chat trigger, then one whose trigger listens to+ # another session, then a disabled one: the message is saved, no run.+ await add_control(chat.database, "wf", "running", nodes=[{"id": "wf:console:1", "type": "console", "data": {}}])+ assert (await send(chat))["run_id"] is None+ await add_control(chat.database, "wf2", "running", nodes=[chat_trigger("wf2")])+ await chat.database.save_node_parameters("wf2:chatTrigger:1", {"session_id": "elsewhere"})+ assert (await send(chat, session_id="wf2"))["run_id"] is None+ await add_control(chat.database, "wf3", "running", nodes=[chat_trigger("wf3", disabled=True)])+ assert (await send(chat, session_id="wf3"))["run_id"] is None+ assert len(await chat.database.get_chat_messages("wf")) == 1+++async def test_the_owner_writes_only_their_own_messages(chat):+ await talking(chat.database)+ assert (await send(chat, role="assistant"))["error"] == "invalid_request"+ assert (await send(chat, message=" "))["error"] == "invalid_request"+ assert chat.dispatched == []+++async def test_only_the_workflows_owner_may_use_its_chat(chat):+ await talking(chat.database)+ await chat.database.save_workflow("wf", "Salon", "Salon_1", {"nodes": [], "edges": [], "owner_id": "alice"})+ stranger = FakeSocket(user_id="mallory")+ owner = FakeSocket(user_id="alice")+ worker = FakeSocket(path="/ws/internal")+ for socket in (stranger, worker):+ assert (await send(chat, socket=socket))["error"] == "access_denied"+ assert (await chat.handlers.handle_get_chat_messages({"session_id": "wf"}, socket))["error"] == "access_denied"+ assert (await chat.handlers.handle_chat_subscribe({"session_id": "wf"}, socket))["error"] == "access_denied"+ assert (await chat.handlers.handle_clear_chat_messages({"session_id": "wf"}, socket))["error"] == "access_denied"+ assert chat.dispatched == []+ assert (await send(chat, socket=owner))["success"] is True+++async def test_reading_the_thread_returns_the_v2_shape(chat):+ await talking(chat.database)+ sent = await send(chat, client_message_id="c-7")+ result = await chat.handlers.handle_get_chat_messages({"session_id": "wf", "all_generations": True}, None)+ assert result["success"] is True and result["protocol_version"] == 1+ [message] = result["messages"]+ assert message["id"] == sent["message_id"] and message["run_id"] == sent["run_id"]+ assert (message["text"], message["message"], message["kind"], message["status"]) == ("Book Saturday",) * 2 + ("text", "complete")+ assert message["client_message_id"] == "c-7" and message["parent_id"] is None+ assert message["siblings"] == {"index": 0, "count": 1, "ids": [message["id"]]}+ assert result["thread"] == {"active_leaf_id": message["id"], "revision": 1}+ [run] = result["active_runs"]+ assert (run["run_id"], run["state"], run["user_message_id"], run["seq"]) == (sent["run_id"], "pending", message["id"], 0)+++async def test_a_failed_read_is_never_an_empty_thread(chat, monkeypatch):+ async def broken(*_args, **_kwargs):+ raise RuntimeError("database is locked")++ monkeypatch.setattr(chat.database, "read_chat_messages", broken)+ result = await chat.handlers.handle_get_chat_messages({"session_id": "wf", "all_generations": True}, None)+ assert result == {"success": False, "error": "read_failed", "session_id": "wf"}+++async def test_subscribing_returns_snapshots_and_then_delivers_events(chat):+ await talking(chat.database)+ sent = await send(chat)+ socket = FakeSocket()+ subscribed = await chat.handlers.handle_chat_subscribe({"session_id": "wf"}, socket)+ assert subscribed["hub_epoch"] == chat.hub.epoch+ assert [run["run_id"] for run in subscribed["active_runs"]] == [sent["run_id"]]++ await ledger.start_run(chat.database, run_id=sent["run_id"], temporal_workflow_id="tw", temporal_run_id="tr")+ await asyncio.sleep(0)+ [event] = socket.events()+ assert event["type"] == "com.opencompany.chat.run.started" and event["data"]["seq"] == 1++ snapshot = await chat.handlers.handle_get_chat_run({"run_id": sent["run_id"]}, socket)+ assert (snapshot["run"]["state"], snapshot["run"]["seq"]) == ("running", 1)++ await chat.handlers.handle_chat_unsubscribe({"session_id": "wf"}, socket)+ await ledger.finish_run(chat.database, run_id=sent["run_id"], temporal_workflow_id="tw", temporal_run_id="tr", success=True)+ await asyncio.sleep(0)+ assert len(socket.events()) == 1+ assert (await chat.handlers.handle_get_chat_run({"run_id": "r_unknown"}, socket))["error"] == "not_found"+++async def test_the_owners_clear_ends_a_live_run(chat):+ await talking(chat.database)+ sent = await send(chat)+ socket = FakeSocket()+ await chat.handlers.handle_chat_subscribe({"session_id": "wf"}, socket)+ cleared = await chat.handlers.handle_clear_chat_messages({"session_id": "wf"}, socket)+ assert cleared["cleared_count"] == 1+ await asyncio.sleep(0)+ [event] = socket.events()+ assert (event["data"]["run_id"], event["data"]["code"]) == (sent["run_id"], "cleared")+ assert await ledger.get_run(chat.database, sent["run_id"]) is None+ # The lane is free: a new conversation starts.+ assert (await send(chat, "Hello again"))["run_id"] is not Nonediff --git a/server/tests/services/chat/test_hub.py b/server/tests/services/chat/test_hub.pynew file mode 100644index 00000000..38b42129--- /dev/null+++ b/server/tests/services/chat/test_hub.py@@ -0,0 +1,208 @@+"""Run events (services/chat/hub.py, reducer.py, events.py): delivered only to+the sockets subscribed to the session, in order, one ``seq`` per run, once+per ``event_key``, nothing after the terminal event, a resync for a socket+that falls behind; and the snapshot folded from a run's events matches the+protocol fixture."""++from __future__ import annotations++import asyncio+import json+from pathlib import Path++import pytest+from starlette.websockets import WebSocketState++from services.chat import reducer+from services.chat.events import RUN_SUFFIXES, chat_run_event, run_event_suffix+from services.chat.hub import ChatRunHub+from tests.services.chat._helpers import FakeSocket++FIXTURE = Path(__file__).resolve().parents[4] / "client" / "src" / "features" / "chat" / "__fixtures__" / "saturday-booking.events.json"+++def publish(hub, suffix, *, run_id="r_1", session_id="wf", **fields):+ event_key = fields.pop("event_key", None)+ return hub.publish(run_id=run_id, session_id=session_id, workflow_id=session_id, suffix=suffix, fields=fields, event_key=event_key)+++async def drained():+ for _ in range(3):+ await asyncio.sleep(0)+++async def test_a_subscriber_gets_its_sessions_events_in_order():+ hub = ChatRunHub(queue_size=16)+ mine, other = FakeSocket(), FakeSocket()+ hub.subscribe(mine, "wf")+ hub.subscribe(other, "another")+ publish(hub, "started", kind="message")+ publish(hub, "step.started", step_id="s1", step_name="Checking")+ publish(hub, "finished", outcome={"type": "success"}, result={"no_reply": True})+ await drained()+ events = mine.events()+ assert [event["type"] for event in events] == [+ "com.opencompany.chat.run.started",+ "com.opencompany.chat.run.step.started",+ "com.opencompany.chat.run.finished",+ ]+ assert [event["data"]["seq"] for event in events] == [1, 2, 3]+ assert [event["id"] for event in events] == ["r_1:1", "r_1:2", "r_1:3"]+ assert all(event["subject"] == "r_1" and event["data"]["hub_epoch"] == hub.epoch for event in events)+ # Scope rides in data, never as top-level extension attributes.+ assert all("workflow_id" not in event for event in events)+ assert other.sent == []+++async def test_repeated_keys_and_events_after_the_end_are_dropped():+ hub = ChatRunHub(queue_size=16)+ socket = FakeSocket()+ hub.subscribe(socket, "wf")+ assert publish(hub, "started", event_key="started") is not None+ assert publish(hub, "started", event_key="started") is None+ publish(hub, "failed", message="no", code="x", event_key="terminal")+ assert publish(hub, "step.started", step_id="late", step_name="Late") is None+ assert hub.ended("r_1") and hub.live_snapshot("r_1") is None and hub.current_seq("r_1") == 0+ await drained()+ assert [event["data"]["seq"] for event in socket.events()] == [1, 2]+++async def test_unsubscribing_and_disconnecting_stop_delivery():+ hub = ChatRunHub(queue_size=16)+ socket = FakeSocket()+ hub.subscribe(socket, "wf")+ hub.subscribe(socket, "second")+ hub.unsubscribe(socket, "wf")+ assert not hub.subscribed(socket, "wf") and hub.subscribed(socket, "second")+ publish(hub, "started")+ publish(hub, "started", run_id="r_2", session_id="second")+ await drained()+ assert [event["data"]["run_id"] for event in socket.events()] == ["r_2"]+ hub.drop_socket(socket)+ assert not hub.subscribed(socket, "second")+++async def test_a_closed_socket_is_dropped_when_written_to():+ hub = ChatRunHub(queue_size=16)+ socket = FakeSocket()+ hub.subscribe(socket, "wf")+ socket.client_state = WebSocketState.DISCONNECTED+ publish(hub, "started")+ await drained()+ assert socket.sent == [] and not hub.subscribed(socket, "wf")+++async def test_a_socket_that_falls_behind_gets_a_resync():+ hub = ChatRunHub(queue_size=3)+ socket = FakeSocket()+ hub.subscribe(socket, "wf")+ # No await between publishes: the writer cannot drain.+ for n in range(5):+ publish(hub, "step.started", step_id=f"s{n}", step_name="Step")+ await drained()+ events = socket.events()+ assert events[0]["data"]["name"] == "opencompany.resync"+ assert events[0]["data"]["session_id"] == "wf"+ # What follows the resync is newer than anything it replaced.+ assert [event["data"]["seq"] for event in events[1:]] == [4, 5]+++async def test_the_disconnect_listener_drops_subscriptions(monkeypatch, hub):+ import services.chat # noqa: F401 - registers the listener+ from services.status_broadcaster import StatusBroadcaster++ socket = FakeSocket()+ hub.subscribe(socket, "wf")+ await StatusBroadcaster().disconnect(socket)+ assert not hub.subscribed(socket, "wf")+++async def test_a_session_snapshot_reads_the_live_runs():+ hub = ChatRunHub(queue_size=16)+ publish(hub, "started", kind="message", reply_message_id="a_r_1")+ publish(hub, "text.started", message_id="r_1.1.1")+ publish(hub, "text.content", message_id="r_1.1.1", delta="Hel")+ publish(hub, "text.content", message_id="r_1.1.1", delta="lo")+ publish(hub, "started", run_id="r_9", session_id="elsewhere")+ live = hub.session_live("wf")+ assert list(live) == ["r_1"]+ assert live["r_1"]["seq"] == 4 and live["r_1"]["state"] == "running"+ assert live["r_1"]["segments"] == [{"message_id": "r_1.1.1", "text": "Hello", "final": None}]+++# ----- events -----+++def test_run_events_are_only_the_protocols():+ with pytest.raises(ValueError):+ chat_run_event(suffix="exploded", run_id="r", seq=1, hub_epoch="e", workflow_id=None, session_id="s")+ assert run_event_suffix("com.opencompany.chat.run.step.finished") == "step.finished"+ assert run_event_suffix("com.opencompany.chat.updated") is None+ # Fields can never override the scope.+ event = chat_run_event(suffix="started", run_id="r", seq=3, hub_epoch="e", workflow_id="w", session_id="s", fields={"seq": 99, "run_id": "x"})+ assert (event.data["seq"], event.data["run_id"]) == (3, "r")+++def test_the_protocol_doc_lists_every_suffix():+ doc = (Path(__file__).resolve().parents[4] / "docs-internal" / "chat_protocol.md").read_text(encoding="utf-8")+ for suffix in RUN_SUFFIXES:+ assert f"`{suffix}`" in doc, suffix+++# ----- the reducer against the protocol fixture -----+++def fixture_events():+ frames = json.loads(FIXTURE.read_text(encoding="utf-8"))+ return [frame["data"] for frame in frames]+++def test_the_fixture_run_folds_to_its_final_state():+ events = [event for event in fixture_events() if event["subject"] == "r_8f2a1c"]+ snapshot = reducer.replay("r_8f2a1c", events)+ assert snapshot["state"] == "finished" and snapshot["seq"] == 29+ assert snapshot["outcome"]["type"] == "interrupt"+ assert [interrupt["id"] for interrupt in snapshot["interrupts"]] == ["ap_1"]+ assert [step["state"] for step in snapshot["steps"]] == ["done", "done", "done"]+ assert snapshot["steps"][0]["name"] == "Checked Google Calendar"+ [segment] = snapshot["segments"]+ assert segment["final"] is True and segment["text"].startswith("Saturday is fairly full")+ assert (snapshot["user_message_id"], snapshot["reply_message_id"]) == ("m_owner_1", "a_r_8f2a1c")+++def test_every_prefix_folds_and_duplicates_change_nothing():+ events = [event for event in fixture_events() if event["subject"] == "r_8f2a1c"]+ snapshot = reducer.empty_snapshot("r_8f2a1c")+ for event in events:+ snapshot = reducer.apply_event(snapshot, event)+ assert reducer.apply_event(snapshot, event) == snapshot+++def test_the_resume_run_folds_on_its_own():+ events = [event for event in fixture_events() if event["subject"] == "r_9b3d2e"]+ snapshot = reducer.replay("r_9b3d2e", events)+ assert (snapshot["kind"], snapshot["parent_run_id"], snapshot["state"]) == ("resume", "r_8f2a1c", "finished")+++def test_a_discarded_segment_and_stopping_fold():+ snapshot = reducer.empty_snapshot("r")+ for seq, (suffix, data) in enumerate(+ [+ ("started", {}),+ ("text.started", {"message_id": "r.1.1"}),+ ("text.content", {"message_id": "r.1.1", "delta": "draft"}),+ ("custom", {"name": "opencompany.segment_discarded", "value": {"message_id": "r.1.1"}}),+ ("custom", {"name": "opencompany.stopping", "value": {}}),+ ],+ start=1,+ ):+ snapshot = reducer.apply_event(snapshot, {"type": f"com.opencompany.chat.run.{suffix}", "data": {"seq": seq, **data}})+ assert snapshot["segments"] == [] and snapshot["state"] == "stopping"+++def test_the_stored_run_wins_once_it_has_ended():+ stored = {**reducer.empty_snapshot("r"), "state": "finished", "seq": 0}+ live = {**reducer.empty_snapshot("r"), "state": "running", "seq": 7, "steps": [{"step_id": "s"}]}+ merged = reducer.merge_snapshot(stored, live)+ assert merged["state"] == "finished" and merged["seq"] == 7 and merged["steps"] == [{"step_id": "s"}]+ assert reducer.merge_snapshot({**stored, "state": "pending"}, live)["state"] == "running"diff --git a/server/tests/services/chat/test_ledger.py b/server/tests/services/chat/test_ledger.pynew file mode 100644index 00000000..a74b12c4--- /dev/null+++ b/server/tests/services/chat/test_ledger.py@@ -0,0 +1,383 @@+"""Chat runs in the database (services/chat/ledger.py): admitted with the+owner's message, one live run per session, claimed by one Temporal workflow,+finished only by it, replies saved under the run's reply id, every message+appended to one chain, and the watchdog ending runs nothing will."""++from __future__ import annotations++import asyncio+from datetime import datetime, timedelta, timezone++import pytest+from sqlalchemy import text++from models.chat import ChatRun+from services.chat import ledger+from tests.services.chat._helpers import add_control+++async def admit(database, session_id="wf", *, text_="Book Saturday", track=True, state="pending", client_message_id=None):+ return await ledger.admit_message(+ database,+ session_id=session_id,+ workflow_id=session_id,+ execution_id="gen-1",+ text=text_,+ track=track,+ state=state,+ client_message_id=client_message_id,+ )+++async def runs_of(database, session_id="wf"):+ async with database.get_session() as session:+ from sqlmodel import select++ result = await session.execute(select(ChatRun).where(ChatRun.session_id == session_id))+ return list(result.scalars().all())+++# ----- admission -----+++async def test_a_message_and_its_run_are_admitted_together(database, hub):+ admission = await admit(database)+ run = admission.run+ assert admission.created is True+ assert run.run_id.startswith("r_") and run.state == "pending" and run.kind == "message"+ assert run.user_message_uid == admission.message["uid"]+ assert run.reply_message_uid == f"a_{run.run_id}"+ assert (run.workflow_id, run.run_key) == ("wf", "gen-1")+ [row] = await database.get_chat_messages("wf")+ assert (row["role"], row["message"], row["run_id"], row["execution_id"]) == ("user", "Book Saturday", run.run_id, "gen-1")+++async def test_one_live_run_per_session(database, hub):+ first = await admit(database)+ with pytest.raises(ledger.RunInProgress) as refused:+ await admit(database, text_="And Sunday?")+ assert refused.value.run.run_id == first.run.run_id+ # Nothing of the refused send was written.+ assert [row["message"] for row in await database.get_chat_messages("wf")] == ["Book Saturday"]+ # Another session has its own lane.+ assert (await admit(database, "other")).run is not None+++async def test_the_lane_index_refuses_a_second_live_run(database):+ await admit(database)+ with pytest.raises(Exception):+ async with database.get_session() as session:+ session.add(ChatRun(run_id="r_sneaky", session_id="wf", state="running", kind="message"))+ await session.commit()+ # A resume run (an approved send executing) never holds the lane.+ async with database.get_session() as session:+ session.add(ChatRun(run_id="r_resume", session_id="wf", state="running", kind="resume"))+ await session.commit()+++async def test_concurrent_sends_admit_one_run(database, hub):+ results = await asyncio.gather(*(admit(database, text_=f"message {n}") for n in range(5)), return_exceptions=True)+ admitted = [result for result in results if isinstance(result, ledger.Admission)]+ refused = [result for result in results if isinstance(result, ledger.RunInProgress)]+ assert len(admitted) == 1 and len(refused) == 4+ assert len(await runs_of(database)) == 1+++async def test_a_resent_message_returns_the_first_send(database, hub):+ first = await admit(database, client_message_id="c-1")+ again = await admit(database, client_message_id="c-1")+ assert again.created is False+ assert again.message["uid"] == first.message["uid"]+ assert again.run.run_id == first.run.run_id+ assert len(await database.get_chat_messages("wf")) == 1+ # The same client id in another chat names another message.+ elsewhere = await admit(database, "other", client_message_id="c-1")+ assert elsewhere.message["uid"] != first.message["uid"]++[email protected]("bad", ["", "a b", "x" * 101, 7, "../etc"])+async def test_a_malformed_client_message_id_is_refused(database, bad):+ with pytest.raises(ValueError):+ await admit(database, client_message_id=bad)+ assert await database.get_chat_messages("wf") == []+++async def test_an_untracked_message_starts_no_run(database):+ admission = await admit(database, track=False)+ assert admission.run is None and admission.message["run_id"] is None+ assert await runs_of(database) == []+++# ----- one chain -----+++async def test_every_message_follows_the_last_one(database):+ for n in range(3):+ await database.add_chat_message("wf", "user", f"m{n}")+ rows = await database.get_chat_messages("wf")+ assert rows[0]["parent_uid"] is None+ assert [row["parent_uid"] for row in rows[1:]] == [row["uid"] for row in rows[:-1]]+ async with database.get_session() as session:+ from models.chat import ChatThread++ thread = await session.get(ChatThread, "wf")+ assert (thread.active_leaf_uid, thread.revision) == (rows[-1]["uid"], 3)+++async def test_concurrent_writes_never_fork_the_chain(database):+ await asyncio.gather(*(database.add_chat_message("wf", "assistant", f"m{n}") for n in range(10)))+ rows = await database.get_chat_messages("wf")+ parents = [row["parent_uid"] for row in rows]+ assert len(set(parents)) == 10, "two messages followed the same one"+ assert sum(parent is None for parent in parents) == 1+++async def test_a_message_id_from_another_chat_is_refused(database):+ await database.add_chat_message("wf", "user", "mine", uid="m_shared")+ assert await database.add_chat_message("other", "user", "theirs", uid="m_shared") is None+ assert await database.get_chat_messages("other") == []+++async def test_older_rows_are_migrated_into_one_chain(database):+ """A database from before chat runs: the migration adds the columns and+ chains each session's rows in order, once."""+ async with database.engine.begin() as conn:+ await conn.execute(text("DROP TABLE chat_messages"))+ await conn.execute(+ text(+ "CREATE TABLE chat_messages (id INTEGER PRIMARY KEY, session_id VARCHAR(255), execution_id VARCHAR(255),"+ " role VARCHAR(20), message VARCHAR(50000), created_at DATETIME)"+ )+ )+ for n, (session_id, moment) in enumerate([("wf", "2026-01-01 10:00:00"), ("other", "2026-01-01 10:00:01"), ("wf", "2026-01-01 10:00:02")], start=1):+ await conn.execute(+ text("INSERT INTO chat_messages (id, session_id, role, message, created_at) VALUES (:id, :s, 'user', :m, :t)"),+ {"id": n, "s": session_id, "m": f"old {n}", "t": moment},+ )+ await database._migrate_chat_messages()+ await database._migrate_chat_messages() # runs again at every startup; changes nothing+ rows = await database.get_chat_messages("wf")+ assert [(row["uid"], row["parent_uid"]) for row in rows] == [("m1", None), ("m3", "m1")]+ assert [(row["kind"], row["status"], row["parts"]) for row in rows] == [("text", "complete", {})] * 2+ # A new message continues the migrated chain.+ added = await database.add_chat_message("wf", "user", "new")+ assert added["parent_uid"] == "m3"+++# ----- start and finish -----+++async def started(database, admission, workflow_id="tw-1", run_id="tr-1"):+ return await ledger.start_run(database, run_id=admission.run.run_id, temporal_workflow_id=workflow_id, temporal_run_id=run_id)+++async def test_the_first_workflow_claims_the_run(database, hub):+ admission = await admit(database)+ run = await started(database, admission)+ assert run.state == "running" and run.started_at is not None+ # A retry by the same workflow still holds it; another trigger's run does not.+ assert (await started(database, admission)) is not None+ assert (await started(database, admission, workflow_id="tw-2")) is None+++async def test_a_queued_run_starts_when_the_employee_resumes(database, hub):+ admission = await admit(database, state="queued")+ assert admission.run.state == "queued"+ assert (await started(database, admission)).state == "running"+++async def test_finishing_with_a_reply_names_it(database, hub):+ admission = await admit(database)+ run = await started(database, admission)+ reply = await ledger.post_reply(database, run=run, node_id="wf:chatReply:1", text="Booked.", execution_id="gen-1")+ assert reply["uid"] == run.reply_message_uid and reply["run_id"] == run.run_id+ finished = await ledger.finish_run(database, run_id=run.run_id, temporal_workflow_id="tw-1", temporal_run_id="tr-1", success=True)+ assert (finished.state, finished.outcome, finished.result) == ("finished", "success", {"reply_message_id": run.reply_message_uid})+ assert finished.finished_at is not None+++async def test_finishing_without_a_reply_says_so(database, hub):+ admission = await admit(database)+ await started(database, admission)+ finished = await ledger.finish_run(database, run_id=admission.run.run_id, temporal_workflow_id="tw-1", temporal_run_id="tr-1", success=True)+ assert finished.result == {"no_reply": True}+++async def test_a_failed_run_keeps_its_error(database, hub):+ admission = await admit(database)+ await started(database, admission)+ failed = await ledger.finish_run(+ database, run_id=admission.run.run_id, temporal_workflow_id="tw-1", temporal_run_id="tr-1",+ success=False, error="Calendar said no", hint="Reconnect Google", requires_user_action=True,+ )+ assert (failed.state, failed.error, failed.error_code) == ("error", "Calendar said no", "run_failed")+ assert failed.result == {"hint": "Reconnect Google", "requires_user_action": True}+ # The lane is free again.+ assert (await admit(database, text_="Try again")).run is not None+++async def test_only_the_claimant_finishes(database, hub):+ admission = await admit(database)+ await started(database, admission)+ assert await ledger.finish_run(database, run_id=admission.run.run_id, temporal_workflow_id="tw-2", temporal_run_id="tr-1", success=True) is None+ assert (await ledger.get_run(database, admission.run.run_id)).state == "running"+++async def test_a_retried_finish_changes_nothing(database, hub):+ admission = await admit(database)+ await started(database, admission)+ args = dict(run_id=admission.run.run_id, temporal_workflow_id="tw-1", temporal_run_id="tr-1")+ first = await ledger.finish_run(database, success=True, **args)+ again = await ledger.finish_run(database, success=False, error="late", **args)+ assert again.state == first.state == "finished" and again.error is None+++async def test_a_second_reply_node_gets_its_own_id_and_a_retry_saves_once(database, hub):+ admission = await admit(database)+ run = await started(database, admission)+ first = await ledger.post_reply(database, run=run, node_id="n1", text="One.", execution_id="gen-1")+ second = await ledger.post_reply(database, run=run, node_id="n2", text="Two.", execution_id="gen-1")+ retry = await ledger.post_reply(database, run=run, node_id="n1", text="One (retried).", execution_id="gen-1")+ assert (first["uid"], second["uid"], retry["uid"]) == (run.reply_message_uid, f"{run.reply_message_uid}.2", run.reply_message_uid)+ assert retry["message"] == "One."+ assert [row["message"] for row in await database.get_chat_messages("wf")] == ["Book Saturday", "One.", "Two."]+++async def test_two_reply_nodes_at_once_never_share_an_id(database, hub):+ admission = await admit(database)+ run = await started(database, admission)+ rows = await asyncio.gather(*(ledger.post_reply(database, run=run, node_id=f"n{n}", text=f"{n}", execution_id="gen-1") for n in range(4)))+ assert len({row["uid"] for row in rows}) == 4+++# ----- events -----+++async def test_transitions_publish_once_each(database, hub):+ from tests.services.chat._helpers import FakeSocket++ socket = FakeSocket()+ hub.subscribe(socket, "wf")+ admission = await admit(database)+ await started(database, admission)+ await started(database, admission) # a retried start publishes nothing new+ args = dict(run_id=admission.run.run_id, temporal_workflow_id="tw-1", temporal_run_id="tr-1", success=True)+ await ledger.finish_run(database, **args)+ await ledger.finish_run(database, **args)+ await asyncio.sleep(0)+ events = socket.events()+ assert [event["type"].rsplit(".", 1)[-1] for event in events] == ["started", "finished"]+ started_event, finished_event = events+ assert started_event["data"]["user_message_id"] == admission.message["uid"]+ assert started_event["data"]["reply_message_id"] == admission.run.reply_message_uid+ assert [event["data"]["seq"] for event in events] == [1, 2]+ assert finished_event["data"]["outcome"] == {"type": "success"}+ assert finished_event["data"]["result"] == {"no_reply": True}+++# ----- the watchdog -----+++NOW = datetime(2026, 10, 3, 12, 0, tzinfo=timezone.utc)+LONG_AGO = NOW - timedelta(days=2)+++async def age(database, run_id, **moments):+ from sqlalchemy import update++ async with database.get_session() as session:+ await session.execute(update(ChatRun).where(ChatRun.run_id == run_id).values(**moments))+ await session.commit()+++async def sweep(database, **kwargs):+ kwargs.setdefault("process_started", LONG_AGO)+ return await ledger.sweep(database, now=NOW, **kwargs)+++async def test_a_run_never_picked_up_ends_as_not_delivered(database, hub):+ admission = await admit(database)+ await age(database, admission.run.run_id, created_at=NOW - timedelta(minutes=5))+ assert await sweep(database) == [admission.run.run_id]+ run = await ledger.get_run(database, admission.run.run_id)+ assert (run.state, run.error_code) == ("error", "not_delivered")+++async def test_a_restart_gives_a_pending_run_its_full_wait_again(database, hub):+ admission = await admit(database)+ await age(database, admission.run.run_id, created_at=NOW - timedelta(minutes=5))+ assert await sweep(database, process_started=NOW - timedelta(seconds=10)) == []+++async def test_a_queued_run_waits_for_resume_and_ends_when_the_employee_stops(database, hub):+ admission = await admit(database, state="queued")+ await age(database, admission.run.run_id, created_at=LONG_AGO)+ await add_control(database, "wf", "paused")+ assert await sweep(database) == []+ async with database.engine.begin() as conn:+ await conn.execute(text("UPDATE workflow_control_executions SET status = 'reset'"))+ assert await sweep(database) == [admission.run.run_id]+++async def test_a_run_past_its_longest_ends_as_timed_out(database, hub):+ admission = await admit(database)+ await started(database, admission)+ await age(database, admission.run.run_id, started_at=LONG_AGO)+ assert await sweep(database) == [admission.run.run_id]+ assert (await ledger.get_run(database, admission.run.run_id)).error_code == "timed_out"++[email protected](("replied", "state", "code"), [(True, "finished", None), (False, "error", "interrupted")])+async def test_a_run_whose_workflow_closed_without_finishing_is_ended(database, hub, replied, state, code):+ admission = await admit(database)+ run = await started(database, admission)+ await age(database, run.run_id, started_at=NOW - timedelta(minutes=5))+ if replied:+ await ledger.post_reply(database, run=run, node_id="n", text="Done.", execution_id="gen-1")+ asked = []++ async def closed(workflow_id, temporal_run_id):+ asked.append((workflow_id, temporal_run_id))+ return True++ assert await sweep(database, temporal_status=closed) == [run.run_id]+ assert asked == [("tw-1", "tr-1")]+ ended = await ledger.get_run(database, run.run_id)+ assert (ended.state, ended.error_code) == (state, code)+++async def test_a_running_workflow_and_an_unknown_answer_keep_the_run(database, hub):+ admission = await admit(database)+ run = await started(database, admission)+ await age(database, run.run_id, started_at=NOW - timedelta(minutes=5))++ async def running(*_):+ return False++ async def unknown(*_):+ return None++ assert await sweep(database, temporal_status=running) == []+ assert await sweep(database, temporal_status=unknown) == []+ # A workflow that only just started is not asked at all.+ await age(database, run.run_id, started_at=NOW - timedelta(seconds=5))++ async def never(*_):+ raise AssertionError("asked too early")++ assert await sweep(database, temporal_status=never) == []+++async def test_clearing_ends_live_runs_for_their_subscribers(database, hub, frames):+ from services import chat_thread+ from tests.services.chat._helpers import FakeSocket++ socket = FakeSocket()+ hub.subscribe(socket, "wf")+ admission = await admit(database)+ assert await chat_thread.clear_chat_thread(database, "wf", reason="cleared") == 1+ await asyncio.sleep(0)+ [event] = socket.events()+ assert event["type"] == "com.opencompany.chat.run.failed"+ assert (event["data"]["run_id"], event["data"]["code"]) == (admission.run.run_id, "cleared")+ assert await runs_of(database) == []diff --git a/server/tests/services/chat/test_watchdog.py b/server/tests/services/chat/test_watchdog.pynew file mode 100644index 00000000..2ffe5f96--- /dev/null+++ b/server/tests/services/chat/test_watchdog.py@@ -0,0 +1,93 @@+"""The chat run watchdog (services/chat/watchdog.py): sweeps at once when it+starts and then every interval, survives a failing sweep, stops cleanly, and+asks Temporal whether a run's workflow has closed."""++from __future__ import annotations++import asyncio+from types import SimpleNamespace++import pytest++from services.chat import watchdog+++async def test_it_sweeps_at_once_and_keeps_sweeping_after_a_failure(monkeypatch):+ sweeps = []++ async def sweep(database, *, temporal_status):+ sweeps.append((database, temporal_status))+ if len(sweeps) == 1:+ raise RuntimeError("database is locked")+ return []++ monkeypatch.setattr(watchdog.ledger, "sweep", sweep)+ dog = watchdog.ChatRunWatchdog("db", interval=0.01)+ dog.start()+ dog.start() # starting twice runs one loop+ for _ in range(50):+ if len(sweeps) >= 3:+ break+ await asyncio.sleep(0.01)+ await dog.stop()+ await dog.stop() # stopping twice is harmless+ assert len(sweeps) >= 3+ assert all(database == "db" and status is watchdog.temporal_workflow_closed for database, status in sweeps)+ count = len(sweeps)+ await asyncio.sleep(0.05)+ assert len(sweeps) == count, "the loop kept running after stop"+++def _container(monkeypatch, client):+ import core.container as container_module++ wrapper = SimpleNamespace(client=client) if client is not None else None+ monkeypatch.setattr(container_module, "container", SimpleNamespace(temporal_client=lambda: wrapper))+++async def test_without_temporal_the_answer_is_unknown(monkeypatch):+ _container(monkeypatch, None)+ assert await watchdog.temporal_workflow_closed("w", "r") is None++[email protected](("status", "closed"), [("RUNNING", False), ("COMPLETED", True), ("TERMINATED", True), ("CANCELED", True)])+async def test_a_workflow_is_closed_once_it_stops_running(monkeypatch, status, closed):+ from temporalio.client import WorkflowExecutionStatus++ asked = []++ class Handle:+ async def describe(self):+ return SimpleNamespace(status=getattr(WorkflowExecutionStatus, status))++ class Client:+ def get_workflow_handle(self, workflow_id, run_id=None):+ asked.append((workflow_id, run_id))+ return Handle()++ _container(monkeypatch, Client())+ assert await watchdog.temporal_workflow_closed("w", "r") is closed+ assert asked == [("w", "r")]+++async def test_a_workflow_temporal_no_longer_knows_is_closed(monkeypatch):+ from temporalio.service import RPCError, RPCStatusCode++ class Handle:+ def __init__(self, status):+ self.status = status++ async def describe(self):+ raise RPCError("nope", self.status, b"")++ class Client:+ status = RPCStatusCode.NOT_FOUND++ def get_workflow_handle(self, workflow_id, run_id=None):+ return Handle(self.status)++ client = Client()+ _container(monkeypatch, client)+ assert await watchdog.temporal_workflow_closed("w", "r") is True+ client.status = RPCStatusCode.UNAVAILABLE+ assert await watchdog.temporal_workflow_closed("w", "r") is Nonediff --git a/server/tests/services/test_chat_thread.py b/server/tests/services/test_chat_thread.pyindex 2d9548ff..7378a4a5 100644--- a/server/tests/services/test_chat_thread.py+++ b/server/tests/services/test_chat_thread.py@@ -1,9 +1,10 @@"""A workflow's chat thread (services/chat_thread.py) and the chat-WebSocket handlers built on it: rows carry the live generation, every-insert and every clear that removed rows is announced on ``chat.updated``,-a message reaches a workflow only while a deployment would read it, the-generation is on every row, a Reset clears the thread through the chat-nodes' Reset hook, and deleting the workflow deletes it."""+WebSocket handlers built on it (services/chat/handlers.py): rows carry the+live generation, every insert and every clear that removed rows is announced+on ``chat.updated``, a message reaches a workflow only while a deployment+would read it, the generation is on every row, a Reset clears the thread+through the chat nodes' Reset hook, and deleting the workflow deletes it.+Chat runs have their own tests in tests/services/chat/."""from __future__ import annotations@@ -16,10 +17,9 @@ from types import SimpleNamespaceimport pytest-import nodes.trigger.chat_trigger._events as chat_events+import services.chat.handlers as chat_handlersimport services.status_broadcaster as status_broadcasterfrom models.database import WorkflowControlExecution-from routers import websocket as ws_routerfrom services import chat_thread@@ -59,15 +59,15 @@ def frames(monkeypatch):@pytest.fixturedef router(monkeypatch, database, frames):- """The router's chat handlers on the test database, with the chat- trigger dispatch captured."""+ """The chat handlers on the test database, with the chat trigger+ dispatch captured."""dispatched: list = []- async def dispatch(event_data, *, workflow_id=None):+ async def dispatch(event_data, *, workflow_id=None, event_id=None):dispatched.append((dict(event_data), workflow_id))- monkeypatch.setattr(ws_router, "container", SimpleNamespace(database=lambda: database))- monkeypatch.setattr(chat_events, "dispatch_chat_message_received", dispatch)+ monkeypatch.setattr(chat_handlers, "container", SimpleNamespace(database=lambda: database))+ monkeypatch.setattr(chat_handlers, "dispatch_chat_message_received", dispatch)# The Clear's listeners are whatever a test registers, never the plugins'.monkeypatch.setattr(chat_thread, "_CLEARED_LISTENERS", [])return SimpleNamespace(database=database, frames=frames, dispatched=dispatched)@@ -123,7 +123,8 @@ def test_rows_belong_to_the_live_generation():async def test_a_recorded_message_is_stamped_and_announced(database, frames):await control(database, "7", "running", generation=2)- assert await chat_thread.record_chat_message(database, "7", "assistant", "Done.") is True+ saved = await chat_thread.record_chat_message(database, "7", "assistant", "Done.")+ assert saved["message"] == "Done." and saved["uid"].startswith("m_")[row] = await database.get_chat_messages("7")assert (row["role"], row["message"], row["execution_id"]) == ("assistant", "Done.", "gen-2")[frame] = frames@@ -198,7 +199,7 @@ async def test_history_is_the_newest_rows_oldest_first_in_utc(database):async def send(message, session_id):- return await ws_router.handle_send_chat_message({"message": message, "session_id": session_id, "timestamp": "t"}, None)+ return await chat_handlers.handle_send_chat_message({"message": message, "session_id": session_id, "timestamp": "t"}, None)async def test_a_workflow_that_is_not_running_takes_no_message(router):@@ -216,13 +217,16 @@ async def test_a_message_goes_now_or_waits_for_resume(router, status, delivery):assert result["success"] is True and result["delivery"] == delivery and result["timestamp"] == "t"[row] = await router.database.get_chat_messages("7")assert (row["role"], row["message"], row["execution_id"]) == ("user", "Book Tuesday", "gen-3")+ assert result["message_id"] == row["uid"]+ # No chat trigger answers this graph, so the message starts no run.+ assert result["run_id"] is None# Only this workflow's triggers hear it.- assert router.dispatched == [({"message": "Book Tuesday", "timestamp": "t", "session_id": "7"}, "7")]+ assert router.dispatched == [({"message": "Book Tuesday", "timestamp": "t", "session_id": "7", "message_id": row["uid"]}, "7")]assert updates(router.frames) == [{"workflow_id": "7", "session_id": "7", "role": "user"}]async def test_the_default_session_keeps_its_old_behaviour(router):- result = await ws_router.handle_send_chat_message({"message": "hi"}, None)+ result = await chat_handlers.handle_send_chat_message({"message": "hi"}, None)assert result["success"] is True and "delivery" not in resultassert datetime.fromisoformat(result["timestamp"]).utcoffset() is not Noneassert [row["message"] for row in await router.database.get_chat_messages("default")] == ["hi"]@@ -236,17 +240,20 @@ async def test_history_is_one_generation_unless_all_are_asked_for(router):await control(router.database, "7", "running", generation=2)await router.database.add_chat_message("7", "user", "after it", execution_id="gen-2")- live = await ws_router.handle_get_chat_messages({"session_id": "7"}, None)+ live = await chat_handlers.handle_get_chat_messages({"session_id": "7"}, None)assert [message["message"] for message in live["messages"]] == ["after it"]- everything = await ws_router.handle_get_chat_messages({"session_id": "7", "all_generations": True, "limit": 200}, None)+ everything = await chat_handlers.handle_get_chat_messages({"session_id": "7", "all_generations": True, "limit": 200}, None)assert [(m["role"], m["message"], m["run_key"]) for m in everything["messages"]] == [("user", "before the restart", "gen-1"),("assistant", "an answer", "gen-1"),("user", "after it", "gen-2"),]- assert all(isinstance(m["id"], int) and datetime.fromisoformat(m["timestamp"]).utcoffset() is not None for m in everything["messages"])- newest = await ws_router.handle_get_chat_messages({"session_id": "7", "all_generations": True, "limit": 2}, None)+ assert all(+ m["id"].startswith("m_") and isinstance(m["legacy_id"], int) and datetime.fromisoformat(m["timestamp"]).utcoffset() is not None+ for m in everything["messages"]+ )+ newest = await chat_handlers.handle_get_chat_messages({"session_id": "7", "all_generations": True, "limit": 2}, None)assert [m["message"] for m in newest["messages"]] == ["an answer", "after it"]@@ -254,8 +261,8 @@ async def test_after_a_reset_the_live_read_is_empty(router):# A row a Reset left (its graph had no chat node to clear it).await router.database.add_chat_message("7", "user", "kept", execution_id="gen-1")await control(router.database, "7", "reset", generation=1)- assert (await ws_router.handle_get_chat_messages({"session_id": "7"}, None))["messages"] == []- assert len((await ws_router.handle_get_chat_messages({"session_id": "7", "all_generations": True}, None))["messages"]) == 1+ assert (await chat_handlers.handle_get_chat_messages({"session_id": "7"}, None))["messages"] == []+ assert len((await chat_handlers.handle_get_chat_messages({"session_id": "7", "all_generations": True}, None))["messages"]) == 1async def test_the_owners_clear_lets_the_agent_forget_too(router):@@ -272,23 +279,23 @@ async def test_the_owners_clear_lets_the_agent_forget_too(router):chat_thread.register_chat_cleared_listener(forget) # registering twice is a no-opawait router.database.add_chat_message("7", "user", "hello", execution_id="gen-1")- cleared = await ws_router.handle_clear_chat_messages({"session_id": "7"}, None)+ cleared = await chat_handlers.handle_clear_chat_messages({"session_id": "7"}, None)assert cleared["cleared_count"] == 1assert await router.database.get_chat_messages("7") == []# A failing listener never fails the clear, nor stops the next one.assert heard == [(router.database, "7")]# The editor's "default" chat is no workflow's: nothing to forget.- await ws_router.handle_clear_chat_messages({"session_id": "default"}, None)+ await chat_handlers.handle_clear_chat_messages({"session_id": "default"}, None)assert heard == [(router.database, "7")]async def test_save_and_clear_go_through_the_thread(router):await control(router.database, "7", "running", generation=4)- saved = await ws_router.handle_save_chat_message({"message": "An answer", "role": "assistant", "session_id": "7"}, None)+ saved = await chat_handlers.handle_save_chat_message({"message": "An answer", "role": "assistant", "session_id": "7"}, None)assert saved["success"] is True[row] = await router.database.get_chat_messages("7")assert row["execution_id"] == "gen-4"- cleared = await ws_router.handle_clear_chat_messages({"session_id": "7"}, None)+ cleared = await chat_handlers.handle_clear_chat_messages({"session_id": "7"}, None)assert cleared["cleared_count"] == 1assert [update["role"] for update in updates(router.frames)] == ["assistant", None]diff --git a/server/tests/temporal/test_machina_chat_run.py b/server/tests/temporal/test_machina_chat_run.pynew file mode 100644index 00000000..1dfffcec--- /dev/null+++ b/server/tests/temporal/test_machina_chat_run.py@@ -0,0 +1,335 @@+"""MachinaWorkflow claims and finishes the chat run the owner's message+started (``chat_run.start`` / ``chat_run.finish``, behind+``machina-chat-run-v1``), and every node of the run carries its scope.++Two layers, like the run-record gate:++- the real ``run()`` body with activity dispatch faked: a run is claimed+ only from an event the chat service produced; a claimed run's nodes carry+ ``run_scope`` and the run is finished with the outcome; an unclaimed run+ (another chat trigger got it) runs untracked; a closed patch (a pre-patch+ history) schedules neither activity;+- an SDK replay gate against Temporal's time-skipping test server: histories+ recorded with the patch closed and open both replay, and forcing the patch+ open on the old history fails replay.+"""++from __future__ import annotations++import asyncio+import subprocess+import sys+from pathlib import Path+from types import SimpleNamespace+from typing import Any, Dict, List+from unittest.mock import MagicMock+from uuid import uuid4++import pytest++import nodes # noqa: F401 -- populate plugin registry++ROUTING = {"version": 1, "agent_workflow_enabled": False, "per_type_dispatch_enabled": False, "worker_pool_enabled": False}+SOURCE = "opencompany://services/chat"+TYPE = "com.opencompany.chat.message.received"+++def _graph(*, source: str = SOURCE, event_type: str = TYPE, run_id: Any = "r_1", spawned: bool = True) -> Dict[str, Any]:+ trigger: Dict[str, Any] = {"id": "wf-cr:chatTrigger:1", "type": "chatTrigger", "data": {"label": "Chat"}}+ if spawned:+ data = {"message": "Book Saturday", "session_id": "wf-cr", "run_id": run_id}+ trigger.update(+ {+ "_pre_executed": True,+ "_trigger_output": {**data, "_event_envelope": {"id": str(run_id), "source": source, "type": event_type, "data": data}},+ }+ )+ return {+ "nodes": [trigger, {"id": "wf-cr:console:1", "type": "console", "data": {"label": "Log"}}],+ "edges": [{"id": "e1", "source": trigger["id"], "target": "wf-cr:console:1", "targetHandle": "input-main"}],+ "workflow_id": "wf-cr",+ "session_id": "wf-cr",+ "execution_id": "wf-cr:execution:1",+ "generation": 2,+ "_temporal_routing_v1": ROUTING,+ }+++# ----- the run() body with fakes -----+++def fakes(monkeypatch):+ from temporalio import workflow as temporal_workflow++ from services.temporal.workflow import CHAT_RUN_PATCH, MachinaWorkflow++ state = SimpleNamespace(+ activities=[],+ contexts=[],+ node_success=True,+ claimed=True,+ open_patches={CHAT_RUN_PATCH, "machina-conditional-edges-v1", "machina-run-record-v1"},+ asked=[],+ )+ monkeypatch.setattr(temporal_workflow, "logger", MagicMock())++ def patched(patch_id):+ state.asked.append(patch_id)+ return patch_id in state.open_patches++ monkeypatch.setattr(temporal_workflow, "patched", patched)+ monkeypatch.setattr(temporal_workflow, "info", lambda: SimpleNamespace(workflow_id="wf-cr-chat-r_1", run_id="run-1"))++ def start_activity(name, **kwargs):+ context = kwargs["args"][0]+ state.contexts.append(context)+ future = asyncio.get_event_loop().create_future()+ future.set_result({"success": state.node_success, "node_id": context["node_id"], "result": {}, "error": None if state.node_success else "boom"})+ return future++ async def execute_activity(name, payload=None, **_kwargs):+ state.activities.append((name, payload))+ if name == "chat_run.start":+ return {"claimed": state.claimed}+ return None++ monkeypatch.setattr(temporal_workflow, "start_activity", start_activity)+ monkeypatch.setattr(temporal_workflow, "execute_activity", execute_activity)+ monkeypatch.setattr(+ MachinaWorkflow,+ "_resolve_dispatch",+ lambda self, node_type, **_kwargs: {"kind": "activity", "name": f"node.{node_type}.v1", "queue": None},+ )+ return state+++async def _run(graph):+ from services.temporal.workflow import MachinaWorkflow++ return await asyncio.wait_for(MachinaWorkflow().run(graph), timeout=5.0)+++def _names(state) -> List[str]:+ return [name for name, _ in state.activities]+++def _payloads(state, name) -> List[Dict[str, Any]]:+ return [payload for activity, payload in state.activities if activity == name]+++async def test_a_chat_message_run_is_claimed_scoped_and_finished(fakes):+ await _run(_graph())+ assert _payloads(fakes, "chat_run.start") == [{"run_id": "r_1", "temporal_workflow_id": "wf-cr-chat-r_1", "temporal_run_id": "run-1"}]+ assert _payloads(fakes, "chat_run.finish") == [+ {"run_id": "r_1", "temporal_workflow_id": "wf-cr-chat-r_1", "temporal_run_id": "run-1", "success": True}+ ]+ assert [context["run_scope"] for context in fakes.contexts] == [{"run_id": "r_1", "session_id": "wf-cr"}]+ # Claimed before the first node is scheduled; finished before the run is recorded.+ names = _names(fakes)+ assert names.index("chat_run.finish") < names.index("workflow_runs.record_completion")+++async def test_a_failed_run_finishes_with_its_error(fakes):+ fakes.node_success = False+ await _run(_graph())+ [finish] = _payloads(fakes, "chat_run.finish")+ assert finish["success"] is False and finish["error"] == "boom"+++async def test_an_unclaimed_run_runs_untracked(fakes):+ fakes.claimed = False+ await _run(_graph())+ assert _payloads(fakes, "chat_run.finish") == []+ assert all("run_scope" not in context for context in fakes.contexts)+++ "graph",+ [+ _graph(source="opencompany://nodes/webhook"),+ _graph(event_type="com.opencompany.webhook.received"),+ _graph(run_id=None),+ _graph(run_id=42),+ _graph(spawned=False),+ ],+)+async def test_a_run_id_is_trusted_only_from_the_chat_service(fakes, graph):+ from services.temporal.workflow import CHAT_RUN_PATCH++ await _run(graph)+ assert "chat_run.start" not in _names(fakes)+ assert CHAT_RUN_PATCH not in fakes.asked+++async def test_a_pre_patch_history_schedules_neither_activity(fakes):+ from services.temporal.workflow import CHAT_RUN_PATCH++ fakes.open_patches.discard(CHAT_RUN_PATCH)+ await _run(_graph())+ assert "chat_run.start" not in _names(fakes) and "chat_run.finish" not in _names(fakes)+ assert all("run_scope" not in context for context in fakes.contexts)+++def test_every_worker_registers_the_activities():+ source = (Path(__file__).parents[2] / "services" / "temporal" / "worker.py").read_text(encoding="utf-8")+ assert source.count("*CHAT_RUN_ACTIVITIES") == 3+ from services.chat.activities import CHAT_RUN_ACTIVITIES++ names = sorted(getattr(fn, "__temporal_activity_definition").name for fn in CHAT_RUN_ACTIVITIES)+ assert names == ["chat_run.finish", "chat_run.start"]+++def test_the_scope_reaches_every_node_path():+ """``run_scope`` must survive each hop a node's context takes: the+ per-type activity extras, the legacy activity's message, the agent+ workflow's inherited keys, and the internal socket's execute handler."""+ import inspect++ from routers import websocket as ws_router+ from services.plugin import base+ from services.temporal import activities, agent_workflow++ assert "run_scope" in agent_workflow._INHERITED_SCOPE_KEYS+ assert '"run_scope"' in inspect.getsource(base)+ assert '"run_scope"' in inspect.getsource(activities.NodeExecutionActivities)+ handler = inspect.getsource(ws_router.handle_execute_node)+ assert '"run_scope"' in handler and "/ws/internal" in handler+++async def test_the_activities_go_through_the_ledger(monkeypatch):+ import core.container as container_module+ import services.chat.ledger as ledger++ from services.chat.activities import finish_chat_run_activity, start_chat_run_activity++ calls = []++ async def start_run(database, **kwargs):+ calls.append(("start", kwargs))+ return SimpleNamespace(state="running")++ async def finish_run(database, **kwargs):+ calls.append(("finish", kwargs))+ return None++ monkeypatch.setattr(ledger, "start_run", start_run)+ monkeypatch.setattr(ledger, "finish_run", finish_run)+ monkeypatch.setattr(container_module, "container", SimpleNamespace(database=lambda: "db"))+ assert await start_chat_run_activity({"run_id": "r", "temporal_workflow_id": "w", "temporal_run_id": "t"}) == {"claimed": True}+ assert await finish_chat_run_activity({"run_id": "r", "temporal_workflow_id": "w", "temporal_run_id": "t", "success": True}) == {"state": None}+ assert calls[0] == ("start", {"run_id": "r", "temporal_workflow_id": "w", "temporal_run_id": "t"})+ assert calls[1][1]["success"] is True and calls[1][1]["error"] is None+++# ----- the SDK replay gate (run in a subprocess, like the run-record gate) -----++TASK_QUEUE = "machina-chat-run-replay-gate"+++async def _replay_gate() -> None:+ from temporalio import activity+ from temporalio import workflow as temporal_workflow+ from temporalio.api.enums.v1 import EventType+ from temporalio.client import WorkflowHistory+ from temporalio.testing import WorkflowEnvironment+ from temporalio.worker import Replayer, Worker++ from services.temporal.workflow import CHAT_RUN_PATCH, MachinaWorkflow++ contexts: List[Dict[str, Any]] = []++ @activity.defn(name="execute_node_activity")+ async def execute_node(context: Dict[str, Any]) -> Dict[str, Any]:+ contexts.append(context)+ return {"success": True, "node_id": context["node_id"], "result": {"logged": True}}++ @activity.defn(name="store_node_output_activity")+ async def store_output(_payload: Dict[str, Any]) -> None:+ return None++ @activity.defn(name="workflow_control.pause_on_failure")+ async def pause(_payload: Dict[str, Any]) -> Dict[str, Any]:+ return {"paused": False}++ @activity.defn(name="workflow_runs.record_completion")+ async def record(_payload: Dict[str, Any]) -> Dict[str, Any]:+ return {"recorded": True}++ @activity.defn(name="chat_run.start")+ async def start(_payload: Dict[str, Any]) -> Dict[str, Any]:+ return {"claimed": True}++ @activity.defn(name="chat_run.finish")+ async def finish(_payload: Dict[str, Any]) -> Dict[str, Any]:+ return {"state": "finished"}++ real_patched = temporal_workflow.patched++ def scheduled(history: WorkflowHistory) -> List[str]:+ return [+ event.activity_task_scheduled_event_attributes.activity_type.name+ for event in history.events+ if event.event_type == EventType.EVENT_TYPE_ACTIVITY_TASK_SCHEDULED+ ]++ async def run_once(environment, *, patch_open: bool) -> WorkflowHistory:+ if not patch_open:+ temporal_workflow.patched = lambda patch_id: False if patch_id == CHAT_RUN_PATCH else real_patched(patch_id)+ try:+ handle = await environment.client.start_workflow(MachinaWorkflow.run, _graph(), id=f"machina-cr-{uuid4()}", task_queue=TASK_QUEUE)+ result = await handle.result()+ assert result["success"] is True+ return await handle.fetch_history()+ finally:+ temporal_workflow.patched = real_patched++ async with await WorkflowEnvironment.start_time_skipping() as environment:+ async with Worker(+ environment.client,+ task_queue=TASK_QUEUE,+ workflows=[MachinaWorkflow],+ activities=[execute_node, store_output, pause, record, start, finish],+ ):+ before = await run_once(environment, patch_open=False)+ assert all("run_scope" not in context for context in contexts)+ contexts.clear()+ after = await run_once(environment, patch_open=True)+ assert [context.get("run_scope") for context in contexts] == [{"run_id": "r_1", "session_id": "wf-cr"}]++ assert "chat_run.start" not in scheduled(before) and "chat_run.finish" not in scheduled(before)+ assert "chat_run.start" in scheduled(after) and "chat_run.finish" in scheduled(after)++ replayer = Replayer(workflows=[MachinaWorkflow])+ for history in (before, after):+ captured = WorkflowHistory.from_json(history.workflow_id, history.to_json())+ replay = await replayer.replay_workflow(captured, raise_on_replay_failure=False)+ assert replay.replay_failure is None, replay.replay_failure++ # Forcing only this patch open on the old history must fail replay: the+ # gate is what keeps old histories working. Every other patch stays real.+ temporal_workflow.patched = lambda patch_id: True if patch_id == CHAT_RUN_PATCH else real_patched(patch_id)+ try:+ captured = WorkflowHistory.from_json(before.workflow_id, before.to_json())+ replay = await Replayer(workflows=[MachinaWorkflow]).replay_workflow(captured, raise_on_replay_failure=False)+ assert replay.replay_failure is not None+ finally:+ temporal_workflow.patched = real_patched+++def test_histories_before_and_after_the_patch_replay() -> None:+ """Run the SDK gate in a clean process with valid Windows I/O handles."""+ completed = subprocess.run(+ [sys.executable, "-m", "tests.temporal.test_machina_chat_run"],+ cwd=Path(__file__).parents[2],+ stdin=subprocess.DEVNULL,+ stdout=subprocess.PIPE,+ stderr=subprocess.STDOUT,+ timeout=180,+ check=False,+ )+ assert completed.returncode == 0, completed.stdout.decode(errors="replace")[-3000:]+++if __name__ == "__main__":+ asyncio.run(_replay_gate())diff --git a/server/tests/test_chat_trigger_canary_producer.py b/server/tests/test_chat_trigger_canary_producer.pyindex 7c9b6d90..83aab996 100644--- a/server/tests/test_chat_trigger_canary_producer.py+++ b/server/tests/test_chat_trigger_canary_producer.py@@ -1,16 +1,15 @@-"""chatTrigger producer canary-emit invariant.--Locks the contract: ``nodes.trigger.chat_trigger._events.dispatch_chat_message_received``-routes through the canary CloudEvents path-(:func:`services.events.dispatch.emit`) ONLY. The legacy-``event_waiter.dispatch`` path was removed in Wave 13 — chatTrigger is-canary-registered, the deployment manager skips ``setup_event_trigger``,-and the legacy collector has zero consumers in production.--Same regex-introspection invariant style as-``tests/test_credential_broadcasts.py`` — source-level assertions catch-the wire contract drifting without paying the cost of standing up-Temporal in CI.+"""The owner's chat message reaches chatTrigger listeners through the canary+path only.++Locks the contract: ``services.chat.events.dispatch_chat_message_received``+routes through :func:`services.events.dispatch.emit` and never through the+legacy ``event_waiter.dispatch`` (removed in Wave 13: chatTrigger is+canary-registered and the legacy collector has no consumers). The envelope+carries the owner's text, so it is never broadcast to every socket.++Source-level assertions, in the style of+``tests/test_credential_broadcasts.py``, catch the wire contract drifting+without standing up Temporal."""from __future__ import annotations@@ -38,41 +37,26 @@ _EVENT_WAITER_DISPATCH_PATTERN = re.compile(r"event_waiter\.dispatch\s*\(")_EVENTS_EMIT_PATTERN = re.compile(r"\bemit\s*\(")-class TestChatTriggerProducerCanaryEmit:- """Producer wrapper emits via the canary CloudEvents path only."""+class TestChatMessageProducerCanaryEmit:+ """The producer emits via the canary CloudEvents path only."""def test_dispatcher_is_async(self):- from nodes.trigger.chat_trigger._events import dispatch_chat_message_received+ from services.chat.events import dispatch_chat_message_received- assert inspect.iscoroutinefunction(dispatch_chat_message_received), (- "dispatch_chat_message_received must be async — it awaits " "services.events.dispatch.emit."- )+ assert inspect.iscoroutinefunction(dispatch_chat_message_received)def test_dispatcher_uses_canary_path_only(self):- from nodes.trigger.chat_trigger import _events-- src = inspect.getsource(_events.dispatch_chat_message_received)+ from services.chat import events- assert _EVENTS_EMIT_PATTERN.search(src), (- "dispatch_chat_message_received must call "- "services.events.dispatch.emit(envelope, ...) — the canary "- "CloudEvents path Signals running TriggerListenerWorkflow "- "consumers AND broadcasts to FE on the chat_message_received "- "wire key."- )+ src = inspect.getsource(events.dispatch_chat_message_received)+ assert _EVENTS_EMIT_PATTERN.search(src), "must call services.events.dispatch.emit(envelope, ...)"assert not _EVENT_WAITER_DISPATCH_PATTERN.search(src), (- "dispatch_chat_message_received must NOT call "- "event_waiter.dispatch — chatTrigger is canary-registered, "- "the legacy collector path has zero consumers, and that "- "call was removed in Wave 13. Reintroducing it would "- "double-dispatch through dead infrastructure."+ "must NOT call event_waiter.dispatch: chatTrigger is canary-registered and the legacy collector has no consumers")@pytest.mark.asyncioasync def test_runtime_emits_canary_envelope(self, monkeypatch):- """Invoking the dispatcher calls dispatch.emit with the right- envelope. The legacy event_waiter is not touched."""- from nodes.trigger.chat_trigger import _events+ from services.chat import eventsfrom services.events import dispatch as dispatch_modemit_calls: List[Any] = []@@ -83,36 +67,30 @@ class TestChatTriggerProducerCanaryEmit:monkeypatch.setattr(dispatch_mod, "emit", fake_emit)- result = await _events.dispatch_chat_message_received(- {- "message": "hello",- "session_id": "sess-1",- "timestamp": "2026-05-14T00:00:00",- }+ result = await events.dispatch_chat_message_received(+ {"message": "hello", "session_id": "sess-1", "timestamp": "2026-05-14T00:00:00"})- # No return value — canary-only emit doesn't carry a waiter count.assert result is None-- assert len(emit_calls) == 1- event = emit_calls[0]["event"]+ [call] = emit_calls+ event = call["event"]assert event.type == "com.opencompany.chat.message.received"+ assert event.source == "opencompany://services/chat"assert event.subject == "sess-1"- assert emit_calls[0]["wire_routing_key"] == "chat_message_received"+ assert call["wire_routing_key"] == "chat_message_received"# The envelope carries the owner's text; it reaches the signalled# workflows only, never every connected socket.- assert emit_calls[0]["broadcast"] is False+ assert call["broadcast"] is False# No scope passed -> unscoped envelope (broadcast semantics).assert event.workflow_id is None@pytest.mark.asyncio- async def test_workflow_scope_rides_the_envelope_verbatim(self, monkeypatch):- """The factory plumbs ``workflow_id`` onto the envelope without- any decision logic — the scoping RULE lives at the core call- site (routers/websocket.py) and the scoped-delivery narrowing in- core dispatch. Without the scope, one workflow's chat message- fired every deployed workflow's chatTrigger."""- from nodes.trigger.chat_trigger import _events+ async def test_scope_and_run_id_ride_the_envelope_verbatim(self, monkeypatch):+ """The factory plumbs ``workflow_id`` and the event id without any+ decision logic: the scoping rule lives in the send handler and the+ narrowing in core dispatch. The event id is the run id, so the run+ a listener spawns has a predictable id."""+ from services.chat import eventsfrom services.events import dispatch as dispatch_modemit_calls: List[Any] = []@@ -123,27 +101,39 @@ class TestChatTriggerProducerCanaryEmit:monkeypatch.setattr(dispatch_mod, "emit", fake_emit)- await _events.dispatch_chat_message_received(- {"message": "hi", "session_id": "wf-1", "timestamp": "t"},+ await events.dispatch_chat_message_received(+ {"message": "hi", "session_id": "wf-1", "timestamp": "t", "run_id": "r_1"},workflow_id="wf-1",+ event_id="r_1",)- assert emit_calls[0]["event"].workflow_id == "wf-1"+ event = emit_calls[0]["event"]+ assert event.workflow_id == "wf-1"+ assert event.id == "r_1"+ assert event.data["run_id"] == "r_1"- def test_router_decides_the_scope_not_the_plugin(self):- """The chat WS handler (core) derives the workflow scope from the- session and the plugin factory carries it verbatim — execution- logic stays out of the node folder."""- from nodes.trigger.chat_trigger import _events- from routers import websocket as ws_router+ def test_the_send_handler_decides_the_scope_not_the_factory(self):+ from services.chat import access, events, handlers- factory_src = inspect.getsource(_events.chat_message_received)- assert '"default"' not in factory_src, (- "The plugin factory must not embed the session!='default' "- "scoping decision — that rule lives in "- "routers/websocket.py:handle_send_chat_message."- )+ factory_src = inspect.getsource(events.chat_message_received)+ assert '"default"' not in factory_src, "the factory must not embed the session != 'default' scoping rule"++ handler_src = inspect.getsource(handlers.handle_send_chat_message)+ assert "workflow_id=scope.workflow_id" in handler_src+ assert "session_id == DEFAULT_SESSION" in inspect.getsource(access.authorize_session)++ def test_the_workflow_trusts_the_same_source_and_type(self):+ """MachinaWorkflow reads a chat run id only from an event this+ module produced; the two must agree."""+ from services.chat import events+ from services.temporal import workflow++ assert workflow.CHAT_MESSAGE_SOURCE == events.SOURCE+ assert workflow.CHAT_MESSAGE_TYPE == events.MESSAGE_RECEIVED_TYPE++ def test_the_trigger_listens_on_the_same_type(self):+ import nodes # noqa: F401 - registers the trigger+ from services.chat import events+ from services.deployment.canary_registry import cloudevent_type_for- handler_src = inspect.getsource(ws_router.handle_send_chat_message)- assert "workflow_id=workflow_scope" in handler_src- assert 'session_id != "default"' in handler_src+ assert cloudevent_type_for("chatTrigger") == events.MESSAGE_RECEIVED_TYPE