Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
feat(chat): stream the answer, show working steps and stop a run
trohitg76e56cfddfc4
diff --git a/CLAUDE.md b/CLAUDE.mdindex a11a6770..b5917f6f 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -14,7 +14,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire| **[Temporal Cleanup & Resilience Plan (Waves 15-18)](./docs-internal/ARCHIVE/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md)** | *Archived — all four waves shipped.* 15 dead-code retirement (Redis-Streams branch in event_waiter, APScheduler cron stack, orphaned `connection_status` factory), 16 per-queue task routing (`TemporalWorkerPool`), 17 resilience hardening (cron `catchup_window`, SDK interceptors, `DEPLOYMENT_MODE`, periodic heartbeats; the one-shot LLM-step retry it records was later reverted to unlimited-with-backoff), 18 worker performance tuning. Current settings live in [Temporal Architecture](./docs-internal/TEMPORAL_ARCHITECTURE.md). || **[Node Allowlist](./docs-internal/node_allowlist.md)** | Single-config UI visibility — `server/config/node_allowlist.json` controls which nodes / credential categories / skill folders show in the UI. Five lists with two enforcement tiers (mode-gated allowlist + absolute blocklist). `useNodeAllowlist` hook exposes `isVisible` / `isBlocked` / `isAllowed` / `isCredentialCategoryDisabled` / `isSkillFolderDisabled`. Adding a new disabled domain = single JSON edit, no code change. || **[Normal Mode (Home)](./docs-internal/normal_mode.md)** | The owner-facing landing screen: describe a job, an LLM on the server drafts a setup screen (a JSON UI spec the client parses, normalises and renders from a fixed catalogue), and Hire builds, validates, saves and starts one workflow per AI employee (`services/employees/`, graph built from `config/employee_apps.json` plus the owner's skill library and a Canvas node, and checked against `node_allowlist.json` `enabled_nodes`). Covers the screen switch (`app/`, `shellMode` / `ui_shell_mode`, `VITE_NORMAL_MODE=false` kill switch, Home shows light/dark only), the employee page (the shared chat, `features/chat`, with the employee named in the header beside New conversation; Pause is in the Workspace header), deleting an employee from the team sidebar (stopped through Reset first, then gone from every tab), the `approvalGate` "ask me before sending" step, run records for "done today" (`machina-run-record-v1`), automatic-pause reasons, Settings (Profile fields; usage-only Billing via `get_employee_usage`; Skills, the user-skills library each new hire copies onto a Skills node; consumer Connectors with `publisher` / `verified`; Plugins, the starter bundles in `hire/starters.json`; the last three share one catalog page), the Workspace dock (an employee's Canvas board, named by the summary's `canvas_node_id`; a live Browser tab with user takeover, shared with Dev mode; Mobile shows the managed local Android phone), the "Web browser" app (read-only under "ask me first" through the tool's `ask_first_params`; while the agent waits in `request_user` the summary's `browser_request` shows Needs you and Help in browser), Talk on the employee page (a thread over the chat session whose id is the workflow id, rows written through `services/chat_thread.py` and, for the owner's messages and their chat runs, `services/chat/ledger.py` (see [Chat Protocol](./docs-internal/chat_protocol.md)); the talk line is chatTrigger → an agent → `chatReply` "Reply in Chat": a chat hire's own agent is its talk agent, other hires get a "Talk with {name}" agent sharing the worker's tools and skills with its own Context, and schedule hires also "Post to Talk"), Turn on Talk (for talk `off`) and Apply (while the summary's `pending_changes` is true) via `enable_employee_talk` / `apply_employee_changes`, each a restart on the saved graph through `services/deployment/restart.py`, self-extension from Talk through the Agent Builder within the Hire rule (`services/employees/policy.py`), the 3D orb, the wire contract and the `employee_lifecycle` / `approval_lifecycle` / `chat.updated` broadcasts. |-| **[Chat Protocol](./docs-internal/chat_protocol.md)** | The wire contract between the chat runtime (`server/services/chat/`) and the shared chat UI: AG-UI's event model on the existing WebSocket, snake_case, in the CloudEvents envelope. A message a deployed chat trigger will answer starts a **chat run** admitted with it in one reserved write transaction (`Database.reserved_session`, SQLite `BEGIN IMMEDIATE`): one live run per session (the lane, also a partial unique index; a second send answers `run_in_progress`), idempotent `client_message_id`, claimed and finished by the MachinaWorkflow that answers it behind `machina-chat-run-v1` (the run id is trusted only from an event whose source is `opencompany://services/chat`; `run_scope` rides every node context; only the claimant finishes), the reply saved as `a_<run id>` by `chatReply`, every message appended after the session's active leaf (`chat_threads`) so concurrent writes never fork the thread. `chat_run_event` frames reach only sockets that passed `chat_subscribe`'s owner check (`services/chat/hub.py`: per-run `seq`, `hub_epoch`, bounded per-socket queues with a resync), never the status broadcast. A watchdog (`services/chat/watchdog.py`, started by `main.py`) ends runs nothing will finish (`not_delivered`, `timed_out`, `interrupted`). Lists every event, handler, part and error code, including the ones later phases build. |+| **[Chat Protocol](./docs-internal/chat_protocol.md)** | The wire contract between the chat runtime (`server/services/chat/`) and the shared chat UI: AG-UI's event model on the existing WebSocket, snake_case, in the CloudEvents envelope. A message a deployed chat trigger will answer starts a **chat run** admitted with it in one reserved write transaction (`Database.reserved_session`, SQLite `BEGIN IMMEDIATE`): one live run per session (the lane, also a partial unique index; a second send answers `run_in_progress`), idempotent `client_message_id`, claimed and finished by the MachinaWorkflow that answers it behind `machina-chat-run-v1` (the run id is trusted only from an event whose source is `opencompany://services/chat`; `run_scope` rides every node context; only the claimant finishes), the reply saved as `a_<run id>` by `chatReply`, every message appended after the session's active leaf (`chat_threads`) so concurrent writes never fork the thread. `chat_run_event` frames reach only sockets that passed `chat_subscribe`'s owner check (`services/chat/hub.py`: per-run `seq`, `hub_epoch`, bounded per-socket queues with a resync), never the status broadcast. A watchdog (`services/chat/watchdog.py`, started by `main.py`) ends runs nothing will finish (`not_delivered`, `timed_out`, `interrupted`). The agent wired into Reply in Chat (`answers_chat_run`) streams its answer as `text.*` events (`services/chat/stream.py`, activity input only, no workflow patch), its tool calls show as steps (`services/chat/steps.py`; label from the plugin's `chat_step`), and `stop_chat_run` stops a run cooperatively: its LLM step polls the run and returns what was shown, unstarted tools are skipped, the reply is saved `stopped`, and the watchdog cancels a run still stopping after `runs.stop_grace_s`. Lists every event, handler, part and error code, including the ones later phases build. || **[Theme System](./docs-internal/theme_system.md)** | 12-way visual theme system — 2 base (light, dark) + 5 utopian (renaissance, greek, edo, steampunk, atomic) + 5 dystopian (cyber, wasteland, rot, plague, surveillance) — driven by `<html data-theme>` + per-theme CSS files in `client/src/themes/`. Token taxonomy (surface / fg / border / accent / typography / motion) in **hex + `color-mix()`**; `@theme inline` bridge maps `--color-X: var(--X)` (no `hsl()` wrapper); per-theme files own the colour hex, base.css owns the shared `--tint-*` alpha scale; action role tokens (incl. `-ink` readable text), **per-theme `--code-*` syntax tokens** (tier 6 — the code editor, console/output JSON viewers, and chat code blocks paint in each theme's own palette: keyword→trigger, string→success, number→agent, function→model, on an adaptive dark/light code surface; replaced the global dracula `--prism-*` block + dead `getPrismTokenCSS`; the `OutputPanel` JSON viewer reads the same vars), per-theme `--pulse-keyframe` animation system + `.opencompany-*` helpers in `animations.css`, decorative-layer wrappers (`.app-frame` / `.canvas-host` / `.modal-frame`), per-component decorative ornaments (panel textures + canvas decorations + node pseudo-element overlays + theme-specific keyframes), **canvas-node visual contract via `--node-color` CSS custom property** (no inline `background` / `border` on node components — base.css + per-theme CSS owns visuals; `NodeStyle` helper type at [types/NodeTypes.ts](./client/src/types/NodeTypes.ts) makes the inline custom-prop typecheck-clean), **`--node-pulse-color` separate from `--node-color`** so executing-node glow uses each theme's highest-contrast accent regardless of plugin accent (Cyber neon cyan, Surveillance REC red, Renaissance ultramarine, etc.), **`data-page-hidden` animation pause** (set by `app/usePageActivitySync.ts`, mounted in `AppShell` on both screens, while the tab is hidden or the window is blurred; base.css declares `html[data-page-hidden] *, *::before, *::after { animation-play-state: paused !important }` to prevent compositor stall on tab return), **per-theme icon glyph system** (290 inline SVG glyph strings across 29 keys × 10 themes via [themedGlyphs.ts](./client/src/assets/icons/themedGlyphs.ts) + theme-aware [NodeIcon.tsx](./client/src/assets/icons/NodeIcon.tsx)), **per-theme canvas-grid + custom cursors** via `--canvas-grid` / `--cursor-default` slots, **decorative HTML primitives** (`<SvgFilterDefs>` mounting `#ink-blot` / `#noise` / `#crt` filter IDs at app root, `<DropCap>` wrapper for Renaissance ornament rule), **parameter panel migrated to Tailwind tokens** (no `useAppTheme()` reads; section headers carry the display-typography triplet; raw `<Button>` swapped to `<ActionButton intent>`), **per-theme scrollbar webkit rules** in all 12 themes, 9-event WebAudio sound system (10 packs via `--sound-pack` token + `useSound()` hook + global hover delegate + sonner toast monkey-patch + `withSound()` HOC + `Sounds.unlock()` gesture-unlock for AudioContext autoplay-policy compliance), `@media (prefers-reduced-motion: reduce)` accessibility, 30 ms throttle on `type` / `hover`, migration recipe, anti-patterns. Read this before adding a new theme, migrating a component to the new contract, or adding a canvas-node component. || **[Design System Bundle](./docs-internal/design-system/IMPLEMENTATION.md)** | Canonical, vendored design-system reference (extracted from this repo; see its `IMPLEMENTATION.md` for the contract). Tokens are **hex + `color-mix()`** (`tokens/{colors,typography,spacing,motion,fonts,base}.css`) — the format the live codebase standardizes on. Includes 34 reference components (`components/`, inline-style reference only — recreate in Tailwind/shadcn idioms), full-app UI kit (`ui_kits/opencompany/index.html`), 12-theme docs (`guidelines/THEMES.md`), **the merged handoff brief (`HANDOFF.md` — glow-ladder traps, step-edge contract, toolbar layout traps, canvas-layer asymmetries, plus the recorded product amendments)**, and **the panel×theme fidelity target (`reference-mockup/Panel Theme Matrix.dc.html` — 17 panels × 12 themes, open in a browser)**. `reference/themes/` no longer vendors CSS snapshots — it points at the authoritative [client/src/themes/](./client/src/themes/). Copy token values verbatim; do not re-derive by eye. Pairs with [Theme System](./docs-internal/theme_system.md). || **[Schema Source of Truth RFC](./docs-internal/ARCHIVE/schema_source_of_truth_rfc.md)** | *Archived — shipped.* Backend is SSOT for node schemas, visual metadata, handlers, palette metadata, icons; `VITE_NODESPEC_BACKEND` defaults on and the frontend node definitions are gone. Plugin pattern: one `BaseNode` subclass in `server/nodes/<group>/<plugin>/__init__.py`. Wire format: `asset:<key>` / `<lib>:<brand>` / URL / emoji. Endpoint: `/api/schemas/nodes/{type}/spec.json`. Current reference: [Plugin System](./docs-internal/plugin_system.md). |@@ -62,7 +62,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire| **[Onboarding Service](./docs-internal/onboarding.md)** | First-launch welcome wizard with 4 steps, database persistence, and replay from Settings || **[CLI Services Integration](./docs-internal/cli_services_integration.md)** | Guide for integrating CLI-based services (Temporal, etc.) with proper lifecycle management || **[Temporal Architecture](./docs-internal/TEMPORAL_ARCHITECTURE.md)** | Distributed workflow execution: activities, FIRST_COMPLETED scheduling, horizontal scaling |-| **[Native LLM SDK](./docs-internal/native_llm_sdk.md)** | Native SDK layer in `services/llm/`: protocol-based providers, config-driven base URLs, and 14 registered providers (11 cloud, Ollama / LM Studio, and `openai_compatible`, which serves any number of user-named endpoints — llama.cpp, vLLM, a LiteLLM proxy — added in the Credentials Modal with no code; xAI is agent-selectable without a standalone chat-model node). A JSON-driven `supports_model_listing: false` flag covers endpoints that speak the OpenAI chat wire format but ship no `/v1/models` route (Sarvam) — `OpenAIProvider.fetch_models` then serves the curated list and probes the key with a one-token completion. Every provider is native; `tests/llm/test_langchain_removed.py` locks the retired-dependency end state. |+| **[Native LLM SDK](./docs-internal/native_llm_sdk.md)** | Native SDK layer in `services/llm/`: protocol-based providers, config-driven base URLs, and 14 registered providers (11 cloud, Ollama / LM Studio, and `openai_compatible`, which serves any number of user-named endpoints — llama.cpp, vLLM, a LiteLLM proxy — added in the Credentials Modal with no code; xAI is agent-selectable without a standalone chat-model node). A JSON-driven `supports_model_listing: false` flag covers endpoints that speak the OpenAI chat wire format but ship no `/v1/models` route (Sarvam) — `OpenAIProvider.fetch_models` then serves the curated list and probes the key with a one-token completion. `chat(on_event=...)` streams `StreamEvent`s only for providers whose block declares `"streaming": true` (Anthropic, OpenAI); the unifier replays the finished response into the sink for the rest, and the returned response never changes (`tests/llm/test_streaming.py`). Every provider is native; `tests/llm/test_langchain_removed.py` locks the retired-dependency end state. || **[RFC-0003 — OpenAI-Compatible Provider Contract](./RFC-0003-OPENAI-COMPATIBLE-PROVIDER-CONTRACT.md)** | Accepted, implemented. "OpenAI-compatible" is a family of divergent shapes with no spec behind it, so per-provider facts are **resolved at save time or declared**, never guessed. A configured `base_url` is copied **verbatim** from vendor docs and never rewritten (`/v1` is neither universal nor positionally predictable — DeepSeek is root-mounted, Groq is `/openai/v1`); a **user-supplied** URL is rooted by `services/llm/endpoints.py::resolve_base_url`, which sends the SDK's own `models.list()` to the URL then to it plus `/v1` and adopts only a body that **is an OpenAI list** — a status alone proves nothing, because LM Studio answers HTTP 200 to routes it does not serve. A failed save writes and broadcasts nothing. One `resolve_credential` sends the stored key or the vendor-documented placeholder (`auth.placeholder_key`) and never `None` (which would make the SDK read `OPENAI_API_KEY` and ship it to a third-party URL); no provider rewrites a key. A 2xx with no `choices` and an error body raises `PROTOCOL` naming the redacted URL. **Named endpoints (D13–D17)**: any server is a provider reference `openai_compatible:<slug>` stored exactly like Ollama (`{ref}` + `{ref}_proxy`), selectable from every agent (the `provider` field is a loader-driven `ProviderRef`, not a `Literal`), from the `openaiCompatibleChatModel` node (its field is `endpoint`, not `provider`, to dodge the panel's stored-key effect) and the global default picker; kind detected once from native routes by body shape; context and price from the server first (Ollama/LM Studio SDK probes, llama.cpp `/props`, vLLM `max_model_len`), then LiteLLM's table for generic servers only, then conservative defaults; local-model entries and the LiteLLM cache live under DATA_DIR, never in the tracked `config/model_registry.json`. RLM cannot route an endpoint and refuses one loudly. Gates AG1–AG20 (AG5 retired), `ProviderSpec` gains no field. || **[Event Waiter System](./docs-internal/event_waiter_system.md)** | In-memory asyncio.Future waiter for push-based trigger nodes on the canvas-Run path (WhatsApp, Telegram, Webhook, Chat, Task completion). Deployed triggers ride the Temporal canary path; the Redis-Streams backend was retired in Wave 15.3 || **[Credentials Encryption](./docs-internal/credentials_encryption.md)** | Fernet + PBKDF2 encryption pipeline, separate credentials.db, two credential systems (OAuth vs API keys), multi-backend abstraction |diff --git a/docs-internal/agent_context_flow.md b/docs-internal/agent_context_flow.mdindex 0c0a1e31..b5752473 100644--- a/docs-internal/agent_context_flow.md+++ b/docs-internal/agent_context_flow.md@@ -114,6 +114,25 @@ checks below, even when the carried transcript is what seeds it. If thatstored row is over the 1 MB seed cap, the resumed run fails with`ConversationTooLarge`.+### Stopped turns and unanswered calls++The owner can stop a chat run mid-answer ([Chat Protocol → Streaming, steps+and Stop](./chat_protocol.md#streaming-steps-and-stop)). The LLM step then+saves the turn as far as it went, like any other step: the exact sent list,+plus the partial answer when the model had written something (its text block+carries `metadata: {"stopped": true}`); with nothing written, the sent list+alone, so the owner's message stays in the conversation unanswered. A tool call+of a stopped run gets a result saying it was not run, so the transcript stays+whole.++A run cancelled while a tool ran (the watchdog cancels a run still stopping+after the grace; a crash) can leave the last assistant turn with calls that+have no result, which no provider will continue from. The load therefore+passes the stored row through `close_unanswered_tool_calls`+(`services/agent_context/conversation.py`), which answers each open call of the+latest assistant turn with `UNANSWERED_TOOL_RESULT`, after the results the turn+already has. A closed conversation is returned unchanged.+## Transcript size: capped results, cleared old resultsThe transcript is the `agent.execute_llm_step` input on every turn and the@@ -297,6 +316,7 @@ workflow, not only the removed Context's agent.| 8 | Specialized bridges record the ORIGINAL prompt, never the augmented one. | Recording the rendered transcript nests the conversation inside itself and grows without bound. || 9 | The store never imports `nodes/`; the plugin registers its broadcaster via `register_conversation_listener`, and a listener failure can never fail a save. | Same layering rule as every plugin registry; a UI notification must not break execution. || 10 | External tool results are capped before they enter the transcript; the latest turn is never cleared or summarized; the pressure rules are chosen by the recorded `context_pressure_version`. | Uncapped results bricked every later firing (`errors.md` #28); an unread turn summarized away loses what the model asked for; a replay must schedule the commands it recorded. |+| 11 | A stopped turn is saved as far as it went, and a load answers any tool call the stored row left open. | A provider refuses a conversation with an unanswered call, so one cancelled run would otherwise break every later firing. |## How this broke (August 2026 regression), and why the journal went awaydiff --git a/docs-internal/chat_protocol.md b/docs-internal/chat_protocol.mdindex 0946cc75..98cdf224 100644--- a/docs-internal/chat_protocol.md+++ b/docs-internal/chat_protocol.md@@ -15,8 +15,9 @@ meaning.Built so far: runs and their storage, the lifecycle events (`started`, `finished`, `failed`), subscriptions,`get_chat_messages` v2 (with each message's `run`), `get_chat_run`, the watchdog and the MachinaWorkflow patch (see-[Runs](#runs)), and the shared chat UI on both hosts (see [Client](#client)). Every other event, handler and part-below is the contract the later phases build to.+[Runs](#runs)); the answer streaming as text events, working steps and Stop (see [Streaming, steps and+Stop](#streaming-steps-and-stop)); and the shared chat UI on both hosts (see [Client](#client)). Every other event,+handler and part below is the contract the later phases build to.## Concepts@@ -53,8 +54,9 @@ internal socket, load the workflow, and compare its owner with the socket's execwith `seq` at or below the snapshot's, and buffer events that arrive before the response.- Each subscriber has a bounded queue (`hub.subscriber_queue_size` in `server/config/chat_defaults.json`). A socketthat falls that far behind has its queue replaced by one `custom` `opencompany.resync` per session it follows- (`data: {session_id, hub_epoch, name, value}`, no `run_id`); the client then takes fresh snapshots. Merging- consecutive text deltas of one segment comes with streaming.+ (`data: {session_id, hub_epoch, name, value}`, no `run_id`); the client then takes fresh snapshots. Text deltas+ are batched before they are published (see [Streaming, steps and Stop](#streaming-steps-and-stop)), not merged in+ the queue.- Delivery is in-process: events published by an activity on a worker in another process do not reach sockets hereyet (a relay is planned).- `chat.updated` remains an identity-only broadcast (`{workflow_id, session_id, role}`) that tells every open thread@@ -70,11 +72,11 @@ Every event's `data` carries `{workflow_id, session_id, run_id, seq, hub_epoch}`| RUN_STARTED | `started` | `kind`, `parent_run_id?`, `user_message_id?`, `reply_message_id`, `started_at` || RUN_FINISHED | `finished` | `outcome` (below), `result {reply_message_id?, no_reply?}`, `duration_ms`, `step_count` || RUN_ERROR | `failed` | `message` (safe to show, at most 500 chars), `code`, `hint?`, `requires_user_action?` |-| STEP_STARTED | `step.started` | `step_id`, `step_name`, `icon?`, `agent_node_id`, `tool_node_id?`, `attempt` |-| STEP_FINISHED | `step.finished` | `step_id`, `step_name` (the done label), `state` (`done`, `failed`, `skipped`), `detail?`, `duration_ms`, `narration?` |-| TEXT_MESSAGE_START | `text.started` | `message_id` (segment id `{run_id}.{iteration}.{attempt}`), `role: "assistant"`, `agent_node_id` |+| STEP_STARTED | `step.started` | `step_id` (the tool call's id), `step_name`, `icon?` |+| STEP_FINISHED | `step.finished` | `step_id`, `step_name`, `state` (`done`, `failed`, `skipped`), `detail?` (at most 200 chars), `duration_ms`, `narration?` |+| TEXT_MESSAGE_START | `text.started` | `message_id` (segment id `{run_id}.{iteration}.{attempt}`), `role: "assistant"` || TEXT_MESSAGE_CONTENT | `text.content` | `message_id`, `delta` (never empty) |-| TEXT_MESSAGE_END | `text.ended` | `message_id`, `final` (bool), `reply_message_id?`, `output_tokens?` |+| TEXT_MESSAGE_END | `text.ended` | `message_id`, `final` (bool), `reply_message_id?` (with `final: true`) || TOOL_CALL_START | `tool_call.started` | `tool_call_id`, `tool_call_name`, `parent_message_id?`, `label` (sending tools only) || TOOL_CALL_ARGS | `tool_call.args` | `tool_call_id`, `delta` || TOOL_CALL_END | `tool_call.ended` | `tool_call_id` |@@ -89,9 +91,9 @@ Every event's `data` carries `{workflow_id, session_id, run_id, seq, hub_epoch}``custom` names: `opencompany.segment_discarded` (`{message_id}`: a retried LLM attempt replaces this segment),`opencompany.retrying` (`{retry_after?, attempt}`), `opencompany.stopping`, `opencompany.resync`.-**Text segments.** Only the agent that answers the owner streams text. A segment that ends with `final: false`-accompanied a tool call: the client folds it into the steps as narration. The segment with `final: true` is the-reply. Text that could still turn out to be `NO_REPLY` or a `<followups>` block is held back on the server and never+**Text segments.** Only the agent that answers the owner streams text. A segment that ends with `final: false` was+written beside tool calls (narration, "Let me check the calendar."); the segment with `final: true` is the reply.+Text that could still turn out to be `NO_REPLY` or a `<followups>` block is held back on the server and neverstreamed.**Outcomes** (`run.finished`):@@ -121,7 +123,7 @@ All are WebSocket request/response handlers with snake_case payloads. Failures a| `chat_subscribe` | `{session_id}` | `{success, session_id, hub_epoch, active_runs: [RunSnapshot]}` || `chat_unsubscribe` | `{session_id}` | `{success, session_id, hub_epoch}` || `get_chat_run` | `{run_id}` | `{success, run: RunSnapshot}`, or `not_found` |-| `stop_chat_run` | `{run_id}` or `{session_id}` | `{success, run_id, state}` |+| `stop_chat_run` | `{run_id}` | `{success, run_id, state: "stopping" \| "stopped"}`; `not_stoppable` (with `state`) for a run that ended otherwise || `edit_chat_message` | `{message_id, text, attachments?, expected_revision}` | `{success, message_id, run_id}` || `regenerate_chat_reply` | `{message_id, expected_revision}` | `{success, run_id}` || `switch_chat_branch` | `{message_id, target_id, expected_revision}` | `{success, thread}` |@@ -170,7 +172,8 @@ summary adds `kind` (`gate` or `tool_call`), `node_type`, `tool_name`, `tool_cal```{id, legacy_id, role, kind, text, message, timestamp, run_key, run_id, parent_id, status, attachments,parts, feedback, siblings: {index, count, ids}, editable, client_message_id?,- run?: {run_id, state, outcome, error?: {message, code, hint?}}}+ run?: {run_id, state, outcome, error?: {message, code, hint?}, steps?: [{step_id, name, state, detail?,+ duration_ms?}], duration_ms?}}```- `id` is the message's stable id (`m_…` for the owner's, `a_<run id>` for a run's reply; rows saved before chat@@ -178,9 +181,10 @@ summary adds `kind` (`gate` or `tool_call`), `node_type`, `tool_name`, `tool_cal- `message` equals `text`; it is kept for older readers.- `client_message_id` is echoed on the owner's messages that carried one, so an optimistic row and its saved rowkeep one key.-- `run` says how the run a message started (or answers) stood when the thread was read: its `state`, `outcome`, and- for a failed run the error with the hint the run recorded. It is how a reload still shows that a message went- unanswered; while the session is subscribed, the run's events are fresher (see [Client](#client)).+- `run` says how the run a message started (or answers) stood when the thread was read: its `state`, `outcome`, for+ a failed run the error with the hint the run recorded, the steps it saved, and once it has ended how long it+ worked (`duration_ms`). It is how a reload still shows that a message went unanswered and what the employee did;+ while the session is subscribed, the run's events are fresher (see [Client](#client)).- Until branches land, `siblings` always holds the message alone and `editable` is false.- `kind`: `text`, `report` (Post to Talk: no parent, not editable), `action` (a button press), `notice`.- `status`: `complete`, `stopped`, `error`.@@ -259,6 +263,43 @@ Approvals are joined live from the approvals store; the part only names them.ends the runs nothing will finish (codes above). A pending run's wait (`runs.pickup_timeout_s`) counts from thelater of its creation and the server's start.+## Streaming, steps and Stop++Settings are in `server/config/chat_defaults.json` (`stream`, `steps`, `runs`).++- **Who streams.** `agent.prepare_payload` gives an agent a `chat_stream` (`services/chat/stream.py`+ `chat_stream_for`) when the run's `run_scope` reached it, it is not working for another agent (no+ `parent_node_id`), and its output goes straight to a node whose plugin declares `answers_chat_run` (Reply in Chat).+ The payload also carries `chat_run_id` for every agent of the run. Both ride the AgentWorkflow's activity inputs+ only (LLM steps and tool calls), so no workflow patch was needed and recorded histories replay unchanged.+- **Text.** `agent.execute_llm_step` passes a `ChatStreamEmitter` as the provider's `on_event` sink (providers that+ declare `streaming` in `llm_defaults.json` stream; for the others the unifier replays the finished response as+ events: see [Native LLM SDK](./native_llm_sdk.md)). Deltas go out every `stream.flush_ms` or `stream.flush_chars`.+ A retried attempt first sends `opencompany.segment_discarded` for each earlier attempt's segment. Streaming never+ changes the step's result.+- **Steps.** `BaseNode.as_activity` wraps every tool call that carries a `chat_stream` and a `tool_call_id`+ (`services/chat/steps.py`): `step.started` when it begins, `step.finished` when it ends, saved on the run as it+ finishes (`ledger.record_step`, at most `steps.max_per_run`). The label is the plugin's `chat_step`+ ("Searched the web"), else "Used <display name>"; plugins with `chat_step_hidden` (the clock, the checklist) show+ none, and skill loads (`agent.skill.invoke`) never pass through it. A tool may put a short line in its result as+ `_step_detail` ("3 events on Saturday"): it is shown under the step and taken out before the model reads the+ result.+- **Stop** (`stop_chat_run`, `ledger.request_stop`). A run nothing has picked up (pending, or queued until Resume)+ ends `stopped` at once and frees the lane; a workflow that picks it up later claims it still `stopped`+ (`chat_run.start`) and its agent answers nothing. A running run moves to `stopping` (`custom`+ `opencompany.stopping`) and stops itself:+ - its agent's model step polls the run every `runs.stop_poll_s` while the provider writes (an activity heartbeat+ arrives too late to stop a stream mid-sentence), drops the call, and returns what the owner saw so far as the+ final answer; a step about to start returns at once, before paying for a request;+ - a tool call not started yet is answered "Not run: the owner stopped this answer." without running;+ - Reply in Chat saves the partial answer with `status: "stopped"`, and the run finishes with outcome `stopped`.++ A run still stopping `runs.stop_grace_s` after Stop (a long tool, a lost workflow) is ended by the watchdog, which+ cancels its Temporal workflow. The conversation keeps the stopped turn: the request and the partial answer (its+ text block marked `stopped`), or the request alone when nothing was written; a tool call a cancelled run left+ without a result is answered when the conversation is next loaded (see+ [Agent Context Flow](./agent_context_flow.md)).+## Client`client/src/features/chat/` is the shared chat; only its `index.ts` is public (an ESLint rule keeps the rest@@ -280,16 +321,24 @@ editor's console pane (`ConsoleChat`, compact, scope `live`) are the two hosts.`active_runs` stand in; after it the store alone says which runs are live, and a message whose `run` still readslive is read with `get_chat_run` (`useThreadRunReconcile`).- **Turns** (`thread/model.ts`): one per message with a divider where `run_key` changes. A run that is going,- failed or was stopped shows on its last answer, or on a turn of its own right after its last message before it- has answered (a live run whose messages are out of view goes last). Failures whose code only says no answer came+ failed or was stopped (or finished, while its answer is on the way) shows on its last answer, or on a turn of its+ own right after its last message before it has answered (a live run whose messages are out of view goes last). Failures whose code only says no answer came(`not_delivered`, `timed_out`, `interrupted`) disappear once an answer lands; `reset` and `cleared` never show. Amessage sent from this tab and its saved row share a key (its `client_message_id`), and a run's first answer takesthe key its run's turn had, so neither remounts.- **Sending** (`data/send.ts`): the message shows at once; the server's answer admits its run into the store(`queued` or `pending`), so the employee shows working before the run's first event. While the lane is held, Send- waits. A refused or failed send takes the message out of the thread and puts its text back in the box+ is Stop. A refused or failed send takes the message out of the thread and puts its text back in the box(`state/composerStore.ts`, a draft per session that survives switching conversations); after a failure in transitthe draft keeps its `client_message_id`, so sending it again is the same message.+- **A working run** (`turns/AssistantTurn.tsx`): skeleton lines until text comes; then the latest segment, muted+ while it is narration, with a caret while it streams (`ReplyMarkdown` renders each finished block once,+ `markdown/blocks.ts`); a status line saying "Thinking", "Writing · N tok/s" or "Stopping…", with an Esc hint. The+ steps disclosure (`turns/StepsDisclosure.tsx`) sits on the run's first turn: "Working…" and open while the run+ works, "Worked for 12s · 3 steps" after; a run read back later starts it closed. A finished run keeps its turn+ until its saved reply lands in the thread, so the streamed answer and the reply stay one element.+- **Stop** (`data/stop.ts`): the Stop button, or Esc anywhere in the pane, sends `stop_chat_run` for the lane's run+ and applies the answer to the store at once; the run's events take it from there.## Notes to the employee@@ -319,7 +368,8 @@ and server tests alike:| `not_running` | send | The employee is not running and cannot queue messages. || `invalid_request` | send, save | A malformed field (`detail` says which): an empty message, a role other than the owner's, or a bad `client_message_id`. || `read_failed` | get_chat_messages, chat_subscribe | The thread could not be read. Never answered as an empty thread. |-| `not_found` | get_chat_run | No such run. |+| `not_found` | get_chat_run, stop_chat_run | No such run. |+| `not_stoppable` | stop_chat_run | The run ended before Stop reached it; `state` says how. || `conflict` | edit, regenerate, switch, set_ask_first | `expected_revision` is stale. || `not_editable` | edit, regenerate, decide | The message belongs to an older generation, its prefix was compacted, or an edited argument is not editable. || `ui_event_rejected` | send | The element or action does not match the saved spec. |diff --git a/docs-internal/plugin_system.md b/docs-internal/plugin_system.mdindex 0c7194b4..a5e79826 100644--- a/docs-internal/plugin_system.md+++ b/docs-internal/plugin_system.md@@ -79,6 +79,8 @@ class SpecializedAgentBase(ActionNode, abstract=True):| `Output` | Pydantic `BaseModel` — runtime output shape. || `usable_as_tool` | `ActionNode` flag — mints a ToolNode adapter for AI invocation. Combined with `component_kind != "model"`, makes the plugin visible to `agentBuilder.add_tool` (catalogue + rebind paths). **Side effect:** setting it auto-sets `hide_input_handle` / `hide_output_handle` to `True` unless the class declares them (`BaseNode.__init_subclass__` in [base.py](../server/services/plugin/base.py), which does the same for `component_kind == "tool"`), and `_metadata_dict` adds an `output-tool` handle. The frontend (`SquareNode`) reads the hide flags only when a spec declares no handles; a spec with handles renders exactly what it declares. So a dual-purpose node that must stay wirable on the canvas declares `input-main` / `output-main` in `handles`; setting the flags `False` alone does not add them. || `needs_canvas` | `ClassVar[bool]` — when `True`, the F4.B `AgentWorkflow` tool-dispatch forwards the parent workflow's `nodes`/`edges` into the per-tool activity payload. Today only `AgentBuilderNode` opts in: `add_tool` compares the run's canvas with the saved graph, so a tool saved earlier but missing from the running snapshot is handed back to be bound. Its calling agent comes from `invoking_agent_node_id` / `parent_node_id`, not from edges. |+| `chat_step` / `chat_step_hidden` | How a call of this tool reads as a working step in the chat while it answers the owner ("Searched the web"; default "Used <display name>"); `chat_step_hidden = True` shows none (the clock, the checklist). A tool may put a short line in its result as `_step_detail`, shown under the step and removed before the model reads the result. See [Chat Protocol](./chat_protocol.md#streaming-steps-and-stop). |+| `answers_chat_run` | `ClassVar[bool]` — the node posts a chat run's answer (Reply in Chat). The agent wired straight into such a node streams its text into the chat. || `task_queue` | Temporal worker pool. See `TaskQueue` constants. || `retry_policy` | `RetryPolicy` dataclass (mirrors `temporalio.common.RetryPolicy`). || `start_to_close_timeout` / `heartbeat_timeout` | Per-node Temporal knobs. |diff --git a/server/config/chat_defaults.json b/server/config/chat_defaults.jsonindex dc364f71..7ca66657 100644--- a/server/config/chat_defaults.json+++ b/server/config/chat_defaults.json@@ -2,9 +2,18 @@"runs": {"pickup_timeout_s": 120,"max_running_s": 86400,- "watchdog_interval_s": 30+ "watchdog_interval_s": 30,+ "stop_poll_s": 1,+ "stop_grace_s": 45},"hub": {"subscriber_queue_size": 512+ },+ "stream": {+ "flush_ms": 50,+ "flush_chars": 160+ },+ "steps": {+ "max_per_run": 100}}diff --git a/server/core/database.py b/server/core/database.pyindex 95968491..402161cf 100644--- a/server/core/database.py+++ b/server/core/database.py@@ -138,6 +138,7 @@ class Database:await self._migrate_workflow_controls()await self._migrate_generation_scoped_runtime_data()await self._migrate_chat_messages()+ await self._migrate_chat_runs()logger.info("Database initialized successfully")@@ -524,6 +525,21 @@ class Database:except Exception as exc:logger.warning(f"Generation runtime-data migration check failed: {exc}")+ async def _migrate_chat_runs(self):+ """Give an older ``chat_runs`` table the columns added since."""+ try:+ async with self.engine.begin() as conn:+ result = await conn.execute(text("PRAGMA table_info(chat_runs)"))+ columns = {row[1] for row in result.fetchall()}+ if not columns:+ return+ additions = {"stop_requested_at": "DATETIME"}+ for column, definition in additions.items():+ if column not in columns:+ await conn.execute(text(f"ALTER TABLE chat_runs ADD COLUMN {column} {definition}"))+ except Exception as exc:+ logger.warning(f"Chat run migration check failed: {exc}")+async def _migrate_chat_messages(self):"""Give older chat rows the columns chat runs need.diff --git a/server/models/chat.py b/server/models/chat.pyindex efb6b9ef..55538e5d 100644--- a/server/models/chat.py+++ b/server/models/chat.py@@ -3,7 +3,8 @@A run is admitted in the same transaction as the owner's message(``services/chat/ledger.py``). It moves ``queued`` (the employee is paused)or ``pending`` -> ``running`` -> ``finished`` | ``error`` | ``stopped``, each-step a compare-and-swap on ``state``. ``kind`` says what started it:+step a compare-and-swap on ``state``; Stop moves a pending or running run to+``stopping`` on the way to ``stopped``. ``kind`` says what started it:``message`` (the owner wrote), ``edit``, ``regenerate``, ``action`` (a buttonin generated UI) or ``resume`` (an approved send executing later).@@ -79,6 +80,9 @@ class ChatRun(SQLModel, table=True):created_at: datetime = Field(default_factory=_utcnow, sa_column=Column(DateTime(timezone=True), nullable=False))started_at: Optional[datetime] = Field(default=None, sa_column=Column(DateTime(timezone=True), nullable=True))finished_at: Optional[datetime] = Field(default=None, sa_column=Column(DateTime(timezone=True), nullable=True))+ #: When the owner pressed Stop; the watchdog ends a run still stopping+ #: ``runs.stop_grace_s`` later.+ stop_requested_at: Optional[datetime] = Field(default=None, sa_column=Column(DateTime(timezone=True), nullable=True))class ChatRunPart(SQLModel, table=True):diff --git a/server/nodes/chat/chat_reply/__init__.py b/server/nodes/chat/chat_reply/__init__.pyindex 4e2063ab..d5f0b9d4 100644--- a/server/nodes/chat/chat_reply/__init__.py+++ b/server/nodes/chat/chat_reply/__init__.py@@ -77,6 +77,9 @@ class ChatReplyNode(ActionNode):hide_output_handle = Trueannotations = {"destructive": False, "readonly": False, "open_world": False}task_queue = TaskQueue.DEFAULT+ # The agent wired into this node answers the chat run: its text streams+ # into the chat as it writes (services/chat/stream.py).+ answers_chat_run = TrueParams = ChatReplyParamsOutput = ChatReplyOutputdiff --git a/server/nodes/search/brave_search/__init__.py b/server/nodes/search/brave_search/__init__.pyindex 06ceee8a..2732ca1e 100644--- a/server/nodes/search/brave_search/__init__.py+++ b/server/nodes/search/brave_search/__init__.py@@ -89,6 +89,7 @@ class BraveSearchNode(ActionNode):credentials = (BraveSearchCredential,)annotations = {"destructive": False, "readonly": True, "open_world": True}task_queue = TaskQueue.REST_API+ chat_step = "Searched the web"usable_as_tool = TrueParams = BraveSearchParamsdiff --git a/server/nodes/search/duckduckgo_search/__init__.py b/server/nodes/search/duckduckgo_search/__init__.pyindex dcc7e70e..229d4a28 100644--- a/server/nodes/search/duckduckgo_search/__init__.py+++ b/server/nodes/search/duckduckgo_search/__init__.py@@ -52,6 +52,7 @@ class DuckDuckGoSearchNode(ToolNode):ui_hints = {"hideRunButton": True}annotations = {"destructive": False, "readonly": True, "open_world": True}task_queue = TaskQueue.REST_API+ chat_step = "Searched the web"Params = DuckDuckGoSearchParamsOutput = DuckDuckGoSearchOutputdiff --git a/server/nodes/search/perplexity_search/__init__.py b/server/nodes/search/perplexity_search/__init__.pyindex 22538cd2..299d05eb 100644--- a/server/nodes/search/perplexity_search/__init__.py+++ b/server/nodes/search/perplexity_search/__init__.py@@ -93,6 +93,7 @@ class PerplexitySearchNode(ActionNode):credentials = (PerplexityCredential,)annotations = {"destructive": False, "readonly": True, "open_world": True}task_queue = TaskQueue.REST_API+ chat_step = "Searched the web"usable_as_tool = TrueParams = PerplexitySearchParamsdiff --git a/server/nodes/search/serper_search/__init__.py b/server/nodes/search/serper_search/__init__.pyindex 3fea2f1f..6feae6a3 100644--- a/server/nodes/search/serper_search/__init__.py+++ b/server/nodes/search/serper_search/__init__.py@@ -88,6 +88,7 @@ class SerperSearchNode(ActionNode):credentials = (SerperCredential,)annotations = {"destructive": False, "readonly": True, "open_world": True}task_queue = TaskQueue.REST_API+ chat_step = "Searched the web"usable_as_tool = TrueParams = SerperSearchParamsdiff --git a/server/nodes/tool/current_time_tool/__init__.py b/server/nodes/tool/current_time_tool/__init__.pyindex 9c70d2f1..f42549da 100644--- a/server/nodes/tool/current_time_tool/__init__.py+++ b/server/nodes/tool/current_time_tool/__init__.py@@ -39,6 +39,7 @@ class CurrentTimeToolNode(ToolNode):ui_hints = {"hideRunButton": True}annotations = {"destructive": False, "readonly": True, "open_world": False}task_queue = TaskQueue.DEFAULT+ chat_step_hidden = TrueParams = CurrentTimeParamsOutput = CurrentTimeOutputdiff --git a/server/nodes/tool/write_todos/__init__.py b/server/nodes/tool/write_todos/__init__.pyindex 4badeb56..cb8ce10c 100644--- a/server/nodes/tool/write_todos/__init__.py+++ b/server/nodes/tool/write_todos/__init__.py@@ -76,6 +76,7 @@ class WriteTodosNode(ToolNode):ui_hints = {"hideRunButton": True, "isTodoEditor": True}annotations = {"destructive": False, "readonly": False, "open_world": False}task_queue = TaskQueue.DEFAULT+ chat_step_hidden = TrueParams = WriteTodosParamsOutput = WriteTodosOutputdiff --git a/server/services/agent_context/conversation.py b/server/services/agent_context/conversation.pyindex 64420eaa..66f61e55 100644--- a/server/services/agent_context/conversation.py+++ b/server/services/agent_context/conversation.py@@ -243,8 +243,47 @@ async def list_conversations(]+#: The result a tool call gets when its run ended before the tool answered.+UNANSWERED_TOOL_RESULT = "This step did not finish: the run ended before its result came back."+++def close_unanswered_tool_calls(messages: List[Dict[str, Any]]) -> List[Dict[str, Any]]:+ """Answer the tool calls a stored conversation left open.++ A run that ends between asking for a tool and getting its result (the+ watchdog cancelling a stopped run mid-tool, a crash) saves a last+ assistant turn whose calls have no result, and every provider refuses to+ continue such a conversation. Each open call gets+ :data:`UNANSWERED_TOOL_RESULT`, after the results the turn already has.+ Returns ``messages`` itself when nothing is open.+ """+ from services.llm.protocol import Message, message_to_wire++ latest = next((i for i in range(len(messages) - 1, -1, -1) if messages[i].get("role") == "assistant"), None)+ if latest is None:+ return messages+ answered = {m.get("tool_call_id") for m in messages[latest + 1:] if m.get("role") == "tool"}+ open_calls = [+ call+ for call in messages[latest].get("tool_calls") or []+ if isinstance(call, dict) and call.get("id") and call.get("id") not in answered+ ]+ if not open_calls:+ return messages+ end = latest + 1+ while end < len(messages) and messages[end].get("role") == "tool":+ end += 1+ closing = [+ dict(message_to_wire(Message(role="tool", content=UNANSWERED_TOOL_RESULT, tool_call_id=call["id"], name=call.get("name"))))+ for call in open_calls+ ]+ return [*messages[:end], *closing, *messages[end:]]++__all__ = [+ "UNANSWERED_TOOL_RESULT","clear_conversation",+ "close_unanswered_tool_calls","list_conversations","load_conversation","save_conversation",diff --git a/server/services/chat/config.py b/server/services/chat/config.pyindex 2a4c45b4..ebfcdc11 100644--- a/server/services/chat/config.py+++ b/server/services/chat/config.py@@ -24,4 +24,13 @@ def hub_setting(name: str) -> int:return int(load_chat_config()["hub"][name])-__all__ = ["CONFIG_PATH", "hub_setting", "load_chat_config", "runs_setting"]+def stream_setting(name: str) -> float:+ """``stream.<name>``: how the answer's text is batched into events."""+ return float(load_chat_config()["stream"][name])+++def steps_setting(name: str) -> int:+ return int(load_chat_config()["steps"][name])+++__all__ = ["CONFIG_PATH", "hub_setting", "load_chat_config", "runs_setting", "steps_setting", "stream_setting"]diff --git a/server/services/chat/handlers.py b/server/services/chat/handlers.pyindex e7f279a8..adf17d31 100644--- a/server/services/chat/handlers.py+++ b/server/services/chat/handlers.py@@ -45,13 +45,20 @@ _DENIED = {"success": False, "error": "access_denied"}def _wire_run(run: Any) -> Dict[str, Any]:"""The run a message started or answers, as a message carries it: enough- to show how it ended after a reload (a failed run leaves no reply)."""+ to show how it ended after a reload (a failed run leaves no reply), what+ the employee did on the way and how long it took."""wire: Dict[str, Any] = {"run_id": run.run_id, "state": run.state, "outcome": run.outcome}if run.state == "error":wire["error"] = {"message": run.error, "code": run.error_code}hint = (run.result or {}).get("hint")if hint:wire["error"]["hint"] = hint+ if run.steps:+ wire["steps"] = [dict(step) for step in run.steps]+ if run.started_at and run.finished_at:+ started = run.started_at if run.started_at.tzinfo else run.started_at.replace(tzinfo=timezone.utc)+ finished = run.finished_at if run.finished_at.tzinfo else run.finished_at.replace(tzinfo=timezone.utc)+ wire["duration_ms"] = max(0, int((finished - started).total_seconds() * 1000))return wire@@ -302,6 +309,26 @@ async def handle_get_chat_run(data: Dict[str, Any], websocket: WebSocket) -> Dicreturn {"success": True, "run": {**snapshot, "hub_epoch": hub.epoch}}+@ws_handler("run_id")+async def handle_stop_chat_run(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:+ """Stop a live run (``ledger.request_stop``). One nothing picked up yet+ ends ``stopped`` at once; a running one stops itself at its next step,+ keeping its reply so far. Answers the run's state: ``stopping`` or+ ``stopped``, ``not_stoppable`` for a run that ended otherwise."""+ database = container.database()+ run = await ledger.get_run(database, str(data["run_id"]))+ if run is None:+ return {"success": False, "error": "not_found"}+ try:+ await authorize_session(database, websocket, run.session_id)+ except ChatAccessDenied:+ return dict(_DENIED)+ run = await ledger.request_stop(database, run.run_id) or run+ if run.state not in ("stopping", "stopped"):+ return {"success": False, "error": "not_stoppable", "run_id": run.run_id, "state": run.state}+ return {"success": True, "run_id": run.run_id, "state": run.state}++@ws_handler()async def handle_clear_chat_messages(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:"""Clear a session's chat, every generation of it, with its runs. For a@@ -339,6 +366,7 @@ WS_HANDLERS = {"chat_subscribe": handle_chat_subscribe,"chat_unsubscribe": handle_chat_unsubscribe,"get_chat_run": handle_get_chat_run,+ "stop_chat_run": handle_stop_chat_run,"clear_chat_messages": handle_clear_chat_messages,"save_chat_message": handle_save_chat_message,}diff --git a/server/services/chat/ledger.py b/server/services/chat/ledger.pyindex 86599aa6..5cecdf89 100644--- a/server/services/chat/ledger.py+++ b/server/services/chat/ledger.py@@ -10,9 +10,11 @@ writing a second one; the id maps to a message id that only this session canproduce (``client_message_uid``).**Start** (``start_run``) moves ``pending``/``queued`` to ``running`` and-records the Temporal workflow that claimed it. Several chatTrigger nodes in-one graph each spawn a run for the same message: the first claims the chat-run, the others run untracked. **Finish** (``finish_run``) is accepted only+records the Temporal workflow that claimed it; a run stopped before anything+picked it up is claimed still ``stopped``, so the workflow that answers it+stops at its first step instead of answering untracked. Several chatTrigger+nodes in one graph each spawn a run for the same message: the first claims+the chat run, the others run untracked. **Finish** (``finish_run``) is accepted onlyfrom the claimant. A retried start or finish publishes nothing new: eventscarry an ``event_key`` the hub publishes once.@@ -24,6 +26,18 @@ running run past ``runs.max_running_s`` (``timed_out``), and a running runwhose Temporal workflow closed without finishing it: finished when its replywas saved, ``interrupted`` otherwise.+**Stop** (``request_stop``) ends a run nothing has picked up at once+(``stopped``, no reply) and moves a running run to ``stopping``. A running+run stops itself: its agent's next model step returns what it has written so+far, and a tool not started yet is skipped (``services/chat/stream.py``,+``services/chat/steps.py``); its reply, if any, is saved ``stopped`` and the+run finishes with outcome ``stopped``. A run still stopping+``runs.stop_grace_s`` later (a tool that takes long, a lost workflow) is+ended by the watchdog, which also cancels its workflow.++**Steps** (``record_step``) are saved on the run as they finish, so a reload+can say how long it worked and what it did.+Every transition commits before its event is published, so a client thatreads the run after an event never sees an older state than the event."""@@ -43,7 +57,7 @@ from sqlmodel import selectfrom core.logging import get_loggerfrom models.chat import LIVE_STATES, TERMINAL_STATES, ChatRunfrom models.database import ChatMessage-from services.chat.config import runs_setting+from services.chat.config import runs_setting, steps_settingfrom services.chat.hub import get_chat_hubfrom services.chat_thread import delivery_for@@ -59,6 +73,9 @@ _CLOSED_GRACE = timedelta(seconds=60)#: ``await status(temporal_workflow_id, temporal_run_id)`` -> True when the#: workflow has closed, False while it runs, None when unknown.TemporalStatus = Callable[[str, Optional[str]], Awaitable[Optional[bool]]]+#: ``await cancel(temporal_workflow_id, temporal_run_id)``: ask Temporal to+#: cancel a workflow (best effort).+TemporalCancel = Callable[[str, Optional[str]], Awaitable[None]]class RunInProgress(Exception):@@ -269,6 +286,8 @@ async def post_reply(uid = run.reply_message_uid or reply_uid(run.run_id)if replies:uid = f"{uid}.{len(replies) + 1}"+ current = await session.get(ChatRun, run.run_id)+ stopped = current is not None and current.state in ("stopping", "stopped")row = await database.append_chat_row(session,session_id=run.session_id,@@ -277,6 +296,7 @@ async def post_reply(execution_id=execution_id,uid=uid,run_id=run.run_id,+ status="stopped" if stopped else "complete",meta={"node_id": node_id},)await session.commit()@@ -362,16 +382,25 @@ async def start_run(temporal_run_id=temporal_run_id,))+ # Stopped before anything picked it up: claimed as it is, so the+ # workflow's agent stops at its first step rather than answering+ # untracked.+ await session.execute(+ update(ChatRun)+ .where(ChatRun.run_id == run_id, ChatRun.state == "stopped", ChatRun.temporal_workflow_id.is_(None))+ .values(temporal_workflow_id=temporal_workflow_id, temporal_run_id=temporal_run_id)+ )await session.commit()run = await get_run(database, run_id)if (run is None- or run.state not in ("running", "stopping")+ or run.state not in ("running", "stopping", "stopped")or run.temporal_workflow_id != temporal_workflow_idor run.temporal_run_id != temporal_run_id):return None- publish_started(run)+ if run.state != "stopped":+ publish_started(run)return run@@ -437,6 +466,74 @@ async def finish_run(return await _settle(database, run, values)+async def request_stop(database: Any, run_id: str) -> Optional[ChatRun]:+ """Stop a live run. Returns the run as it now is (unchanged when it was+ not stoppable), None when it does not exist.++ A run nothing has picked up yet (pending, or queued until the employee+ resumes) ends ``stopped`` at once, which frees the lane; a workflow that+ picks it up later claims it stopped (``start_run``) and answers nothing.+ A running run moves to ``stopping`` and stops itself."""+ now = _utcnow()+ async with database.get_session() as session:+ unclaimed = await session.execute(+ update(ChatRun)+ .where(+ ChatRun.run_id == run_id,+ ChatRun.state.in_(("pending", "queued")),+ ChatRun.temporal_workflow_id.is_(None),+ )+ .values(+ state="stopped",+ outcome="stopped",+ result={"no_reply": True},+ stop_requested_at=now,+ finished_at=now,+ )+ )+ ended = bool(unclaimed.rowcount)+ stopping = False+ if not ended:+ result = await session.execute(+ update(ChatRun)+ .where(ChatRun.run_id == run_id, ChatRun.state.in_(("pending", "queued", "running")))+ .values(state="stopping", stop_requested_at=now)+ )+ stopping = bool(result.rowcount)+ await session.commit()+ run = await get_run(database, run_id)+ if run is not None and ended:+ publish_terminal(run)+ elif run is not None and stopping:+ _publish(run, "custom", {"name": "opencompany.stopping", "value": {}}, "stopping")+ return run+++async def is_stopping(database: Any, run_id: str) -> bool:+ """Whether the owner asked the run to stop (or it already has)."""+ run = await get_run(database, run_id)+ return run is not None and run.state in ("stopping", "stopped")+++async def record_step(database: Any, run_id: str, step: Dict[str, Any]) -> None:+ """Save a finished step on the run (replacing an earlier save of the+ same step), up to ``steps.max_per_run``."""+ step_id = step.get("step_id")+ if not step_id:+ return+ async with database.reserved_session() as session:+ run = await session.get(ChatRun, run_id)+ if run is None:+ return+ steps = [dict(item) for item in (run.steps or []) if item.get("step_id") != step_id]+ if len(steps) >= steps_setting("max_per_run"):+ return+ steps.append(dict(step))+ run.steps = steps+ session.add(run)+ await session.commit()++async def fail_run(database: Any, run: ChatRun, *, code: str, message: str) -> Optional[ChatRun]:"""End a live run with an error. None when it moved meanwhile."""if run.state not in LIVE_STATES:@@ -461,6 +558,7 @@ async def sweep(now: Optional[datetime] = None,process_started: datetime = PROCESS_STARTED,temporal_status: Optional[TemporalStatus] = None,+ temporal_cancel: Optional[TemporalCancel] = None,) -> List[str]:"""End the live runs nothing will end. Returns their ids."""now = now or _utcnow()@@ -469,7 +567,10 @@ async def sweep(ended: List[str] = []for run in await live_runs(database):try:- closed = await _sweep_one(database, run, now, process_started, pickup, longest, temporal_status)+ if run.state == "stopping":+ closed = await _sweep_stopping(database, run, now, temporal_cancel)+ else:+ closed = await _sweep_one(database, run, now, process_started, pickup, longest, temporal_status)except Exception: # noqa: BLE001 - one bad row must not stop the sweeplogger.warning("Chat run sweep failed for a run", run_id=run.run_id, exc_info=True)continue@@ -480,6 +581,22 @@ async def sweep(return ended+async def _sweep_stopping(+ database: Any, run: ChatRun, now: datetime, temporal_cancel: Optional[TemporalCancel]+) -> Optional[ChatRun]:+ """End a run still stopping ``runs.stop_grace_s`` after Stop, cancelling+ its workflow: its reply so far, if any, stays."""+ requested = _aware(run.stop_requested_at) or _aware(run.started_at) or _aware(run.created_at) or now+ if now - requested < timedelta(seconds=runs_setting("stop_grace_s")):+ return None+ if temporal_cancel is not None and run.temporal_workflow_id:+ try:+ await temporal_cancel(run.temporal_workflow_id, run.temporal_run_id)+ except Exception: # noqa: BLE001 - the run ends either way+ logger.warning("Could not cancel a stopped run's workflow", run_id=run.run_id, exc_info=True)+ return await _settle(database, run, await _success_values(database, run, "stopped"))++async def _sweep_one(database: Any,run: ChatRun,@@ -529,6 +646,7 @@ __all__ = ["fail_run","finish_run","get_run",+ "is_stopping","lane_run","live_runs","new_run_id",@@ -536,7 +654,9 @@ __all__ = ["publish_cleared","publish_started","publish_terminal",+ "record_step","reply_uid",+ "request_stop","runs_by_id","start_run","sweep",diff --git a/server/services/chat/steps.py b/server/services/chat/steps.pynew file mode 100644index 00000000..4b514847--- /dev/null+++ b/server/services/chat/steps.py@@ -0,0 +1,159 @@+"""Working steps: the tool calls of the agent answering a chat run, shown in+the chat while it works ("Checked Google Calendar", docs-internal/+chat_protocol.md).++``BaseNode.as_activity`` calls these around a tool call. A call is a step+only when it comes from the agent that answers the run: its payload carries+``chat_stream`` (``agent.prepare_payload`` -> AgentWorkflow), and a tool+call carries a ``tool_call_id``, which is the step id.++- **The label** is the plugin's ``chat_step``, else "Used <display name>";+ ``chat_step_hidden`` plugins (the clock, a checklist) show nothing.+- **Detail.** A tool may put a short line in its result as ``_step_detail``+ ("3 events on Saturday"); it is shown under the step and taken out of the+ result before the model reads it.+- **Saved** on the run as it finishes (``ledger.record_step``), so a reload+ can say what the employee did.+- **Stopped runs.** A tool call of a run the owner stopped is not run: it+ gets a result saying so, which keeps the conversation whole (every call+ answered) while the agent's next model step ends the run.++Nothing here fails a tool call: publishing and saving are best effort.+"""++from __future__ import annotations++import time+from dataclasses import dataclass+from typing import Any, Dict, Mapping, Optional++from core.logging import get_logger++logger = get_logger(__name__)++#: The key a tool result may carry its step detail under.+STEP_DETAIL_KEY = "_step_detail"+#: What a tool call of a stopped run returns instead of running.+STOPPED_RESULT = {"stopped": True, "message": "Not run: the owner stopped this answer."}+++def step_label(node_class: Any) -> Optional[str]:+ """How a call of this tool reads as a step; None when it shows none."""+ if getattr(node_class, "chat_step_hidden", False):+ return None+ label = getattr(node_class, "chat_step", "") or ""+ if label:+ return label+ name = getattr(node_class, "display_name", "") or getattr(node_class, "type", "") or "a tool"+ return f"Used {name}"+++@dataclass+class Step:+ run_id: str+ session_id: str+ workflow_id: Optional[str]+ step_id: str+ name: str+ started: float+++def _stream_of(context: Mapping[str, Any]) -> Optional[Mapping[str, Any]]:+ stream = context.get("chat_stream")+ if not isinstance(stream, Mapping):+ return None+ if not isinstance(stream.get("run_id"), str) or not isinstance(stream.get("session_id"), str):+ return None+ return stream+++def _publish(step: Step, suffix: str, fields: Dict[str, Any], event_key: str) -> None:+ try:+ from services.chat.hub import get_chat_hub++ get_chat_hub().publish(+ run_id=step.run_id,+ session_id=step.session_id,+ workflow_id=step.workflow_id,+ suffix=suffix,+ fields=fields,+ event_key=event_key,+ )+ except Exception: # noqa: BLE001 - a step is a courtesy+ logger.warning("Chat step event could not be published", run_id=step.run_id, exc_info=True)+++def begin(context: Mapping[str, Any], node_class: Any) -> Optional[Step]:+ """Announce a tool call as a step; None when it is not one."""+ stream = _stream_of(context)+ step_id = context.get("tool_call_id")+ if stream is None or not isinstance(step_id, str) or not step_id:+ return None+ name = step_label(node_class)+ if name is None:+ return None+ workflow_id = stream.get("workflow_id")+ step = Step(+ run_id=stream["run_id"],+ session_id=stream["session_id"],+ workflow_id=workflow_id if isinstance(workflow_id, str) else None,+ step_id=step_id,+ name=name,+ started=time.monotonic(),+ )+ _publish(step, "step.started", {"step_id": step_id, "step_name": name}, f"step:{step_id}:start")+ return step+++async def end(step: Optional[Step], *, state: str, detail: Optional[str] = None) -> None:+ """Close a step (``done``, ``failed`` or ``skipped``) and save it."""+ if step is None:+ return+ fields: Dict[str, Any] = {+ "step_id": step.step_id,+ "step_name": step.name,+ "state": state,+ "duration_ms": int((time.monotonic() - step.started) * 1000),+ }+ if detail:+ fields["detail"] = detail[:200]+ _publish(step, "step.finished", fields, f"step:{step.step_id}:end")+ try:+ from core.container import container+ from services.chat import ledger++ saved = {key: fields[key] for key in ("step_id", "state", "duration_ms", "detail") if key in fields}+ saved["name"] = step.name+ await ledger.record_step(container.database(), step.run_id, saved)+ except Exception: # noqa: BLE001+ logger.warning("Chat step could not be saved", run_id=step.run_id, exc_info=True)+++async def run_stopped(context: Mapping[str, Any]) -> bool:+ """Whether the call belongs to a chat run the owner stopped."""+ run_id = context.get("chat_run_id")+ if not isinstance(run_id, str) or not run_id or not context.get("tool_call_id"):+ return False+ try:+ from core.container import container+ from services.chat import ledger++ return await ledger.is_stopping(container.database(), run_id)+ except Exception: # noqa: BLE001 - never skip a call over a failed read+ logger.warning("Could not read whether a chat run is stopping", exc_info=True)+ return False+++def take_detail(result: Any) -> Optional[str]:+ """Remove a tool's ``_step_detail`` from its result, wherever the+ plugin's result shape puts it, and return it."""+ if not isinstance(result, dict):+ return None+ detail = result.pop(STEP_DETAIL_KEY, None)+ inner = result.get("result")+ if isinstance(inner, dict):+ detail = inner.pop(STEP_DETAIL_KEY, None) or detail+ return detail if isinstance(detail, str) and detail.strip() else None+++__all__ = ["STEP_DETAIL_KEY", "STOPPED_RESULT", "Step", "begin", "end", "run_stopped", "step_label", "take_detail"]diff --git a/server/services/chat/stream.py b/server/services/chat/stream.pynew file mode 100644index 00000000..652ca543--- /dev/null+++ b/server/services/chat/stream.py@@ -0,0 +1,238 @@+"""The employee's answer, streamed into the chat as it is written.++One :class:`ChatStreamEmitter` per LLM step of the agent that answers a chat+run (``chat_stream`` in the step's payload, set by ``agent.prepare_payload``+for the agent whose output reaches a reply node). It turns the provider's+:class:`~services.llm.protocol.StreamEvent` deltas into the run's text+events (docs-internal/chat_protocol.md):++- **One segment per attempt.** The segment id is ``<run>.<iteration>.<attempt>``.+ A retried step first discards the segments its earlier attempts streamed+ (``custom`` ``opencompany.segment_discarded``), so a client never shows+ text twice.+- **Batched.** Deltas go out every ``stream.flush_ms`` or ``stream.flush_chars``+ (``config/chat_defaults.json``), and whatever is left when the step ends.+- **Held back.** Text that could still turn out to be ``NO_REPLY`` (the agent+ had nothing to say, so the reply node posts nothing) is not shown until it+ stops matching; a ``<followups>`` block, and anything after it, never+ streams (the server reads it when the run ends).+- **Reply or narration.** ``text.ended`` says ``final: true`` when the step+ answered, ``final: false`` when tool calls followed (narration beside them).++Publishing never fails the step: the hub drops what it cannot deliver, and a+problem here is logged, not raised.+"""++from __future__ import annotations++import time+from typing import Any, Callable, Dict, List, Mapping, Optional++from core.logging import get_logger+from services.approvals.contract import NO_REPLY+from services.chat.config import stream_setting+from services.llm.protocol import StreamEvent++logger = get_logger(__name__)++#: Opens the block the agent ends a reply with to offer follow-up questions.+FOLLOWUPS_OPEN = "<followups>"++Publish = Callable[..., Any]+++def _default_publish(**kwargs: Any) -> Any:+ from services.chat.hub import get_chat_hub++ return get_chat_hub().publish(**kwargs)+++def segment_id(run_id: str, iteration: int, attempt: int) -> str:+ return f"{run_id}.{iteration}.{attempt}"+++def chat_run_id_of(context: Mapping[str, Any]) -> Optional[str]:+ """The chat run a node works for (its ``run_scope``), or None."""+ scope = context.get("run_scope")+ run_id = scope.get("run_id") if isinstance(scope, Mapping) else None+ return run_id if isinstance(run_id, str) and run_id else None+++def chat_stream_for(context: Mapping[str, Any]) -> Optional[Dict[str, Any]]:+ """Where an agent's text streams, or None when it does not answer a chat+ run.++ An agent answers the run when the run's scope reached it (MachinaWorkflow+ puts ``run_scope`` on every node of the run), it is not working for+ another agent (a delegated child carries ``parent_node_id``), and its+ output goes straight to a node that posts the run's answer+ (``answers_chat_run``, Reply in Chat). Decided by the plugin attribute,+ never by a node type name.+ """+ scope = context.get("run_scope")+ if not isinstance(scope, Mapping) or context.get("parent_node_id"):+ return None+ run_id = chat_run_id_of(context)+ session_id = scope.get("session_id")+ if run_id is None or not isinstance(session_id, str) or not session_id:+ return None+ node_id = context.get("node_id")+ nodes = {str(node.get("id")): node for node in context.get("nodes") or [] if isinstance(node, Mapping)}+ from services.chat.ledger import reply_uid+ from services.node_registry import get_node_class++ for edge in context.get("edges") or []:+ if not isinstance(edge, Mapping) or edge.get("source") != node_id:+ continue+ target = nodes.get(str(edge.get("target")))+ node_class = get_node_class(str(target.get("type") or "")) if target else None+ if node_class is not None and getattr(node_class, "answers_chat_run", False):+ workflow_id = context.get("workflow_id")+ return {+ "run_id": run_id,+ "session_id": session_id,+ "workflow_id": workflow_id if isinstance(workflow_id, str) and workflow_id else session_id,+ "reply_message_id": reply_uid(run_id),+ }+ return None+++def shown_text(text: str) -> str:+ """The part of ``text`` a client may show: nothing while it could still+ be ``NO_REPLY``, and nothing from a ``<followups>`` block on (nor a tag+ that is still arriving)."""+ stripped = text.lstrip()+ if stripped and NO_REPLY.startswith(stripped):+ return ""+ cut = text.find(FOLLOWUPS_OPEN)+ if cut != -1:+ return text[:cut]+ # A tail that could still become the tag waits for the next delta.+ for length in range(min(len(FOLLOWUPS_OPEN) - 1, len(text)), 0, -1):+ if text.endswith(FOLLOWUPS_OPEN[:length]):+ return text[:-length]+ return text+++class ChatStreamEmitter:+ """Streams one step's text to a chat run's subscribers."""++ def __init__(+ self,+ *,+ run_id: str,+ session_id: str,+ workflow_id: Optional[str],+ segment: str,+ discarded: List[str],+ reply_message_id: Optional[str],+ flush_ms: Optional[float] = None,+ flush_chars: Optional[int] = None,+ publish: Publish = _default_publish,+ clock: Callable[[], float] = time.monotonic,+ ) -> None:+ self.run_id = run_id+ self.session_id = session_id+ self.workflow_id = workflow_id+ self.segment = segment+ self.reply_message_id = reply_message_id+ self._discarded = list(discarded)+ self._flush_s = (flush_ms if flush_ms is not None else stream_setting("flush_ms")) / 1000.0+ self._flush_chars = int(flush_chars if flush_chars is not None else stream_setting("flush_chars"))+ self._publish = publish+ self._clock = clock+ self._text = ""+ self._published = ""+ self._opened = False+ self._last_flush = clock()++ @classmethod+ def from_payload(cls, payload: Mapping[str, Any], *, attempt: int, **kwargs: Any) -> Optional["ChatStreamEmitter"]:+ """The emitter for an LLM step's payload, or None when the step does+ not answer a chat run."""+ stream = payload.get("chat_stream")+ if not isinstance(stream, Mapping):+ return None+ run_id = stream.get("run_id")+ session_id = stream.get("session_id")+ if not isinstance(run_id, str) or not run_id or not isinstance(session_id, str) or not session_id:+ return None+ iteration = int(payload.get("iteration") or 0)+ attempt = max(1, int(attempt))+ return cls(+ run_id=run_id,+ session_id=session_id,+ workflow_id=stream.get("workflow_id") if isinstance(stream.get("workflow_id"), str) else None,+ segment=segment_id(run_id, iteration, attempt),+ discarded=[segment_id(run_id, iteration, earlier) for earlier in range(1, attempt)],+ reply_message_id=stream.get("reply_message_id") if isinstance(stream.get("reply_message_id"), str) else None,+ **kwargs,+ )++ # ---- publishing ------------------------------------------------------++ def _send(self, suffix: str, fields: Dict[str, Any], event_key: Optional[str] = None) -> None:+ try:+ self._publish(+ run_id=self.run_id,+ session_id=self.session_id,+ workflow_id=self.workflow_id,+ suffix=suffix,+ fields=fields,+ event_key=event_key,+ )+ except Exception: # noqa: BLE001 - streaming is a courtesy; the answer is saved anyway+ logger.warning("Chat stream event could not be published", run_id=self.run_id, suffix=suffix, exc_info=True)++ def begin(self) -> None:+ """Withdraw what earlier attempts of this step streamed."""+ for segment in self._discarded:+ self._send(+ "custom",+ {"name": "opencompany.segment_discarded", "value": {"message_id": segment}},+ event_key=f"discard:{segment}",+ )++ def _flush(self) -> None:+ visible = shown_text(self._text)+ if len(visible) <= len(self._published) or not visible.startswith(self._published):+ return+ delta = visible[len(self._published):]+ if not self._opened:+ if not visible.strip():+ return+ self._opened = True+ self._send("text.started", {"message_id": self.segment, "role": "assistant"}, event_key=f"start:{self.segment}")+ self._published = visible+ self._last_flush = self._clock()+ self._send("text.content", {"message_id": self.segment, "delta": delta})++ async def __call__(self, event: StreamEvent) -> None:+ if event.kind != "text" or not event.delta:+ return+ self._text += event.delta+ pending = len(self._text) - len(self._published)+ if pending >= self._flush_chars or self._clock() - self._last_flush >= self._flush_s:+ self._flush()++ def end(self, *, final: bool) -> None:+ """The step finished: send what is left and close the segment."""+ self._flush()+ if not self._opened:+ return+ fields: Dict[str, Any] = {"message_id": self.segment, "final": bool(final)}+ if final and self.reply_message_id:+ fields["reply_message_id"] = self.reply_message_id+ self._send("text.ended", fields, event_key=f"end:{self.segment}")++ @property+ def text(self) -> str:+ """Everything the provider wrote so far, held-back parts included."""+ return self._text++ @property+ def published(self) -> str:+ return self._published+++__all__ = ["ChatStreamEmitter", "FOLLOWUPS_OPEN", "chat_run_id_of", "chat_stream_for", "segment_id", "shown_text"]diff --git a/server/services/chat/watchdog.py b/server/services/chat/watchdog.pyindex ebc98d10..23b5dcf2 100644--- a/server/services/chat/watchdog.py+++ b/server/services/chat/watchdog.py@@ -44,6 +44,27 @@ async def temporal_workflow_closed(workflow_id: str, run_id: Optional[str]) -> Oreturn description.status != WorkflowExecutionStatus.RUNNING+async def cancel_temporal_workflow(workflow_id: str, run_id: Optional[str]) -> None:+ """Ask Temporal to cancel a workflow (a stopped run's). Does nothing when+ Temporal cannot be reached; a workflow that already closed is fine."""+ try:+ from core.container import container++ wrapper = container.temporal_client()+ except Exception: # noqa: BLE001 - Temporal is optional+ return+ client = getattr(wrapper, "client", None) if wrapper is not None else None+ if client is None:+ return+ from temporalio.service import RPCError++ try:+ await client.get_workflow_handle(workflow_id, run_id=run_id).cancel()+ except RPCError:+ # Already closed, or unknown: nothing left to cancel.+ return++class ChatRunWatchdog:def __init__(self, database: Any, *, interval: Optional[float] = None) -> None:self.database = database@@ -67,7 +88,11 @@ class ChatRunWatchdog:async def _loop(self) -> None:while True:try:- await ledger.sweep(self.database, temporal_status=temporal_workflow_closed)+ await ledger.sweep(+ self.database,+ temporal_status=temporal_workflow_closed,+ temporal_cancel=cancel_temporal_workflow,+ )except asyncio.CancelledError:raiseexcept Exception: # noqa: BLE001 - the next sweep tries again@@ -75,4 +100,4 @@ class ChatRunWatchdog:await asyncio.sleep(self.interval)-__all__ = ["ChatRunWatchdog", "temporal_workflow_closed"]+__all__ = ["ChatRunWatchdog", "cancel_temporal_workflow", "temporal_workflow_closed"]diff --git a/server/services/plugin/base.py b/server/services/plugin/base.pyindex b083af18..4c43c2a1 100644--- a/server/services/plugin/base.py+++ b/server/services/plugin/base.py@@ -274,6 +274,17 @@ class BaseNode:tool_name: ClassVar[str] = ""tool_description: ClassVar[str] = ""+ # Chat (docs-internal/chat_protocol.md). ``answers_chat_run``: this node+ # posts the answer to a chat run (Reply in Chat), so the agent whose+ # output it takes streams its text into the chat as it writes.+ # ``chat_step``: how a call of this tool reads as a working step while+ # the employee answers ("Checked Google Calendar"); empty falls back to+ # "Used <display name>". ``chat_step_hidden``: housekeeping (the clock,+ # a checklist) that shows no step.+ answers_chat_run: ClassVar[bool] = False+ chat_step: ClassVar[str] = ""+ chat_step_hidden: ClassVar[bool] = False+# Set by __init_subclass__: {op_name: OperationSpec}_operations: ClassVar[Dict[str, OperationSpec]] = {}# Flag so concrete subclasses auto-register; abstract kinds don't.@@ -1086,6 +1097,29 @@ class BaseNode:)return result+ # Chat (services/chat/steps.py): a tool call of a run the owner+ # stopped is answered without running; a tool call of the agent+ # answering a chat run shows as a working step.+ from services.chat import steps as chat_steps++ if await chat_steps.run_stopped(context):+ result = {+ "success": True,+ "node_id": node_id,+ "node_type": cls.type,+ "result": dict(chat_steps.STOPPED_RESULT),+ "execution_id": execution_id,+ "timestamp": datetime.now().isoformat(),+ }+ await broadcaster.update_node_status(+ node_id,+ "skipped",+ {"stopped": True, "execution_id": execution_id},+ workflow_id=workflow_id,+ )+ return result+ chat_step = chat_steps.begin(context, cls)+# Broadcast executing — UI cyan-glow.await broadcaster.update_node_status(node_id,@@ -1180,7 +1214,12 @@ class BaseNode:# / TriggerNode return the {success, result} envelope.# cls.interpret_result() normalizes both into (success,# payload, error_message).+ step_detail = chat_steps.take_detail(result)success, payload, error = cls.interpret_result(result)+ if chat_step is not None and not (+ not success and result.get("error_type") == NODE_WAIT_INTERRUPTED+ ):+ await chat_steps.end(chat_step, state="done" if success else "failed", detail=step_detail)if not success and result.get("error_type") == NODE_WAIT_INTERRUPTED:# The body stopped waiting because this worker is going# away, not because the work failed. Fail the attempt@@ -1229,6 +1268,7 @@ class BaseNode:raiseerror_msg = f"{type(e).__name__}: {e}"activity.logger.error(f"Node {node_id} crashed: {error_msg}")+ await chat_steps.end(chat_step, state="failed")await broadcaster.update_node_status(node_id,"error",diff --git a/server/services/temporal/agent_activities.py b/server/services/temporal/agent_activities.pyindex 57a338fb..9d7ff417 100644--- a/server/services/temporal/agent_activities.py+++ b/server/services/temporal/agent_activities.py@@ -317,11 +317,86 @@ async def _await_with_llm_heartbeats(await asyncio.gather(task, return_exceptions=True)+#: ``_until_chat_run_stops`` result when the owner stopped the run.+_STOPPED = object()+++async def _chat_run_stopping(run_id: str) -> bool:+ """Whether the owner asked this chat run to stop. A failed read is+ "no": stopping is never worth failing a step over."""+ try:+ from core.container import container+ from services.chat import ledger++ return await ledger.is_stopping(container.database(), run_id)+ except Exception: # noqa: BLE001+ logger.warning("Could not read whether a chat run is stopping", exc_info=True)+ return False+++async def _until_chat_run_stops(call: Any, run_id: Optional[str]) -> Any:+ """Await a model call, or return ``_STOPPED`` as soon as the chat run it+ works for is stopped (polled every ``runs.stop_poll_s``: a heartbeat-+ borne cancel arrives too late to stop a stream mid-sentence)."""+ if not run_id:+ return await call+ from services.chat.config import runs_setting++ poll = max(0.05, runs_setting("stop_poll_s"))++ async def watch() -> None:+ while not await _chat_run_stopping(run_id):+ await asyncio.sleep(poll)++ task = asyncio.ensure_future(call)+ watcher = asyncio.ensure_future(watch())+ try:+ done, _ = await asyncio.wait({task, watcher}, return_when=asyncio.FIRST_COMPLETED)+ if task in done:+ return task.result()+ return _STOPPED+ finally:+ for pending in (task, watcher):+ if not pending.done():+ pending.cancel()+ await asyncio.gather(task, watcher, return_exceptions=True)+++async def _stopped_llm_step(payload: Dict[str, Any], messages: List[Any], *, partial: str) -> Dict[str, Any]:+ """The step's result when the owner stopped the run: what was written so+ far (possibly nothing) as the final answer. The conversation keeps the+ turn as far as it went: the request (its tool results included, so no+ call is left unanswered) and the partial answer, marked stopped; nothing+ is added when nothing was written."""+ from services.llm.protocol import ContentBlock, Message, Usage, message_to_wire++ sent = [dict(message_to_wire(message)) for message in messages]+ assistant = Message(+ role="assistant",+ content=partial,+ blocks=[ContentBlock(type="text", text=partial, metadata={"stopped": True})] if partial else [],+ )+ assistant_wire = dict(message_to_wire(assistant))+ await _save_conversation(payload, sent=sent, assistant_wire=assistant_wire if partial else None)+ result: Dict[str, Any] = {+ "kind": "final",+ "assistant_message": assistant_wire,+ "content": partial,+ "thinking": None,+ "usage": asdict(Usage()),+ "stopped": True,+ }+ if payload.get("include_finish_reason"):+ result["finish_reason"] = "stopped"+ return result++async def _execute_native_llm_step(payload: Dict[str, Any]) -> Dict[str, Any]:"""Run one native-SDK model turn and retain the v1 activity envelope."""from core.container import containerfrom services.agent_runtime import run_native_llm_step+ from services.chat.stream import ChatStreamEmitter, shown_textfrom services.llm.messages import filter_empty_messagesfrom services.llm.protocol import (LLMError,@@ -357,27 +432,39 @@ async def _execute_native_llm_step(payload: Dict[str, Any]) -> Dict[str, Any]:if isinstance(thinking_data, dict)else None)+ # A chat run the owner stopped ends at this step, before paying for it.+ chat_run_id = payload.get("chat_run_id")+ if isinstance(chat_run_id, str) and chat_run_id and await _chat_run_stopping(chat_run_id):+ return await _stopped_llm_step(payload, messages, partial="")+ emitter = ChatStreamEmitter.from_payload(payload, attempt=_llm_activity_attempt())+ if emitter is not None:+ emitter.begin()+api_key = await _resolve_activity_api_key(payload)try:response = await _await_with_llm_heartbeats(- run_native_llm_step(- container.chat_unifier(),- provider=payload["provider"],- api_key=api_key,- messages=messages,- model=payload["model"],- temperature=payload.get("temperature", 0.7),- max_tokens=payload.get("max_tokens", 4096),- thinking=thinking,- tools=tool_defs,- context_management=payload.get("context_management"),- # Temporal owns retries; nested SDK retries would bypass- # quota classification and hide the wait from the UI.- sdk_max_retries=0,- explicit_max_retries=0,- # Keep structured metadata until this activity boundary,- # where it is converted to a safe Temporal failure.- translate_errors=False,+ _until_chat_run_stops(+ run_native_llm_step(+ container.chat_unifier(),+ provider=payload["provider"],+ api_key=api_key,+ messages=messages,+ model=payload["model"],+ temperature=payload.get("temperature", 0.7),+ max_tokens=payload.get("max_tokens", 4096),+ thinking=thinking,+ tools=tool_defs,+ context_management=payload.get("context_management"),+ # Temporal owns retries; nested SDK retries would bypass+ # quota classification and hide the wait from the UI.+ sdk_max_retries=0,+ explicit_max_retries=0,+ # Keep structured metadata until this activity boundary,+ # where it is converted to a safe Temporal failure.+ translate_errors=False,+ on_event=emitter,+ ),+ chat_run_id if isinstance(chat_run_id, str) else None,),detail=f"LLM step waiting: {payload.get('model')}",)@@ -396,6 +483,17 @@ async def _execute_native_llm_step(payload: Dict[str, Any]) -> Dict[str, Any]:# SDK bodies in chained tracebacks can include prompts or credentials.raise failure from None+ if response is _STOPPED:+ # What the owner saw so far is the answer: the reply node posts it,+ # saved as stopped, and the run finishes ``stopped``.+ partial = shown_text(emitter.text).rstrip() if emitter is not None else ""+ if emitter is not None:+ emitter.end(final=True)+ return await _stopped_llm_step(payload, messages, partial=partial)+ if emitter is not None:+ # Text beside tool calls is narration; without them it is the reply.+ emitter.end(final=not response.tool_calls)+assistant = response.assistant_message or Message(role="assistant",content=response.content or "",@@ -456,7 +554,7 @@ async def _save_conversation(payload: Dict[str, Any],*,sent: List[Dict[str, Any]],- assistant_wire: Dict[str, Any],+ assistant_wire: Optional[Dict[str, Any]],) -> None:"""Persist the full transcript for this turn, when a key is attached.@@ -464,7 +562,8 @@ async def _save_conversation(exact list handed to the provider; ``sent + [assistant]`` IS theconversation after this turn, saved whole (last-write-wins upsert perkey). Living inside the LLM activity means per-turn durability with- zero extra activities and zero Temporal-history payload.+ zero extra activities and zero Temporal-history payload. A step stopped+ before the model wrote anything saves ``sent`` alone."""key = payload.get("conversation_key")if not isinstance(key, dict):@@ -473,7 +572,7 @@ async def _save_conversation(from core.container import containerfrom services.agent_context import save_conversation- messages = [*sent, assistant_wire]+ messages = [*sent, assistant_wire] if assistant_wire is not None else list(sent)saved_bytes = len(json.dumps(messages, default=str).encode("utf-8"))if saved_bytes > _SEED_TRANSCRIPT_MAX_BYTES // 2:# The next firing refuses to load a row over the cap@@ -1169,10 +1268,15 @@ async def prepare_agent_payload(context: Dict[str, Any]) -> Dict[str, Any]:}try:from services.agent_context import load_conversation-- conversation = await load_conversation(- database,- **conversation_key,+ from services.agent_context.conversation import close_unanswered_tool_calls++ # A run stopped or crashed mid-tool left calls without results,+ # which no provider accepts: answer them before continuing.+ conversation = close_unanswered_tool_calls(+ await load_conversation(+ database,+ **conversation_key,+ ))except Exception as exc:raise ApplicationError(@@ -1381,8 +1485,13 @@ async def prepare_agent_payload(context: Dict[str, Any]) -> Dict[str, Any]:CONTEXT_PRESSURE_VERSION,transcript_budget_bytes,)+ from services.chat.stream import chat_run_id_of, chat_stream_forreturn {+ # The chat run this agent works for (stops when the owner presses+ # Stop), and, for the agent that answers it, where its text streams.+ "chat_run_id": chat_run_id_of(context),+ "chat_stream": chat_stream_for(context),"node_id": node_id,"node_type": node_type,"workflow_id": workflow_id,diff --git a/server/services/temporal/agent_workflow.py b/server/services/temporal/agent_workflow.pyindex 6156ab83..e0e36d3a 100644--- a/server/services/temporal/agent_workflow.py+++ b/server/services/temporal/agent_workflow.py@@ -901,6 +901,20 @@ class AgentWorkflow:# indistinguishable from a normal completion and gets# returned to the user as a truncated final answer."include_finish_reason": True,+ # The chat run this agent works for: the step stops when the+ # owner presses Stop, and the agent that answers the run+ # streams its text into the chat (services/chat/stream.py).+ # Activity input only, so recorded histories replay as before.+ **(+ {"chat_run_id": payload["chat_run_id"]}+ if payload.get("chat_run_id")+ else {}+ ),+ **(+ {"chat_stream": payload["chat_stream"]}+ if payload.get("chat_stream")+ else {}+ ),}# Transient provider failures (429 rate limit, 5xx, network)@@ -1785,6 +1799,19 @@ class AgentWorkflow:call.get("id") or f"{iteration + 1}:{call_index + 1}"),**call_metadata,+ # Chat (services/chat/steps.py): a call of a stopped run+ # is not run, and a call of the agent answering the run+ # shows as a working step. Activity input only.+ **(+ {"chat_run_id": payload["chat_run_id"]}+ if payload.get("chat_run_id")+ else {}+ ),+ **(+ {"chat_stream": payload["chat_stream"]}+ if payload.get("chat_stream")+ else {}+ ),}tool_activity_name = (diff --git a/server/tests/services/agent_context/test_conversation.py b/server/tests/services/agent_context/test_conversation.pyindex e14f0b45..360159f5 100644--- a/server/tests/services/agent_context/test_conversation.py+++ b/server/tests/services/agent_context/test_conversation.py@@ -114,6 +114,31 @@ async def test_blocked_tool_turn_is_saved_once_and_stale_results_are_ignored(conassert len(await load_conversation(conversation_database, **_KEY)) == 1+def test_calls_a_stopped_run_left_open_are_answered():+ """A run ended between asking for tools and getting their results (Stop+ while a tool ran, a crash) leaves calls no provider will continue from;+ each open call gets a result after the ones the turn already has."""+ from services.agent_context.conversation import UNANSWERED_TOOL_RESULT, close_unanswered_tool_calls+ from services.llm.protocol import ToolCall++ assistant = dict(message_to_wire(Message(+ role="assistant",+ tool_calls=[ToolCall(id="c1", name="search", args={}), ToolCall(id="c2", name="calendar", args={})],+ )))+ answered = dict(message_to_wire(Message(role="tool", tool_call_id="c1", name="search", content="3 results")))+ stored = [_wire("user", "Book Saturday"), assistant, answered]+ repaired = close_unanswered_tool_calls(stored)+ assert [(m["role"], m.get("tool_call_id")) for m in repaired] == [+ ("user", None), ("assistant", None), ("tool", "c1"), ("tool", "c2"),+ ]+ assert (repaired[-1]["content"], repaired[-1]["name"]) == (UNANSWERED_TOOL_RESULT, "calendar")+ # A closed turn, or no assistant turn at all, is returned as it is.+ assert close_unanswered_tool_calls(repaired) is repaired+ plain = [_wire("user", "hi"), _wire("assistant", "hello")]+ assert close_unanswered_tool_calls(plain) is plain+ assert close_unanswered_tool_calls([]) == []++def _sans_ts(messages: list[dict]) -> list[dict]:return [{k: v for k, v in m.items() if k != "ts"} for m in messages]diff --git a/server/tests/services/chat/test_handlers.py b/server/tests/services/chat/test_handlers.pyindex 1fcf821c..aac12324 100644--- a/server/tests/services/chat/test_handlers.py+++ b/server/tests/services/chat/test_handlers.py@@ -128,15 +128,24 @@ async def test_each_message_says_how_its_run_ended(chat):await ledger.post_reply(chat.database, run=run, node_id="n", text="Booked.", execution_id="gen-1")await ledger.finish_run(chat.database, run_id=answered["run_id"], temporal_workflow_id="tw2", temporal_run_id="tr", success=True)+ await ledger.record_step(chat.database, answered["run_id"], {"step_id": "c1", "state": "done", "name": "Checked Google Calendar"})+result = await chat.handlers.handle_get_chat_messages({"session_id": "wf", "all_generations": True}, None)first, second, reply = result["messages"]+ # How long each run worked comes with it.+ assert all(message["run"].pop("duration_ms") >= 0 for message in (first, second, reply))assert first["run"] == {"run_id": failed["run_id"],"state": "error","outcome": None,"error": {"message": "Calendar said no", "code": "run_failed", "hint": "Reconnect Google"},}- assert second["run"] == reply["run"] == {"run_id": answered["run_id"], "state": "finished", "outcome": "success"}+ assert second["run"] == reply["run"] == {+ "run_id": answered["run_id"],+ "state": "finished",+ "outcome": "success",+ "steps": [{"step_id": "c1", "state": "done", "name": "Checked Google Calendar"}],+ }assert result["active_runs"] == []diff --git a/server/tests/services/chat/test_step_activity.py b/server/tests/services/chat/test_step_activity.pynew file mode 100644index 00000000..43f7204c--- /dev/null+++ b/server/tests/services/chat/test_step_activity.py@@ -0,0 +1,107 @@+"""A tool call run as a node activity (``BaseNode.as_activity``) for the+agent answering a chat run: it shows as a working step with the detail the+tool gave, the detail never reaches the model, and a call of a run the owner+stopped is answered without running."""++from __future__ import annotations++from typing import Any, Dict, List+from unittest.mock import AsyncMock, MagicMock++import pytest+from temporalio.testing import ActivityEnvironment++import nodes # noqa: F401 -- populate plugin registry+from services.chat import ledger, steps+from services.node_registry import get_node_class++STREAM = {"run_id": "r_1", "session_id": "wf", "workflow_id": "wf", "reply_message_id": "a_r_1"}+CONTEXT = {+ "node_id": "wf:braveSearch:1",+ "workflow_id": "wf",+ "node_data": {},+ "tool_call_id": "call_1",+ "chat_run_id": "r_1",+ "chat_stream": STREAM,+}+++def surroundings(monkeypatch):+ from core.container import container+ import services.chat.hub as hub_module++ workflow_service = MagicMock()+ workflow_service.execute_node = AsyncMock(+ return_value={"success": True, "result": {"results": ["a", "b", "c"], "_step_detail": "3 results"}}+ )+ monkeypatch.setattr(container, "workflow_service", lambda: workflow_service)+ monkeypatch.setattr(container, "database", lambda: "db")++ broadcaster = MagicMock()+ broadcaster.update_node_status = AsyncMock()+ broadcaster.update_node_output = AsyncMock()+ monkeypatch.setattr("services.status_broadcaster.get_status_broadcaster", lambda: broadcaster)++ published: List[Dict[str, Any]] = []++ class Hub:+ def publish(self, **kwargs):+ published.append(kwargs)++ monkeypatch.setattr(hub_module, "get_chat_hub", lambda: Hub())+ saved: List[Any] = []++ async def record_step(database, run_id, step):+ saved.append((run_id, step))++ stopping = {"value": False}++ async def is_stopping(database, run_id):+ return stopping["value"]++ monkeypatch.setattr(ledger, "record_step", record_step)+ monkeypatch.setattr(ledger, "is_stopping", is_stopping)+ return type(+ "Surroundings",+ (),+ {"service": workflow_service, "broadcaster": broadcaster, "published": published, "saved": saved, "stopping": stopping},+ )+++async def test_a_tool_call_shows_as_a_step(surroundings):+ activity_fn = get_node_class("braveSearch").as_activity()+ result = await ActivityEnvironment().run(activity_fn, dict(CONTEXT))+ # The model reads the result without the detail.+ assert result["result"] == {"results": ["a", "b", "c"]}+ started, finished = surroundings.published+ assert (started["suffix"], started["fields"]["step_name"]) == ("step.started", "Searched the web")+ assert (finished["suffix"], finished["fields"]["state"], finished["fields"]["detail"]) == ("step.finished", "done", "3 results")+ [(run_id, step)] = surroundings.saved+ assert run_id == "r_1" and step["detail"] == "3 results"+++async def test_a_failed_tool_call_is_a_failed_step(surroundings):+ surroundings.service.execute_node.return_value = {"success": False, "error": "quota", "error_type": "NodeUserError"}+ activity_fn = get_node_class("braveSearch").as_activity()+ await ActivityEnvironment().run(activity_fn, dict(CONTEXT))+ assert surroundings.published[-1]["fields"]["state"] == "failed"+++async def test_a_call_of_a_stopped_run_is_not_run(surroundings):+ surroundings.stopping["value"] = True+ activity_fn = get_node_class("braveSearch").as_activity()+ result = await ActivityEnvironment().run(activity_fn, dict(CONTEXT))+ assert result["success"] is True and result["result"] == steps.STOPPED_RESULT+ surroundings.service.execute_node.assert_not_awaited()+ assert surroundings.published == []+ statuses = [call.args[1] for call in surroundings.broadcaster.update_node_status.await_args_list]+ assert statuses == ["skipped"]+++async def test_a_node_outside_a_chat_run_is_untouched(surroundings):+ surroundings.stopping["value"] = True+ activity_fn = get_node_class("braveSearch").as_activity()+ result = await ActivityEnvironment().run(activity_fn, {"node_id": "wf:braveSearch:1", "workflow_id": "wf", "node_data": {}})+ surroundings.service.execute_node.assert_awaited_once()+ assert result["success"] is True and surroundings.published == []diff --git a/server/tests/services/chat/test_steps.py b/server/tests/services/chat/test_steps.pynew file mode 100644index 00000000..a1510228--- /dev/null+++ b/server/tests/services/chat/test_steps.py@@ -0,0 +1,133 @@+"""Working steps (services/chat/steps.py): a tool call of the agent answering+a chat run shows as a step labelled by its plugin, hidden tools show none,+the step detail is taken out of the result, finished steps are saved on the+run, and a tool call of a stopped run is not run."""++from __future__ import annotations++from types import SimpleNamespace+from typing import Any, Dict, List++import pytest++from services.chat import ledger, steps++STREAM = {"run_id": "r_1", "session_id": "wf", "workflow_id": "wf", "reply_message_id": "a_r_1"}+++def published(monkeypatch):+ sent: List[Dict[str, Any]] = []++ class Hub:+ def publish(self, **kwargs):+ sent.append(kwargs)++ import services.chat.hub as hub_module++ monkeypatch.setattr(hub_module, "get_chat_hub", lambda: Hub())+ return sent+++def saved(monkeypatch):+ calls: List[Any] = []++ async def record_step(database, run_id, step):+ calls.append((run_id, step))++ monkeypatch.setattr(ledger, "record_step", record_step)+ import core.container as container_module++ monkeypatch.setattr(container_module, "container", SimpleNamespace(database=lambda: "db"))+ return calls+++def test_labels_come_from_the_plugin():+ assert steps.step_label(SimpleNamespace(chat_step="Searched the web", display_name="Brave Search")) == "Searched the web"+ assert steps.step_label(SimpleNamespace(chat_step="", display_name="Gmail")) == "Used Gmail"+ assert steps.step_label(SimpleNamespace(chat_step_hidden=True, display_name="Current Time")) is None+++def test_the_shipped_labels():+ from nodes.search.brave_search import BraveSearchNode+ from nodes.tool.current_time_tool import CurrentTimeToolNode+ from nodes.tool.write_todos import WriteTodosNode++ assert steps.step_label(BraveSearchNode) == "Searched the web"+ assert steps.step_label(CurrentTimeToolNode) is None+ assert steps.step_label(WriteTodosNode) is None+++async def test_a_tool_call_of_the_answering_agent_is_a_step(published, saved):+ tool = SimpleNamespace(chat_step="Searched the web")+ step = steps.begin({"chat_stream": STREAM, "tool_call_id": "call_1"}, tool)+ assert step is not None+ await steps.end(step, state="done", detail="3 results for salons near me" + "." * 300)+ started, finished = published+ assert (started["suffix"], started["fields"], started["event_key"]) == (+ "step.started",+ {"step_id": "call_1", "step_name": "Searched the web"},+ "step:call_1:start",+ )+ assert finished["suffix"] == "step.finished" and finished["event_key"] == "step:call_1:end"+ assert finished["fields"]["state"] == "done" and len(finished["fields"]["detail"]) == 200+ assert finished["fields"]["duration_ms"] >= 0+ [(run_id, record)] = saved+ assert run_id == "r_1"+ assert record["name"] == "Searched the web" and record["step_id"] == "call_1" and record["state"] == "done"+++def test_other_calls_are_not_steps(published):+ tool = SimpleNamespace(chat_step="Searched the web")+ # No chat stream (another agent, a canvas run), no call id, a hidden tool.+ assert steps.begin({"tool_call_id": "call_1"}, tool) is None+ assert steps.begin({"chat_stream": STREAM}, tool) is None+ assert steps.begin({"chat_stream": {"run_id": "r_1"}, "tool_call_id": "call_1"}, tool) is None+ assert steps.begin({"chat_stream": STREAM, "tool_call_id": "call_1"}, SimpleNamespace(chat_step_hidden=True)) is None+ assert published == []+++async def test_ending_no_step_does_nothing(saved):+ await steps.end(None, state="done")+ assert saved == []+++ ("result", "remaining"),+ [+ ({"success": True, "_step_detail": "Found 3", "x": 1}, {"success": True, "x": 1}),+ ({"success": True, "result": {"_step_detail": "Found 3", "x": 1}}, {"success": True, "result": {"x": 1}}),+ ],+)+def test_the_detail_is_taken_out_of_the_result(result, remaining):+ assert steps.take_detail(result) == "Found 3"+ assert result == remaining+++def test_no_detail():+ assert steps.take_detail({"success": True}) is None+ assert steps.take_detail({"_step_detail": " "}) is None+ assert steps.take_detail("plain") is None+++async def test_a_call_of_a_stopped_run_is_not_run(monkeypatch):+ import core.container as container_module++ monkeypatch.setattr(container_module, "container", SimpleNamespace(database=lambda: "db"))+ stopping = {"r_1": True, "r_2": False}++ async def is_stopping(database, run_id):+ return stopping[run_id]++ monkeypatch.setattr(ledger, "is_stopping", is_stopping)+ assert await steps.run_stopped({"chat_run_id": "r_1", "tool_call_id": "call_1"}) is True+ assert await steps.run_stopped({"chat_run_id": "r_2", "tool_call_id": "call_1"}) is False+ # Only an agent's tool call is skipped: a node of the run itself runs.+ assert await steps.run_stopped({"chat_run_id": "r_1"}) is False++ async def broken(*_args):+ raise RuntimeError("database is locked")++ monkeypatch.setattr(ledger, "is_stopping", broken)+ assert await steps.run_stopped({"chat_run_id": "r_1", "tool_call_id": "call_1"}) is Falsediff --git a/server/tests/services/chat/test_stop.py b/server/tests/services/chat/test_stop.pynew file mode 100644index 00000000..6d25dc80--- /dev/null+++ b/server/tests/services/chat/test_stop.py@@ -0,0 +1,221 @@+"""Stopping a chat run (services/chat/ledger.py, services/chat/handlers.py):+a run nothing picked up ends at once and frees the lane, and a workflow that+picks it up later claims it stopped; a running run moves to ``stopping``, its+reply so far is saved stopped and it finishes ``stopped``; the watchdog ends a+run still stopping after the grace and cancels its workflow. Steps are saved+on the run as they finish."""++from __future__ import annotations++import asyncio+from datetime import datetime, timedelta, timezone++from sqlalchemy import update++from models.chat import ChatRun+from services.chat import ledger+from tests.services.chat._helpers import FakeSocket, talking++NOW = datetime(2026, 10, 4, 12, 0, tzinfo=timezone.utc)+CLAIM = dict(temporal_workflow_id="tw-1", temporal_run_id="tr-1")+++async def admit(database, *, state="pending", session_id="wf"):+ admission = await ledger.admit_message(+ database, session_id=session_id, workflow_id=session_id, execution_id="gen-1",+ text="Book Saturday", track=True, state=state,+ )+ return admission.run+++async def running(database):+ run = await admit(database)+ return await ledger.start_run(database, run_id=run.run_id, **CLAIM)+++def kinds(socket):+ return [event["type"].removeprefix("com.opencompany.chat.run.") for event in socket.events()]+++# ----- a run nothing picked up -----+++async def test_stopping_a_pending_run_ends_it_at_once(database, hub):+ socket = FakeSocket()+ hub.subscribe(socket, "wf")+ run = await admit(database)+ stopped = await ledger.request_stop(database, run.run_id)+ assert (stopped.state, stopped.outcome, stopped.result) == ("stopped", "stopped", {"no_reply": True})+ assert stopped.stop_requested_at is not None and stopped.finished_at is not None+ await asyncio.sleep(0)+ [event] = socket.events()+ assert event["type"] == "com.opencompany.chat.run.finished"+ assert event["data"]["outcome"] == {"type": "stopped"}+ # The lane is free at once.+ assert await admit(database) is not None+++async def test_stopping_a_queued_run_ends_it_before_resume(database, hub):+ run = await admit(database, state="queued")+ assert (await ledger.request_stop(database, run.run_id)).state == "stopped"+++async def test_a_workflow_that_picks_up_a_stopped_run_claims_it_stopped(database, hub):+ socket = FakeSocket()+ hub.subscribe(socket, "wf")+ run = await admit(database)+ await ledger.request_stop(database, run.run_id)+ claimed = await ledger.start_run(database, run_id=run.run_id, **CLAIM)+ # Claimed as it is, so the agent stops at its first step instead of+ # answering untracked; nothing says it started.+ assert claimed is not None and claimed.state == "stopped"+ assert (claimed.temporal_workflow_id, claimed.temporal_run_id) == ("tw-1", "tr-1")+ assert await ledger.is_stopping(database, run.run_id) is True+ # Another workflow does not get it, and the claimant's finish changes nothing.+ assert await ledger.start_run(database, run_id=run.run_id, temporal_workflow_id="tw-2", temporal_run_id="tr-1") is None+ finished = await ledger.finish_run(database, run_id=run.run_id, success=True, **CLAIM)+ assert (finished.state, finished.outcome) == ("stopped", "stopped")+ await asyncio.sleep(0)+ assert kinds(socket) == ["finished"]+++# ----- a running run -----+++async def test_stopping_a_running_run_lets_it_stop_itself(database, hub):+ socket = FakeSocket()+ hub.subscribe(socket, "wf")+ run = await running(database)+ stopping = await ledger.request_stop(database, run.run_id)+ assert stopping.state == "stopping" and stopping.stop_requested_at is not None+ assert await ledger.is_stopping(database, run.run_id) is True+ # A second press changes nothing and says nothing new.+ assert (await ledger.request_stop(database, run.run_id)).state == "stopping"+ await asyncio.sleep(0)+ assert kinds(socket) == ["started", "custom"]+ assert socket.events()[-1]["data"]["name"] == "opencompany.stopping"+++async def test_a_stopped_runs_reply_so_far_is_saved_stopped(database, hub):+ run = await running(database)+ await ledger.request_stop(database, run.run_id)+ reply = await ledger.post_reply(database, run=run, node_id="wf:chatReply:1", text="I was about to", execution_id="gen-1")+ assert (reply["uid"], reply["status"]) == (run.reply_message_uid, "stopped")+ finished = await ledger.finish_run(database, run_id=run.run_id, success=True, **CLAIM)+ assert (finished.state, finished.outcome) == ("stopped", "stopped")+ assert finished.result == {"reply_message_id": run.reply_message_uid}+ # The lane is free again.+ assert await admit(database) is not None+++async def test_a_run_that_stopped_before_writing_ends_without_a_reply(database, hub):+ run = await running(database)+ await ledger.request_stop(database, run.run_id)+ # A failure while stopping is still a stop: the owner asked for it.+ finished = await ledger.finish_run(database, run_id=run.run_id, success=False, error="cancelled", **CLAIM)+ assert (finished.state, finished.outcome, finished.result) == ("stopped", "stopped", {"no_reply": True})+++async def test_a_run_that_ended_is_not_stopped(database, hub):+ run = await running(database)+ await ledger.finish_run(database, run_id=run.run_id, success=True, **CLAIM)+ assert (await ledger.request_stop(database, run.run_id)).state == "finished"+ assert await ledger.request_stop(database, "r_unknown") is None+ assert await ledger.is_stopping(database, "r_unknown") is False+++# ----- the watchdog -----+++async def stop_requested(database, run_id, moment):+ async with database.get_session() as session:+ await session.execute(update(ChatRun).where(ChatRun.run_id == run_id).values(stop_requested_at=moment))+ await session.commit()+++async def test_the_watchdog_ends_a_run_still_stopping_after_the_grace(database, hub):+ from services.chat.config import runs_setting++ run = await running(database)+ await ledger.request_stop(database, run.run_id)+ await ledger.post_reply(database, run=run, node_id="n", text="Half an answer", execution_id="gen-1")+ cancelled = []++ async def cancel(workflow_id, temporal_run_id):+ cancelled.append((workflow_id, temporal_run_id))++ grace = timedelta(seconds=runs_setting("stop_grace_s"))+ await stop_requested(database, run.run_id, NOW - grace + timedelta(seconds=5))+ assert await ledger.sweep(database, now=NOW, temporal_cancel=cancel) == []+ assert cancelled == []++ await stop_requested(database, run.run_id, NOW - grace - timedelta(seconds=1))+ assert await ledger.sweep(database, now=NOW, temporal_cancel=cancel) == [run.run_id]+ assert cancelled == [("tw-1", "tr-1")]+ ended = await ledger.get_run(database, run.run_id)+ assert (ended.state, ended.outcome, ended.result) == ("stopped", "stopped", {"reply_message_id": run.reply_message_uid})+++async def test_a_failed_cancel_still_ends_the_run(database, hub):+ run = await running(database)+ await ledger.request_stop(database, run.run_id)+ await stop_requested(database, run.run_id, NOW - timedelta(days=1))++ async def broken(*_):+ raise RuntimeError("temporal is down")++ assert await ledger.sweep(database, now=NOW, temporal_cancel=broken) == [run.run_id]+ assert (await ledger.get_run(database, run.run_id)).state == "stopped"+++# ----- steps -----+++async def test_steps_are_saved_once_each_up_to_the_cap(database, hub, monkeypatch):+ run = await running(database)+ await ledger.record_step(database, run.run_id, {"step_id": "c1", "state": "done", "name": "Searched the web"})+ await ledger.record_step(database, run.run_id, {"step_id": "c2", "state": "failed", "name": "Used Gmail"})+ # A retried step replaces its earlier save.+ await ledger.record_step(database, run.run_id, {"step_id": "c1", "state": "done", "name": "Searched the web", "duration_ms": 900})+ saved = (await ledger.get_run(database, run.run_id)).steps+ assert [(step["step_id"], step.get("duration_ms")) for step in saved] == [("c2", None), ("c1", 900)]++ monkeypatch.setattr(ledger, "steps_setting", lambda name: 2)+ await ledger.record_step(database, run.run_id, {"step_id": "c3", "state": "done", "name": "Used Drive"})+ assert len((await ledger.get_run(database, run.run_id)).steps) == 2+ # Nothing to save without an id, or for a run that is gone.+ await ledger.record_step(database, run.run_id, {"state": "done"})+ await ledger.record_step(database, "r_unknown", {"step_id": "c4"})+++# ----- the command -----+++async def test_the_stop_command(chat):+ await talking(chat.database)+ sent = await chat.handlers.handle_send_chat_message({"message": "Book Saturday", "session_id": "wf"}, None)+ stopped = await chat.handlers.handle_stop_chat_run({"run_id": sent["run_id"]}, None)+ assert stopped == {"success": True, "run_id": sent["run_id"], "state": "stopped"}+ # Pressed again, it answers the same.+ assert (await chat.handlers.handle_stop_chat_run({"run_id": sent["run_id"]}, None))["state"] == "stopped"++ second = await chat.handlers.handle_send_chat_message({"message": "And Sunday?", "session_id": "wf"}, None)+ await ledger.start_run(chat.database, run_id=second["run_id"], **CLAIM)+ assert (await chat.handlers.handle_stop_chat_run({"run_id": second["run_id"]}, None))["state"] == "stopping"+ await ledger.finish_run(chat.database, run_id=second["run_id"], success=True, **CLAIM)++ third = await chat.handlers.handle_send_chat_message({"message": "Thanks", "session_id": "wf"}, None)+ await ledger.start_run(chat.database, run_id=third["run_id"], temporal_workflow_id="tw-3", temporal_run_id="tr-3")+ await ledger.finish_run(chat.database, run_id=third["run_id"], temporal_workflow_id="tw-3", temporal_run_id="tr-3", success=True)+ refused = await chat.handlers.handle_stop_chat_run({"run_id": third["run_id"]}, None)+ assert refused == {"success": False, "error": "not_stoppable", "run_id": third["run_id"], "state": "finished"}+ assert (await chat.handlers.handle_stop_chat_run({"run_id": "r_unknown"}, None))["error"] == "not_found"+++async def test_only_the_workflows_owner_may_stop_its_runs(chat):+ await talking(chat.database)+ await chat.database.save_workflow("wf", "Salon", "Salon_1", {"nodes": [], "edges": [], "owner_id": "alice"})+ sent = await chat.handlers.handle_send_chat_message({"message": "Book Saturday", "session_id": "wf"}, FakeSocket(user_id="alice"))+ for socket in (FakeSocket(user_id="mallory"), FakeSocket(path="/ws/internal")):+ assert (await chat.handlers.handle_stop_chat_run({"run_id": sent["run_id"]}, socket))["error"] == "access_denied"+ assert (await ledger.get_run(chat.database, sent["run_id"])).state == "pending"diff --git a/server/tests/services/chat/test_stream.py b/server/tests/services/chat/test_stream.pynew file mode 100644index 00000000..83b1962a--- /dev/null+++ b/server/tests/services/chat/test_stream.py@@ -0,0 +1,214 @@+"""The employee's answer streamed into the chat (services/chat/stream.py):+which agent streams, what text may be shown while it is written, and the text+events one step publishes (one segment per attempt, batched, held back while+it could still be NO_REPLY or a follow-ups block, reply or narration)."""++from __future__ import annotations++from types import SimpleNamespace+from typing import Any, Dict, List++import pytest++from services.chat import stream+from services.llm.protocol import StreamEvent+++# ----- which agent streams -----+++def _graph(agent_id="wf:aiAgent:1", target_type="chatReply"):+ nodes = [+ {"id": agent_id, "type": "aiAgent"},+ {"id": "wf:out:1", "type": target_type},+ ]+ edges = [{"source": agent_id, "target": "wf:out:1"}]+ return nodes, edges+++def registry(monkeypatch):+ import services.node_registry as node_registry++ classes = {"chatReply": SimpleNamespace(answers_chat_run=True), "console": SimpleNamespace()}+ monkeypatch.setattr(node_registry, "get_node_class", lambda node_type: classes.get(node_type))+++def _context(**overrides):+ nodes, edges = _graph()+ context = {+ "node_id": "wf:aiAgent:1",+ "workflow_id": "wf",+ "nodes": nodes,+ "edges": edges,+ "run_scope": {"run_id": "r_1", "session_id": "wf"},+ }+ context.update(overrides)+ return context+++def test_the_agent_whose_answer_is_the_reply_streams(registry):+ assert stream.chat_stream_for(_context()) == {+ "run_id": "r_1",+ "session_id": "wf",+ "workflow_id": "wf",+ "reply_message_id": "a_r_1",+ }+ assert stream.chat_run_id_of(_context()) == "r_1"+++def test_no_other_agent_streams(registry):+ nodes, edges = _graph(target_type="console")+ # Its output goes elsewhere; it works for another agent; no chat run.+ assert stream.chat_stream_for(_context(nodes=nodes, edges=edges)) is None+ assert stream.chat_stream_for(_context(parent_node_id="wf:lead:1")) is None+ assert stream.chat_stream_for(_context(run_scope=None)) is None+ assert stream.chat_stream_for(_context(run_scope={"run_id": "r_1"})) is None+ assert stream.chat_run_id_of({"run_scope": {"run_id": ""}}) is None+++def test_reply_in_chat_answers_the_run():+ from nodes.chat.chat_reply import ChatReplyNode++ assert ChatReplyNode.answers_chat_run is True+++# ----- what may be shown -----+++ ("text", "shown"),+ [+ ("", ""),+ ("NO", ""),+ (" NO_REP", ""),+ ("NO_REPLY", ""),+ ("No problem", "No problem"),+ ("NOTE: done", "NOTE: done"),+ ("Booked.\n<followups>[\"Change it\"]</followups>", "Booked.\n"),+ ("Booked.\n<follow", "Booked.\n"),+ ("Booked.\n<", "Booked.\n"),+ ("a < b", "a < b"),+ ],+)+def test_text_that_could_still_be_hidden_waits(text, shown):+ assert stream.shown_text(text) == shown+++# ----- the events -----+++class Clock:+ def __init__(self):+ self.now = 0.0++ def __call__(self):+ return self.now+++def _emitter(published: List[Dict[str, Any]], *, attempt=1, flush_chars=10, clock=None, iteration=2):+ def publish(**kwargs):+ published.append(kwargs)++ return stream.ChatStreamEmitter.from_payload(+ {+ "iteration": iteration,+ "chat_stream": {"run_id": "r_1", "session_id": "wf", "workflow_id": "wf", "reply_message_id": "a_r_1"},+ },+ attempt=attempt,+ publish=publish,+ flush_ms=50,+ flush_chars=flush_chars,+ clock=clock or Clock(),+ )+++def _text(published):+ return "".join(event["fields"]["delta"] for event in published if event["suffix"] == "text.content")+++async def test_a_reply_streams_in_batches_and_ends_final():+ published: List[Dict[str, Any]] = []+ clock = Clock()+ emitter = _emitter(published, clock=clock)+ emitter.begin()+ for piece in ["Your ", "booking ", "is set ", "for 10:00."]:+ await emitter(StreamEvent("text", piece))+ await emitter(StreamEvent("reasoning", "ignored"))+ emitter.end(final=True)+ suffixes = [event["suffix"] for event in published]+ assert suffixes[0] == "text.started" and suffixes[-1] == "text.ended"+ assert published[0]["fields"] == {"message_id": "r_1.2.1", "role": "assistant"}+ assert _text(published) == "Your booking is set for 10:00."+ # Batched: four deltas, fewer events than deltas.+ assert suffixes.count("text.content") < 4+ assert published[-1]["fields"] == {"message_id": "r_1.2.1", "final": True, "reply_message_id": "a_r_1"}+ assert {event["run_id"] for event in published} == {"r_1"}+++async def test_time_flushes_a_short_delta():+ published: List[Dict[str, Any]] = []+ clock = Clock()+ emitter = _emitter(published, clock=clock, flush_chars=1000)+ await emitter(StreamEvent("text", "Hi"))+ assert published == []+ clock.now = 0.06+ await emitter(StreamEvent("text", " there"))+ assert _text(published) == "Hi there"+++async def test_narration_beside_tool_calls_is_not_the_reply():+ published: List[Dict[str, Any]] = []+ emitter = _emitter(published)+ await emitter(StreamEvent("text", "Let me check the calendar."))+ emitter.end(final=False)+ assert published[-1]["fields"] == {"message_id": "r_1.2.1", "final": False}+++async def test_no_reply_never_shows():+ published: List[Dict[str, Any]] = []+ emitter = _emitter(published, flush_chars=1)+ for piece in ["NO", "_RE", "PLY"]:+ await emitter(StreamEvent("text", piece))+ emitter.end(final=True)+ assert published == []+++async def test_a_followups_block_never_streams():+ published: List[Dict[str, Any]] = []+ emitter = _emitter(published, flush_chars=1)+ for piece in ["Booked.", " <fol", "lowups>[\"Move it\"]", "</followups>"]:+ await emitter(StreamEvent("text", piece))+ emitter.end(final=True)+ assert _text(published) == "Booked. "+ assert emitter.text.endswith("</followups>")+++async def test_a_retry_withdraws_what_earlier_attempts_streamed():+ published: List[Dict[str, Any]] = []+ emitter = _emitter(published, attempt=3)+ emitter.begin()+ discarded = [event for event in published if event["suffix"] == "custom"]+ assert [event["fields"]["value"]["message_id"] for event in discarded] == ["r_1.2.1", "r_1.2.2"]+ assert {event["fields"]["name"] for event in discarded} == {"opencompany.segment_discarded"}+ assert [event["event_key"] for event in discarded] == ["discard:r_1.2.1", "discard:r_1.2.2"]+ await emitter(StreamEvent("text", "Third time lucky."))+ emitter.end(final=True)+ assert published[-1]["fields"]["message_id"] == "r_1.2.3"+++async def test_a_publishing_failure_never_fails_the_step():+ def broken(**_kwargs):+ raise RuntimeError("hub is gone")++ emitter = stream.ChatStreamEmitter.from_payload(+ {"chat_stream": {"run_id": "r_1", "session_id": "wf"}}, attempt=1, publish=broken, flush_chars=1+ )+ await emitter(StreamEvent("text", "Still answering"))+ emitter.end(final=True)+ assert emitter.text == "Still answering"+++def test_a_step_without_a_chat_stream_has_no_emitter():+ assert stream.ChatStreamEmitter.from_payload({}, attempt=1) is None+ assert stream.ChatStreamEmitter.from_payload({"chat_stream": {"run_id": "r_1"}}, attempt=1) is Nonediff --git a/server/tests/services/chat/test_watchdog.py b/server/tests/services/chat/test_watchdog.pyindex 2ffe5f96..567bb5b6 100644--- a/server/tests/services/chat/test_watchdog.py+++ b/server/tests/services/chat/test_watchdog.py@@ -1,6 +1,6 @@"""The chat run watchdog (services/chat/watchdog.py): sweeps at once when it-starts and then every interval, survives a failing sweep, stops cleanly, and-asks Temporal whether a run's workflow has closed."""+starts and then every interval, survives a failing sweep, stops cleanly, asks+Temporal whether a run's workflow has closed, and cancels a stopped run's."""from __future__ import annotations@@ -15,8 +15,8 @@ from services.chat import watchdogasync def test_it_sweeps_at_once_and_keeps_sweeping_after_a_failure(monkeypatch):sweeps = []- async def sweep(database, *, temporal_status):- sweeps.append((database, temporal_status))+ async def sweep(database, *, temporal_status, temporal_cancel):+ sweeps.append((database, temporal_status, temporal_cancel))if len(sweeps) == 1:raise RuntimeError("database is locked")return []@@ -32,7 +32,10 @@ async def test_it_sweeps_at_once_and_keeps_sweeping_after_a_failure(monkeypatch)await dog.stop()await dog.stop() # stopping twice is harmlessassert len(sweeps) >= 3- assert all(database == "db" and status is watchdog.temporal_workflow_closed for database, status in sweeps)+ assert all(+ database == "db" and status is watchdog.temporal_workflow_closed and cancel is watchdog.cancel_temporal_workflow+ for database, status, cancel in sweeps+ )count = len(sweeps)await asyncio.sleep(0.05)assert len(sweeps) == count, "the loop kept running after stop"@@ -91,3 +94,31 @@ async def test_a_workflow_temporal_no_longer_knows_is_closed(monkeypatch):assert await watchdog.temporal_workflow_closed("w", "r") is Trueclient.status = RPCStatusCode.UNAVAILABLEassert await watchdog.temporal_workflow_closed("w", "r") is None+++async def test_cancelling_a_stopped_runs_workflow(monkeypatch):+ from temporalio.service import RPCError, RPCStatusCode++ cancelled = []++ class Handle:+ def __init__(self, workflow_id, run_id):+ self.ids = (workflow_id, run_id)++ async def cancel(self):+ if self.ids[0] == "gone":+ raise RPCError("already closed", RPCStatusCode.NOT_FOUND, b"")+ cancelled.append(self.ids)++ class Client:+ def get_workflow_handle(self, workflow_id, run_id=None):+ return Handle(workflow_id, run_id)++ _container(monkeypatch, Client())+ await watchdog.cancel_temporal_workflow("w", "r")+ # A workflow that already closed has nothing left to cancel.+ await watchdog.cancel_temporal_workflow("gone", "r")+ assert cancelled == [("w", "r")]+ # Without Temporal there is nothing to do.+ _container(monkeypatch, None)+ await watchdog.cancel_temporal_workflow("w", "r")diff --git a/server/tests/temporal/test_agent_chat_stream.py b/server/tests/temporal/test_agent_chat_stream.pynew file mode 100644index 00000000..9620332f--- /dev/null+++ b/server/tests/temporal/test_agent_chat_stream.py@@ -0,0 +1,196 @@+"""The LLM step of an agent answering a chat run (agent.execute_llm_step,+services/temporal/agent_activities.py): its text streams to the run, a run the+owner stopped ends at the next step without paying for it, a stream stopped+midway keeps what was shown as the answer, and the conversation keeps the+turn as far as it went."""++from __future__ import annotations++import asyncio+from typing import Any, Dict, List++import pytest++from services.llm.protocol import LLMResponse, Message, StreamEvent, ToolCall, Usage, message_to_wire++STREAM = {"run_id": "r_1", "session_id": "wf", "workflow_id": "wf", "reply_message_id": "a_r_1"}+++def _payload(**extra: Any) -> Dict[str, Any]:+ return {+ "provider": "anthropic",+ "model": "test-model",+ "api_key": "secret",+ "iteration": 0,+ "messages": [+ message_to_wire(Message(role="system", content="Be brief.")),+ message_to_wire(Message(role="user", content="Book Saturday")),+ ],+ "tools": [],+ "conversation_key": {"workflow_id": "wf", "generation": 1, "agent_node_id": "wf:aiAgent:1"},+ "chat_run_id": "r_1",+ "chat_stream": dict(STREAM),+ "include_finish_reason": True,+ **extra,+ }+++def env(monkeypatch):+ """The step's surroundings: the hub, the conversation store and the stop+ flag, all recorded."""+ import core.container as container_module+ import services.chat.config as chat_config+ import services.chat.hub as hub_module+ from services.temporal import agent_activities++ published: List[Dict[str, Any]] = []+ saved: List[Dict[str, Any]] = []+ state = {"stopping": False, "polls": 0}++ class Hub:+ def publish(self, **kwargs):+ published.append(kwargs)++ async def save(payload, *, sent, assistant_wire):+ saved.append({"sent": sent, "assistant": assistant_wire})++ async def stopping(run_id):+ state["polls"] += 1+ return state["stopping"]++ real_runs_setting = chat_config.runs_setting+ monkeypatch.setattr(hub_module, "get_chat_hub", lambda: Hub())+ monkeypatch.setattr(agent_activities, "_save_conversation", save)+ monkeypatch.setattr(agent_activities, "_chat_run_stopping", stopping)+ monkeypatch.setattr(chat_config, "runs_setting", lambda name: 0.0 if name == "stop_poll_s" else real_runs_setting(name))+ monkeypatch.setattr(container_module.container, "chat_unifier", lambda: object())+ return type("Env", (), {"published": published, "saved": saved, "state": state})+++def _run_step(monkeypatch, step):+ import services.agent_runtime as runtime_module++ monkeypatch.setattr(runtime_module, "run_native_llm_step", step)+++def _texts(published):+ return "".join(event["fields"]["delta"] for event in published if event["suffix"] == "text.content")+++async def test_the_answer_streams_and_the_result_is_unchanged(env, monkeypatch):+ from services.temporal.agent_activities import _execute_native_llm_step++ async def step(unifier, **kwargs):+ sink = kwargs["on_event"]+ for piece in ["Booked ", "for 10:00."]:+ await sink(StreamEvent("text", piece))+ return LLMResponse(content="Booked for 10:00.", finish_reason="stop", usage=Usage(input_tokens=3, output_tokens=4))++ _run_step(monkeypatch, step)+ result = await _execute_native_llm_step(_payload())+ assert (result["kind"], result["content"], result["finish_reason"]) == ("final", "Booked for 10:00.", "stop")+ assert "stopped" not in result+ assert _texts(env.published) == "Booked for 10:00."+ assert env.published[-1]["suffix"] == "text.ended"+ assert env.published[-1]["fields"] == {"message_id": "r_1.0.1", "final": True, "reply_message_id": "a_r_1"}+ [turn] = env.saved+ assert turn["assistant"]["content"] == "Booked for 10:00."+++async def test_text_beside_tool_calls_streams_as_narration(env, monkeypatch):+ from services.temporal.agent_activities import _execute_native_llm_step++ async def step(unifier, **kwargs):+ await kwargs["on_event"](StreamEvent("text", "Checking the calendar."))+ return LLMResponse(+ content="Checking the calendar.",+ tool_calls=[ToolCall(id="call_1", name="calendar", args={})],+ finish_reason="tool_calls",+ usage=Usage(),+ )++ _run_step(monkeypatch, step)+ result = await _execute_native_llm_step(_payload())+ assert result["kind"] == "tool_calls"+ assert env.published[-1]["fields"]["final"] is False+++async def test_a_step_that_does_not_answer_the_run_streams_nothing(env, monkeypatch):+ from services.temporal.agent_activities import _execute_native_llm_step++ sinks = []++ async def step(unifier, **kwargs):+ sinks.append(kwargs["on_event"])+ return LLMResponse(content="Done.", usage=Usage())++ _run_step(monkeypatch, step)+ payload = _payload()+ payload.pop("chat_stream")+ payload.pop("chat_run_id")+ await _execute_native_llm_step(payload)+ assert sinks == [None] and env.published == []+ assert env.state["polls"] == 0+++async def test_a_stopped_run_ends_before_paying_for_a_step(env, monkeypatch):+ from services.temporal.agent_activities import _execute_native_llm_step++ async def step(unifier, **kwargs):+ raise AssertionError("the model was called for a stopped run")++ _run_step(monkeypatch, step)+ env.state["stopping"] = True+ result = await _execute_native_llm_step(_payload())+ assert (result["kind"], result["content"], result["stopped"], result["finish_reason"]) == ("final", "", True, "stopped")+ assert result["usage"]["total_tokens"] == 0+ # The request is kept (its tool results answer every call); no answer is added.+ [turn] = env.saved+ assert [message["role"] for message in turn["sent"]] == ["system", "user"]+ assert turn["assistant"] is None+ assert env.published == []+++async def test_stopping_midway_keeps_what_was_shown(env, monkeypatch):+ from services.temporal.agent_activities import _execute_native_llm_step++ cancelled = asyncio.Event()++ async def step(unifier, **kwargs):+ sink = kwargs["on_event"]+ await sink(StreamEvent("text", "Your Saturday slot is "))+ await sink(StreamEvent("text", "10:00 <follow"))+ env.state["stopping"] = True+ try:+ await asyncio.sleep(30)+ except asyncio.CancelledError:+ cancelled.set()+ raise+ return LLMResponse(content="never", usage=Usage())++ _run_step(monkeypatch, step)+ result = await asyncio.wait_for(_execute_native_llm_step(_payload()), timeout=5)+ assert cancelled.is_set(), "the model call kept running after Stop"+ assert (result["kind"], result["content"], result["stopped"]) == ("final", "Your Saturday slot is 10:00", True)+ # The partial answer is the reply: its segment ends final.+ assert env.published[-1]["suffix"] == "text.ended" and env.published[-1]["fields"]["final"] is True+ [turn] = env.saved+ assert turn["assistant"]["content"] == "Your Saturday slot is 10:00"+ assert turn["assistant"]["blocks"][0]["metadata"] == {"stopped": True}+++async def test_a_retried_step_withdraws_the_earlier_attempts_text(env, monkeypatch):+ from services.temporal import agent_activities+ from services.temporal.agent_activities import _execute_native_llm_step++ async def step(unifier, **kwargs):+ await kwargs["on_event"](StreamEvent("text", "Second try."))+ return LLMResponse(content="Second try.", usage=Usage())++ _run_step(monkeypatch, step)+ monkeypatch.setattr(agent_activities, "_llm_activity_attempt", lambda: 2)+ await _execute_native_llm_step(_payload(iteration=3))+ first = env.published[0]+ assert (first["suffix"], first["fields"]["value"]) == ("custom", {"message_id": "r_1.3.1"})+ assert env.published[1]["fields"]["message_id"] == "r_1.3.2"