Skip to content

Self-improving AI Employees for everyone

docs(chat): standalone workers, at-most-once sends, and provider streaming

trohitge4aeea660114
diff --git a/CLAUDE.md b/CLAUDE.md
index 2720d2b5..651f1eb8 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -14,7 +14,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire
| **[Temporal Cleanup & Resilience Plan (Waves 15-18)](./docs-internal/ARCHIVE/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md)** | *Archived — all four waves shipped.* 15 dead-code retirement (Redis-Streams branch in event_waiter, APScheduler cron stack, orphaned `connection_status` factory), 16 per-queue task routing (`TemporalWorkerPool`), 17 resilience hardening (cron `catchup_window`, SDK interceptors, `DEPLOYMENT_MODE`, periodic heartbeats; the one-shot LLM-step retry it records was later reverted to unlimited-with-backoff), 18 worker performance tuning. Current settings live in [Temporal Architecture](./docs-internal/TEMPORAL_ARCHITECTURE.md). |
| **[Node Allowlist](./docs-internal/node_allowlist.md)** | Single-config UI visibility — `server/config/node_allowlist.json` controls which nodes / credential categories / skill folders show in the UI. Five lists with two enforcement tiers (mode-gated allowlist + absolute blocklist). `useNodeAllowlist` hook exposes `isVisible` / `isBlocked` / `isAllowed` / `isCredentialCategoryDisabled` / `isSkillFolderDisabled`. Adding a new disabled domain = single JSON edit, no code change. |
| **[Normal Mode (Home)](./docs-internal/normal_mode.md)** | The owner-facing landing screen: describe a job, an LLM on the server drafts a setup screen (a JSON UI spec the client parses, normalises and renders from a fixed catalogue), and Hire builds, validates, saves and starts one workflow per AI employee (`services/employees/`, graph built from `config/employee_apps.json` plus the owner's skill library and a Canvas node, and checked against `node_allowlist.json` `enabled_nodes`). Covers the screen switch (`app/`, `shellMode` / `ui_shell_mode`, `VITE_NORMAL_MODE=false` kill switch, Home shows light/dark only), the employee page (the shared chat, `features/chat`, with the employee named in the header beside New conversation; Pause is in the Workspace header), deleting an employee from the team sidebar (stopped through Reset first, then gone from every tab), the live Ask first rule (`workflow_rules`, the chat's Ask first chip via `set_ask_first`; app replies wait in `approvalGate`, sending tool calls are held as drafts by `services/approvals/tool_calls.py` and sent after a 5 s Undo window by `ApprovedToolCallWorkflow`; cards in the chat; builder v3 gates every app reply, attaches every app tool and gives the talk agent `chatUi` and the registry's `talk_send` tools, with instructions identical whatever the rule; older employees are upgraded on Apply / Turn on Talk / Start by `services/employees/upgrade.py`), run records for "done today" (`machina-run-record-v1`), automatic-pause reasons, Settings (Profile fields; usage-only Billing via `get_employee_usage`; Skills, the user-skills library each new hire copies onto a Skills node; consumer Connectors with `publisher` / `verified`; Plugins, the starter bundles in `hire/starters.json`; the last three share one catalog page), the Workspace dock (an employee's Canvas board, named by the summary's `canvas_node_id`; a live Browser tab with user takeover, shared with Dev mode; Mobile shows the managed local Android phone), the "Web browser" app (read-only under "ask me first" through the tool's `ask_first_params`; while the agent waits in `request_user` the summary's `browser_request` shows Needs you and Help in browser), Talk on the employee page (a thread over the chat session whose id is the workflow id, rows written through `services/chat_thread.py` and, for the owner's messages and their chat runs, `services/chat/ledger.py` (see [Chat Protocol](./docs-internal/chat_protocol.md)); the talk line is chatTrigger → an agent → `chatReply` "Reply in Chat": a chat hire's own agent is its talk agent, other hires get a "Talk with {name}" agent sharing the worker's tools and skills with its own Context, and schedule hires also "Post to Talk"), Turn on Talk (for talk `off`) and Apply (while the summary's `pending_changes` is true) via `enable_employee_talk` / `apply_employee_changes`, each a restart on the saved graph through `services/deployment/restart.py`, self-extension from Talk through the Agent Builder within the Hire rule (`services/employees/policy.py`), the 3D orb, the wire contract and the `employee_lifecycle` / `approval_lifecycle` / `chat.updated` broadcasts. |
-| **[Chat Protocol](./docs-internal/chat_protocol.md)** | The wire contract between the chat runtime (`server/services/chat/`) and the shared chat UI: AG-UI's event model on the existing WebSocket, snake_case, in the CloudEvents envelope. A message a deployed chat trigger will answer starts a **chat run** admitted with it in one reserved write transaction (`Database.reserved_session`, SQLite `BEGIN IMMEDIATE`): one live run per session (the lane, also a partial unique index; a second send answers `run_in_progress`), idempotent `client_message_id`, claimed and finished by the MachinaWorkflow that answers it behind `machina-chat-run-v1` (the run id is trusted only from an event whose source is `opencompany://services/chat`; `run_scope` rides every node context; only the claimant finishes), the reply saved as `a_<run id>` by `chatReply`, every message appended after the session's active leaf (`chat_threads`) so concurrent writes never fork the thread. `chat_run_event` frames reach only sockets that passed `chat_subscribe`'s owner check (`services/chat/hub.py`: per-run `seq`, `hub_epoch`, bounded per-socket queues with a resync), never the status broadcast. A watchdog (`services/chat/watchdog.py`, started by `main.py`) ends runs nothing will finish (`not_delivered`, `timed_out`, `interrupted`). The agent wired into Reply in Chat (`answers_chat_run`) streams its answer as `text.*` events (`services/chat/stream.py`, activity input only, no workflow patch), its tool calls show as steps (`services/chat/steps.py`; label from the plugin's `chat_step`), and `stop_chat_run` stops a run cooperatively: its LLM step polls the run and returns what was shown, unstarted tools are skipped, the reply is saved `stopped`, and the watchdog cancels a run still stopping after `runs.stop_grace_s`. Generated UI: the `show_ui` tool (`chatUi`) is checked against `config/chat_genui_catalog.json` (`services/genui/spec.py`; refused with every reason so the model retries), saved as a run part, streamed as `activity.*` RFC 6902 patches and sealed into the reply (`services/chat/parts.py`); `chat_ui_state` keeps what the owner set, and a button press comes back as `send_chat_message{ui_event}` -> a `[ui-event]` turn. The answering agent's system prompt ends with a fixed guide (`services/chat/guide.py`), what the owner set reaches its next turn as a `[ui-state]` note (`chat_notes`, `services/chat/notes.py`: claimed by the payload, told when the run answered), a trailing `<followups>` block becomes `parts.followups`, and a `chat_sources` tool's results are numbered per conversation for `[n]` citations (`services/chat/sources.py`). Approvals (`services/approvals/`, its own section): a plugin's `approval` spec marks a tool call that sends; while the workflow asks first it is held as a `tool_call` draft, shown as a card on the reply that made it, sent once after the Undo window by `ApprovedToolCallWorkflow`, with the outcome told to the employee as an `[update]` note. Branches (`services/chat/branches.py`): `get_chat_messages` returns the path to the thread's active leaf, each message with its versions (`siblings`), `editable` and the owner's rating; `edit_chat_message` / `regenerate_chat_reply` / `switch_chat_branch` move the path in one reserved transaction under the conversation store's lock, taking each agent's stored conversation back to the cursor its run recorded at `prepare_payload` (`chat_runs.context_cursors`) or restoring the branch's kept copy (`chat_branch_snapshots`), refused on `revision_conflict`, `run_in_progress`, `cannot_rewind`, `branch_unavailable` or `older_generation`; `set_chat_feedback` keeps a rating and tells the next turn (`[feedback]` note). Attachments (`services/chat/attachments.py`): the chat uploads files under `uploads/` and sends only their paths, the server rebuilds each reference from the file (refusing anything else, at most six), the answering agent reads an `[attachments]` line and images ride the opening user message for a provider with `vision.user_images`; `options.web: false` drops the `search` tools for that run; `get_chat_context` gives slash commands (generic plus the apps' `commands`) and capabilities; dictation is the speech plugin's `dictation_status` / `transcribe_audio`. Lists every event, handler, part and error code, including the ones later phases build. |
+| **[Chat Protocol](./docs-internal/chat_protocol.md)** | The wire contract between the chat runtime (`server/services/chat/`) and the shared chat UI: AG-UI's event model on the existing WebSocket, snake_case, in the CloudEvents envelope. A message a deployed chat trigger will answer starts a **chat run** admitted with it in one reserved write transaction (`Database.reserved_session`, SQLite `BEGIN IMMEDIATE`): one live run per session (the lane, also a partial unique index; a second send answers `run_in_progress`), idempotent `client_message_id`, claimed and finished by the MachinaWorkflow that answers it behind `machina-chat-run-v1` (the run id is trusted only from an event whose source is `opencompany://services/chat`; `run_scope` rides every node context; only the claimant finishes), the reply saved as `a_<run id>` by `chatReply`, every message appended after the session's active leaf (`chat_threads`) so concurrent writes never fork the thread. `chat_run_event` frames reach only sockets that passed `chat_subscribe`'s owner check (`services/chat/hub.py`: per-run `seq`, `hub_epoch`, bounded per-socket queues with a resync), never the status broadcast; a standalone worker sends its run events, `chat.updated` and `approval_lifecycle` to the backend over `/ws/internal` (`services/chat/relay.py`, `chat_run_publish`, ordered, dropped events resynced). A watchdog (`services/chat/watchdog.py`, started by `main.py`) ends runs nothing will finish (`not_delivered`, `timed_out`, `interrupted`). The agent wired into Reply in Chat (`answers_chat_run`) streams its answer as `text.*` events (`services/chat/stream.py`, activity input only, no workflow patch), its tool calls show as steps (`services/chat/steps.py`; label from the plugin's `chat_step`), and `stop_chat_run` stops a run cooperatively: its LLM step polls the run and returns what was shown, unstarted tools are skipped, the reply is saved `stopped`, and the watchdog cancels a run still stopping after `runs.stop_grace_s`. Generated UI: the `show_ui` tool (`chatUi`) is checked against `config/chat_genui_catalog.json` (`services/genui/spec.py`; refused with every reason so the model retries), saved as a run part, streamed as `activity.*` RFC 6902 patches and sealed into the reply (`services/chat/parts.py`); `chat_ui_state` keeps what the owner set, and a button press comes back as `send_chat_message{ui_event}` -> a `[ui-event]` turn. The answering agent's system prompt ends with a fixed guide (`services/chat/guide.py`), what the owner set reaches its next turn as a `[ui-state]` note (`chat_notes`, `services/chat/notes.py`: claimed by the payload, told when the run answered), a trailing `<followups>` block becomes `parts.followups`, and a `chat_sources` tool's results are numbered per conversation for `[n]` citations (`services/chat/sources.py`). Approvals (`services/approvals/`, its own section): a plugin's `approval` spec marks a tool call that sends; while the workflow asks first it is held as a `tool_call` draft, shown as a card on the reply that made it, sent once after the Undo window by `ApprovedToolCallWorkflow`, with the outcome told to the employee as an `[update]` note. Branches (`services/chat/branches.py`): `get_chat_messages` returns the path to the thread's active leaf, each message with its versions (`siblings`), `editable` and the owner's rating; `edit_chat_message` / `regenerate_chat_reply` / `switch_chat_branch` move the path in one reserved transaction under the conversation store's lock, taking each agent's stored conversation back to the cursor its run recorded at `prepare_payload` (`chat_runs.context_cursors`) or restoring the branch's kept copy (`chat_branch_snapshots`), refused on `revision_conflict`, `run_in_progress`, `cannot_rewind`, `branch_unavailable` or `older_generation`; `set_chat_feedback` keeps a rating and tells the next turn (`[feedback]` note). Attachments (`services/chat/attachments.py`): the chat uploads files under `uploads/` and sends only their paths, the server rebuilds each reference from the file (refusing anything else, at most six), the answering agent reads an `[attachments]` line and images ride the opening user message for a provider with `vision.user_images`; `options.web: false` drops the `search` tools for that run; `get_chat_context` gives slash commands (generic plus the apps' `commands`) and capabilities; dictation is the speech plugin's `dictation_status` / `transcribe_audio`. Lists every event, handler, part and error code, including the ones later phases build. |
| **[Theme System](./docs-internal/theme_system.md)** | 12-way visual theme system — 2 base (light, dark) + 5 utopian (renaissance, greek, edo, steampunk, atomic) + 5 dystopian (cyber, wasteland, rot, plague, surveillance) — driven by `<html data-theme>` + per-theme CSS files in `client/src/themes/`. Token taxonomy (surface / fg / border / accent / typography / motion) in **hex + `color-mix()`**; `@theme inline` bridge maps `--color-X: var(--X)` (no `hsl()` wrapper); per-theme files own the colour hex, base.css owns the shared `--tint-*` alpha scale; action role tokens (incl. `-ink` readable text), **per-theme `--code-*` syntax tokens** (tier 6 — the code editor, console/output JSON viewers, and chat code blocks paint in each theme's own palette: keyword→trigger, string→success, number→agent, function→model, on an adaptive dark/light code surface; replaced the global dracula `--prism-*` block + dead `getPrismTokenCSS`; the `OutputPanel` JSON viewer reads the same vars), per-theme `--pulse-keyframe` animation system + `.opencompany-*` helpers in `animations.css`, decorative-layer wrappers (`.app-frame` / `.canvas-host` / `.modal-frame`), per-component decorative ornaments (panel textures + canvas decorations + node pseudo-element overlays + theme-specific keyframes), **canvas-node visual contract via `--node-color` CSS custom property** (no inline `background` / `border` on node components — base.css + per-theme CSS owns visuals; `NodeStyle` helper type at [types/NodeTypes.ts](./client/src/types/NodeTypes.ts) makes the inline custom-prop typecheck-clean), **`--node-pulse-color` separate from `--node-color`** so executing-node glow uses each theme's highest-contrast accent regardless of plugin accent (Cyber neon cyan, Surveillance REC red, Renaissance ultramarine, etc.), **`data-page-hidden` animation pause** (set by `app/usePageActivitySync.ts`, mounted in `AppShell` on both screens, while the tab is hidden or the window is blurred; base.css declares `html[data-page-hidden] *, *::before, *::after { animation-play-state: paused !important }` to prevent compositor stall on tab return), **per-theme icon glyph system** (290 inline SVG glyph strings across 29 keys × 10 themes via [themedGlyphs.ts](./client/src/assets/icons/themedGlyphs.ts) + theme-aware [NodeIcon.tsx](./client/src/assets/icons/NodeIcon.tsx)), **per-theme canvas-grid + custom cursors** via `--canvas-grid` / `--cursor-default` slots, **decorative HTML primitives** (`<SvgFilterDefs>` mounting `#ink-blot` / `#noise` / `#crt` filter IDs at app root, `<DropCap>` wrapper for Renaissance ornament rule), **parameter panel migrated to Tailwind tokens** (no `useAppTheme()` reads; section headers carry the display-typography triplet; raw `<Button>` swapped to `<ActionButton intent>`), **per-theme scrollbar webkit rules** in all 12 themes, 9-event WebAudio sound system (10 packs via `--sound-pack` token + `useSound()` hook + global hover delegate + sonner toast monkey-patch + `withSound()` HOC + `Sounds.unlock()` gesture-unlock for AudioContext autoplay-policy compliance), `@media (prefers-reduced-motion: reduce)` accessibility, 30 ms throttle on `type` / `hover`, migration recipe, anti-patterns. Read this before adding a new theme, migrating a component to the new contract, or adding a canvas-node component. |
| **[Design System Bundle](./docs-internal/design-system/IMPLEMENTATION.md)** | Canonical, vendored design-system reference (extracted from this repo; see its `IMPLEMENTATION.md` for the contract). Tokens are **hex + `color-mix()`** (`tokens/{colors,typography,spacing,motion,fonts,base}.css`) — the format the live codebase standardizes on. Includes 34 reference components (`components/`, inline-style reference only — recreate in Tailwind/shadcn idioms), full-app UI kit (`ui_kits/opencompany/index.html`), 12-theme docs (`guidelines/THEMES.md`), **the merged handoff brief (`HANDOFF.md` — glow-ladder traps, step-edge contract, toolbar layout traps, canvas-layer asymmetries, plus the recorded product amendments)**, and **the panel×theme fidelity target (`reference-mockup/Panel Theme Matrix.dc.html` — 17 panels × 12 themes, open in a browser)**. `reference/themes/` no longer vendors CSS snapshots — it points at the authoritative [client/src/themes/](./client/src/themes/). Copy token values verbatim; do not re-derive by eye. Pairs with [Theme System](./docs-internal/theme_system.md). |
| **[Schema Source of Truth RFC](./docs-internal/ARCHIVE/schema_source_of_truth_rfc.md)** | *Archived — shipped.* Backend is SSOT for node schemas, visual metadata, handlers, palette metadata, icons; `VITE_NODESPEC_BACKEND` defaults on and the frontend node definitions are gone. Plugin pattern: one `BaseNode` subclass in `server/nodes/<group>/<plugin>/__init__.py`. Wire format: `asset:<key>` / `<lib>:<brand>` / URL / emoji. Endpoint: `/api/schemas/nodes/{type}/spec.json`. Current reference: [Plugin System](./docs-internal/plugin_system.md). |
@@ -2779,7 +2779,7 @@ This function:
- **Telegram inbound content types**: `_format_message` extracts detail dicts for all 11 types via the `_CONTENT_PROBES` / `_DETAIL_EXTRACTORS` tables, plus a normalized `media` block (`kind` / `file_id` / `mime_type` / `file_name` / …) that downstream nodes read instead of branching per type; MIME and filename are synthesised where Telegram omits them. **Probe order is load-bearing**: Telegram sets `message.document` on animation messages too, so `animation` and `video_note` must be probed before `document` and `video` or every GIF classifies as a document. The trigger never downloads — media travels as a `file_id`, never bytes or base64, because node results are persisted, broadcast, and replayed into the LLM conversation every turn.
- **Google OAuth scope expansion**: Google's authorisation server legitimately returns a wider scope set than requested when the OAuth Client's "Data Access" page lists extra scopes (commonly `cloud-platform`) or when `include_granted_scopes` replays a previously-granted scope. `oauthlib` does strict set-equality and aborts with `Warning: Scope has changed`. As of 2026 (`google-auth-oauthlib` 1.2.4, `oauthlib` upstream issue #562 still open), no constructor flag, context manager, or `expected_scopes` argument exists — the documented relief is the env var. `server/nodes/google/_oauth.py` sets `OAUTHLIB_RELAX_TOKEN_SCOPE=1` via `os.environ.setdefault` BEFORE the `google_auth_oauthlib.flow` import (oauthlib reads it once at parameters-module import; request-time setting races under uvicorn workers), paired with `warnings.filterwarnings(message=r"Scope has changed.*")` to keep the operator log clean. Long-term root cause: audit the Cloud Console Data Access page and remove `cloud-platform` if no handler uses it.
- **Code-executor error mapping** (`pythonExecutor` / `javascriptExecutor` / `typescriptExecutor`): wrap user-code execution and sidecar calls in `try/except`. Python: detect `ImportError("__import__ not found")` (the LLM tried `import X` against the sandboxed builtins) and surface the pre-injected names list (`math, json, datetime, timedelta, re, random, Counter, defaultdict`) plus the suggestion to use `process_manager` for unsupported modules; other exceptions get formatted as `<ErrorName> at line N: <message>` (line N walked from `<string>` frame in the traceback) plus any captured stdout. JS/TS: detect `aiohttp.ClientConnectorError` / the spawn `RuntimeError` (bun missing or `dist/index.js` not built) and surface "JavaScript executor is unavailable (bun sidecar at <NODEJS_EXECUTOR_URL>). Fall back to python_executor for similar logic." All raise `NodeUserError` so the framework logs one WARN line — no aiohttp/CreateProcessW noise in the operator log.
-- **Workflow-scoped chat + console history**: chat messages persist with `chat_messages.session_id == <workflow_id>` (or `"default"` when no workflow is open); console logs persist with `console_logs.workflow_id`. Backend `database.get_console_logs(limit, workflow_id=None)` and `clear_console_logs(workflow_id=None)` filter by workflow when given. `handle_clear_console_logs` broadcasts `console_logs_cleared` carrying the `workflow_id` so the existing frontend filter in `WebSocketContext` keeps other workflows' panels intact. Frontend reads `currentWorkflow.id` via the documented Zustand escape hatch (`useAppStore.getState().currentWorkflow?.id`) at call time for `clearChatMessages` / `clearConsoleLogs` / `sendChatMessage` so the callbacks don't re-create on every workflow switch. A dedicated `useEffect([currentWorkflowId, isReady])` resets local `chatMessages` / `consoleLogs` and refetches both when the user opens / switches workflow. Incoming `console_log` broadcasts are filtered by `currentWorkflow.id` so a parallel run on another workflow never bleeds into the active panel. Legacy logs without `workflow_id` still surface (transition guard). The console's Chat pane no longer reads the context's `chatMessages` (it is the shared chat, `features/chat`, reading `get_chat_messages` through TanStack Query); the get-started checklist (`hooks/useGetStarted.ts`) still does, until it moves too.
+- **Workflow-scoped chat + console history**: chat messages persist with `chat_messages.session_id == <workflow_id>` (or `"default"` when no workflow is open); console logs persist with `console_logs.workflow_id`. Backend `database.get_console_logs(limit, workflow_id=None)` and `clear_console_logs(workflow_id=None)` filter by workflow when given. `handle_clear_console_logs` broadcasts `console_logs_cleared` carrying the `workflow_id` so the existing frontend filter in `WebSocketContext` keeps other workflows' panels intact. Frontend reads `currentWorkflow.id` via the documented Zustand escape hatch (`useAppStore.getState().currentWorkflow?.id`) at call time for `clearConsoleLogs` so the callback doesn't re-create on every workflow switch. A dedicated `useEffect([currentWorkflowId, isReady])` resets local `consoleLogs` and refetches them when the user opens / switches workflow. Incoming `console_log` broadcasts are filtered by `currentWorkflow.id` so a parallel run on another workflow never bleeds into the active panel. Legacy logs without `workflow_id` still surface (transition guard). `WebSocketContext` keeps no chat state: every reader (the console's Chat pane, Home's employee page, the get-started checklist in `hooks/useGetStarted.ts`) reads `get_chat_messages` through the shared thread query (`useChatThread` from `features/chat`), which `chat.updated`, a runtime reset and a reconnect invalidate.
- **Auto-derived `isConfigNode` uiHint**: `_derive_auto_ui_hints(group)` in `services/plugin/base.py` automatically sets `uiHints.isConfigNode: True` on any plugin whose `group` tuple contains `memory` or `tool` (centralized as `_CONFIG_NODE_GROUPS = frozenset({"memory", "tool"})`). Explicit `cls.ui_hints` always wins (merge order: auto first, then `dict.update`). Frontend `InputSection.tsx` consumes the flag via `definition?.uiHints?.isConfigNode === true` — the old `groups.includes('memory') || groups.includes('tool')` heuristic is gone. Pytest invariant `test_ui_hints_only_carry_known_flags` locks the flag name. Adding a new auxiliary node type costs zero per-plugin code; opting out costs one line (`ui_hints = {"isConfigNode": False}`).
- **`isMasterSkillEditor` uiHint replaces `node.type === 'masterSkill'` checks**: the frontend callsites (`useAutoSkillEdges.ts` constant + edge filter, `MiddleSection.tsx` × 3; Dashboard's component-dispatch branch went with the unused `ToolkitNode`, because masterSkill declares `component_kind = "tool"` and renders through SquareNode) read `getCachedNodeSpec(type)?.uiHints?.isMasterSkillEditor === true` instead of comparing the type string. The `MasterSkillNode` plugin already declared the hint — no backend change. Renaming the plugin's `type` is now a single backend edit followed by a NodeSpec deploy; the frontend never needs to know the string.
- **`outputMode: "terminal"` uiHint — spec-driven CLI output rendering**: the output panel (`components/output/OutputPanel.tsx`, the active renderer — `ui/OutputDisplayPanel.tsx` is legacy/unimported) renders string responses through ReactMarkdown by default, which whitespace-collapses CLI text. CLI-wrapper plugins (`githubAction`, `vercelAction`, `shell`) declare `ui_hints = {"outputMode": "terminal"}`; the panel resolves the spec via `useNodeSpec(selectedNode?.type)` and renders their text in a `<pre>` on the per-theme `--code-*` surface, routes wholly-JSON strings to the JSON tree via the shared `tryParseJson` (`utils/formatters.ts`), and surfaces object/array `result` payloads (server-side-parsed CLI JSON; arrays survive `unwrap` un-peeled) in the Response section. Pair with the `_shape` convention: parsed JSON in `result` OR text in `stdout` — never both — empty keys omitted. Locked by `client/src/components/__tests__/OutputPanel.test.tsx` (render tests: real `\n`/`\t` preserved) + the uiHints known-set in `test_node_spec.py`. Cache note: nodeSpecs are session-sticky (`staleTime: FOREVER`, revision-busted at page load) — after a backend restart a hard browser reload is needed before new uiHints reach an open canvas.
diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.md
index 6f7061ec..14f855c3 100644
--- a/docs-internal/TEMPORAL_ARCHITECTURE.md
+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md
@@ -48,7 +48,11 @@ cd server
python -m services.temporal.worker
```
-This invokes `run_standalone_worker()` from `services/temporal/worker.py`.
+This invokes `run_standalone_worker()` from `services/temporal/worker.py`. It also starts the chat relay
+(`services/chat/relay.py`): the worker has no sockets of its own, so the chat run events and the `chat.updated` /
+`approval_lifecycle` broadcasts its activities make go to the backend over `/ws/internal`
+([Chat Protocol → Standalone workers](./chat_protocol.md#standalone-workers)). Other broadcasts from its activities
+(node status, agent progress) still reach only its own process.
## System Architecture
diff --git a/docs-internal/authentication.md b/docs-internal/authentication.md
index f89800f7..ab57aa2c 100644
--- a/docs-internal/authentication.md
+++ b/docs-internal/authentication.md
@@ -231,9 +231,10 @@ Recorded explicitly because each of these is easy to assume is handled.
Docker and by `company deploy`), or any web page open on the same machine,
could run a `shell` node. The handshake now requires the
`X-OpenCompany-Internal-Token` header, an HMAC of `SECRET_KEY`
- (`internal_socket_token` in `services/authz/ws_surface.py`); both worker
- clients (`services/temporal/activities.py`, `ws_client.py`) send it, and any
- new internal client must too. A loopback-address check would not do: a
+ (`internal_socket_token` in `services/authz/ws_surface.py`); the worker
+ clients (`services/temporal/activities.py`, `ws_client.py`, and a standalone
+ worker's chat relay, `services/chat/relay.py`) send it, and any new internal
+ client must too. A loopback-address check would not do: a
reverse proxy on the same host makes public traffic arrive from 127.0.0.1,
and a page's socket to `localhost` is itself loopback. Even with the token
the socket reaches only the deny-by-default allowlist
diff --git a/docs-internal/chat_protocol.md b/docs-internal/chat_protocol.md
index 040b37d6..024cb467 100644
--- a/docs-internal/chat_protocol.md
+++ b/docs-internal/chat_protocol.md
@@ -58,8 +58,9 @@ internal socket, load the workflow, and compare its owner with the socket's exec
(`data: {session_id, hub_epoch, name, value}`, no `run_id`); the client then takes fresh snapshots. Text deltas
are batched before they are published (see [Streaming, steps and Stop](#streaming-steps-and-stop)), not merged in
the queue.
-- Delivery is in-process: events published by an activity on a worker in another process do not reach sockets here
- yet (a relay is planned).
+- Publishers call `publish_run_event` (`services/chat/hub.py`), which delivers in-process. A standalone worker
+ (`python -m services.temporal.worker`) has no sockets, so it relays instead; see
+ [Standalone workers](#standalone-workers).
- `chat.updated` remains an identity-only broadcast (`{workflow_id, session_id, role}`) that tells every open thread
to refetch `get_chat_messages`.
@@ -177,6 +178,34 @@ tool_call_id?, ui_part_id?, agent_node_id?, deployment_state?}`. `status` is `pe
| `dictation_status` | `nodes/speech` | `{session_id}` | `{success, available, provider}` |
| `transcribe_audio` | `nodes/speech` | `{session_id, path, language?}` (`path` under `uploads/`) | `{success, text, language, provider}`; `speech_unavailable` |
+### Worker relay (`server/services/chat/relay.py`)
+
+Internal: the backend accepts it on `/ws/internal` only (the worker token, `services/authz/ws_surface.py`
+`INTERNAL_SOCKET_HANDLERS`), and a client socket gets `access_denied`. The worker sends it without a `request_id`.
+
+| Handler | Request | Response |
+|---|---|---|
+| `chat_run_publish` | `{items}` (at most 500): `{kind: "run_event", run_id, session_id, workflow_id, suffix, fields, event_key}`, `{kind: "broadcast", type: "chat.updated" \| "approval_lifecycle", data: <CloudEvent>, session_id?}`, `{kind: "resync", session_ids}` | `{success, published, refused}`; `invalid_items` |
+
+## Standalone workers
+
+The hub delivers to the sockets of the process it runs in. The backend's embedded workers share that process, but
+a standalone worker (`run_standalone_worker`, for horizontal scaling) does not. It starts a relay at boot
+(`start_relay`), and from then on `publish_run_event`, `announce_chat_updated` (`services/chat_thread.py`) and
+`broadcast_approval_change` (`services/approvals/events.py`) queue what they would have published. One writer sends
+the queue in order over the worker's own `/ws/internal` connection as `chat_run_publish` frames, at most
+`hub.relay_batch_size` items to a frame (`server/config/chat_defaults.json`), reconnecting with backoff when the
+backend restarts. The backend publishes each run event into its own hub, so `seq`, `hub_epoch` and the `event_key`
+dedupe are the backend's, exactly as for its own events. It sends each broadcast to its sockets only when the
+CloudEvent type matches the wire key.
+
+Nothing waits on the relay, and nothing is resent. Past `hub.relay_queue_size` queued items an event is dropped, and
+so is a frame the connection lost. The next frame that gets through ends with a `resync` item for the sessions
+those belonged to, and the backend sends their subscribers `opencompany.resync`. Their clients take fresh
+snapshots, in which a stored terminal state wins. The rows are the record: text deltas a resync cannot restore are in
+the saved reply. Other broadcasts from a standalone worker's activities (node status, agent progress, Canvas and
+Memory updates) still reach only that process.
+
## Messages
`get_chat_messages` returns the active path, oldest first:
@@ -439,6 +468,12 @@ What an employee sends to someone waits for the owner's OK while its workflow as
the outcome (`approvals.record_outcome`): `sent`, `not_sent`, or `unknown` when the activity broke off. The node's
activity runs only under that claim. The first time the drafts are listed after a start, sends that never started
are started again and sends that never reported end `failed` with an unknown outcome.
+- **At most once.** Temporal runs a tool activity again only when an attempt broke off (the worker stopped, a
+ timeout); a failure the node reports comes back as a result and is not retried. A later attempt of an agent's call
+ that sends (Ask first off, or no rule) therefore does not send again (`tool_calls.resend_refusal`, checked in
+ `BaseNode.as_activity` after the gate): the model reads `SendOutcomeUnknown` (it may have gone out), and a call
+ answering the owner in the chat is recorded with outcome `unknown`, so the card offers Try again behind a
+ confirmation. Held calls are drafts, made once per call, and still retry.
- **The employee hears how it went**: an `[update]{...}[/update]` note (`approval:<id>`) at the start of its next
turn in the chat, for a send that went, failed or was discarded.
- **In the chat.** A draft made answering the owner is recorded on the run (`parts.approvals`) and shown at once (an
diff --git a/docs-internal/native_llm_sdk.md b/docs-internal/native_llm_sdk.md
index 17b4b17f..eb7f98d0 100644
--- a/docs-internal/native_llm_sdk.md
+++ b/docs-internal/native_llm_sdk.md
@@ -267,6 +267,29 @@ either way, only later. How each declared provider streams:
Turning streaming on for another provider means declaring the flag after a
recorded-stream test shows its response is unchanged (`tests/llm/test_streaming.py`).
+None has a recording yet. Each needs one stream recorded with a real key (a
+request with tool calls and, where the model reasons, reasoning on). Checked
+against each vendor's docs (2026-10-04), this is what stands in the way:
+
+- **Gemini** (`generate_content_stream`): its thinking guide says a client must
+ send thought parts back exactly as received, and a stream arrives as text
+ fragments, with the signature possibly on a part of its own. The streamed
+ parts therefore cannot be folded into the parts the plain call returns
+ (`content` joins parts with a newline; tool-call ids fall back to the part's
+ index), and keeping them as received changes `provider_state`. A recording
+ decides between merging them and relaxing the contract for `provider_state`.
+- **OpenRouter** (an `OpenAIProvider`): streamed `reasoning_details` entries
+ carry `index` only optionally and repeat `id` and `format` in every delta. The
+ SDK's accumulator fails on an entry without `index` and joins repeated
+ strings ("anthropic-claude-v1anthropic-claude-v1"), which would corrupt the
+ details replayed on the next tool turn. It needs its own fold first.
+- **OpenAI-compatible providers** go through the same chat-completions path and
+ need `stream_options.include_usage`, or the response would report no usage.
+ DeepSeek documents it, along with `reasoning_content` deltas, and is the
+ nearest. Groq's reference does not say where streamed usage goes (top-level
+ or `x_groq.usage`). For Mistral, Kimi, Cerebras, xAI and Sarvam, the docs
+ read did not settle it either. A user-named endpoint (`openai_compatible`)
+ can be any server, so it does not stream.
## Registry, Unifier, and Lazy SDK Clients
diff --git a/docs-internal/theme_system.md b/docs-internal/theme_system.md
index 622da390..41ea4817 100644
--- a/docs-internal/theme_system.md
+++ b/docs-internal/theme_system.md
@@ -327,7 +327,7 @@ Wave 15 added structural classNames across the React tree so the handoff CSS sel
| Sidebar | `.sidebar` | WorkflowSidebar |
| Component palette | `.palette` | ComponentPalette |
| Console / chat panel | `.chat` | ConsolePanel (root) |
-| Chat turns (Home and Dev) | `.chat-msg .chat-msg-user` on the owner's bubble; `.chat-msg .chat-msg-bot` on a reply's text only; `.chat-turn-user`, `.chat-turn-bot` on each turn; `.chat-composer` on the message box | features/chat (`UserTurn`, `AssistantTurn`, `Composer`) |
+| Chat turns (Home and Dev) | `.chat-msg .chat-msg-user` on the owner's bubble; `.chat-msg .chat-msg-bot` on a reply's text only; `.chat-turn-user`, `.chat-turn-bot` on each turn; `.chat-composer` on the message box; `.chat-approval` on a draft's card; `.chat-genui` on a generated interface | features/chat (`UserTurn`, `AssistantTurn`, `Composer`, `approval/ApprovalCard`, `turns/GeneratedUiBlock`) |
| Status bar | `.statusbar`, `.pip` (connection dot) | StatusBar |
| Modal | `.modal`, `.modal-frame`, `.modal-head` | Modal |
| Collapsible section | `.cat`, `.cat-head`, `.cat-body` | CollapsibleSection |