Skip to content

Self-improving AI Employees for everyone

fix(chat): a chat run past 24 h cancels its workflow (1b.14)

trohitge6fb360ac204
diff --git a/CLAUDE.md b/CLAUDE.md
index 89cb7747..0fca18e9 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -16,7 +16,7 @@ OpenCompany is an open-source operating system for self-improving AI employees.
| **[Temporal Cleanup & Resilience Plan (Waves 15-18)](./docs-internal/ARCHIVE/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md)** | *Archived — all four waves shipped.* 15 dead-code retirement (Redis-Streams branch in event_waiter, APScheduler cron stack, orphaned `connection_status` factory), 16 per-queue task routing (`TemporalWorkerPool`), 17 resilience hardening (cron `catchup_window`, SDK interceptors, `DEPLOYMENT_MODE`, periodic heartbeats; the one-shot LLM-step retry it records was later reverted to unlimited-with-backoff), 18 worker performance tuning. Current settings live in [Temporal Architecture](./docs-internal/TEMPORAL_ARCHITECTURE.md). |
| **[Node Allowlist](./docs-internal/node_allowlist.md)** | Single-config UI visibility — `server/config/node_allowlist.json` controls which nodes / credential categories / skill folders show in the UI. Five lists with two enforcement tiers (mode-gated allowlist + absolute blocklist). `useNodeAllowlist` hook exposes `isVisible` / `isBlocked` / `isAllowed` / `isCredentialCategoryDisabled` / `isSkillFolderDisabled`. Adding a new disabled domain = single JSON edit, no code change. |
| **[Normal Mode (Home)](./docs-internal/normal_mode.md)** | The owner-facing landing screen: describe a job, an LLM on the server drafts a setup screen (a JSON UI spec the client parses, normalises and renders from a fixed catalogue), and Hire builds, validates, saves and starts one workflow per AI employee (`services/employees/`, graph built from `config/employee_apps.json` plus the owner's skill library and a Canvas node, and checked against `node_allowlist.json` `enabled_nodes`). Covers the screen switch (`app/`, `shellMode` / `ui_shell_mode`, `VITE_NORMAL_MODE=false` kill switch, the same three themes and theme button as the editor), the employee page (the shared chat, `features/chat`, with the employee named in the header beside New conversation; Stop is in the Workspace header), deleting an employee from the team sidebar (stopped through Reset first, then gone from every tab), the live Ask first rule (`workflow_rules`, the chat's Ask first chip via `set_ask_first`; app replies wait in `approvalGate`, sending tool calls are held as drafts by `services/approvals/tool_calls.py` and sent after a 5 s Undo window by `ApprovedToolCallWorkflow`; cards in the chat; builder v3 gates every app reply, attaches every app tool and gives the talk agent `chatUi` and the registry's `talk_send` tools, with instructions identical whatever the rule; older employees are upgraded on Apply / Turn on Talk / Start by `services/employees/upgrade.py`; in a generation that waits for the owner (RFC-0007, `approval_wait_version` 1, which Start does not write yet) Ask first becomes a mode over a class table (`services/approvals/classes.py`) minus always-allow rules (`allows.py`), the agent answering a chat run waits in its turn on `ApprovalWorkflow` and runs the allowed call itself, and the gate no longer holds Stop), run records for "done today" (`machina-run-record-v1`), automatic-pause reasons, Settings (Profile fields; usage-only Billing via `get_employee_usage`; Skills, the user-skills library each new hire copies onto a Skills node; consumer Connectors with `publisher` / `verified`; Plugins, the starter bundles in `hire/starters.json`; the last three share one catalog page), the Workspace dock (an employee's Canvas board, named by the summary's `canvas_node_id`; a live Browser tab with user takeover, shared with Dev mode; Mobile shows the managed local Android phone; a footer timeline of the employee's steps from the step log `services/workspace_steps.py`, read with `workspace_steps_list` and again on `workspace_step`, and Take over, which takes the screen and stops the employee until Hand back), the "Web browser" app (attached whole, each call read-only while the owner asks first through its approval spec's `restrict_while_asking`; while the agent waits in `request_user` the summary's `browser_request` shows Needs you and Help in browser), Talk on the employee page (a thread over the chat session whose id is the workflow id, rows written through `services/chat_thread.py` and, for the owner's messages and their chat runs, `services/chat/ledger.py` (see [Chat Protocol](./docs-internal/chat_protocol.md)); the talk line is chatTrigger → an agent → `chatReply` "Reply in Chat": a chat hire's own agent is its talk agent, other hires get a "Talk with {name}" agent sharing the worker's tools and skills with its own Context, and schedule hires also "Post to Talk"), Turn on Talk (for talk `off`) and Apply (while the summary's `pending_changes` is true) via `enable_employee_talk` / `apply_employee_changes`, each a restart on the saved graph through `services/deployment/restart.py`, self-extension from Talk through the Agent Builder within the Hire rule (`services/employees/policy.py`), the 3D orb, the wire contract and the `employee_lifecycle` / `approval_lifecycle` / `chat.updated` / `workspace_step` broadcasts. |
-| **[Chat Protocol](./docs-internal/chat_protocol.md)** | The wire contract between the chat runtime (`server/services/chat/`) and the shared chat UI: AG-UI's event model on the existing WebSocket, snake_case, in the CloudEvents envelope. A message a deployed chat trigger will answer starts a **chat run** admitted with it in one reserved write transaction (`Database.reserved_session`, SQLite `BEGIN IMMEDIATE`): one live run per session (the lane, also a partial unique index; a second send answers `run_in_progress`), idempotent `client_message_id`, claimed and finished by the MachinaWorkflow that answers it behind `machina-chat-run-v1` (the run id is trusted only from an event whose source is `opencompany://services/chat`; `run_scope` rides every node context; only the claimant finishes), the reply saved as `a_<run id>` by `chatReply`, every message appended after the session's active leaf (`chat_threads`) so concurrent writes never fork the thread. `chat_run_event` frames reach only sockets that passed `chat_subscribe`'s owner check (`services/chat/hub.py`: per-run `seq`, `hub_epoch`, bounded per-socket queues with a resync), never the status broadcast; a standalone worker sends its run events, `chat.updated` and `approval_lifecycle` to the backend over `/ws/internal` (`services/chat/relay.py`, `chat_run_publish`, ordered, dropped events resynced). A watchdog (`services/chat/watchdog.py`, started by `main.py`) ends runs nothing will finish (`not_delivered`, `timed_out`, `interrupted`). The agent wired into Reply in Chat (`answers_chat_run`) streams its answer as `text.*` events (`services/chat/stream.py`, activity input only, no workflow patch), its tool calls show as steps (`services/chat/steps.py`; label from the plugin's `chat_step`), and `stop_chat_run` stops a run cooperatively: its LLM step polls the run and returns what was shown, unstarted tools are skipped, the reply is saved `stopped`, and the watchdog cancels a run still stopping after `runs.stop_grace_s`. Generated UI: the `show_ui` tool (`chatUi`) is checked against `config/chat_genui_catalog.json` (`services/genui/spec.py`; refused with every reason so the model retries), saved as a run part, streamed as `activity.*` RFC 6902 patches and sealed into the reply (`services/chat/parts.py`); `chat_ui_state` keeps what the owner set, and a button press comes back as `send_chat_message{ui_event}` -> a `[ui-event]` turn. The answering agent's system prompt ends with a fixed guide (`services/chat/guide.py`), what the owner set reaches its next turn as a `[ui-state]` note (`chat_notes`, `services/chat/notes.py`: claimed by the payload, told when the run answered), a trailing `<followups>` block becomes `parts.followups`, and a `chat_sources` tool's results are numbered per conversation for `[n]` citations (`services/chat/sources.py`). Approvals (`services/approvals/`, its own section): a plugin's `approval` spec marks a tool call that sends; while the workflow asks first it is held as a `tool_call` draft, shown as a card on the reply that made it, sent once after the Undo window by `ApprovedToolCallWorkflow`, with the outcome told to the employee as an `[update]` note; in a generation that waits for the owner (RFC-0007, not yet written by Start) the mode and the call's class decide what asks, the agent answering the run waits in its turn for Allow or Decline (`services/approvals/waiting.py`; a message sent meanwhile declines the call and answers `declined`), and the watchdog does not time the wait out. Branches (`services/chat/branches.py`): `get_chat_messages` returns the path to the thread's active leaf, each message with its versions (`siblings`), `editable` and the owner's rating; `edit_chat_message` / `regenerate_chat_reply` / `switch_chat_branch` move the path in one reserved transaction under the conversation store's lock, taking each agent's stored conversation back to the cursor its run recorded at `prepare_payload` (`chat_runs.context_cursors`) or restoring the branch's kept copy (`chat_branch_snapshots`), refused on `revision_conflict`, `run_in_progress`, `cannot_rewind`, `branch_unavailable` or `older_generation`; `set_chat_feedback` keeps a rating and tells the next turn (`[feedback]` note). Attachments (`services/chat/attachments.py`): the chat uploads files under `uploads/` and sends only their paths, the server rebuilds each reference from the file (refusing anything else, at most six), the answering agent reads an `[attachments]` line and images ride the opening user message for a provider with `vision.user_images`; `options.web: false` drops the `search` tools for that run; Home's model picker (`services/chat/choice.py`, `get_chat_models`) sends `options.model` / `options.effort`, resolved at admission (Auto = the owner's default model; refused with `model_unavailable`, never swapped) and applied only to the answering agent, effort only to a provider's `effort_models`; `get_chat_context` gives slash commands (generic plus the apps' `commands`) and capabilities; dictation is the speech plugin's `dictation_status` / `transcribe_audio`. Lists every event, handler, part and error code (among them `engine_unavailable`: with Temporal not connected a workflow's chat saves nothing, since nothing could deliver it). |
+| **[Chat Protocol](./docs-internal/chat_protocol.md)** | The wire contract between the chat runtime (`server/services/chat/`) and the shared chat UI: AG-UI's event model on the existing WebSocket, snake_case, in the CloudEvents envelope. A message a deployed chat trigger will answer starts a **chat run** admitted with it in one reserved write transaction (`Database.reserved_session`, SQLite `BEGIN IMMEDIATE`): one live run per session (the lane, also a partial unique index; a second send answers `run_in_progress`), idempotent `client_message_id`, claimed and finished by the MachinaWorkflow that answers it behind `machina-chat-run-v1` (the run id is trusted only from an event whose source is `opencompany://services/chat`; `run_scope` rides every node context; only the claimant finishes), the reply saved as `a_<run id>` by `chatReply`, every message appended after the session's active leaf (`chat_threads`) so concurrent writes never fork the thread. `chat_run_event` frames reach only sockets that passed `chat_subscribe`'s owner check (`services/chat/hub.py`: per-run `seq`, `hub_epoch`, bounded per-socket queues with a resync), never the status broadcast; a standalone worker sends its run events, `chat.updated` and `approval_lifecycle` to the backend over `/ws/internal` (`services/chat/relay.py`, `chat_run_publish`, ordered, dropped events resynced). A watchdog (`services/chat/watchdog.py`, started by `main.py`) ends runs nothing will finish (`not_delivered`, `timed_out`, `interrupted`). The agent wired into Reply in Chat (`answers_chat_run`) streams its answer as `text.*` events (`services/chat/stream.py`, activity input only, no workflow patch), its tool calls show as steps (`services/chat/steps.py`; label from the plugin's `chat_step`), and `stop_chat_run` stops a run cooperatively: its LLM step polls the run and returns what was shown, unstarted tools are skipped, the reply is saved `stopped`, and the watchdog cancels the workflow of a run still stopping after `runs.stop_grace_s`, and of one running past `runs.max_running_s` (`timed_out`). Generated UI: the `show_ui` tool (`chatUi`) is checked against `config/chat_genui_catalog.json` (`services/genui/spec.py`; refused with every reason so the model retries), saved as a run part, streamed as `activity.*` RFC 6902 patches and sealed into the reply (`services/chat/parts.py`); `chat_ui_state` keeps what the owner set, and a button press comes back as `send_chat_message{ui_event}` -> a `[ui-event]` turn. The answering agent's system prompt ends with a fixed guide (`services/chat/guide.py`), what the owner set reaches its next turn as a `[ui-state]` note (`chat_notes`, `services/chat/notes.py`: claimed by the payload, told when the run answered), a trailing `<followups>` block becomes `parts.followups`, and a `chat_sources` tool's results are numbered per conversation for `[n]` citations (`services/chat/sources.py`). Approvals (`services/approvals/`, its own section): a plugin's `approval` spec marks a tool call that sends; while the workflow asks first it is held as a `tool_call` draft, shown as a card on the reply that made it, sent once after the Undo window by `ApprovedToolCallWorkflow`, with the outcome told to the employee as an `[update]` note; in a generation that waits for the owner (RFC-0007, not yet written by Start) the mode and the call's class decide what asks, the agent answering the run waits in its turn for Allow or Decline (`services/approvals/waiting.py`; a message sent meanwhile declines the call and answers `declined`), and the watchdog does not time the wait out. Branches (`services/chat/branches.py`): `get_chat_messages` returns the path to the thread's active leaf, each message with its versions (`siblings`), `editable` and the owner's rating; `edit_chat_message` / `regenerate_chat_reply` / `switch_chat_branch` move the path in one reserved transaction under the conversation store's lock, taking each agent's stored conversation back to the cursor its run recorded at `prepare_payload` (`chat_runs.context_cursors`) or restoring the branch's kept copy (`chat_branch_snapshots`), refused on `revision_conflict`, `run_in_progress`, `cannot_rewind`, `branch_unavailable` or `older_generation`; `set_chat_feedback` keeps a rating and tells the next turn (`[feedback]` note). Attachments (`services/chat/attachments.py`): the chat uploads files under `uploads/` and sends only their paths, the server rebuilds each reference from the file (refusing anything else, at most six), the answering agent reads an `[attachments]` line and images ride the opening user message for a provider with `vision.user_images`; `options.web: false` drops the `search` tools for that run; Home's model picker (`services/chat/choice.py`, `get_chat_models`) sends `options.model` / `options.effort`, resolved at admission (Auto = the owner's default model; refused with `model_unavailable`, never swapped) and applied only to the answering agent, effort only to a provider's `effort_models`; `get_chat_context` gives slash commands (generic plus the apps' `commands`) and capabilities; dictation is the speech plugin's `dictation_status` / `transcribe_audio`. Lists every event, handler, part and error code (among them `engine_unavailable`: with Temporal not connected a workflow's chat saves nothing, since nothing could deliver it). |
| **[Theme System](./docs-internal/theme_system.md)** | Three themes — the base `light` and `dark` plus one stylized skin, `atomic` (the design handoff's nine other skins were removed on 2026-10-12) — driven by `<html data-theme>` + per-theme CSS files in `client/src/themes/`. Home, the editor and the sign-in screen all show the chosen theme, and one theme button ([components/ui/ThemeButton.tsx](./client/src/components/ui/ThemeButton.tsx)) in the Home header, the editor's toolbar and the sign-in screen steps Light → Dark → Atomic (`cycleTheme`; every theme name comes from `THEME_LABEL` in `ThemeContext.tsx`). Token taxonomy (surface / fg / border / accent / typography / motion) in **hex + `color-mix()`**; `@theme inline` bridge maps `--color-X: var(--X)` (no `hsl()` wrapper); per-theme files own the colour hex, base.css owns the shared `--tint-*` alpha scale; action role tokens (incl. `-ink` readable text), **per-theme `--code-*` syntax tokens** (tier 6 — the code editor, console/output JSON viewers, and chat code blocks paint in each theme's own palette: keyword→trigger, string→success, number→agent, function→model, on an adaptive dark/light code surface; replaced the global dracula `--prism-*` block + dead `getPrismTokenCSS`; the `OutputPanel` JSON viewer reads the same vars), the `--pulse-keyframe` animation system + `.opencompany-*` helpers in `animations.css`, per-component decoration hooks (`.canvas-host`, `.toolbar`, `.sq-node`, …; Atomic's panel edges, starburst canvas and node styling), **canvas-node visual contract via `--node-color` CSS custom property** (no inline `background` / `border` on node components — base.css + per-theme CSS owns visuals; `NodeStyle` helper type at [types/NodeTypes.ts](./client/src/types/NodeTypes.ts) makes the inline custom-prop typecheck-clean), **`--node-pulse-color` separate from `--node-color`** so executing-node glow uses each theme's highest-contrast accent regardless of plugin accent, **`data-page-hidden` animation pause** (set by `app/usePageActivitySync.ts`, mounted in `App.tsx` above the sign-in gate so it covers every screen, while the page is hidden per the Page Visibility API, never merely because the window lost focus; base.css declares `html[data-page-hidden] *, *::before, *::after { animation-play-state: paused !important }` to prevent compositor stall on tab return), **per-theme icon glyphs** (Atomic's 29 inline SVG glyphs in [themedGlyphs.ts](./client/src/assets/icons/themedGlyphs.ts), read by theme-aware [NodeIcon.tsx](./client/src/assets/icons/NodeIcon.tsx)), **per-theme canvas-grid + custom cursors** via `--canvas-grid` / `--cursor-default` slots, **parameter panel migrated to Tailwind tokens** (no `useAppTheme()` reads; section headers carry the display-typography triplet; raw `<Button>` swapped to `<ActionButton intent>`), scrollbars from the global token rules plus Atomic's own, 9-event WebAudio sound system (3 packs — parchment, terminal, vibraphone — via `--sound-pack` token + `useSound()` hook + global hover delegate + sonner toast monkey-patch + `withSound()` HOC + `Sounds.unlock()` gesture-unlock for AudioContext autoplay-policy compliance), `@media (prefers-reduced-motion: reduce)` accessibility, 30 ms throttle on `type` / `hover`, migration recipe, anti-patterns. Read this before adding a new theme, migrating a component to the new contract, or adding a canvas-node component. |
| **[Design System Bundle](./docs-internal/design-system/IMPLEMENTATION.md)** | Canonical, vendored design-system reference (extracted from this repo; see its `IMPLEMENTATION.md` for the contract). Tokens are **hex + `color-mix()`** (`tokens/{colors,typography,spacing,motion,fonts,base}.css`) — the format the live codebase standardizes on. Includes 34 reference components (`components/`, inline-style reference only — recreate in Tailwind/shadcn idioms), full-app UI kit (`ui_kits/opencompany/index.html`), theme docs (`guidelines/THEMES.md`), **the merged handoff brief (`HANDOFF.md` — glow-ladder traps, step-edge contract, toolbar layout traps, canvas-layer asymmetries, plus the recorded product amendments)**, and **the panel×theme fidelity target (`reference-mockup/Panel Theme Matrix.dc.html` — 17 panels × 3 themes, open in a browser)**. `reference/themes/` no longer vendors CSS snapshots — it points at the authoritative [client/src/themes/](./client/src/themes/). Copy token values verbatim; do not re-derive by eye. Pairs with [Theme System](./docs-internal/theme_system.md). |
| **[Schema Source of Truth RFC](./docs-internal/ARCHIVE/schema_source_of_truth_rfc.md)** | *Archived — shipped.* Backend is SSOT for node schemas, visual metadata, handlers, palette metadata, icons; `VITE_NODESPEC_BACKEND` defaults on and the frontend node definitions are gone. Plugin pattern: one `BaseNode` subclass in `server/nodes/<group>/<plugin>/__init__.py`. Wire format: `asset:<key>` / `<lib>:<brand>` / URL / emoji. Endpoint: `/api/schemas/nodes/{type}/spec.json`. Current reference: [Plugin System](./docs-internal/plugin_system.md). |
diff --git a/docs-internal/chat_protocol.md b/docs-internal/chat_protocol.md
index 600da556..9fa6913d 100644
--- a/docs-internal/chat_protocol.md
+++ b/docs-internal/chat_protocol.md
@@ -102,7 +102,8 @@ streamed.
| `stopped` | A legacy or uncontrolled chat run ended after Stop (an OpenCompany addition to AG-UI's outcomes). Controlled Stop does not emit this terminal outcome. |
Runs nothing will finish end with `run.failed`: code `not_delivered` (never picked up, or the employee stopped
-first), `timed_out` (running longer than its current `runs.max_running_s` window), `interrupted` (its workflow closed without finishing
+first), `timed_out` (running longer than its current `runs.max_running_s` window; the watchdog cancels its Temporal
+workflow first, so nothing keeps working on the answer), `interrupted` (its workflow closed without finishing
it and no reply was saved; with a saved reply it finishes instead). A Reset ends live runs with code `reset`, the
owner's Clear with `cleared`, and both delete them. Intentional suspension of a controlled generation excludes
its live runs from pickup and running expiry; Resume starts a fresh timeout window, as described below.
@@ -365,7 +366,9 @@ newer version wins in place.
running runs are excluded from age expiry. After successful Resume, the existing generation manifest stores
`last_resumed_at`: pending pickup counts from the latest of creation, server start and Resume; queued pickup
follows the resumed delivery window; running expiry counts from the later of run start and Resume. Stop does
- not move controlled runs into the legacy `stopping` state, so they bypass grace-period cancellation. Temporal
+ not move controlled runs into the legacy `stopping` state, so they bypass grace-period cancellation. A running
+ run past `runs.max_running_s`, controlled or not, has its workflow cancelled before it ends `timed_out`, unless
+ its generation is paused or a turn of it waits for the owner. Temporal
closure checks follow the current Workflow ID and compare execution chains across continue-as-new.
## Streaming, steps and Stop
diff --git a/server/services/chat/ledger.py b/server/services/chat/ledger.py
index ff2efc4c..e94aa449 100644
--- a/server/services/chat/ledger.py
+++ b/server/services/chat/ledger.py
@@ -22,9 +22,10 @@ carry an ``event_key`` the hub publishes once.
picked up (``not_delivered``, counted from the later of its creation and
this process's start, so a restart does not expire runs it held up), a
queued run whose deployment stopped or did not pick it up after resuming, a
-running run past ``runs.max_running_s`` (``timed_out``), and a running run
-whose Temporal workflow closed without finishing it: finished when its reply
-was saved, ``interrupted`` otherwise.
+running run past ``runs.max_running_s`` (``timed_out``; its Temporal workflow
+is cancelled first, so nothing keeps working on an answer no one waits for),
+and a running run whose Temporal workflow closed without finishing it:
+finished when its reply was saved, ``interrupted`` otherwise.
**Controlled Stop** suspends the owning generation while keeping the run
and its lane alive. The watchdog excludes intentionally suspended generations
@@ -705,7 +706,7 @@ async def sweep(
if run.state == "stopping":
closed = await _sweep_stopping(database, run, now, temporal_cancel)
else:
- closed = await _sweep_one(database, run, now, process_started, pickup, longest, temporal_status)
+ closed = await _sweep_one(database, run, now, process_started, pickup, longest, temporal_status, temporal_cancel)
except Exception: # noqa: BLE001 - one bad row must not stop the sweep
logger.warning("Chat run sweep failed for a run", run_id=run.run_id, exc_info=True)
continue
@@ -724,15 +725,24 @@ async def _sweep_stopping(
requested = _aware(run.stop_requested_at) or _aware(run.started_at) or _aware(run.created_at) or now
if now - requested < timedelta(seconds=runs_setting("stop_grace_s")):
return None
- if temporal_cancel is not None and run.temporal_workflow_id:
- try:
- await temporal_cancel(run.temporal_workflow_id, run.temporal_run_id)
- except Exception: # noqa: BLE001 - the run ends either way
- logger.warning("Could not cancel a stopped run's workflow", run_id=run.run_id, exc_info=True)
+ await _cancel_workflow(run, temporal_cancel)
await _seal(database, run)
return await _settle(database, run, await _success_values(database, run, "stopped"))
+async def _cancel_workflow(run: ChatRun, temporal_cancel: Optional[TemporalCancel]) -> None:
+ """Ask Temporal to cancel the workflow answering ``run``, best effort: the
+ caller ends the run either way (a stopped run, or one past
+ ``runs.max_running_s``). Asked before the row ends, so a sweep that
+ dies in between asks again on its next round."""
+ if temporal_cancel is None or not run.temporal_workflow_id:
+ return
+ try:
+ await temporal_cancel(run.temporal_workflow_id, run.temporal_run_id)
+ except Exception: # noqa: BLE001 - the run ends either way
+ logger.warning("Could not cancel a chat run's workflow", run_id=run.run_id, exc_info=True)
+
+
async def _sweep_one(
database: Any,
run: ChatRun,
@@ -741,6 +751,7 @@ async def _sweep_one(
pickup: timedelta,
longest: timedelta,
temporal_status: Optional[TemporalStatus],
+ temporal_cancel: Optional[TemporalCancel],
) -> Optional[ChatRun]:
control = await owning_control(database, run)
controlled = controlled_generation(control)
@@ -772,6 +783,7 @@ async def _sweep_one(
from services.approvals.waiting import run_waited
if not await run_waited(database, run.run_id, since=now - longest):
+ await _cancel_workflow(run, temporal_cancel)
return await fail_run(database, run, code="timed_out", message="The employee took too long to answer.")
if temporal_status is None or not run.temporal_workflow_id or now - started < _CLOSED_GRACE:
return None
diff --git a/server/services/chat/watchdog.py b/server/services/chat/watchdog.py
index a7a719c2..8b3575a2 100644
--- a/server/services/chat/watchdog.py
+++ b/server/services/chat/watchdog.py
@@ -61,8 +61,9 @@ async def temporal_workflow_closed(workflow_id: str, run_id: Optional[str]) -> O
async def cancel_temporal_workflow(workflow_id: str, run_id: Optional[str]) -> None:
- """Ask Temporal to cancel a workflow (a stopped run's). Does nothing when
- Temporal cannot be reached; a workflow that already closed is fine."""
+ """Ask Temporal to cancel a workflow (a stopped or timed-out run's). Does
+ nothing when Temporal cannot be reached; a workflow that already closed is
+ fine."""
try:
from core.container import container
diff --git a/server/tests/services/chat/test_ledger.py b/server/tests/services/chat/test_ledger.py
index 8d1207ec..286508ad 100644
--- a/server/tests/services/chat/test_ledger.py
+++ b/server/tests/services/chat/test_ledger.py
@@ -356,6 +356,36 @@ async def test_a_run_past_its_longest_ends_as_timed_out(database, hub):
assert (await ledger.get_run(database, admission.run.run_id)).error_code == "timed_out"
+async def test_a_timed_out_run_has_its_workflow_cancelled_before_the_row_ends(database, hub):
+ # 1b.14: nothing keeps working on an answer no one waits for.
+ admission = await admit(database)
+ run = await started(database, admission)
+ await age(database, run.run_id, started_at=LONG_AGO)
+ asked = []
+
+ async def cancel(workflow_id, temporal_run_id):
+ asked.append((workflow_id, temporal_run_id, (await ledger.get_run(database, run.run_id)).state))
+
+ assert await sweep(database, temporal_cancel=cancel) == [run.run_id]
+ assert asked == [("tw-1", "tr-1", "running")] # asked while the row was still live
+ assert (await ledger.get_run(database, run.run_id)).error_code == "timed_out"
+ # Ended, it is neither swept nor cancelled again.
+ assert await sweep(database, temporal_cancel=cancel) == []
+ assert len(asked) == 1
+
+
+async def test_a_failed_cancel_still_times_the_run_out(database, hub):
+ admission = await admit(database)
+ run = await started(database, admission)
+ await age(database, run.run_id, started_at=LONG_AGO)
+
+ async def broken(*_):
+ raise RuntimeError("temporal is down")
+
+ assert await sweep(database, temporal_cancel=broken) == [run.run_id]
+ assert (await ledger.get_run(database, run.run_id)).error_code == "timed_out"
+
+
async def answered(database, approval_id, at):
"""The owner answered a held call at ``at``."""
from sqlalchemy import update
@@ -369,15 +399,24 @@ async def answered(database, approval_id, at):
async def test_a_turn_waiting_for_the_owner_is_not_timed_out(database, hub):
# RFC-0007 D2: the wait does not count, and the answer starts a fresh window.
+ # A run that waited 30 h and was then answered is not timed out, nor is
+ # its workflow cancelled (1b.14), until its fresh window passes too.
admission = await admit(database)
run = await started(database, admission)
- await age(database, run.run_id, started_at=LONG_AGO)
+ await age(database, run.run_id, started_at=NOW - timedelta(hours=30))
await held_call(database, "ap_1", run.run_id)
- assert await sweep(database) == []
+ cancelled = []
+
+ async def cancel(workflow_id, temporal_run_id):
+ cancelled.append((workflow_id, temporal_run_id))
+
+ assert await sweep(database, temporal_cancel=cancel) == []
await answered(database, "ap_1", NOW - timedelta(hours=1))
- assert await sweep(database) == []
+ assert await sweep(database, temporal_cancel=cancel) == []
+ assert cancelled == []
await answered(database, "ap_1", LONG_AGO)
- assert await sweep(database) == [run.run_id]
+ assert await sweep(database, temporal_cancel=cancel) == [run.run_id]
+ assert cancelled == [("tw-1", "tr-1")]
assert (await ledger.get_run(database, run.run_id)).error_code == "timed_out"
diff --git a/server/tests/services/chat/test_stop.py b/server/tests/services/chat/test_stop.py
index 67769a73..c37d671e 100644
--- a/server/tests/services/chat/test_stop.py
+++ b/server/tests/services/chat/test_stop.py
@@ -289,10 +289,18 @@ async def test_long_controlled_pause_does_not_expire_and_resume_grants_a_fresh_w
started_at=NOW - timedelta(days=30), created_at=NOW - timedelta(days=30),
))
await session.commit()
+ cancelled = []
+
+ async def cancel(workflow_id, temporal_run_id):
+ cancelled.append((workflow_id, temporal_run_id))
+
+ # A paused run is never timed out, so its workflow is never cancelled (1b.14).
for state in ("pausing", "paused", "resuming"):
await version_control(database, status=state)
- assert await ledger.sweep(database, now=NOW, process_started=NOW - timedelta(days=30)) == []
+ assert await ledger.sweep(database, now=NOW, process_started=NOW - timedelta(days=30), temporal_cancel=cancel) == []
await version_control(database, status="running", resumed=NOW)
- assert await ledger.sweep(database, now=NOW + longest - timedelta(seconds=1), process_started=NOW) == []
- assert await ledger.sweep(database, now=NOW + longest + timedelta(seconds=1), process_started=NOW) == [run.run_id]
+ assert await ledger.sweep(database, now=NOW + longest - timedelta(seconds=1), process_started=NOW, temporal_cancel=cancel) == []
+ assert cancelled == []
+ assert await ledger.sweep(database, now=NOW + longest + timedelta(seconds=1), process_started=NOW, temporal_cancel=cancel) == [run.run_id]
+ assert cancelled == [(run.temporal_workflow_id, run.temporal_run_id)]
assert (await ledger.get_run(database, run.run_id)).error_code == "timed_out"