Skip to content

Self-improving AI Employees for everyone

feat(temporal): the retry alert, with the attention store and one Home line (1b.30)

trohitgbc26edc8cde8
diff --git a/CLAUDE.md b/CLAUDE.md
index 25b47fdb..89cb7747 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -752,7 +752,7 @@ Service orchestration lives in the Python CLI (`company dev` / `start` / `stop`,
### Temporal Distributed Execution
-Workflows execute via Temporal for durability and horizontal scaling, gated by `TEMPORAL_ENABLED`. Three dispatch paths exist — legacy single `execute_node_activity` (fallback), per-type `node.{type}.v{version}` activities (F4.A, `TEMPORAL_PER_TYPE_DISPATCH=true`), and Agent-as-child-workflow (F4.B, `TEMPORAL_AGENT_WORKFLOW_ENABLED=true`) — with `rlm_agent`/`claude_code_agent` always bypassing AgentWorkflow. **Per-queue activity routing is production-default (Wave 16)**: a `TemporalWorkerPool` runs one activity-only worker per plugin-declared `cls.task_queue` (ai-heavy / browser / code-exec / rest-api / messaging / ...) with per-queue concurrency + rate limits + resource-based tuning for ai-heavy/browser; rollback via `TEMPORAL_WORKER_POOL_ENABLED=false`. Resilience knobs (Waves 17-18): `DEPLOYMENT_MODE` (local halves concurrency; sizes the sticky cache), unlimited-with-exponential-backoff retry on `agent.execute_llm_step` (`LLM_STEP_RETRY`, `maximum_attempts=0`, 5 s to 5 min; terminal provider categories still fail fast as non-retryable; the earlier one-shot policy was reverted), observability interceptors (`activity_retry` WARN on re-dispatch), 30s periodic heartbeat during long activity bodies, cron `catchup_window=24h`, poller autoscaling. **Conditional edges are evaluated on this path too, as of the `machina-conditional-edges-v1` patch.** They previously were not: `MachinaWorkflow._find_ready_nodes` scheduled purely on dependency completion, so a condition set in the editor rendered an edge label and then did nothing once execution routed through Temporal — silently, with both branches running. The skip mirrors `WorkflowExecutor._evaluate_incoming_conditions` exactly rather than improving on it: **OR-any** across a node's conditional incoming edges, and **skipping is not transitive** (a skipped node counts as completed, matching `get_completed_nodes` including `SKIPPED`, so an unconditional downstream node still runs). `evaluate_condition` is imported from `services.execution.conditions` **directly, never via the package `__init__`**, which also pulls in the executor / cache / recovery sweeper / DLQ; the module itself is pure (`re` + comparisons, no IO, no clock, no randomness) so it is safe to evaluate inside a workflow. The `workflow.patched(...)` gate is mandatory — the skip changes which activities get scheduled, so an ungated change would break replay for in-flight workflows. Locked by `tests/temporal/test_conditional_edges.py`, whose patch-closed case asserts the pre-fix command sequence is still reproduced verbatim.
+Workflows execute via Temporal for durability and horizontal scaling, gated by `TEMPORAL_ENABLED`. Three dispatch paths exist — legacy single `execute_node_activity` (fallback), per-type `node.{type}.v{version}` activities (F4.A, `TEMPORAL_PER_TYPE_DISPATCH=true`), and Agent-as-child-workflow (F4.B, `TEMPORAL_AGENT_WORKFLOW_ENABLED=true`) — with `rlm_agent`/`claude_code_agent` always bypassing AgentWorkflow. **Per-queue activity routing is production-default (Wave 16)**: a `TemporalWorkerPool` runs one activity-only worker per plugin-declared `cls.task_queue` (ai-heavy / browser / code-exec / rest-api / messaging / ...) with per-queue concurrency + rate limits + resource-based tuning for ai-heavy/browser; rollback via `TEMPORAL_WORKER_POOL_ENABLED=false`. Resilience knobs (Waves 17-18): `DEPLOYMENT_MODE` (local halves concurrency; sizes the sticky cache), unlimited-with-exponential-backoff retry on `agent.execute_llm_step` (`LLM_STEP_RETRY`, `maximum_attempts=0`, 5 s to 5 min; terminal provider categories still fail fast as non-retryable; the earlier one-shot policy was reverted), observability interceptors (`activity_retry` WARN at the first retry, the attempt limit and every 10th attempt after it; an activity still retrying past its limit shows on Home as a `stuck_retry` attention item, `services/attention.py`), 30s periodic heartbeat during long activity bodies, cron `catchup_window=24h`, poller autoscaling. **Conditional edges are evaluated on this path too, as of the `machina-conditional-edges-v1` patch.** They previously were not: `MachinaWorkflow._find_ready_nodes` scheduled purely on dependency completion, so a condition set in the editor rendered an edge label and then did nothing once execution routed through Temporal — silently, with both branches running. The skip mirrors `WorkflowExecutor._evaluate_incoming_conditions` exactly rather than improving on it: **OR-any** across a node's conditional incoming edges, and **skipping is not transitive** (a skipped node counts as completed, matching `get_completed_nodes` including `SKIPPED`, so an unconditional downstream node still runs). `evaluate_condition` is imported from `services.execution.conditions` **directly, never via the package `__init__`**, which also pulls in the executor / cache / recovery sweeper / DLQ; the module itself is pure (`re` + comparisons, no IO, no clock, no randomness) so it is safe to evaluate inside a workflow. The `workflow.patched(...)` gate is mandatory — the skip changes which activities get scheduled, so an ungated change would break replay for in-flight workflows. Locked by `tests/temporal/test_conditional_edges.py`, whose patch-closed case asserts the pre-fix command sequence is still reproduced verbatim.
Execution routing falls back Temporal → sequential; the Temporal dev server + embedded worker are managed in-process under `server/services/temporal/` (pooch-downloaded CLI, single `temporal server start-dev` process, gRPC `TEMPORAL_FRONTEND_GRPC_PORT` + Web UI `TEMPORAL_UI_PORT`, SQLite at `~/.opencompany/temporal.db`). Full architecture, dispatch matrix, per-node + agent-loop lifecycle, the F4.B `agent.*` activities (unsuffixed since the AgentWorkflow fold; read the live set from `collect_agent_activities()` rather than a hardcoded count), heartbeat semantics, delegation input contract, the worker tuning recipe, and all `.env` tunables live in [docs-internal/TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md).
@@ -2554,6 +2554,7 @@ Updated `client_left` and presence handlers use `has_real_android_devices()` ins
| `chat_run_event` | CloudEvents v1.0 envelope (type=`com.opencompany.chat.run.<suffix>`, id `<run id>:<seq>`, subject = run id) — AG-UI run events, sent by `services/chat/hub.py` only to sockets subscribed to the session through `chat_subscribe`, never on the broadcast; see [chat_protocol.md](./docs-internal/chat_protocol.md) |
| `workflow_lifecycle` | CloudEvents v1.0 envelope for workflow stages (`deployment.*`, `lock.*`, `execution.*`, `imported`, `renamed`, `created`, `deleted`) — other tabs invalidate their workflow lists |
| `workspace_step` | CloudEvents v1.0 envelope (type=`com.opencompany.workspace.step`, subject = workflow id, data `{workflow_id, surface, step_id}`) — sent directly by `services/workspace_steps.py` when a Browser, Canvas or phone step is recorded; identity only, Home's Workspace timeline refetches via `workspace_steps_list` |
+| `attention_lifecycle` | CloudEvents v1.0 envelope (type=`com.opencompany.attention.{opened,resolved}`, subject = the item's kind, data `{kind}`) — sent directly by `services/attention.py` when an attention item opens or is resolved; identity only, Home's attention line refetches via `list_attention_items` (and once a minute, since an item drops off an hour after it was last seen) |
| `deployment_snapshot` | CloudEvents v1.0 envelope (type=`workflow.deployment.snapshot`) — pushed once per WS connect from `broadcaster._send_deployment_snapshot`; lets the FE reconcile stale `deploymentStatus.isRunning=true` after a backend restart wiped `DeploymentManager._deployments`. Empty list is meaningful (forces reset). |
| `variable_update` | Single variable value change |
| `workflow_status` | Workflow execution progress |
diff --git a/client/src/features/home/HomeShell.tsx b/client/src/features/home/HomeShell.tsx
index 0d5e772b..32f3ac7b 100644
--- a/client/src/features/home/HomeShell.tsx
+++ b/client/src/features/home/HomeShell.tsx
@@ -5,7 +5,8 @@
* current view, either hiring a new employee or one employee's page.
*
* The shell owns what spans views: the employee broadcasts that keep the
- * team current, the orb behind the content, the Workspace dock on the
+ * team current, the line for work that keeps retrying (header/AttentionLine),
+ * the orb behind the content, the Workspace dock on the
* right, the Settings dialog, the Welcome guide (onboarding/, which
* opens over the hire view on first launch) and, once the guide is
* finished, the Get started checklist. Connection actions open the app shell's
@@ -25,6 +26,7 @@ import { useApprovalLifecycle } from './approvals/data';
import { useEmployeeLifecycle, useEmployeesQuery } from './data/employees';
import type { EmployeeSummary } from './data/schemas';
import { EmployeeView } from './employee/EmployeeView';
+import { AttentionLine } from './header/AttentionLine';
import { HomeHeader } from './header/HomeHeader';
import { HireView } from './hire/HireView';
import { GetStartedChecklist } from './onboarding/GetStartedChecklist';
@@ -89,6 +91,7 @@ export default function HomeShell() {
employee={employee}
scrolled={scrolled}
/>
+ <AttentionLine />
{view.kind === 'employee' ? (
<div className="relative z-10 flex min-h-0 flex-1 flex-col">
<div ref={viewRef} key={viewKey} className="flex min-h-0 w-full flex-1 flex-col">
diff --git a/client/src/features/home/__tests__/attention.test.tsx b/client/src/features/home/__tests__/attention.test.tsx
new file mode 100644
index 00000000..59282197
--- /dev/null
+++ b/client/src/features/home/__tests__/attention.test.tsx
@@ -0,0 +1,78 @@
+/**
+ * Home's attention line (header/AttentionLine): nothing while no item is
+ * open; one warning line while work keeps retrying; gone after the
+ * `attention_lifecycle` broadcast and the refetch it starts.
+ */
+
+import { beforeEach, describe, expect, it, vi } from 'vitest';
+import { act, render, screen, waitFor } from '@testing-library/react';
+import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
+
+const sendRequest = vi.fn();
+const listeners = new Map<string, Set<(data: unknown) => void>>();
+
+vi.mock('@/contexts/WebSocketContext', async (importOriginal) => ({
+ ...(await importOriginal<typeof import('@/contexts/WebSocketContext')>()),
+ useWebSocketActions: () => ({
+ sendRequest,
+ isReady: true,
+ addEventListener: (type: string, listener: (data: unknown) => void) => {
+ const set = listeners.get(type) ?? new Set();
+ set.add(listener);
+ listeners.set(type, set);
+ return () => set.delete(listener);
+ },
+ }),
+}));
+
+import { AttentionLine } from '../header/AttentionLine';
+
+let items: unknown[] = [];
+
+function renderLine() {
+ const client = new QueryClient({ defaultOptions: { queries: { retry: false } } });
+ render(
+ <QueryClientProvider client={client}>
+ <AttentionLine />
+ </QueryClientProvider>,
+ );
+}
+
+function emit(type: string, data: unknown) {
+ act(() => {
+ for (const listener of listeners.get(type) ?? []) listener(data);
+ });
+}
+
+const listCalls = () => sendRequest.mock.calls.filter(([type]) => type === 'list_attention_items');
+
+beforeEach(() => {
+ listeners.clear();
+ items = [];
+ sendRequest.mockReset().mockImplementation(async (type: string) =>
+ type === 'list_attention_items' ? { success: true, items } : { success: true },
+ );
+});
+
+describe('the attention line on Home', () => {
+ it('shows nothing while nothing is open', async () => {
+ renderLine();
+ await waitFor(() => expect(listCalls()).toHaveLength(1));
+ expect(screen.queryByRole('alert')).toBeNull();
+ });
+
+ it('shows one line while work keeps retrying, and clears after the broadcast', async () => {
+ items = [
+ { id: 'stuck_retry:wf:a', kind: 'stuck_retry', count: 3 },
+ { id: 'stuck_retry:wf:b', kind: 'stuck_retry', count: 1 },
+ ];
+ renderLine();
+ expect(await screen.findByRole('alert')).toHaveTextContent('2 tasks keep retrying and may be stuck.');
+
+ items = [];
+ emit('attention_lifecycle', { specversion: '1.0', type: 'com.opencompany.attention.resolved', data: { kind: 'stuck_retry' } });
+
+ await waitFor(() => expect(screen.queryByRole('alert')).toBeNull());
+ expect(listCalls()).toHaveLength(2);
+ });
+});
diff --git a/client/src/features/home/data/attention.ts b/client/src/features/home/data/attention.ts
new file mode 100644
index 00000000..45a0c53e
--- /dev/null
+++ b/client/src/features/home/data/attention.ts
@@ -0,0 +1,55 @@
+/**
+ * Home's attention items (server: services/attention.py): work that keeps
+ * retrying past its limit, open until it gets through and dropped an hour
+ * after it was last seen. Read through `list_attention_items`, again on each
+ * `attention_lifecycle` broadcast (identity only), and once a minute, since
+ * nothing announces the drop-off.
+ */
+
+import { useQuery, useQueryClient } from '@tanstack/react-query';
+import { useEffect } from 'react';
+import { z } from 'zod';
+import { useWebSocketActions } from '@/contexts/WebSocketContext';
+
+export interface AttentionItem {
+ id: string;
+ /** `stuck_retry`: an activity still retrying past its limit. */
+ kind: string;
+ /** Sightings since it opened. */
+ count: number;
+}
+
+const itemSchema = z.object({ id: z.string(), kind: z.string(), count: z.number().int() });
+
+const listSchema = z.object({ items: z.array(z.unknown()).catch([]) }).transform((raw) =>
+ raw.items.flatMap((item): AttentionItem[] => {
+ const parsed = itemSchema.safeParse(item);
+ return parsed.success ? [parsed.data] : [];
+ }),
+);
+
+export const ATTENTION_QUERY_KEY = ['attentionItems'] as const;
+
+/** Items drop off an hour after they were last seen, which no event says. */
+const REFETCH_MS = 60_000;
+
+export function useAttentionItems() {
+ const { sendRequest, isReady, addEventListener } = useWebSocketActions();
+ const queryClient = useQueryClient();
+ useEffect(
+ () => addEventListener('attention_lifecycle', () => void queryClient.invalidateQueries({ queryKey: ATTENTION_QUERY_KEY })),
+ [addEventListener, queryClient],
+ );
+ return useQuery<AttentionItem[], Error>({
+ queryKey: ATTENTION_QUERY_KEY,
+ enabled: isReady,
+ refetchOnMount: 'always',
+ staleTime: 0,
+ refetchInterval: REFETCH_MS,
+ queryFn: async () => {
+ const reply = await sendRequest<{ success?: boolean; error?: string }>('list_attention_items', {});
+ if (reply?.success === false) throw new Error(reply.error || 'read_failed');
+ return listSchema.parse(reply);
+ },
+ });
+}
diff --git a/client/src/features/home/header/AttentionLine.tsx b/client/src/features/home/header/AttentionLine.tsx
new file mode 100644
index 00000000..cfd800f9
--- /dev/null
+++ b/client/src/features/home/header/AttentionLine.tsx
@@ -0,0 +1,31 @@
+/**
+ * One line under the Home header while work keeps retrying past its limit
+ * (data/attention.ts). It speaks of tasks, not employees: an item names a
+ * Temporal activity. Nothing shows while nothing is open.
+ */
+
+import { TriangleAlert } from 'lucide-react';
+import { Alert, AlertDescription } from '@/components/ui/alert';
+import { useAttentionItems, type AttentionItem } from '../data/attention';
+
+function describe(items: AttentionItem[]): string {
+ const stuck = items.filter((item) => item.kind === 'stuck_retry').length;
+ const other = items.length - stuck;
+ const parts: string[] = [];
+ if (stuck) parts.push(stuck === 1 ? 'A task keeps retrying and may be stuck.' : `${stuck} tasks keep retrying and may be stuck.`);
+ if (other) parts.push(other === 1 ? 'Something else needs a look.' : `${other} other things need a look.`);
+ return parts.join(' ');
+}
+
+export function AttentionLine() {
+ const { data: items } = useAttentionItems();
+ if (!items?.length) return null;
+ return (
+ <div className="relative z-10 shrink-0 px-3.5 pb-2">
+ <Alert variant="warning">
+ <TriangleAlert />
+ <AlertDescription>{describe(items)}</AlertDescription>
+ </Alert>
+ </div>
+ );
+}
diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.md
index e7f3c770..1fe3949a 100644
--- a/docs-internal/TEMPORAL_ARCHITECTURE.md
+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md
@@ -484,7 +484,7 @@ Env overrides: `TEMPORAL_<QUEUE>_CONCURRENCY` (int) and `TEMPORAL_<QUEUE>_RATE_L
- **Poller autoscaling** (Wave 18.3): `PollerBehaviorAutoscaling` — manager activity pollers 1-10 (initial 2), workflow pollers 1-20 (initial 2); pool workers 1-5 (initial 1). Invariant per the [worker-performance docs](https://docs.temporal.io/develop/worker-performance): pollers stay below executor slots.
- **Resource-based slot supplier** (Wave 18.4): `ai-heavy` + `browser` (unpredictable workloads) use `ResourceBasedSlotSupplier` targeting 80% host CPU + memory, `minimum_slots=1`, `maximum_slots` = the per-queue concurrency. Requires `temporalio>=1.25.0`; the floor is now 1.34.0 (RFC-0004 D5).
- **SDK tracing interceptor ownership**: `TracingInterceptor` is registered exactly once on the shared `Client.connect(...)`. The Temporal Python SDK automatically prepends compatible client interceptors to every `Worker` built from that client. Worker constructors must not repeat `TracingInterceptor`; doing so creates paired OpenTelemetry spans for one execution.
-- **Application observability interceptor** (Wave 17.3, `services/temporal/_interceptors.py`): workers explicitly register `ObservabilityWorkerInterceptor` for `activity_retry` WARN logs when `activity.info().attempt > 1` — the "worker died and Temporal re-dispatched" signal — and replay-guarded `workflow_start` logs. This is separate from SDK tracing and remains worker-owned.
+- **Application observability interceptor** (Wave 17.3, `services/temporal/_interceptors.py`): workers explicitly register `ObservabilityWorkerInterceptor` for `activity_retry` logs — the "worker died and Temporal re-dispatched" signal: WARN at the first retry, at the attempt limit (8 for the LLM step and compaction, 10 otherwise) and every 10th attempt after it, DEBUG otherwise, and always DEBUG for planned waits (a sub-agent permit, the approval gate, the Pi driver) — and replay-guarded `workflow_start` logs. An attempt past its limit touches a `stuck_retry` attention item, which Home shows as one line until a successful attempt resolves it or an hour passes without a sighting (`services/attention.py`). This is separate from SDK tracing and remains worker-owned.
- **Periodic activity heartbeat** (Wave 17.6, `plugin/base.py::as_activity`): 30s background beat during long bodies so a laptop-sleep crash is detected within one `heartbeat_timeout` (2 min) instead of `start_to_close` (24 h).
- **Cron catch-up bound** (Wave 17.1, `services/temporal/schedules.py`): `SchedulePolicy(catchup_window=24h)` + `SKIP` overlap — a laptop offline a week does not replay 168 hourly ticks on wake.
- **LLM-step retry** (Wave 17.2, since reverted): `LLM_STEP_RETRY` on `agent.execute_llm_step` is now **unlimited** (`maximum_attempts=0`, 5 s → 5 min exponential backoff) — see the docstring in `services/temporal/_retry_policies.py`. The one-shot policy Wave 17.2 shipped meant one transient provider error killed a months-long deployment and tripped the circuit breaker; terminal outcomes still fail fast because `_as_temporal_llm_error` marks non-retryable categories (invalid_request, authentication, ...) with `non_retryable=True`, and provider `retry_after` hints ride `ApplicationError.next_retry_delay`. Accepted trade-off: an ambiguous loss can re-bill a prompt. `agent.refresh_tools` keeps the 3-attempt default (idempotent canvas rebuild).
@@ -996,7 +996,7 @@ server/services/temporal/
├── schedules.py # Temporal Schedule creation for cronScheduler
├── search_attributes.py # EVENT_SEARCH_ATTRIBUTES (7 custom SAs, incl. ControlEventTypes)
├── _retry_policies.py # DEFAULT_ACTIVITY_RETRY / QUICK_ACTIVITY_RETRY / LLM_STEP_RETRY / PERMIT_WAIT_RETRY / ...
-├── _interceptors.py # ObservabilityWorkerInterceptor (activity_retry WARN, workflow_start logs)
+├── _interceptors.py # ObservabilityWorkerInterceptor (activity_retry log, stuck_retry attention items, workflow_start logs)
├── plugin_registry.py # Temporal plugin registry
├── worker.py # TemporalWorkerManager + TemporalWorkerPool
│ ├── start() # Start embedded worker
diff --git a/docs-internal/native_llm_sdk.md b/docs-internal/native_llm_sdk.md
index ef6623aa..44965dec 100644
--- a/docs-internal/native_llm_sdk.md
+++ b/docs-internal/native_llm_sdk.md
@@ -384,9 +384,12 @@ Failed streamed attempts are withdrawn immediately, while an owner's Stop
keeps the intentionally saved partial reply. A planned Temporal retry is a
benign `ApplicationError` (`category=BENIGN`): the SDK logs it at DEBUG,
without a traceback (`core/logging.ExpectedLLMRetryFilter`), and counts no
-failure metric; the retry itself shows at WARNING when the next attempt
-starts (the `activity_retry` interceptor). Terminal quota, credential, and
-unexpected programming failures keep their tracebacks.
+failure metric. The retry itself shows at WARNING at the first retry, at
+attempt 8 and every 10th attempt after it, at DEBUG otherwise (the
+`activity_retry` interceptor and the activity's "LLM retry scheduled" line),
+and a step still retrying past attempt 8 shows on Home as an attention item
+until it gets through. Terminal quota, credential, and unexpected
+programming failures keep their tracebacks.
The legacy `factory.py` (`create_provider` / `is_native_provider` /
`NATIVE_PROVIDERS`) was **removed** — `ChatUnifier` + `registry.py` is the
diff --git a/docs-internal/normal_mode.md b/docs-internal/normal_mode.md
index 01259653..8905c196 100644
--- a/docs-internal/normal_mode.md
+++ b/docs-internal/normal_mode.md
@@ -1083,6 +1083,7 @@ WebSocket requests (snake_case; failures come back as `success: false` with an
| `stop_chat_run` | `{run_id, expected_revision, idempotency_key}` for controlled runs; `{run_id}` for legacy | Controlled: generation Stop payload with `run_id` and `resumable: true`; the owning root is checked. Legacy: terminal chat `stopping` / `stopped`. Full acknowledgement and error contract: [chat_protocol.md](./chat_protocol.md#controlled-stop-acknowledgement) |
| `get_chat_models` | `{session_id: <workflow_id>}` | `{auto, models, efforts}`: the model picker's rows, each `{id, name, short, description, effort, effort_note, available, reason}`; the owner's choice is `chat_model` / `chat_effort` in `get_user_settings`, saved with `save_user_settings` (refused with `chat_choice_refused` and `detail`) |
| `workspace_steps_list` | `{workflow_id}` | `{workflow_id, steps}`, oldest first, each `{id, surface: "browser" \| "canvas" \| "mobile", text, node_id, at}` (`at` an ISO time). Only the workflow's owner reads them (`access_denied` otherwise, and always for the internal worker socket) |
+| `list_attention_items` | `{}` | `{items}`, newest first, each `{id, kind, severity, ref, count, first_seen, last_seen}`: the open attention items seen in the last hour (`services/attention.py`; `stuck_retry` is an activity still retrying past its limit). The owner's sockets only, never the internal worker socket |
| `list_approvals` | `{workflow_id?, status?, limit <= 100}` | `{approvals, counts, server_time}` |
| `decide_approval` | `{approval_id, decision, text?, subject?, note?, always?, decision_key}` | `{approval, will_send_on_resume}` |
| `delete_workflow` | `{workflow_id}` | `{workflow_id, contexts_archived, context_archives_pending}`; `DELETE /api/database/workflows/{id}` is the same handler. Error `workflow_shutdown_failed` (with `detail`) when stopping the employee failed: nothing was deleted |
@@ -1098,6 +1099,7 @@ every frame but `workflow_ops_apply` carries the whole envelope:
| `workflow_lifecycle` | gains `created` and `deleted` stages | So open editors refresh their workflow lists. `deleted` also makes every tab forget the workflow and its employee (`forgetWorkflow`, see [Deleting an employee](#deleting-an-employee)) |
| `chat.updated` | `com.opencompany.chat.updated` | Sent after every chat insert and clear (`services/chat_thread.py`), and when a chat run ends. Data `{workflow_id, session_id, role}`, identity only: `role` is null for a clear or a run's end, `workflow_id` null for session `"default"`. Home's thread and the editor's chat pane refetch |
| `workspace_step` | `com.opencompany.workspace.step` | A step was recorded (`services/workspace_steps.py`). Subject is the workflow id; data `{workflow_id, surface, step_id}`, identity only. The Workspace's timeline for that workflow refetches |
+| `attention_lifecycle` | `com.opencompany.attention.{opened,resolved}` | An attention item opened or was resolved (`services/attention.py`). Subject is the item's kind; data `{kind}`, identity only. Home's attention line under the header (`header/AttentionLine.tsx`) refetches; it also refetches once a minute, because an item drops off an hour after it was last seen without an event, and a standalone worker's broadcast stays in its process |
| `workflow_ops_apply` | `com.opencompany.workflow.ops.applied` | The frame is the event's flat data, `{workflow_id, caller_node_id, operations, persisted?}`, not the envelope. `persisted: true` marks a batch the server already saved (`apply_graph_additions`: Turn on Talk, the Agent Builder), whose ops carry the server's ids; editors adopt it without saving. See [Workflow Operations Protocol](./workflow_ops_protocol.md#persisted-batches) |
## Tests
diff --git a/server/core/database.py b/server/core/database.py
index 63d089ba..22e44f15 100644
--- a/server/core/database.py
+++ b/server/core/database.py
@@ -62,6 +62,7 @@ from models.employees import ( # noqa: F401 - registers SQLModel tables
WorkflowQueuedEvent,
)
from models.employee_conversion import EmployeeConversion # noqa: F401
+from models.attention import AttentionItem # noqa: F401 - registers SQLModel tables
from models.approvals import ( # noqa: F401 - registers SQLModel tables
ApprovalDecision,
ApprovalRequest,
diff --git a/server/main.py b/server/main.py
index 17d83e0e..5bced2ce 100644
--- a/server/main.py
+++ b/server/main.py
@@ -183,6 +183,10 @@ async def lifespan(app: FastAPI):
# Workspace timeline) and its cleanup on workflow delete.
import services.workspace_steps # noqa: F401
+ # services/attention.py self-registers list_attention_items (Home's
+ # line for work that keeps retrying).
+ import services.attention # noqa: F401
+
# Wave 13.8: services/pricing_handlers.py self-registers the 3
# pricing handlers (get_pricing_config / save_pricing_config /
# get_api_usage_summary). Flat module (sibling to services/pricing.py)
diff --git a/server/models/attention.py b/server/models/attention.py
new file mode 100644
index 00000000..9417910b
--- /dev/null
+++ b/server/models/attention.py
@@ -0,0 +1,37 @@
+"""Attention items: something the owner should look at (services/attention.py).
+
+One row per thing, keyed ``<kind>:<ref>``: ``stuck_retry`` and
+``<Temporal workflow id>:<activity id>`` for an activity that keeps
+retrying. Every sighting touches it, and Home shows the items seen in the
+last hour, so a row stranded by a Reset or a deletion drops off by itself.
+"""
+
+from __future__ import annotations
+
+from datetime import datetime, timezone
+
+from sqlalchemy import Column, DateTime
+from sqlmodel import Field, SQLModel
+
+
+def _utcnow() -> datetime:
+ return datetime.now(timezone.utc)
+
+
+class AttentionItem(SQLModel, table=True):
+ __tablename__ = "attention_items"
+
+ #: ``<kind>:<ref>``. Temporal ids run to about 500 characters.
+ id: str = Field(primary_key=True, max_length=1100)
+ kind: str = Field(max_length=64)
+ severity: str = Field(default="warning", max_length=16)
+ ref: str = Field(max_length=1024)
+ #: ``open`` or ``resolved``.
+ state: str = Field(default="open", max_length=16)
+ #: Sightings since it last opened.
+ count: int = Field(default=0)
+ first_seen: datetime = Field(default_factory=_utcnow, sa_column=Column(DateTime(timezone=True), nullable=False))
+ last_seen: datetime = Field(default_factory=_utcnow, sa_column=Column(DateTime(timezone=True), nullable=False, index=True))
+
+
+__all__ = ["AttentionItem"]
diff --git a/server/services/attention.py b/server/services/attention.py
new file mode 100644
index 00000000..de6d5c1d
--- /dev/null
+++ b/server/services/attention.py
@@ -0,0 +1,161 @@
+"""Attention items: something the owner should look at, one line on Home.
+
+An item is a kind and a reference: ``stuck_retry`` and ``<Temporal workflow
+id>:<activity id>`` for an activity that keeps retrying past its limit
+(services/temporal/_interceptors.py). Every sighting :func:`touch`-es it, a
+recovery :func:`resolve`-s it, and :func:`open_items` returns the open ones
+seen in the last hour, so an item stranded by a Reset or a deletion drops
+off by itself. Both writes are best effort and never raise: the work never
+fails over its alert.
+
+Home reads the items through the ``list_attention_items`` command. The
+identity-only ``attention_lifecycle`` broadcast tells it to look again when
+an item opens or is resolved; nothing announces the one-hour drop-off, so
+Home also polls.
+"""
+
+from __future__ import annotations
+
+from datetime import datetime, timedelta, timezone
+from typing import Any, Dict, List, Optional
+
+from fastapi import WebSocket
+from sqlmodel import select
+
+from core.logging import get_logger
+from models.attention import AttentionItem
+from services.events.envelope import WorkflowEvent
+from services.ws_handler_registry import ws_handler
+
+logger = get_logger(__name__)
+
+STUCK_RETRY = "stuck_retry"
+#: An item drops off this long after it was last seen.
+WINDOW = timedelta(hours=1)
+WIRE_KEY = "attention_lifecycle"
+SOURCE = "opencompany://services/attention"
+
+
+def _utcnow() -> datetime:
+ return datetime.now(timezone.utc)
+
+
+def _aware(moment: datetime) -> datetime:
+ # SQLite hands back naive datetimes; they were written in UTC.
+ return moment if moment.tzinfo else moment.replace(tzinfo=timezone.utc)
+
+
+def _database() -> Any:
+ from core.container import container
+
+ return container.database()
+
+
+def attention_lifecycle(stage: str, kind: str) -> WorkflowEvent:
+ """An item opened or was resolved: identity only, Home reads it back."""
+ return WorkflowEvent(source=SOURCE, type=f"com.opencompany.attention.{stage}", subject=kind, data={"kind": kind})
+
+
+async def _announce(stage: str, kind: str) -> None:
+ from services.status_broadcaster import get_status_broadcaster
+
+ event = attention_lifecycle(stage, kind)
+ await get_status_broadcaster().broadcast({"type": WIRE_KEY, "data": event.model_dump(mode="json", exclude_none=True)})
+
+
+async def touch(kind: str, ref: str, *, database: Any = None, now: Optional[datetime] = None) -> None:
+ """Record a sighting: open the item, or count it while it stays open.
+ A resolved item, or one not seen for an hour, opens again."""
+ now = now or _utcnow()
+ try:
+ database = database or _database()
+ async with database.reserved_session() as session:
+ item = await session.get(AttentionItem, f"{kind}:{ref}")
+ opened = item is None or item.state != "open" or _aware(item.last_seen) < now - WINDOW
+ if item is None:
+ item = AttentionItem(id=f"{kind}:{ref}", kind=kind, ref=ref)
+ if opened:
+ item.state, item.first_seen, item.count = "open", now, 0
+ item.count += 1
+ item.last_seen = now
+ session.add(item)
+ await session.commit()
+ if opened:
+ await _announce("opened", kind)
+ except Exception:
+ logger.warning("Attention item not recorded", kind=kind, exc_info=True)
+
+
+async def resolve(kind: str, ref: str, *, database: Any = None) -> None:
+ """Close an open item."""
+ try:
+ database = database or _database()
+ async with database.reserved_session() as session:
+ item = await session.get(AttentionItem, f"{kind}:{ref}")
+ if item is None or item.state != "open":
+ return
+ item.state = "resolved"
+ session.add(item)
+ await session.commit()
+ await _announce("resolved", kind)
+ except Exception:
+ logger.warning("Attention item not resolved", kind=kind, exc_info=True)
+
+
+async def open_items(database: Any, *, now: Optional[datetime] = None) -> List[Dict[str, Any]]:
+ """The open items seen in the last hour, newest first."""
+ since = (now or _utcnow()) - WINDOW
+ async with database.get_session() as session:
+ rows = (
+ (
+ await session.execute(
+ select(AttentionItem)
+ .where(AttentionItem.state == "open", AttentionItem.last_seen >= since)
+ .order_by(AttentionItem.last_seen.desc())
+ )
+ )
+ .scalars()
+ .all()
+ )
+ return [
+ {
+ "id": row.id,
+ "kind": row.kind,
+ "severity": row.severity,
+ "ref": row.ref,
+ "count": row.count,
+ "first_seen": _aware(row.first_seen).isoformat(),
+ "last_seen": _aware(row.last_seen).isoformat(),
+ }
+ for row in rows
+ ]
+
+
+@ws_handler()
+async def handle_list_attention_items(data: Dict[str, Any], websocket: WebSocket) -> Dict[str, Any]:
+ """``{success, items: [{id, kind, severity, ref, count, first_seen,
+ last_seen}]}``. The owner's sockets only: the internal worker socket
+ reaches no command outside its allowlist."""
+ return {"items": await open_items(_database())}
+
+
+WS_HANDLERS = {"list_attention_items": handle_list_attention_items}
+
+
+def _register() -> None:
+ from services.ws_handler_registry import register_ws_handlers
+
+ register_ws_handlers(WS_HANDLERS)
+
+
+_register()
+
+__all__ = [
+ "STUCK_RETRY",
+ "WINDOW",
+ "attention_lifecycle",
+ "handle_list_attention_items",
+ "open_items",
+ "resolve",
+ "touch",
+]
diff --git a/server/services/temporal/_interceptors.py b/server/services/temporal/_interceptors.py
index 1ef2f0cb..cc53db34 100644
--- a/server/services/temporal/_interceptors.py
+++ b/server/services/temporal/_interceptors.py
@@ -4,9 +4,11 @@ On a local-PC deployment the worker dies without SIGTERM (laptop sleep,
hard kill, battery). Temporal re-dispatches the interrupted activity on
the next poll — invisibly, unless something logs the attempt number.
These interceptors give the operator log a structured
-``activity_retry`` line whenever ``activity.info().attempt > 1`` so
-"first attempt" vs "re-dispatch after worker restart" is
-distinguishable without opening the Temporal Web UI.
+``activity_retry`` line (WARN at the first retry, at the attempt limit
+and every 10th attempt after it; DEBUG otherwise) so "first attempt" vs
+"re-dispatch after worker restart" is distinguishable without opening
+the Temporal Web UI, and an activity still retrying past its limit shows
+on Home as an attention item (services/attention.py).
Registration: both Worker constructions in ``worker.py`` pass
``interceptors=[ObservabilityWorkerInterceptor()]``.
@@ -38,6 +40,24 @@ from core.logging import get_logger, log_context
logger = get_logger(__name__)
+#: Retries past these attempts mean an activity is stuck: an LLM call
+#: (unlimited retries through an outage) sooner than the rest.
+RETRY_LIMITS = {"agent.execute_llm_step": 8, "agent.compact_context": 8}
+DEFAULT_RETRY_LIMIT = 10
+#: Activities that wait by retrying, or whose attempts end on a timer: their
+#: attempts are no sign of trouble (names or prefixes).
+PLANNED_WAITS = ("agent.acquire_subagent_permit", "node.approvalGate.", "coding.pi_drive_turn")
+
+
+def retry_limit(activity_type: str) -> int:
+ return RETRY_LIMITS.get(activity_type, DEFAULT_RETRY_LIMIT)
+
+
+def retry_warns(attempt: int, limit: int) -> bool:
+ """A retry is WARN-worthy at the first one, at the limit and every 10th
+ attempt after it; DEBUG otherwise."""
+ return attempt == 2 or attempt == limit or (attempt > limit and attempt % 10 == 0)
+
class ActivityObservabilityInterceptor(ActivityInboundInterceptor):
"""Log start / retry / end (with outcome) for every activity.
@@ -45,6 +65,10 @@ class ActivityObservabilityInterceptor(ActivityInboundInterceptor):
Also binds the Temporal identifiers to the log context for the whole
activity, under their own keys (``workflow_id`` elsewhere is the graph
id). The LLM gateway's attribution reads them (services.gateway).
+
+ An attempt past its retry limit touches a ``stuck_retry`` attention
+ item (one line on Home), and its success resolves it. Planned waits
+ never count. The store calls are best effort.
"""
async def execute_activity(self, input: ExecuteActivityInput) -> Any:
@@ -55,13 +79,20 @@ class ActivityObservabilityInterceptor(ActivityInboundInterceptor):
"workflow_id": info.workflow_id,
"attempt": info.attempt,
}
- if info.attempt > 1:
+ planned = info.activity_type.startswith(PLANNED_WAITS)
+ limit = retry_limit(info.activity_type)
+ stuck = not planned and info.attempt > limit
+ if info.attempt > 1 and not planned and retry_warns(info.attempt, limit):
# Re-dispatch: prior attempt died (worker crash / laptop
# sleep / timeout) or failed retryably. WARN so ops can
# count these without DEBUG noise.
- logger.warning("activity_retry", **extra)
+ logger.warning("activity_retry", limit=limit, **extra)
else:
- logger.debug("activity_start", **extra)
+ logger.debug("activity_retry" if info.attempt > 1 else "activity_start", **extra)
+ if stuck:
+ from services import attention
+
+ await attention.touch(attention.STUCK_RETRY, f"{info.workflow_id}:{info.activity_id}")
bound = {
"temporal_workflow_id": info.workflow_id,
"temporal_run_id": info.workflow_run_id,
@@ -80,6 +111,10 @@ class ActivityObservabilityInterceptor(ActivityInboundInterceptor):
)
raise
logger.debug("activity_end", outcome="success", **extra)
+ if stuck:
+ from services import attention
+
+ await attention.resolve(attention.STUCK_RETRY, f"{info.workflow_id}:{info.activity_id}")
return result
diff --git a/server/services/temporal/agent_activities.py b/server/services/temporal/agent_activities.py
index fc5daf8d..242f9323 100644
--- a/server/services/temporal/agent_activities.py
+++ b/server/services/temporal/agent_activities.py
@@ -575,7 +575,12 @@ async def _execute_native_llm_step(payload: Dict[str, Any]) -> Dict[str, Any]:
failure = _as_temporal_llm_error(error, attempt=attempt)
if error.retryable:
await _publish_llm_attempt_status(payload, error=error, delay=failure.next_retry_delay)
- logger.warning(
+ # Logged like the interceptor logs the retry it schedules: WARN at
+ # the first, at the limit and every 10th attempt after it.
+ from ._interceptors import retry_limit, retry_warns
+
+ log = logger.warning if retry_warns(attempt + 1, retry_limit("agent.execute_llm_step")) else logger.debug
+ log(
"LLM retry scheduled: provider=%s category=%s attempt=%s delay_seconds=%s",
error.provider, error.category.value, attempt,
failure.next_retry_delay.total_seconds() if failure.next_retry_delay is not None else None,
diff --git a/server/tests/services/test_attention.py b/server/tests/services/test_attention.py
new file mode 100644
index 00000000..84a9d2ab
--- /dev/null
+++ b/server/tests/services/test_attention.py
@@ -0,0 +1,78 @@
+"""The attention store behind Home's line for work that keeps retrying (1b.30)."""
+
+from datetime import datetime, timedelta, timezone
+from types import SimpleNamespace
+
+import pytest
+
+from services import attention
+from services.authz.ws_surface import INTERNAL_SOCKET_HANDLERS
+from services.ws_handler_registry import get_ws_handlers
+
+NOW = datetime(2026, 10, 27, 12, 0, tzinfo=timezone.utc)
+
+
+def announced(monkeypatch):
+ stages = []
+
+ async def announce(stage, kind):
+ stages.append((stage, kind))
+
+ monkeypatch.setattr(attention, "_announce", announce)
+ return stages
+
+
+async def test_touch_opens_once_and_counts_and_resolve_closes(real_database, announced):
+ later = NOW + timedelta(minutes=5)
+ await attention.touch("stuck_retry", "wf:llm-step-3", database=real_database, now=NOW)
+ await attention.touch("stuck_retry", "wf:llm-step-3", database=real_database, now=later)
+
+ [item] = await attention.open_items(real_database, now=later)
+ assert (item["kind"], item["ref"], item["count"], item["severity"]) == ("stuck_retry", "wf:llm-step-3", 2, "warning")
+ assert announced == [("opened", "stuck_retry")]
+
+ await attention.resolve("stuck_retry", "wf:llm-step-3", database=real_database)
+ assert await attention.open_items(real_database, now=later) == []
+ assert announced[-1] == ("resolved", "stuck_retry")
+
+ await attention.touch("stuck_retry", "wf:llm-step-3", database=real_database, now=later)
+ [item] = await attention.open_items(real_database, now=later)
+ assert item["count"] == 1
+ assert announced[-1] == ("opened", "stuck_retry")
+
+
+async def test_an_item_drops_off_an_hour_after_it_was_last_seen(real_database, announced):
+ await attention.touch("stuck_retry", "wf:a", database=real_database, now=NOW)
+ assert len(await attention.open_items(real_database, now=NOW + timedelta(minutes=59))) == 1
+ assert await attention.open_items(real_database, now=NOW + timedelta(minutes=61)) == []
+
+ # Seen again after the window, it opens again instead of counting on.
+ await attention.touch("stuck_retry", "wf:a", database=real_database, now=NOW + timedelta(minutes=90))
+ [item] = await attention.open_items(real_database, now=NOW + timedelta(minutes=90))
+ assert item["count"] == 1
+ assert announced == [("opened", "stuck_retry"), ("opened", "stuck_retry")]
+
+
+async def test_a_store_failure_is_logged_never_raised():
+ def unavailable():
+ raise RuntimeError("database down")
+
+ broken = SimpleNamespace(reserved_session=unavailable)
+ await attention.touch("stuck_retry", "wf:a", database=broken)
+ await attention.resolve("stuck_retry", "wf:a", database=broken)
+
+
+def test_the_command_is_registered_and_not_internal():
+ assert "list_attention_items" in get_ws_handlers()
+ assert "list_attention_items" not in INTERNAL_SOCKET_HANDLERS
+
+
+async def test_the_command_lists_the_open_items(real_database, announced, monkeypatch):
+ monkeypatch.setattr(attention, "_database", lambda: real_database)
+ await attention.touch("stuck_retry", "wf:a", database=real_database)
+
+ reply = await attention.handle_list_attention_items({}, None)
+
+ assert reply["success"] is True
+ assert [item["ref"] for item in reply["items"]] == ["wf:a"]
diff --git a/server/tests/temporal/test_activity_interceptor.py b/server/tests/temporal/test_activity_interceptor.py
new file mode 100644
index 00000000..eedf191b
--- /dev/null
+++ b/server/tests/temporal/test_activity_interceptor.py
@@ -0,0 +1,84 @@
+"""The retry alert (1b.30): a quieter retry log, and stuck work on Home."""
+
+from dataclasses import replace
+from unittest.mock import AsyncMock, MagicMock
+
+import pytest
+from temporalio.testing import ActivityEnvironment
+from temporalio.worker import ExecuteActivityInput
+
+from services import attention
+from services.temporal import _interceptors
+from services.temporal._interceptors import ActivityObservabilityInterceptor, retry_warns
+
+
[email protected](("limit", "warned"), [(10, [2, 10, 20, 30]), (8, [2, 8, 10, 20, 30])])
+def test_retries_warn_at_the_first_the_limit_and_every_tenth(limit, warned):
+ assert [attempt for attempt in range(2, 35) if retry_warns(attempt, limit)] == warned
+
+
+class _Next:
+ def __init__(self, failure=None):
+ self.failure = failure
+
+ async def execute_activity(self, input):
+ if self.failure is not None:
+ raise self.failure
+ return "done"
+
+
+def store(monkeypatch):
+ monkeypatch.setattr(attention, "touch", AsyncMock())
+ monkeypatch.setattr(attention, "resolve", AsyncMock())
+ monkeypatch.setattr(_interceptors, "logger", MagicMock())
+ return attention
+
+
+async def _attempt(activity_type, number, failure=None):
+ env = ActivityEnvironment()
+ env.info = replace(env.info, activity_type=activity_type, attempt=number, workflow_id="wf", activity_id="act-1")
+ interceptor = ActivityObservabilityInterceptor(_Next(failure))
+
+ async def call():
+ return await interceptor.execute_activity(ExecuteActivityInput(fn=None, args=[], executor=None, headers={}))
+
+ return await env.run(call)
+
+
+async def test_an_llm_step_past_its_limit_opens_the_item_and_its_success_resolves_it(store):
+ assert await _attempt("agent.execute_llm_step", 9) == "done"
+ store.touch.assert_awaited_once_with("stuck_retry", "wf:act-1")
+ store.resolve.assert_awaited_once_with("stuck_retry", "wf:act-1")
+
+
+async def test_a_failed_attempt_leaves_the_item_open(store):
+ with pytest.raises(RuntimeError):
+ await _attempt("agent.execute_llm_step", 9, RuntimeError("still down"))
+ store.touch.assert_awaited_once()
+ store.resolve.assert_not_awaited()
+
+
+ ("activity_type", "number"),
+ [
+ ("agent.execute_llm_step", 8), # at its limit, not past it
+ ("node.httpRequest.v1", 10),
+ ("agent.acquire_subagent_permit", 50),
+ ("node.approvalGate.v1", 50),
+ ("coding.pi_drive_turn", 50),
+ ],
+)
+async def test_no_item_up_to_the_limit_or_for_a_planned_wait(store, activity_type, number):
+ await _attempt(activity_type, number)
+ store.touch.assert_not_awaited()
+ store.resolve.assert_not_awaited()
+
+
+async def test_the_retry_log_is_quiet_between_its_warnings(store):
+ await _attempt("node.httpRequest.v1", 2)
+ _interceptors.logger.warning.assert_called_once()
+ _interceptors.logger.warning.reset_mock()
+ await _attempt("node.httpRequest.v1", 3)
+ await _attempt("agent.acquire_subagent_permit", 2)
+ _interceptors.logger.warning.assert_not_called()
diff --git a/server/tests/temporal/test_llm_retry_status.py b/server/tests/temporal/test_llm_retry_status.py
index c73ef7c6..5ccce6ff 100644
--- a/server/tests/temporal/test_llm_retry_status.py
+++ b/server/tests/temporal/test_llm_retry_status.py
@@ -1,6 +1,7 @@
"""A provider outage stays visible until the next attempt; quota fails fast."""
import asyncio
+import logging
from dataclasses import replace
from datetime import timedelta
from types import SimpleNamespace
@@ -214,6 +215,21 @@ async def test_activity_cancellation_withdraws_partial_and_cancels_provider(step
run.assert_awaited_once()
[email protected](("attempt", "warned"), [(1, True), (2, False), (7, True), (8, False), (9, True)])
+async def test_a_scheduled_retry_warns_by_the_quiet_rule(step, caplog, attempt, warned):
+ # 1b.30: judged on the attempt it schedules, like the interceptor's line
+ # (WARN at the first retry, at the limit 8 and every 10th attempt).
+ env, run, _broadcaster, payload = step
+ env.info = replace(env.info, attempt=attempt)
+ run.side_effect = LLMError("private", "gemini", category=LLMErrorCategory.SERVER, retryable=True)
+ with caplog.at_level(logging.DEBUG, logger="services.temporal.agent_activities"):
+ with pytest.raises(ApplicationError):
+ await env.run(execute_llm_step, payload)
+ [line] = [record for record in caplog.records if record.getMessage().startswith("LLM retry scheduled")]
+ assert (line.levelno == logging.WARNING) is warned
+
+
def test_a_result_too_big_for_temporal_is_not_retried():
# 1b.26: the next attempt would be as big, and would pay for it again.
from services.plugin.scaling import RetryPolicy