Skip to content

Self-improving AI Employees for everyone

feat(temporal): Wave 16.4 — per-queue routing default-on + pool worker crash fix

trohitg190c89611c2e
diff --git a/.env.template b/.env.template
index ffa363e7..64c0baff 100644
--- a/.env.template
+++ b/.env.template
@@ -56,13 +56,14 @@ TEMPORAL_NAMESPACE=default
TEMPORAL_TASK_QUEUE=machina-tasks
TEMPORAL_PER_TYPE_DISPATCH=true
TEMPORAL_AGENT_WORKFLOW_ENABLED=true
-# Wave 16: per-queue activity routing. When true, a TemporalWorkerPool
-# starts one activity-only Worker per plugin-declared task queue
-# (ai-heavy / browser / code-exec / rest-api / messaging / ...) and
-# per-type activities route to their specialised queue instead of the
-# single 100-slot pool on machina-tasks. Rollback channel: set false +
-# restart — activities route back to the manager worker unchanged.
-TEMPORAL_WORKER_POOL_ENABLED=false
+# Per-queue activity routing (Wave 16, default-on since 16.4). A
+# TemporalWorkerPool starts one activity-only Worker per plugin-declared
+# task queue (ai-heavy / browser / code-exec / rest-api / messaging /
+# ...) and per-type activities route to their specialised queue instead
+# of the single 100-slot pool on machina-tasks. Rollback channel: set
+# false + restart — activities route back to the manager worker
+# unchanged.
+TEMPORAL_WORKER_POOL_ENABLED=true
# Deployment topology hint: local (dev laptop; sleeps; halved per-queue
# concurrency) | cloud (machina deploy VM) | self_hosted. Drives worker
# identity strings in the Temporal Web UI and per-mode concurrency
diff --git a/CLAUDE.md b/CLAUDE.md
index 93e827a8..d7f755a6 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -13,7 +13,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire
| **[UI Migration Plan](./docs-internal/ui_migration_plan.md)** | antd → shadcn/ui migration plan + completion log. Waves 1–10 done. Full frontend is schema-driven (backend SSOT); remaining DIY widget registry (ex-Phase 6) is the one deferred item. |
| **[Temporal Cleanup & Resilience Plan (Waves 15-18)](./docs-internal/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md)** | RFC for the four forward waves: 15 dead-code retirement (Redis-Streams branch in event_waiter, APScheduler cron stack, orphaned `connection_status` factory — ~556 LOC), 16 per-queue task routing activation (`TemporalWorkerPool` + `TEMPORAL_WORKER_POOL_ENABLED` gate), 17 resilience hardening (cron `catchup_window`, one-shot LLM-step retry, SDK interceptors, `DEPLOYMENT_MODE`, periodic heartbeats), 18 worker performance tuning (per-queue rate limits, sticky cache, poller autoscaling, resource-based slot supplier). Ship order, rollback channels, verified line citations. |
| **[Node Allowlist](./docs-internal/node_allowlist.md)** | Single-config UI visibility — `server/config/node_allowlist.json` controls which nodes / credential categories / skill folders show in the UI. Five lists with two enforcement tiers (mode-gated allowlist + absolute blocklist). `useNodeAllowlist` hook exposes `isVisible` / `isBlocked` / `isAllowed` / `isCredentialCategoryDisabled` / `isSkillFolderDisabled`. Adding a new disabled domain = single JSON edit, no code change. |
-| **[Theme System](./docs-internal/theme_system.md)** | 10-way visual theme system — 5 utopian (light, dark, renaissance, greek, edo, steampunk, atomic) + 5 dystopian (cyber, wasteland, rot, plague, surveillance) — driven by `<html data-theme>` + per-theme CSS files in `client/src/themes/`. Token taxonomy (surface / fg / border / accent / typography / motion) in **hex + `color-mix()`**; `@theme inline` bridge maps `--color-X: var(--X)` (no `hsl()` wrapper); per-theme files own the colour hex, base.css owns the shared `--tint-*` alpha scale; action role tokens (incl. `-ink` readable text), **per-theme `--code-*` syntax tokens** (tier 6 — the code editor, console/output JSON viewers, and chat code blocks paint in each theme's own palette: keyword→trigger, string→success, number→agent, function→model, on an adaptive dark/light code surface; replaced the global dracula `--prism-*` block + dead `getPrismTokenCSS`; the `OutputPanel` JSON viewer reads the same vars), per-theme `--pulse-keyframe` animation system + `.machina-*` helpers in `animations.css`, decorative-layer wrappers (`.app-frame` / `.canvas-host` / `.modal-frame`), per-component decorative ornaments (panel textures + canvas decorations + node pseudo-element overlays + theme-specific keyframes), **canvas-node visual contract via `--node-color` CSS custom property** (no inline `background` / `border` on node components — base.css + per-theme CSS owns visuals; `NodeStyle` helper type at [types/NodeTypes.ts](./client/src/types/NodeTypes.ts) makes the inline custom-prop typecheck-clean), **`--node-pulse-color` separate from `--node-color`** so executing-node glow uses each theme's highest-contrast accent regardless of plugin accent (Cyber neon cyan, Surveillance REC red, Renaissance ultramarine, etc.), **`data-page-hidden` animation pause** (toggled by Dashboard's `visibilitychange` listener; base.css declares `html[data-page-hidden] *, *::before, *::after { animation-play-state: paused !important }` to prevent compositor stall on tab return), **per-theme icon glyph system** (290 SVGs across 29 keys × 10 themes via [themedGlyphs.ts](./client/src/assets/icons/themedGlyphs.ts) + theme-aware [NodeIcon.tsx](./client/src/assets/icons/NodeIcon.tsx)), **per-theme canvas-grid + custom cursors** via `--canvas-grid` / `--cursor-default` slots, **decorative HTML primitives** (`<SvgFilterDefs>` mounting `#ink-blot` / `#noise` / `#crt` filter IDs at app root, `<DropCap>` wrapper for Renaissance ornament rule), **parameter panel migrated to Tailwind tokens** (no `useAppTheme()` reads; section headers carry the display-typography triplet; raw `<Button>` swapped to `<ActionButton intent>`), **per-theme scrollbar webkit rules** in all 10 themes, 9-event WebAudio sound system (10 packs via `--sound-pack` token + `useSound()` hook + global hover delegate + sonner toast monkey-patch + `withSound()` HOC + `Sounds.unlock()` gesture-unlock for AudioContext autoplay-policy compliance), `@media (prefers-reduced-motion: reduce)` accessibility, 30 ms throttle on `type` / `hover`, migration recipe, anti-patterns. Read this before adding a new theme, migrating a component to the new contract, or adding a canvas-node component. |
+| **[Theme System](./docs-internal/theme_system.md)** | 12-way visual theme system — 2 base (light, dark) + 5 utopian (renaissance, greek, edo, steampunk, atomic) + 5 dystopian (cyber, wasteland, rot, plague, surveillance) — driven by `<html data-theme>` + per-theme CSS files in `client/src/themes/`. Token taxonomy (surface / fg / border / accent / typography / motion) in **hex + `color-mix()`**; `@theme inline` bridge maps `--color-X: var(--X)` (no `hsl()` wrapper); per-theme files own the colour hex, base.css owns the shared `--tint-*` alpha scale; action role tokens (incl. `-ink` readable text), **per-theme `--code-*` syntax tokens** (tier 6 — the code editor, console/output JSON viewers, and chat code blocks paint in each theme's own palette: keyword→trigger, string→success, number→agent, function→model, on an adaptive dark/light code surface; replaced the global dracula `--prism-*` block + dead `getPrismTokenCSS`; the `OutputPanel` JSON viewer reads the same vars), per-theme `--pulse-keyframe` animation system + `.machina-*` helpers in `animations.css`, decorative-layer wrappers (`.app-frame` / `.canvas-host` / `.modal-frame`), per-component decorative ornaments (panel textures + canvas decorations + node pseudo-element overlays + theme-specific keyframes), **canvas-node visual contract via `--node-color` CSS custom property** (no inline `background` / `border` on node components — base.css + per-theme CSS owns visuals; `NodeStyle` helper type at [types/NodeTypes.ts](./client/src/types/NodeTypes.ts) makes the inline custom-prop typecheck-clean), **`--node-pulse-color` separate from `--node-color`** so executing-node glow uses each theme's highest-contrast accent regardless of plugin accent (Cyber neon cyan, Surveillance REC red, Renaissance ultramarine, etc.), **`data-page-hidden` animation pause** (toggled by Dashboard's `visibilitychange` listener; base.css declares `html[data-page-hidden] *, *::before, *::after { animation-play-state: paused !important }` to prevent compositor stall on tab return), **per-theme icon glyph system** (290 SVGs across 29 keys × 10 themes via [themedGlyphs.ts](./client/src/assets/icons/themedGlyphs.ts) + theme-aware [NodeIcon.tsx](./client/src/assets/icons/NodeIcon.tsx)), **per-theme canvas-grid + custom cursors** via `--canvas-grid` / `--cursor-default` slots, **decorative HTML primitives** (`<SvgFilterDefs>` mounting `#ink-blot` / `#noise` / `#crt` filter IDs at app root, `<DropCap>` wrapper for Renaissance ornament rule), **parameter panel migrated to Tailwind tokens** (no `useAppTheme()` reads; section headers carry the display-typography triplet; raw `<Button>` swapped to `<ActionButton intent>`), **per-theme scrollbar webkit rules** in all 12 themes, 9-event WebAudio sound system (10 packs via `--sound-pack` token + `useSound()` hook + global hover delegate + sonner toast monkey-patch + `withSound()` HOC + `Sounds.unlock()` gesture-unlock for AudioContext autoplay-policy compliance), `@media (prefers-reduced-motion: reduce)` accessibility, 30 ms throttle on `type` / `hover`, migration recipe, anti-patterns. Read this before adding a new theme, migrating a component to the new contract, or adding a canvas-node component. |
| **[Design System Bundle](./docs-internal/design-system/IMPLEMENTATION.md)** | Canonical, vendored design-system reference (extracted from this repo; see its `IMPLEMENTATION.md` for the contract). Tokens are **hex + `color-mix()`** (`tokens/{colors,typography,spacing,motion,fonts,base}.css`) — the format the live codebase standardizes on. Includes 34 reference components (`components/`, inline-style reference only — recreate in Tailwind/shadcn idioms), full-app UI kit (`ui_kits/machinaos/index.html`), and 12-theme docs (`guidelines/THEMES.md`, `reference/themes/`). Copy token values verbatim; do not re-derive by eye. Pairs with [Theme System](./docs-internal/theme_system.md). |
| **[Schema Source of Truth RFC](./docs-internal/schema_source_of_truth_rfc.md)** | Backend is SSOT for node schemas, visual metadata, handlers, palette metadata, icons. Plugin pattern: one `BaseNode` subclass in `server/nodes/<group>/<plugin>/__init__.py`. Wire format: `asset:<key>` / `<lib>:<brand>` / URL / emoji. Endpoint: `/api/schemas/nodes/{type}/spec.json`. Live invariant total via `pytest --collect-only`. |
| **[Plugin System (Wave 11)](./docs-internal/plugin_system.md)** | Class-based plugin-first architecture. `BaseNode` / `ActionNode` / `TriggerNode` / `ToolNode` + `@Operation` decorator. Pydantic `Params`/`Output`. Declarative `Routing` DSL + `Connection` facade (Nango pattern). 18 `Credential` subclasses live in each node folder's `_credentials.py` (or inline for single-use). `TaskQueue` constants route to Temporal worker pools. Plugins live across 9 queues (live count via `glob server/nodes/**/__init__.py`); handler bodies fully inlined (`services/handlers/` shrank 12.8K → 1.1K LOC across 16 → 4 files; only cross-cutting orchestration remains: `tools.py` AI-tool dispatch + agent delegation, `google_auth.py`, `triggers.py`). **Wave 11.H added "self-contained plugin folders"** — up to six generic registries (`ws_handler_registry`, `register_router`, `event_waiter.{register_filter_builder,register_trigger_precheck}`, `status_broadcaster.register_service_refresh`, `node_output_schemas.register_output_schema`) so rich plugins like telegram own their entire surface area without core-services edits. |
@@ -594,7 +594,7 @@ AI model nodes route through `SquareNode` via `Dashboard.tsx`'s `COMPONENT_BY_KI
### Theme System
-10-way visual theme system (5 utopian: light, dark, renaissance, greek, edo, steampunk, atomic + 5 dystopian: cyber, wasteland, rot, plague, surveillance) driven by `<html data-theme>` (set by [ThemeContext.tsx](./client/src/contexts/ThemeContext.tsx), which also toggles `.dark` for DARK_FAMILY themes) + per-theme CSS in `client/src/themes/`. Token VALUES are hex + `color-mix()` (never HSL): per-theme files own shadcn/dracula/node/action hex, `base.css` owns the shared `--tint-*` scale, `index.css` is plumbing (`@theme inline` maps `--color-X: var(--X)`). Six token tiers, the per-theme `--pulse-keyframe` animation system, decorative-layer wrappers, the 10-pack WebAudio sound system, and the canvas-wide edge/node status rules (`canvasAnimations.ts`) are all documented in **[Theme System](./docs-internal/theme_system.md)** — read it before adding a theme or a canvas-node component. The strict frontend theme RULES remain normative under "Frontend Design + Theme System (strict)" above.
+12-way visual theme system (2 base: light, dark + 5 utopian: renaissance, greek, edo, steampunk, atomic + 5 dystopian: cyber, wasteland, rot, plague, surveillance) driven by `<html data-theme>` (set by [ThemeContext.tsx](./client/src/contexts/ThemeContext.tsx), which also toggles `.dark` for DARK_FAMILY themes) + per-theme CSS in `client/src/themes/`. Token VALUES are hex + `color-mix()` (never HSL): per-theme files own shadcn/dracula/node/action hex, `base.css` owns the shared `--tint-*` scale, `index.css` is plumbing (`@theme inline` maps `--color-X: var(--X)`). Six token tiers, the per-theme `--pulse-keyframe` animation system, decorative-layer wrappers, the 10-pack WebAudio sound system, and the canvas-wide edge/node status rules (`canvasAnimations.ts`) are all documented in **[Theme System](./docs-internal/theme_system.md)** — read it before adding a theme or a canvas-node component. The strict frontend theme RULES remain normative under "Frontend Design + Theme System (strict)" above.
### WebSocket-First Architecture
The project uses WebSocket as the primary communication method between frontend and backend, replacing most REST API calls:
@@ -681,7 +681,7 @@ Node groups (palette categories): agent, model, skill, tool, trigger, workflow,
### Temporal Distributed Execution
-Workflows execute via Temporal for durability and horizontal scaling, gated by `TEMPORAL_ENABLED`. Three dispatch paths exist — legacy single `execute_node_activity` (default), per-type `node.{type}.v{version}` activities (F4.A, `TEMPORAL_PER_TYPE_DISPATCH=true`), and Agent-as-child-workflow (F4.B, `TEMPORAL_AGENT_WORKFLOW_ENABLED=true`) — with `rlm_agent`/`claude_code_agent` always bypassing AgentWorkflow. Execution routing falls back Temporal → Redis-parallel → sequential; the Temporal dev server + embedded worker are managed in-process under `server/services/temporal/` (pooch-downloaded CLI, single `temporal server start-dev` process, gRPC 7233 + Web UI 8080, SQLite at `~/.machina/temporal.db`). Full architecture, dispatch matrix, per-node + agent-loop lifecycle, the 7 F4.B `agent.*.v1` activities, heartbeat semantics, delegation input contract, resumption toggle (`TEMPORAL_TERMINATE_RUNNING_ON_STARTUP`), and all `.env` tunables live in [docs-internal/TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md).
+Workflows execute via Temporal for durability and horizontal scaling, gated by `TEMPORAL_ENABLED`. Three dispatch paths exist — legacy single `execute_node_activity` (fallback), per-type `node.{type}.v{version}` activities (F4.A, `TEMPORAL_PER_TYPE_DISPATCH=true`), and Agent-as-child-workflow (F4.B, `TEMPORAL_AGENT_WORKFLOW_ENABLED=true`) — with `rlm_agent`/`claude_code_agent` always bypassing AgentWorkflow. **Per-queue activity routing is production-default (Wave 16)**: a `TemporalWorkerPool` runs one activity-only worker per plugin-declared `cls.task_queue` (ai-heavy / browser / code-exec / rest-api / messaging / ...) with per-queue concurrency + rate limits + resource-based tuning for ai-heavy/browser; rollback via `TEMPORAL_WORKER_POOL_ENABLED=false`. Resilience knobs (Waves 17-18): `DEPLOYMENT_MODE` (local halves concurrency; sizes the sticky cache), one-shot retry on `agent.execute_llm_step.v1` (LLM calls are not idempotent), observability interceptors (`activity_retry` WARN on re-dispatch), 30s periodic heartbeat during long activity bodies, cron `catchup_window=24h`, poller autoscaling. Execution routing falls back Temporal → Redis-parallel → sequential; the Temporal dev server + embedded worker are managed in-process under `server/services/temporal/` (pooch-downloaded CLI, single `temporal server start-dev` process, gRPC 7233 + Web UI 8080, SQLite at `~/.machina/temporal.db`). Full architecture, dispatch matrix, per-node + agent-loop lifecycle, the 7 F4.B `agent.*.v1` activities, heartbeat semantics, delegation input contract, resumption toggle (`TEMPORAL_TERMINATE_RUNNING_ON_STARTUP`), the worker tuning recipe, and all `.env` tunables live in [docs-internal/TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md).
## Development Commands
diff --git a/README.md b/README.md
index c7d86cd4..590eec5d 100644
--- a/README.md
+++ b/README.md
@@ -38,7 +38,7 @@ The `dev` task starts the Python backend, Vite client, WhatsApp service, and Tem
</details>
-## Quick Hello World Setup for Machinaos ↓.
+## Quick Hello World Setup for MachinaOS ↓
https://github.com/user-attachments/assets/a5a5583f-bb5f-4d27-a387-8522c556e89e
@@ -120,7 +120,7 @@ RAG pipeline out of the box: parse PDFs and HTML, chunk into searchable pieces,
Local providers (Ollama, LM Studio) are first-class — context length, vision support, and tool-use capability are detected automatically from your running server. No paid API needed.
-### 17 specialized agent types
+### 16 specialized agent types
Pick the right agent for the job:
@@ -150,7 +150,7 @@ Skills are short markdown files that teach an agent how to do something well —
### Memory that scales with your context window
-Agents track token usage and automatically compact long conversations when you hit half your model's context limit. Compaction summarizes in five sections — Task Overview, Current State, Important Discoveries, Next Steps, Context to Preserve — so the agent picks up exactly where it left off. For Anthropic and OpenAI, native API compaction is used; everywhere else, the agent summarizes itself.
+Agents track token usage and automatically compact long conversations as you approach your model's context limit (80% by default, configurable). Compaction summarizes in five sections — Task Overview, Current State, Important Discoveries, Next Steps, Context to Preserve — so the agent picks up exactly where it left off. For Anthropic and OpenAI, native API compaction is used; everywhere else, the agent summarizes itself.
### Cost tracking, built in
@@ -158,7 +158,7 @@ Every LLM call and API request is tracked with USD cost. See per-provider spend
## The Canvas
-- **10 visual themes** — light, dark, Renaissance, Greek, Edo, Steampunk, Atomic, Cyber, Wasteland, Rot, Plague, Surveillance — each with its own icon set, sound pack, and decorative ornaments. Pick the vibe that matches your workflow.
+- **12 visual themes** — light, dark, Renaissance, Greek, Edo, Steampunk, Atomic, Cyber, Wasteland, Rot, Plague, Surveillance — each with its own icon set, sound pack, and decorative ornaments. Pick the vibe that matches your workflow.
- **Drag-to-map outputs** from one node's output directly onto another's input fields.
- **Live execution animations** — nodes glow while running, show iteration count for AI agents, and surface errors inline.
- **Multi-tab Console** — chat with trigger nodes, watch console logs, and view terminal output side by side.
diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.md
index 59640901..b0444d07 100644
--- a/docs-internal/TEMPORAL_ARCHITECTURE.md
+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md
@@ -6,8 +6,8 @@ Each workflow node executes as a **Temporal activity** with its own isolated con
| Dispatch | Trigger | Use case |
|---|---|---|
-| **Legacy single activity** (`execute_node_activity`) | Default | Every node routed through one dispatcher activity. WebSocket round-trip back to the FastAPI server. Stable since Wave 11. |
-| **Per-type activity** (`node.{type}.v{version}`) | `TEMPORAL_PER_TYPE_DISPATCH=true` | Each plugin gets its own `@activity.defn`. Per-plugin retry / timeout / heartbeat configs apply. Worker pool can later specialise per `cls.task_queue` (browser / code-exec / ai-heavy / ...). Shipped in F4.A (commit `8261b05`). |
+| **Legacy single activity** (`execute_node_activity`) | `TEMPORAL_PER_TYPE_DISPATCH=false` | Every node routed through one dispatcher activity. WebSocket round-trip back to the FastAPI server. Stable since Wave 11; kept as the fallback path. |
+| **Per-type activity** (`node.{type}.v{version}`) | `TEMPORAL_PER_TYPE_DISPATCH=true` (production default) | Each plugin gets its own `@activity.defn`. Per-plugin retry / timeout / heartbeat configs apply. With `TEMPORAL_WORKER_POOL_ENABLED=true` (default since Wave 16.4) the activity also carries `task_queue=cls.task_queue`, landing it on its specialised `TemporalWorkerPool` worker (browser / code-exec / ai-heavy / ...). Shipped in F4.A (commit `8261b05`); queue routing activated in Wave 16. |
| **Agent-as-child-workflow** (`AgentWorkflow`) | `TEMPORAL_AGENT_WORKFLOW_ENABLED=true` | AI Agents (aiAgent, chatAgent, 11 specialized agents, 2 team leads) run as Temporal child workflows. Each LLM turn = activity; each tool call = per-type activity. Mirrors Temporal's AI Cookbook canonical pattern. F4.B infrastructure shipped (commit `a4d009e`); per-agent migrations follow. |
`rlm_agent`, `claude_code_agent` are intentionally excluded from AgentWorkflow — their externalised loops (RLM REPL / Claude CLI `--resume`) require single-process state continuity.
@@ -44,28 +44,31 @@ This invokes `run_standalone_worker()` from `services/temporal/worker.py`.
```
TEMPORAL SERVER (port 7233)
|
- Task Queue: machina-tasks
+ Task Queue: machina-tasks (workflows + framework activities)
+---------------------------------------------------------------+
| Workflow: MachinaWorkflow (orchestrator only) |
| - Parses graph structure from React Flow |
| - Filters config nodes (tools, memory, services) |
| - Resolves activity per node (legacy / per-type / agent-wf) |
+ | - Per-type activities carry task_queue=cls.task_queue |
+ | (Wave 16; TEMPORAL_WORKER_POOL_ENABLED, default on) |
| - Schedules activities (FIRST_COMPLETED pattern) |
| - Collects results and routes outputs to dependent nodes |
+---------------------------------------------------------------+
|
- Activity / child-workflow scheduling
- +--------+ +--------+ +--------+ +----------------+
- | Node A | | Node B | | Node C | | AgentWorkflow | (F4.B child wf)
- +--------+ +--------+ +--------+ +----------------+
- | | | |
- v v v v
- +----------+ +----------+ +----------+ +-----------------------+
- | Worker 1 | | Worker 2 | | Worker 3 | | LLM step + tool steps |
- | aiAgent | | timer | | console | | (per-type activities) |
- +----------+ +----------+ +----------+ +-----------------------+
- | | | |
- +----------+-----------+--------------+
+ Activity / child-workflow scheduling (routed by queue)
+ |
+ machina-tasks specialised queues (TemporalWorkerPool)
+ | ai-heavy code-exec browser rest-api ...
+ +-----------+ +--------+ +--------+ +-------+ +-------+
+ | Manager | | Pool | | Pool | | Pool | | Pool |
+ | worker | | worker | | worker | | worker| | worker|
+ | (wf tasks | | aiAgent| | python | |browser| | gmail |
+ | + legacy | | chatA | | js/ts | | | | brave |
+ | dispatch | | deepA | | | | | | ... |
+ | + AgentWf)| +--------+ +--------+ +-------+ +-------+
+ +-----------+ | | | |
+ +----------------+-----------+----------+----------+
|
v
In-process call (F4.A) OR
@@ -272,9 +275,9 @@ Benefits:
Add more workers = handle more concurrent nodes
```
-### Specialized Worker Pools (Wave 16 — gated by `TEMPORAL_WORKER_POOL_ENABLED`)
+### Specialized Worker Pools (Wave 16 — `TEMPORAL_WORKER_POOL_ENABLED`, default on since 16.4)
-`TemporalWorkerPool` is wired into the `main.py` lifespan (starts right after `TemporalWorkerManager`, stops before it). When `TEMPORAL_WORKER_POOL_ENABLED=true`, one activity-only Worker per plugin-declared queue polls its specialised queue, and `MachinaWorkflow._resolve_activity` returns `(activity_name, cls.task_queue)` so per-type activities land there. Flag off (default until the stability window) = queue `None` = everything routes to the single manager worker on `machina-tasks`. **The flag is the rollback channel.**
+`TemporalWorkerPool` is wired into the `main.py` lifespan (starts right after `TemporalWorkerManager`, stops before it). One activity-only Worker per plugin-declared queue polls its specialised queue, and `MachinaWorkflow._resolve_activity` returns `(activity_name, cls.task_queue)` so per-type activities land there. Setting `TEMPORAL_WORKER_POOL_ENABLED=false` stops the pool and routes every activity back to the single manager worker on `machina-tasks` — **the flag is the rollback channel** (locked default-on by `test_task_queue_coverage.py::TestWorkerPoolDefaultOn`).
```
Queue: rest-api Queue: ai-heavy Queue: code-exec
diff --git a/docs-internal/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md b/docs-internal/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md
index ab38aa62..4d412883 100644
--- a/docs-internal/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md
+++ b/docs-internal/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md
@@ -16,7 +16,7 @@
| 16.1 — task-queue coverage invariants | ✅ shipped `3262fb0` |
| 16.2 — pool wired into lifespan (gated off) | ✅ shipped `18df875` |
| 16.3 — `_resolve_activity` returns `cls.task_queue` when pool on | ✅ shipped `d4b55c4` |
-| 16.4 — default flip | ⏳ parked for one-release stability window (`TEMPORAL_WORKER_POOL_ENABLED` stays default-false; flip + `test_temporal_worker_pool_enabled_defaults_true` invariant afterwards) |
+| 16.4 — default flip | ✅ shipped (this commit) — `temporal_worker_pool_enabled` default `True`; opt-out via env var only, locked by `test_task_queue_coverage.py::TestWorkerPoolDefaultOn` |
| 17.1 — cron `catchup_window=24h` | ✅ shipped `5bf9000` (note: `catchup_window` is a `SchedulePolicy` field in the Python SDK, not `ScheduleSpec` as §6 originally said) |
| 17.2 — one-shot LLM-step retry | ✅ shipped `03c2bf6` (`LLM_STEP_RETRY` constant; refresh_tools keeps 3 attempts, documented at callsite) |
| 17.3 — observability interceptors | ✅ shipped `9ecd743` (replay guard via `workflow.unsafe.is_replaying()`) |
diff --git a/docs-internal/frontend_architecture.md b/docs-internal/frontend_architecture.md
index 35247b87..de5253ef 100644
--- a/docs-internal/frontend_architecture.md
+++ b/docs-internal/frontend_architecture.md
@@ -189,7 +189,7 @@ client/src/
## Tokens + theming
-The frontend ships **10 themes** organised utopian / dystopian: `light` · `dark` · `renaissance` · `greek` · `edo` · `steampunk` · `atomic` · `cyber` · `wasteland` · `rot` · `plague` · `surveillance`. Active theme is `<html data-theme="...">` set by `<ThemeProvider>` (see [contexts/ThemeContext.tsx](../client/src/contexts/ThemeContext.tsx)). Per-theme blocks live in [client/src/themes/](../client/src/themes/) and own all colour VALUES as **hex + `color-mix()`** (never HSL triplets); [src/index.css](../client/src/index.css) is **plumbing-only** (the `@theme inline` `var()` bridge — no literal colours). Full token contract, sound + decorative-layer wiring, and the 12-theme migration playbook in **[theme_system.md](./theme_system.md)** — read that before adding a theme or migrating a component.
+The frontend ships **12 themes** organised base / utopian / dystopian: `light` · `dark` · `renaissance` · `greek` · `edo` · `steampunk` · `atomic` · `cyber` · `wasteland` · `rot` · `plague` · `surveillance`. Active theme is `<html data-theme="...">` set by `<ThemeProvider>` (see [contexts/ThemeContext.tsx](../client/src/contexts/ThemeContext.tsx)). Per-theme blocks live in [client/src/themes/](../client/src/themes/) and own all colour VALUES as **hex + `color-mix()`** (never HSL triplets); [src/index.css](../client/src/index.css) is **plumbing-only** (the `@theme inline` `var()` bridge — no literal colours). Full token contract, sound + decorative-layer wiring, and the 12-theme migration playbook in **[theme_system.md](./theme_system.md)** — read that before adding a theme or migrating a component.
The bridge ([src/index.css](../client/src/index.css)) maps each Tailwind colour token to a raw `var(--X)`.
@@ -508,6 +508,6 @@ The `MACHINAOS_INSTALLING=true` env var suppresses the recursive project postins
This architecture is the post-migration state. Pre-migration was antd + `styled-components` + a custom theme.ts-driven palette. See [ui_migration_plan.md](./ui_migration_plan.md) for the phase-by-phase transition and the 17 commits that executed it.
-**`useAppTheme()` powers the canvas + maps surface across all 10 themes.** The hook returns a `theme` object with the legacy `Colors` shape (`theme.colors.X`, `theme.isDarkMode`) so existing call sites don't change. Under non-light/dark themes it merges a per-theme overlay (primary, focus, action palette, edge stroke / selection / executing / completed / error) on top of the chosen base pack (`lightColors` for utopian-bright themes, `darkColors` for dystopian / dark themes). Adding a new theme overlay is a single entry in the `THEME_OVERRIDES` map in [hooks/useAppTheme.ts](../client/src/hooks/useAppTheme.ts).
+**`useAppTheme()` powers the canvas + maps surface across all 12 themes.** The hook returns a `theme` object with the legacy `Colors` shape (`theme.colors.X`, `theme.isDarkMode`) so existing call sites don't change. Under non-light/dark themes it merges a per-theme overlay (primary, focus, action palette, edge stroke / selection / executing / completed / error) on top of the chosen base pack (`lightColors` for utopian-bright themes, `darkColors` for dystopian / dark themes). Adding a new theme overlay is a single entry in the `THEME_OVERRIDES` map in [hooks/useAppTheme.ts](../client/src/hooks/useAppTheme.ts).
Read sites: every canvas node component (`AIAgentNode`, `SquareNode`, `TriggerNode`, `StartNode`, `ToolkitNode`, `TeamMonitorNode`, `GenericNode`), `EdgeConditionEditor`, and the Maps surface (`MapSelector`, `GoogleMapsPicker`, `MapsPreviewPanel`) — they interpolate per-definition `nodeColor` and JS-side hex values that Tailwind classes can't express. Every other surface uses Tailwind + the token tiers above and retints automatically through the per-theme `[data-theme="..."]` block.
diff --git a/docs-internal/theme_system.md b/docs-internal/theme_system.md
index b0889a18..8abdf8e5 100644
--- a/docs-internal/theme_system.md
+++ b/docs-internal/theme_system.md
@@ -1,4 +1,4 @@
-# Theme System — design-handoff token contract + 10 themes
+# Theme System — design-handoff token contract + 12 themes
The MachinaOs frontend supports **ten visual themes**, organised as a utopian / dystopian taxonomy from the design handoff:
@@ -144,7 +144,7 @@ Concrete swaps when migrating an existing component:
### Display-typography helper
-Heading surfaces under all 10 themes use the same triplet. Keep them grouped in the same className for clarity:
+Heading surfaces under all 12 themes use the same triplet. Keep them grouped in the same className for clarity:
```tsx
<span className="font-display tracking-[var(--type-tracking-display)] text-fg-default [text-transform:var(--type-uppercase)]">
@@ -227,16 +227,16 @@ Result by theme (display font / tracking / case):
### Done
- **`--node-pulse-color` contract** — executing-node glow color is independent of `--node-color` (the plugin accent). Each theme overrides `--node-pulse-color` to its highest-contrast accent (Cyber neon cyan, Surveillance REC red, Renaissance ultramarine `--lapis-bright`, Atomic turquoise, etc.) so the glow stays visible against any canvas background. Defined in [base.css](../client/src/themes/base.css) at `:root`, overridden in each per-theme file. The executing keyframes (`node-pulse` + per-theme `cyber-pulse-exec` / `surv-pulse-exec` / `ren-pulse-exec`) and the per-theme `--pulse-keyframe` / `--pulse-duration` / `--pulse-timing` tokens live in [animations.css](../client/src/themes/animations.css); base.css's `.sq-node[data-executing] .sq-node-box` / `.react-flow__node.executing .{sq-node-box,node}` rule reads `var(--pulse-keyframe, node-pulse)`, so a theme picks its pulse by overriding ONE token (Cyber → `cyber-pulse-exec`, Surveillance → `surv-pulse-exec`, Renaissance → `ren-pulse-exec`) — no high-specificity per-theme animation rule. `node-pulse` is a triple-layer expanding box-shadow (12px ring + 32px mid halo + 56px outer halo) consuming `var(--node-pulse-color)` + the shared `--tint-pulse-*` alphas. Trigger nodes use a separate continuous `.machina-trigger-armed` "listening" heartbeat (`trigger-listening` keyframe) while waiting, plus `.machina-bolt` on the ⚡ badge — distinct from the one-shot execution pulse. **Never animate `opacity` on whole-node selectors** — it fades the node icon + content, not just the glow.
- **`data-page-hidden` animation pause** — [Dashboard.tsx](../client/src/Dashboard.tsx) mounts a `visibilitychange` listener that toggles `data-page-hidden` on `<html>`; [base.css](../client/src/themes/base.css) declares `html[data-page-hidden] *, *::before, *::after { animation-play-state: paused !important; }`. Without this, paused CSS keyframes accumulate frames in the compositor queue while the tab is hidden; on tab return all 50+ executing nodes' triple-layer pulses + Cyber's full-viewport `cyber-flicker` / `cyber-roll` resume simultaneously, stalling the GPU compositor 100-200ms and blocking input dispatch (first-click-feels-frozen pattern). The unpause is deferred via `requestAnimationFrame` so input dispatch wins the frame before composite resumes.
-- **Parameter panel theme contract** — MasterSkillEditor, OutputPanel, MiddleSection, ToolSchemaEditor, ParameterPanel removed `useAppTheme()`. Every surface renders against Tailwind tokens + new-contract CSS custom props. Section headers carry the display-typography triplet (`font-display tracking-[var(--type-tracking-display)] [text-transform:var(--type-uppercase)] text-fg-default`); action buttons use `<ActionButton intent="run|stop|save|config|tools">`; backgrounds use `bg-bg-elevated` / `bg-bg-panel` / `bg-bg-input` from the surface tier table above. EditableNodeLabel emits both `sq-node-label` and `node-label` classNames so per-theme typography rules fire under either topology. Per-theme scrollbar webkit rules (`::-webkit-scrollbar-thumb` etc.) declared in all 10 themes — gold (Renaissance), square-cornered (Atomic / Edo / Greek / Wasteland / Plague), metallic (Steampunk), phosphor (Rot), neon (Cyber), REC-red (Surveillance), shadcn neutral (Light / Dark).
+- **Parameter panel theme contract** — MasterSkillEditor, OutputPanel, MiddleSection, ToolSchemaEditor, ParameterPanel removed `useAppTheme()`. Every surface renders against Tailwind tokens + new-contract CSS custom props. Section headers carry the display-typography triplet (`font-display tracking-[var(--type-tracking-display)] [text-transform:var(--type-uppercase)] text-fg-default`); action buttons use `<ActionButton intent="run|stop|save|config|tools">`; backgrounds use `bg-bg-elevated` / `bg-bg-panel` / `bg-bg-input` from the surface tier table above. EditableNodeLabel emits both `sq-node-label` and `node-label` classNames so per-theme typography rules fire under either topology. Per-theme scrollbar webkit rules (`::-webkit-scrollbar-thumb` etc.) declared in all 12 themes — gold (Renaissance), square-cornered (Atomic / Edo / Greek / Wasteland / Plague), metallic (Steampunk), phosphor (Rot), neon (Cyber), REC-red (Surveillance), shadcn neutral (Light / Dark).
- **Canvas-node class topology alignment** (W26) — TriggerNode, StartNode, ToolkitNode migrated from `.node` (rectangular spec-card) topology to `.sq-node` / `.sq-node-box` (square-icon node) topology so per-theme decorations (Steampunk brass rivets, Edo hanko seal, Surveillance REC LED, Renaissance gold emblem) reach them. Status pips, gear buttons, and React Flow handles on every node component now carry `.sq-node-pip` / `.sq-node-gear` / `.sq-node-handle.in/.out` (square nodes) or `.node-pip` / `.node-gear` / `.node-handle.in/.out` (rectangular nodes — AIAgentNode, GenericNode, TeamMonitorNode). Pip background is data-driven via `data-status="idle | executing | waiting | success | error"` — base.css picks the colour from shadcn semantic tokens; per-theme files override. Inline JS-computed `backgroundColor` / `border` / `animation` on these elements stripped — CSS owns visuals. Cyber's `.node-trigger` rule extended to dual-target both rectangular and square topologies (`.sq-node.node-trigger .sq-node-box`). Orphan `.node-output` rule removed. EditableNodeLabel now emits both `sq-node-label` and `node-label` classNames so per-theme typography rules fire. StartNode reads `nodeColor` from `definition?.defaults?.color` instead of hardcoded `theme.dracula.cyan`.
- **Audio fix** (W20) — `--sound-pack: chime` typo in `light.css` + `dark.css` was silently falling through to `'none'` (`chime` not in the engine's pack registry), making sound silent for the most-used themes. Renamed `light` → `parchment`, `dark` → `terminal`. Added `Sounds.unlock()` exported from [lib/sound.ts](../client/src/lib/sound.ts) plus a one-shot `pointerdown / keydown / touchstart` capture-phase listener in `useSoundSync()` ([hooks/useSound.ts](../client/src/hooks/useSound.ts)) so the AudioContext resumes on the user's first gesture (Chrome / Safari autoplay policy compliance).
- **Canvas-node visual contract** (W21) — every node component (`AIAgentNode`, `SquareNode`, `GenericNode`, `TriggerNode`, `StartNode`, `ToolkitNode`, `TeamMonitorNode`) stripped its inline `background` / `border` / `borderRadius` / `boxShadow` / `animation` props. Each now exposes the per-definition accent hex via `style={{ '--node-color': accentColor }}` on its outer wrapper. Visual styling lives in [base.css](../client/src/themes/base.css) defaults + per-theme overrides; `var(--node-color)` carries the plugin's `visuals.json` accent through CSS without fighting specificity. New `NodeStyle` helper type in [client/src/types/NodeTypes.ts](../client/src/types/NodeTypes.ts) (`React.CSSProperties & Record<\`--${string}\`, string | number>`) makes the inline custom-prop typecheck-clean.
- **Per-theme icons** (W23) — [client/src/assets/icons/themedGlyphs.ts](../client/src/assets/icons/themedGlyphs.ts) ports all 290 SVG glyphs from `design_handoff_machinaos_themes/app/icons.js` (29 keys × 10 themes). [NodeIcon.tsx](../client/src/assets/icons/NodeIcon.tsx) consults `THEMED_GLYPHS[activeTheme][key]` first; falls through to `lucide-react` / `lobehub:` / `asset:` dispatch on miss. Renaissance gets heraldic shields, Cyber wireframe + glow, Plague woodcut hatching, etc. Light + dark themes have no entries — they fall through to existing dispatch.
- **Canvas grid + cursors** (W24) — `--canvas-grid` and `--cursor-default` slots declared in [base.css](../client/src/themes/base.css) and bound to `.canvas-host`. Per-theme grids: Cyber 24px magenta+cyan grid, Surveillance CCTV crosshair + REC overlay, Renaissance fleur-de-lis cartography, Greek key meander, Steampunk brass bolt grid, Atomic mid-century starburst, Wasteland cracked-earth fissures, Rot flagstone, Plague broadsheet red-X. Custom cursors: Cyber crosshair reticle, Surveillance snooper reticle, Renaissance gold-leaf quill.
- **Decorative HTML primitives** (W25) — [client/src/components/SvgFilterDefs.tsx](../client/src/components/SvgFilterDefs.tsx) mounts a hidden inline `<svg><defs>` at app root exposing `#ink-blot` (Renaissance edge warble), `#noise` (Wasteland turbulence), `#crt` (Cyber chromatic aberration) so per-theme `filter: url(#...)` rules resolve. [client/src/components/ui/DropCap.tsx](../client/src/components/ui/DropCap.tsx) wraps content with `v-display drop-cap` className so the Renaissance `::first-letter` ornament rule fires.
-- **Token contract**: all 10 themes ported in [client/src/themes/](../client/src/themes) (5 utopian + 5 dystopian). 13 CSS files, ~3,750 LOC total.
-- **ThemeProvider** ([client/src/contexts/ThemeContext.tsx](../client/src/contexts/ThemeContext.tsx)) — 10-way `ThemeName`, `DARK_FAMILY` ⊃ `{dark, cyber, wasteland, rot, surveillance, steampunk}`, legacy `darkMode` localStorage migration on first load
-- **ThemeSwitcher** — DropdownMenu grouped into System / Utopian / Dystopian sections, all 10 themes
+- **Token contract**: all 12 themes in [client/src/themes/](../client/src/themes) (2 base + 5 utopian + 5 dystopian; the 10 decorative ones ported from the handoff). 13 CSS files, ~3,750 LOC total.
+- **ThemeProvider** ([client/src/contexts/ThemeContext.tsx](../client/src/contexts/ThemeContext.tsx)) — 12-way `ThemeName`, `DARK_FAMILY` ⊃ `{dark, cyber, wasteland, rot, surveillance, steampunk}`, legacy `darkMode` localStorage migration on first load
+- **ThemeSwitcher** — DropdownMenu grouped into System / Utopian / Dystopian sections, all 12 themes
- **Tailwind v4 `@theme inline`** exposing new-contract utilities (`bg-bg-app`, `text-fg-default`, `border-border-default`, `font-display`, `font-body`, `rounded-pill`)
- **Decorative wrappers** — Dashboard root carries `app-frame`, the React Flow host carries `canvas-host` (aliased as `canvas` so handoff selectors resolve), every Modal carries `modal-frame`. Per-theme CSS targets these classes for outer ornaments (gilded corners, scanlines, riveted borders, REC dot, etc.)
- **Per-component structural classNames** (W15) — every chrome / interactive / canvas component carries the handoff selector co-class so per-theme decorative rules resolve without component-level branching. See [Per-component decorations](#per-component-decorations) for the full list.
@@ -247,7 +247,7 @@ Result by theme (display font / tracking / case):
- **Sound throttling** (W19) — `type` and `hover` events throttled to a 30 ms last-fire window inside the engine to prevent OscillatorNode flooding. See [Throttling](#throttling).
- **Migrated chrome**: TopToolbar, WorkflowSidebar, ComponentPalette + ComponentItem + CollapsibleSection, ConsolePanel chrome, SettingsPanel, Modal, ParameterPanel modal title, AIResultModal title, OutputDisplayPanel title, InputSection title
- **New shell components**: StatusBar (fixed-bottom system line with WS connection / workflow / theme / clock), CommandPalette (`⌘K`), CommandPaletteHost (canonical command list with Workflow / Run / Open / View / Theme groups)
-- **Google Fonts** — deferred-load `<link>` covers all 10 themes' typefaces (Cinzel, Cormorant Garamond, IM Fell English / SC, JetBrains Mono, Major Mono Display, VT323, Shippori Mincho, Sawarabi Mincho, Special Elite, Bevan, Lato, Pirata One, EB Garamond, UnifrakturCook, Anonymous Pro, IBM Plex Mono, Courier Prime, Space Mono)
+- **Google Fonts** — deferred-load `<link>` covers all 12 themes' typefaces (Cinzel, Cormorant Garamond, IM Fell English / SC, JetBrains Mono, Major Mono Display, VT323, Shippori Mincho, Sawarabi Mincho, Special Elite, Bevan, Lato, Pirata One, EB Garamond, UnifrakturCook, Anonymous Pro, IBM Plex Mono, Courier Prime, Space Mono)
- **Sound contract** — [client/src/lib/sound.ts](../client/src/lib/sound.ts) ports the upstream WebAudio engine with all 10 packs (`parchment`, `marble`, `ink`, `clockwork`, `vibraphone`, `terminal`, `scrap`, `crypt`, `bell`, `telex`). [client/src/hooks/useSound.ts](../client/src/hooks/useSound.ts) reads `--sound-pack` from `:root` on theme change and mirrors the `soundEnabled` Zustand slice into `Sounds.setEnabled()`. Settings panel ships the toggle (Audio section). Persists to `localStorage['machinaos-sound']`; **default ON** (browsers gesture-gate WebAudio without a separate permission API — the AudioContext starts suspended and resumes on the user's first pointerdown / keydown / touchstart via `Sounds.unlock()`, registered as a one-shot capture-phase listener in `useSoundSync()`).
- **Canvas overlay packs** — [client/src/hooks/useAppTheme.ts](../client/src/hooks/useAppTheme.ts) extended from 2-way (`{light, dark}`) to 10-way: a `THEME_OVERRIDES` map applies a small overlay (primary, focus, focusRing, action colours, edge palette) on top of `lightColors` / `darkColors`. Existing call sites continue to read `theme.colors.X` and `theme.isDarkMode` unchanged; canvas selection rings, action buttons, and edge strokes pick up the active theme's accents under any of the 10 themes.
@@ -519,7 +519,7 @@ cd client && npx eslint <files>
| [client/src/themes/animations.css](../client/src/themes/animations.css) | Per-theme `--pulse-keyframe` tokens + executing / trigger keyframes + `.machina-*` helpers |
| [client/tailwind.config.js](../client/tailwind.config.js) | Colour bridge (`theme.extend.colors` → `var(--X)`, no `hsl()`) |
| [client/src/index.css](../client/src/index.css) | Plumbing: global resets + `@theme inline` `var()` bridge (no literal colours) |
-| [client/src/contexts/ThemeContext.tsx](../client/src/contexts/ThemeContext.tsx) | `<ThemeProvider>` + `useTheme()` (10-way) |
+| [client/src/contexts/ThemeContext.tsx](../client/src/contexts/ThemeContext.tsx) | `<ThemeProvider>` + `useTheme()` (12-way) |
| [client/src/hooks/useAppTheme.ts](../client/src/hooks/useAppTheme.ts) | Canvas overlay packs (10 themes × Colors override) |
| [client/src/lib/sound.ts](../client/src/lib/sound.ts) | WebAudio engine — 10 sound packs, 9 events |
| [client/src/hooks/useSound.ts](../client/src/hooks/useSound.ts) | `useSoundSync()` + `useSound()` React glue |
@@ -534,7 +534,7 @@ cd client && npx eslint <files>
| [client/src/components/SvgFilterDefs.tsx](../client/src/components/SvgFilterDefs.tsx) | Inline `<svg><defs>` carrying `#ink-blot` / `#noise` / `#crt` filter IDs — Wave 25 |
| [client/src/components/ui/DropCap.tsx](../client/src/components/ui/DropCap.tsx) | `v-display drop-cap` wrapper for Renaissance ornament rule — Wave 25 |
| [client/src/types/NodeTypes.ts](../client/src/types/NodeTypes.ts) | `NodeStyle` helper type for inline `--*` custom props — Wave 21 |
-| [client/index.html](../client/index.html) | Google Fonts loading (all 10 themes' typefaces) |
+| [client/index.html](../client/index.html) | Google Fonts loading (all 12 themes' typefaces) |
| `design_handoff_machinaos_themes/` (external design handoff, not in repo) | Reference HTML mocks + token spec |
| `design_handoff_machinaos_themes/app/icons.js` (external) | Upstream icon source (ported by Wave 23) |
| `design_handoff_machinaos_themes/MIGRATION_PLAYBOOK.md` (external) | Upstream recipe — all originally-deferred items now landed (W14–W25) |
diff --git a/server/core/config.py b/server/core/config.py
index fd7a127a..0ba1f9ff 100644
--- a/server/core/config.py
+++ b/server/core/config.py
@@ -101,11 +101,16 @@ class Settings(BaseSettings):
# task queue) alongside the TemporalWorkerManager, and
# MachinaWorkflow._resolve_activity returns cls.task_queue so
# per-type activities land on their specialised pool worker.
+ #
+ # Wave 16.4: default flipped to True — per-queue routing is the
+ # production default (LLM / browser / code-exec / REST workloads
+ # get isolated slot pools instead of sharing one 100-slot queue).
# Rollback: TEMPORAL_WORKER_POOL_ENABLED=false + restart — the pool
# stops and every activity routes back to the single manager worker
- # on "machina-tasks". Default off until the 16.4 stability window.
+ # on "machina-tasks". Opt out via the env var, never by reverting
+ # this default (locked by test_task_queue_coverage.py).
temporal_worker_pool_enabled: bool = Field(
- default=False,
+ default=True,
env="TEMPORAL_WORKER_POOL_ENABLED",
)
# Wave 17.4: deployment topology hint. Drives worker identity
diff --git a/server/main.py b/server/main.py
index e09b8175..364de950 100644
--- a/server/main.py
+++ b/server/main.py
@@ -351,10 +351,11 @@ async def lifespan(app: FastAPI):
await worker_manager.start()
app.state.temporal_worker_manager = worker_manager
- # Wave 16.2: per-queue activity worker pool. Gated —
- # inert until TEMPORAL_WORKER_POOL_ENABLED=true. Starts
- # AFTER the manager so workflow registration is in
- # place before specialised activity workers poll.
+ # Wave 16: per-queue activity worker pool (default-on
+ # since 16.4; TEMPORAL_WORKER_POOL_ENABLED=false is
+ # the rollback channel). Starts AFTER the manager so
+ # workflow registration is in place before
+ # specialised activity workers poll.
if settings.temporal_worker_pool_enabled:
from services.temporal.worker import TemporalWorkerPool
diff --git a/server/services/temporal/worker.py b/server/services/temporal/worker.py
index 62c23d21..dd6e4c99 100644
--- a/server/services/temporal/worker.py
+++ b/server/services/temporal/worker.py
@@ -466,7 +466,6 @@ class TemporalWorkerPool:
worker_kwargs: dict = dict(
task_queue=queue,
activities=activities,
- max_concurrent_workflow_tasks=10,
graceful_shutdown_timeout=_graceful_shutdown_timeout(),
identity=_worker_identity(queue),
interceptors=[ObservabilityWorkerInterceptor()],
@@ -477,13 +476,19 @@ class TemporalWorkerPool:
# lower task volume than the manager's default queue.
activity_task_poller_behavior=PollerBehaviorAutoscaling(initial=1, minimum=1, maximum=5),
)
- # Wave 18.4: the Worker rejects tuner + max_concurrent_activities
- # together — resource-tuned queues get the tuner, the rest keep
- # fixed sizing.
+ # Wave 18.4: the Worker rejects `tuner` alongside ANY of
+ # `max_concurrent_workflow_tasks` / `max_concurrent_activities` /
+ # `max_concurrent_local_activities` / `max_concurrent_nexus_tasks`
+ # — the SDK enforces "size via tuner OR via max_concurrent_*,
+ # never both" (temporalio.worker.Worker.__init__). Resource-tuned
+ # queues get the tuner (which already carries a
+ # FixedSizeSlotSupplier(10) for workflow slots via
+ # `_tuner_for`); the rest keep fixed sizing.
if tuner is not None:
worker_kwargs["tuner"] = tuner
else:
worker_kwargs["max_concurrent_activities"] = concurrency
+ worker_kwargs["max_concurrent_workflow_tasks"] = 10
worker = Worker(self.client, **worker_kwargs)
task = asyncio.create_task(worker.run(), name=f"worker-{queue}")
diff --git a/server/tests/test_task_queue_coverage.py b/server/tests/test_task_queue_coverage.py
index 0dcfaea5..30685d4c 100644
--- a/server/tests/test_task_queue_coverage.py
+++ b/server/tests/test_task_queue_coverage.py
@@ -87,3 +87,41 @@ class TestTaskQueueCoverage:
"update docs-internal/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md "
"if a queue was deliberately removed."
)
+
+
+class TestWorkerPoolDefaultOn:
+ """Wave 16.4: per-queue routing is the production default.
+
+ Source-introspection rather than ``Settings()`` instantiation because
+ conftest stubs ``core.config.Settings`` as a MagicMock for the test
+ session — same pattern as
+ ``test_event_framework_phase_a.py::test_event_framework_enabled_defaults_true``.
+ """
+
+ def test_temporal_worker_pool_enabled_defaults_true(self):
+ import re
+ from pathlib import Path
+
+ config_src = (Path(__file__).parent.parent / "core" / "config.py").read_text(encoding="utf-8")
+ default_true = re.compile(
+ r"default=True[\s,]+env=\"TEMPORAL_WORKER_POOL_ENABLED\"",
+ re.DOTALL,
+ )
+ default_false = re.compile(
+ r"default=False[\s,]+env=\"TEMPORAL_WORKER_POOL_ENABLED\"",
+ re.DOTALL,
+ )
+ assert default_true.search(config_src) is not None, (
+ "temporal_worker_pool_enabled default flipped to True in "
+ "Wave 16.4. The declaration "
+ '``Field(default=True, env="TEMPORAL_WORKER_POOL_ENABLED")`` '
+ "is missing from core/config.py. Opt out via "
+ "TEMPORAL_WORKER_POOL_ENABLED=false in .env — not by "
+ "reverting the default."
+ )
+ assert default_false.search(config_src) is None, (
+ "temporal_worker_pool_enabled reverted to default=False. "
+ "Wave 16.4 flipped the default to True. Use the env var "
+ "(TEMPORAL_WORKER_POOL_ENABLED=false) for opt-out, not a "
+ "code revert."
+ )