Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
feat(temporal): Wave 16.4 — per-queue routing default-on + pool worker crash fix
trohitg190c89611c2e
diff --git a/.env.template b/.env.templateindex ffa363e7..64c0baff 100644--- a/.env.template+++ b/.env.template@@ -56,13 +56,14 @@ TEMPORAL_NAMESPACE=defaultTEMPORAL_TASK_QUEUE=machina-tasksTEMPORAL_PER_TYPE_DISPATCH=trueTEMPORAL_AGENT_WORKFLOW_ENABLED=true-# Wave 16: per-queue activity routing. When true, a TemporalWorkerPool-# starts one activity-only Worker per plugin-declared task queue-# (ai-heavy / browser / code-exec / rest-api / messaging / ...) and-# per-type activities route to their specialised queue instead of the-# single 100-slot pool on machina-tasks. Rollback channel: set false +-# restart — activities route back to the manager worker unchanged.-TEMPORAL_WORKER_POOL_ENABLED=false+# Per-queue activity routing (Wave 16, default-on since 16.4). A+# TemporalWorkerPool starts one activity-only Worker per plugin-declared+# task queue (ai-heavy / browser / code-exec / rest-api / messaging /+# ...) and per-type activities route to their specialised queue instead+# of the single 100-slot pool on machina-tasks. Rollback channel: set+# false + restart — activities route back to the manager worker+# unchanged.+TEMPORAL_WORKER_POOL_ENABLED=true# Deployment topology hint: local (dev laptop; sleeps; halved per-queue# concurrency) | cloud (machina deploy VM) | self_hosted. Drives worker# identity strings in the Temporal Web UI and per-mode concurrencydiff --git a/CLAUDE.md b/CLAUDE.mdindex 93e827a8..d7f755a6 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -13,7 +13,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire| **[UI Migration Plan](./docs-internal/ui_migration_plan.md)** | antd → shadcn/ui migration plan + completion log. Waves 1–10 done. Full frontend is schema-driven (backend SSOT); remaining DIY widget registry (ex-Phase 6) is the one deferred item. || **[Temporal Cleanup & Resilience Plan (Waves 15-18)](./docs-internal/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md)** | RFC for the four forward waves: 15 dead-code retirement (Redis-Streams branch in event_waiter, APScheduler cron stack, orphaned `connection_status` factory — ~556 LOC), 16 per-queue task routing activation (`TemporalWorkerPool` + `TEMPORAL_WORKER_POOL_ENABLED` gate), 17 resilience hardening (cron `catchup_window`, one-shot LLM-step retry, SDK interceptors, `DEPLOYMENT_MODE`, periodic heartbeats), 18 worker performance tuning (per-queue rate limits, sticky cache, poller autoscaling, resource-based slot supplier). Ship order, rollback channels, verified line citations. || **[Node Allowlist](./docs-internal/node_allowlist.md)** | Single-config UI visibility — `server/config/node_allowlist.json` controls which nodes / credential categories / skill folders show in the UI. Five lists with two enforcement tiers (mode-gated allowlist + absolute blocklist). `useNodeAllowlist` hook exposes `isVisible` / `isBlocked` / `isAllowed` / `isCredentialCategoryDisabled` / `isSkillFolderDisabled`. Adding a new disabled domain = single JSON edit, no code change. |-| **[Theme System](./docs-internal/theme_system.md)** | 10-way visual theme system — 5 utopian (light, dark, renaissance, greek, edo, steampunk, atomic) + 5 dystopian (cyber, wasteland, rot, plague, surveillance) — driven by `<html data-theme>` + per-theme CSS files in `client/src/themes/`. Token taxonomy (surface / fg / border / accent / typography / motion) in **hex + `color-mix()`**; `@theme inline` bridge maps `--color-X: var(--X)` (no `hsl()` wrapper); per-theme files own the colour hex, base.css owns the shared `--tint-*` alpha scale; action role tokens (incl. `-ink` readable text), **per-theme `--code-*` syntax tokens** (tier 6 — the code editor, console/output JSON viewers, and chat code blocks paint in each theme's own palette: keyword→trigger, string→success, number→agent, function→model, on an adaptive dark/light code surface; replaced the global dracula `--prism-*` block + dead `getPrismTokenCSS`; the `OutputPanel` JSON viewer reads the same vars), per-theme `--pulse-keyframe` animation system + `.machina-*` helpers in `animations.css`, decorative-layer wrappers (`.app-frame` / `.canvas-host` / `.modal-frame`), per-component decorative ornaments (panel textures + canvas decorations + node pseudo-element overlays + theme-specific keyframes), **canvas-node visual contract via `--node-color` CSS custom property** (no inline `background` / `border` on node components — base.css + per-theme CSS owns visuals; `NodeStyle` helper type at [types/NodeTypes.ts](./client/src/types/NodeTypes.ts) makes the inline custom-prop typecheck-clean), **`--node-pulse-color` separate from `--node-color`** so executing-node glow uses each theme's highest-contrast accent regardless of plugin accent (Cyber neon cyan, Surveillance REC red, Renaissance ultramarine, etc.), **`data-page-hidden` animation pause** (toggled by Dashboard's `visibilitychange` listener; base.css declares `html[data-page-hidden] *, *::before, *::after { animation-play-state: paused !important }` to prevent compositor stall on tab return), **per-theme icon glyph system** (290 SVGs across 29 keys × 10 themes via [themedGlyphs.ts](./client/src/assets/icons/themedGlyphs.ts) + theme-aware [NodeIcon.tsx](./client/src/assets/icons/NodeIcon.tsx)), **per-theme canvas-grid + custom cursors** via `--canvas-grid` / `--cursor-default` slots, **decorative HTML primitives** (`<SvgFilterDefs>` mounting `#ink-blot` / `#noise` / `#crt` filter IDs at app root, `<DropCap>` wrapper for Renaissance ornament rule), **parameter panel migrated to Tailwind tokens** (no `useAppTheme()` reads; section headers carry the display-typography triplet; raw `<Button>` swapped to `<ActionButton intent>`), **per-theme scrollbar webkit rules** in all 10 themes, 9-event WebAudio sound system (10 packs via `--sound-pack` token + `useSound()` hook + global hover delegate + sonner toast monkey-patch + `withSound()` HOC + `Sounds.unlock()` gesture-unlock for AudioContext autoplay-policy compliance), `@media (prefers-reduced-motion: reduce)` accessibility, 30 ms throttle on `type` / `hover`, migration recipe, anti-patterns. Read this before adding a new theme, migrating a component to the new contract, or adding a canvas-node component. |+| **[Theme System](./docs-internal/theme_system.md)** | 12-way visual theme system — 2 base (light, dark) + 5 utopian (renaissance, greek, edo, steampunk, atomic) + 5 dystopian (cyber, wasteland, rot, plague, surveillance) — driven by `<html data-theme>` + per-theme CSS files in `client/src/themes/`. Token taxonomy (surface / fg / border / accent / typography / motion) in **hex + `color-mix()`**; `@theme inline` bridge maps `--color-X: var(--X)` (no `hsl()` wrapper); per-theme files own the colour hex, base.css owns the shared `--tint-*` alpha scale; action role tokens (incl. `-ink` readable text), **per-theme `--code-*` syntax tokens** (tier 6 — the code editor, console/output JSON viewers, and chat code blocks paint in each theme's own palette: keyword→trigger, string→success, number→agent, function→model, on an adaptive dark/light code surface; replaced the global dracula `--prism-*` block + dead `getPrismTokenCSS`; the `OutputPanel` JSON viewer reads the same vars), per-theme `--pulse-keyframe` animation system + `.machina-*` helpers in `animations.css`, decorative-layer wrappers (`.app-frame` / `.canvas-host` / `.modal-frame`), per-component decorative ornaments (panel textures + canvas decorations + node pseudo-element overlays + theme-specific keyframes), **canvas-node visual contract via `--node-color` CSS custom property** (no inline `background` / `border` on node components — base.css + per-theme CSS owns visuals; `NodeStyle` helper type at [types/NodeTypes.ts](./client/src/types/NodeTypes.ts) makes the inline custom-prop typecheck-clean), **`--node-pulse-color` separate from `--node-color`** so executing-node glow uses each theme's highest-contrast accent regardless of plugin accent (Cyber neon cyan, Surveillance REC red, Renaissance ultramarine, etc.), **`data-page-hidden` animation pause** (toggled by Dashboard's `visibilitychange` listener; base.css declares `html[data-page-hidden] *, *::before, *::after { animation-play-state: paused !important }` to prevent compositor stall on tab return), **per-theme icon glyph system** (290 SVGs across 29 keys × 10 themes via [themedGlyphs.ts](./client/src/assets/icons/themedGlyphs.ts) + theme-aware [NodeIcon.tsx](./client/src/assets/icons/NodeIcon.tsx)), **per-theme canvas-grid + custom cursors** via `--canvas-grid` / `--cursor-default` slots, **decorative HTML primitives** (`<SvgFilterDefs>` mounting `#ink-blot` / `#noise` / `#crt` filter IDs at app root, `<DropCap>` wrapper for Renaissance ornament rule), **parameter panel migrated to Tailwind tokens** (no `useAppTheme()` reads; section headers carry the display-typography triplet; raw `<Button>` swapped to `<ActionButton intent>`), **per-theme scrollbar webkit rules** in all 12 themes, 9-event WebAudio sound system (10 packs via `--sound-pack` token + `useSound()` hook + global hover delegate + sonner toast monkey-patch + `withSound()` HOC + `Sounds.unlock()` gesture-unlock for AudioContext autoplay-policy compliance), `@media (prefers-reduced-motion: reduce)` accessibility, 30 ms throttle on `type` / `hover`, migration recipe, anti-patterns. Read this before adding a new theme, migrating a component to the new contract, or adding a canvas-node component. || **[Design System Bundle](./docs-internal/design-system/IMPLEMENTATION.md)** | Canonical, vendored design-system reference (extracted from this repo; see its `IMPLEMENTATION.md` for the contract). Tokens are **hex + `color-mix()`** (`tokens/{colors,typography,spacing,motion,fonts,base}.css`) — the format the live codebase standardizes on. Includes 34 reference components (`components/`, inline-style reference only — recreate in Tailwind/shadcn idioms), full-app UI kit (`ui_kits/machinaos/index.html`), and 12-theme docs (`guidelines/THEMES.md`, `reference/themes/`). Copy token values verbatim; do not re-derive by eye. Pairs with [Theme System](./docs-internal/theme_system.md). || **[Schema Source of Truth RFC](./docs-internal/schema_source_of_truth_rfc.md)** | Backend is SSOT for node schemas, visual metadata, handlers, palette metadata, icons. Plugin pattern: one `BaseNode` subclass in `server/nodes/<group>/<plugin>/__init__.py`. Wire format: `asset:<key>` / `<lib>:<brand>` / URL / emoji. Endpoint: `/api/schemas/nodes/{type}/spec.json`. Live invariant total via `pytest --collect-only`. || **[Plugin System (Wave 11)](./docs-internal/plugin_system.md)** | Class-based plugin-first architecture. `BaseNode` / `ActionNode` / `TriggerNode` / `ToolNode` + `@Operation` decorator. Pydantic `Params`/`Output`. Declarative `Routing` DSL + `Connection` facade (Nango pattern). 18 `Credential` subclasses live in each node folder's `_credentials.py` (or inline for single-use). `TaskQueue` constants route to Temporal worker pools. Plugins live across 9 queues (live count via `glob server/nodes/**/__init__.py`); handler bodies fully inlined (`services/handlers/` shrank 12.8K → 1.1K LOC across 16 → 4 files; only cross-cutting orchestration remains: `tools.py` AI-tool dispatch + agent delegation, `google_auth.py`, `triggers.py`). **Wave 11.H added "self-contained plugin folders"** — up to six generic registries (`ws_handler_registry`, `register_router`, `event_waiter.{register_filter_builder,register_trigger_precheck}`, `status_broadcaster.register_service_refresh`, `node_output_schemas.register_output_schema`) so rich plugins like telegram own their entire surface area without core-services edits. |@@ -594,7 +594,7 @@ AI model nodes route through `SquareNode` via `Dashboard.tsx`'s `COMPONENT_BY_KI### Theme System-10-way visual theme system (5 utopian: light, dark, renaissance, greek, edo, steampunk, atomic + 5 dystopian: cyber, wasteland, rot, plague, surveillance) driven by `<html data-theme>` (set by [ThemeContext.tsx](./client/src/contexts/ThemeContext.tsx), which also toggles `.dark` for DARK_FAMILY themes) + per-theme CSS in `client/src/themes/`. Token VALUES are hex + `color-mix()` (never HSL): per-theme files own shadcn/dracula/node/action hex, `base.css` owns the shared `--tint-*` scale, `index.css` is plumbing (`@theme inline` maps `--color-X: var(--X)`). Six token tiers, the per-theme `--pulse-keyframe` animation system, decorative-layer wrappers, the 10-pack WebAudio sound system, and the canvas-wide edge/node status rules (`canvasAnimations.ts`) are all documented in **[Theme System](./docs-internal/theme_system.md)** — read it before adding a theme or a canvas-node component. The strict frontend theme RULES remain normative under "Frontend Design + Theme System (strict)" above.+12-way visual theme system (2 base: light, dark + 5 utopian: renaissance, greek, edo, steampunk, atomic + 5 dystopian: cyber, wasteland, rot, plague, surveillance) driven by `<html data-theme>` (set by [ThemeContext.tsx](./client/src/contexts/ThemeContext.tsx), which also toggles `.dark` for DARK_FAMILY themes) + per-theme CSS in `client/src/themes/`. Token VALUES are hex + `color-mix()` (never HSL): per-theme files own shadcn/dracula/node/action hex, `base.css` owns the shared `--tint-*` scale, `index.css` is plumbing (`@theme inline` maps `--color-X: var(--X)`). Six token tiers, the per-theme `--pulse-keyframe` animation system, decorative-layer wrappers, the 10-pack WebAudio sound system, and the canvas-wide edge/node status rules (`canvasAnimations.ts`) are all documented in **[Theme System](./docs-internal/theme_system.md)** — read it before adding a theme or a canvas-node component. The strict frontend theme RULES remain normative under "Frontend Design + Theme System (strict)" above.### WebSocket-First ArchitectureThe project uses WebSocket as the primary communication method between frontend and backend, replacing most REST API calls:@@ -681,7 +681,7 @@ Node groups (palette categories): agent, model, skill, tool, trigger, workflow,### Temporal Distributed Execution-Workflows execute via Temporal for durability and horizontal scaling, gated by `TEMPORAL_ENABLED`. Three dispatch paths exist — legacy single `execute_node_activity` (default), per-type `node.{type}.v{version}` activities (F4.A, `TEMPORAL_PER_TYPE_DISPATCH=true`), and Agent-as-child-workflow (F4.B, `TEMPORAL_AGENT_WORKFLOW_ENABLED=true`) — with `rlm_agent`/`claude_code_agent` always bypassing AgentWorkflow. Execution routing falls back Temporal → Redis-parallel → sequential; the Temporal dev server + embedded worker are managed in-process under `server/services/temporal/` (pooch-downloaded CLI, single `temporal server start-dev` process, gRPC 7233 + Web UI 8080, SQLite at `~/.machina/temporal.db`). Full architecture, dispatch matrix, per-node + agent-loop lifecycle, the 7 F4.B `agent.*.v1` activities, heartbeat semantics, delegation input contract, resumption toggle (`TEMPORAL_TERMINATE_RUNNING_ON_STARTUP`), and all `.env` tunables live in [docs-internal/TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md).+Workflows execute via Temporal for durability and horizontal scaling, gated by `TEMPORAL_ENABLED`. Three dispatch paths exist — legacy single `execute_node_activity` (fallback), per-type `node.{type}.v{version}` activities (F4.A, `TEMPORAL_PER_TYPE_DISPATCH=true`), and Agent-as-child-workflow (F4.B, `TEMPORAL_AGENT_WORKFLOW_ENABLED=true`) — with `rlm_agent`/`claude_code_agent` always bypassing AgentWorkflow. **Per-queue activity routing is production-default (Wave 16)**: a `TemporalWorkerPool` runs one activity-only worker per plugin-declared `cls.task_queue` (ai-heavy / browser / code-exec / rest-api / messaging / ...) with per-queue concurrency + rate limits + resource-based tuning for ai-heavy/browser; rollback via `TEMPORAL_WORKER_POOL_ENABLED=false`. Resilience knobs (Waves 17-18): `DEPLOYMENT_MODE` (local halves concurrency; sizes the sticky cache), one-shot retry on `agent.execute_llm_step.v1` (LLM calls are not idempotent), observability interceptors (`activity_retry` WARN on re-dispatch), 30s periodic heartbeat during long activity bodies, cron `catchup_window=24h`, poller autoscaling. Execution routing falls back Temporal → Redis-parallel → sequential; the Temporal dev server + embedded worker are managed in-process under `server/services/temporal/` (pooch-downloaded CLI, single `temporal server start-dev` process, gRPC 7233 + Web UI 8080, SQLite at `~/.machina/temporal.db`). Full architecture, dispatch matrix, per-node + agent-loop lifecycle, the 7 F4.B `agent.*.v1` activities, heartbeat semantics, delegation input contract, resumption toggle (`TEMPORAL_TERMINATE_RUNNING_ON_STARTUP`), the worker tuning recipe, and all `.env` tunables live in [docs-internal/TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md).## Development Commandsdiff --git a/README.md b/README.mdindex c7d86cd4..590eec5d 100644--- a/README.md+++ b/README.md@@ -38,7 +38,7 @@ The `dev` task starts the Python backend, Vite client, WhatsApp service, and Tem</details>-## Quick Hello World Setup for Machinaos ↓.+## Quick Hello World Setup for MachinaOS ↓https://github.com/user-attachments/assets/a5a5583f-bb5f-4d27-a387-8522c556e89e@@ -120,7 +120,7 @@ RAG pipeline out of the box: parse PDFs and HTML, chunk into searchable pieces,Local providers (Ollama, LM Studio) are first-class — context length, vision support, and tool-use capability are detected automatically from your running server. No paid API needed.-### 17 specialized agent types+### 16 specialized agent typesPick the right agent for the job:@@ -150,7 +150,7 @@ Skills are short markdown files that teach an agent how to do something well —### Memory that scales with your context window-Agents track token usage and automatically compact long conversations when you hit half your model's context limit. Compaction summarizes in five sections — Task Overview, Current State, Important Discoveries, Next Steps, Context to Preserve — so the agent picks up exactly where it left off. For Anthropic and OpenAI, native API compaction is used; everywhere else, the agent summarizes itself.+Agents track token usage and automatically compact long conversations as you approach your model's context limit (80% by default, configurable). Compaction summarizes in five sections — Task Overview, Current State, Important Discoveries, Next Steps, Context to Preserve — so the agent picks up exactly where it left off. For Anthropic and OpenAI, native API compaction is used; everywhere else, the agent summarizes itself.### Cost tracking, built in@@ -158,7 +158,7 @@ Every LLM call and API request is tracked with USD cost. See per-provider spend## The Canvas-- **10 visual themes** — light, dark, Renaissance, Greek, Edo, Steampunk, Atomic, Cyber, Wasteland, Rot, Plague, Surveillance — each with its own icon set, sound pack, and decorative ornaments. Pick the vibe that matches your workflow.+- **12 visual themes** — light, dark, Renaissance, Greek, Edo, Steampunk, Atomic, Cyber, Wasteland, Rot, Plague, Surveillance — each with its own icon set, sound pack, and decorative ornaments. Pick the vibe that matches your workflow.- **Drag-to-map outputs** from one node's output directly onto another's input fields.- **Live execution animations** — nodes glow while running, show iteration count for AI agents, and surface errors inline.- **Multi-tab Console** — chat with trigger nodes, watch console logs, and view terminal output side by side.diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.mdindex 59640901..b0444d07 100644--- a/docs-internal/TEMPORAL_ARCHITECTURE.md+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md@@ -6,8 +6,8 @@ Each workflow node executes as a **Temporal activity** with its own isolated con| Dispatch | Trigger | Use case ||---|---|---|-| **Legacy single activity** (`execute_node_activity`) | Default | Every node routed through one dispatcher activity. WebSocket round-trip back to the FastAPI server. Stable since Wave 11. |-| **Per-type activity** (`node.{type}.v{version}`) | `TEMPORAL_PER_TYPE_DISPATCH=true` | Each plugin gets its own `@activity.defn`. Per-plugin retry / timeout / heartbeat configs apply. Worker pool can later specialise per `cls.task_queue` (browser / code-exec / ai-heavy / ...). Shipped in F4.A (commit `8261b05`). |+| **Legacy single activity** (`execute_node_activity`) | `TEMPORAL_PER_TYPE_DISPATCH=false` | Every node routed through one dispatcher activity. WebSocket round-trip back to the FastAPI server. Stable since Wave 11; kept as the fallback path. |+| **Per-type activity** (`node.{type}.v{version}`) | `TEMPORAL_PER_TYPE_DISPATCH=true` (production default) | Each plugin gets its own `@activity.defn`. Per-plugin retry / timeout / heartbeat configs apply. With `TEMPORAL_WORKER_POOL_ENABLED=true` (default since Wave 16.4) the activity also carries `task_queue=cls.task_queue`, landing it on its specialised `TemporalWorkerPool` worker (browser / code-exec / ai-heavy / ...). Shipped in F4.A (commit `8261b05`); queue routing activated in Wave 16. || **Agent-as-child-workflow** (`AgentWorkflow`) | `TEMPORAL_AGENT_WORKFLOW_ENABLED=true` | AI Agents (aiAgent, chatAgent, 11 specialized agents, 2 team leads) run as Temporal child workflows. Each LLM turn = activity; each tool call = per-type activity. Mirrors Temporal's AI Cookbook canonical pattern. F4.B infrastructure shipped (commit `a4d009e`); per-agent migrations follow. |`rlm_agent`, `claude_code_agent` are intentionally excluded from AgentWorkflow — their externalised loops (RLM REPL / Claude CLI `--resume`) require single-process state continuity.@@ -44,28 +44,31 @@ This invokes `run_standalone_worker()` from `services/temporal/worker.py`.```TEMPORAL SERVER (port 7233)|- Task Queue: machina-tasks+ Task Queue: machina-tasks (workflows + framework activities)+---------------------------------------------------------------+| Workflow: MachinaWorkflow (orchestrator only) || - Parses graph structure from React Flow || - Filters config nodes (tools, memory, services) || - Resolves activity per node (legacy / per-type / agent-wf) |+ | - Per-type activities carry task_queue=cls.task_queue |+ | (Wave 16; TEMPORAL_WORKER_POOL_ENABLED, default on) || - Schedules activities (FIRST_COMPLETED pattern) || - Collects results and routes outputs to dependent nodes |+---------------------------------------------------------------+|- Activity / child-workflow scheduling- +--------+ +--------+ +--------+ +----------------+- | Node A | | Node B | | Node C | | AgentWorkflow | (F4.B child wf)- +--------+ +--------+ +--------+ +----------------+- | | | |- v v v v- +----------+ +----------+ +----------+ +-----------------------+- | Worker 1 | | Worker 2 | | Worker 3 | | LLM step + tool steps |- | aiAgent | | timer | | console | | (per-type activities) |- +----------+ +----------+ +----------+ +-----------------------+- | | | |- +----------+-----------+--------------++ Activity / child-workflow scheduling (routed by queue)+ |+ machina-tasks specialised queues (TemporalWorkerPool)+ | ai-heavy code-exec browser rest-api ...+ +-----------+ +--------+ +--------+ +-------+ +-------++ | Manager | | Pool | | Pool | | Pool | | Pool |+ | worker | | worker | | worker | | worker| | worker|+ | (wf tasks | | aiAgent| | python | |browser| | gmail |+ | + legacy | | chatA | | js/ts | | | | brave |+ | dispatch | | deepA | | | | | | ... |+ | + AgentWf)| +--------+ +--------+ +-------+ +-------++ +-----------+ | | | |+ +----------------+-----------+----------+----------+|vIn-process call (F4.A) OR@@ -272,9 +275,9 @@ Benefits:Add more workers = handle more concurrent nodes```-### Specialized Worker Pools (Wave 16 — gated by `TEMPORAL_WORKER_POOL_ENABLED`)+### Specialized Worker Pools (Wave 16 — `TEMPORAL_WORKER_POOL_ENABLED`, default on since 16.4)-`TemporalWorkerPool` is wired into the `main.py` lifespan (starts right after `TemporalWorkerManager`, stops before it). When `TEMPORAL_WORKER_POOL_ENABLED=true`, one activity-only Worker per plugin-declared queue polls its specialised queue, and `MachinaWorkflow._resolve_activity` returns `(activity_name, cls.task_queue)` so per-type activities land there. Flag off (default until the stability window) = queue `None` = everything routes to the single manager worker on `machina-tasks`. **The flag is the rollback channel.**+`TemporalWorkerPool` is wired into the `main.py` lifespan (starts right after `TemporalWorkerManager`, stops before it). One activity-only Worker per plugin-declared queue polls its specialised queue, and `MachinaWorkflow._resolve_activity` returns `(activity_name, cls.task_queue)` so per-type activities land there. Setting `TEMPORAL_WORKER_POOL_ENABLED=false` stops the pool and routes every activity back to the single manager worker on `machina-tasks` — **the flag is the rollback channel** (locked default-on by `test_task_queue_coverage.py::TestWorkerPoolDefaultOn`).```Queue: rest-api Queue: ai-heavy Queue: code-execdiff --git a/docs-internal/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md b/docs-internal/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.mdindex ab38aa62..4d412883 100644--- a/docs-internal/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md+++ b/docs-internal/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md@@ -16,7 +16,7 @@| 16.1 — task-queue coverage invariants | ✅ shipped `3262fb0` || 16.2 — pool wired into lifespan (gated off) | ✅ shipped `18df875` || 16.3 — `_resolve_activity` returns `cls.task_queue` when pool on | ✅ shipped `d4b55c4` |-| 16.4 — default flip | ⏳ parked for one-release stability window (`TEMPORAL_WORKER_POOL_ENABLED` stays default-false; flip + `test_temporal_worker_pool_enabled_defaults_true` invariant afterwards) |+| 16.4 — default flip | ✅ shipped (this commit) — `temporal_worker_pool_enabled` default `True`; opt-out via env var only, locked by `test_task_queue_coverage.py::TestWorkerPoolDefaultOn` || 17.1 — cron `catchup_window=24h` | ✅ shipped `5bf9000` (note: `catchup_window` is a `SchedulePolicy` field in the Python SDK, not `ScheduleSpec` as §6 originally said) || 17.2 — one-shot LLM-step retry | ✅ shipped `03c2bf6` (`LLM_STEP_RETRY` constant; refresh_tools keeps 3 attempts, documented at callsite) || 17.3 — observability interceptors | ✅ shipped `9ecd743` (replay guard via `workflow.unsafe.is_replaying()`) |diff --git a/docs-internal/frontend_architecture.md b/docs-internal/frontend_architecture.mdindex 35247b87..de5253ef 100644--- a/docs-internal/frontend_architecture.md+++ b/docs-internal/frontend_architecture.md@@ -189,7 +189,7 @@ client/src/## Tokens + theming-The frontend ships **10 themes** organised utopian / dystopian: `light` · `dark` · `renaissance` · `greek` · `edo` · `steampunk` · `atomic` · `cyber` · `wasteland` · `rot` · `plague` · `surveillance`. Active theme is `<html data-theme="...">` set by `<ThemeProvider>` (see [contexts/ThemeContext.tsx](../client/src/contexts/ThemeContext.tsx)). Per-theme blocks live in [client/src/themes/](../client/src/themes/) and own all colour VALUES as **hex + `color-mix()`** (never HSL triplets); [src/index.css](../client/src/index.css) is **plumbing-only** (the `@theme inline` `var()` bridge — no literal colours). Full token contract, sound + decorative-layer wiring, and the 12-theme migration playbook in **[theme_system.md](./theme_system.md)** — read that before adding a theme or migrating a component.+The frontend ships **12 themes** organised base / utopian / dystopian: `light` · `dark` · `renaissance` · `greek` · `edo` · `steampunk` · `atomic` · `cyber` · `wasteland` · `rot` · `plague` · `surveillance`. Active theme is `<html data-theme="...">` set by `<ThemeProvider>` (see [contexts/ThemeContext.tsx](../client/src/contexts/ThemeContext.tsx)). Per-theme blocks live in [client/src/themes/](../client/src/themes/) and own all colour VALUES as **hex + `color-mix()`** (never HSL triplets); [src/index.css](../client/src/index.css) is **plumbing-only** (the `@theme inline` `var()` bridge — no literal colours). Full token contract, sound + decorative-layer wiring, and the 12-theme migration playbook in **[theme_system.md](./theme_system.md)** — read that before adding a theme or migrating a component.The bridge ([src/index.css](../client/src/index.css)) maps each Tailwind colour token to a raw `var(--X)`.@@ -508,6 +508,6 @@ The `MACHINAOS_INSTALLING=true` env var suppresses the recursive project postinsThis architecture is the post-migration state. Pre-migration was antd + `styled-components` + a custom theme.ts-driven palette. See [ui_migration_plan.md](./ui_migration_plan.md) for the phase-by-phase transition and the 17 commits that executed it.-**`useAppTheme()` powers the canvas + maps surface across all 10 themes.** The hook returns a `theme` object with the legacy `Colors` shape (`theme.colors.X`, `theme.isDarkMode`) so existing call sites don't change. Under non-light/dark themes it merges a per-theme overlay (primary, focus, action palette, edge stroke / selection / executing / completed / error) on top of the chosen base pack (`lightColors` for utopian-bright themes, `darkColors` for dystopian / dark themes). Adding a new theme overlay is a single entry in the `THEME_OVERRIDES` map in [hooks/useAppTheme.ts](../client/src/hooks/useAppTheme.ts).+**`useAppTheme()` powers the canvas + maps surface across all 12 themes.** The hook returns a `theme` object with the legacy `Colors` shape (`theme.colors.X`, `theme.isDarkMode`) so existing call sites don't change. Under non-light/dark themes it merges a per-theme overlay (primary, focus, action palette, edge stroke / selection / executing / completed / error) on top of the chosen base pack (`lightColors` for utopian-bright themes, `darkColors` for dystopian / dark themes). Adding a new theme overlay is a single entry in the `THEME_OVERRIDES` map in [hooks/useAppTheme.ts](../client/src/hooks/useAppTheme.ts).Read sites: every canvas node component (`AIAgentNode`, `SquareNode`, `TriggerNode`, `StartNode`, `ToolkitNode`, `TeamMonitorNode`, `GenericNode`), `EdgeConditionEditor`, and the Maps surface (`MapSelector`, `GoogleMapsPicker`, `MapsPreviewPanel`) — they interpolate per-definition `nodeColor` and JS-side hex values that Tailwind classes can't express. Every other surface uses Tailwind + the token tiers above and retints automatically through the per-theme `[data-theme="..."]` block.diff --git a/docs-internal/theme_system.md b/docs-internal/theme_system.mdindex b0889a18..8abdf8e5 100644--- a/docs-internal/theme_system.md+++ b/docs-internal/theme_system.md@@ -1,4 +1,4 @@-# Theme System — design-handoff token contract + 10 themes+# Theme System — design-handoff token contract + 12 themesThe MachinaOs frontend supports **ten visual themes**, organised as a utopian / dystopian taxonomy from the design handoff:@@ -144,7 +144,7 @@ Concrete swaps when migrating an existing component:### Display-typography helper-Heading surfaces under all 10 themes use the same triplet. Keep them grouped in the same className for clarity:+Heading surfaces under all 12 themes use the same triplet. Keep them grouped in the same className for clarity:```tsx<span className="font-display tracking-[var(--type-tracking-display)] text-fg-default [text-transform:var(--type-uppercase)]">@@ -227,16 +227,16 @@ Result by theme (display font / tracking / case):### Done- **`--node-pulse-color` contract** — executing-node glow color is independent of `--node-color` (the plugin accent). Each theme overrides `--node-pulse-color` to its highest-contrast accent (Cyber neon cyan, Surveillance REC red, Renaissance ultramarine `--lapis-bright`, Atomic turquoise, etc.) so the glow stays visible against any canvas background. Defined in [base.css](../client/src/themes/base.css) at `:root`, overridden in each per-theme file. The executing keyframes (`node-pulse` + per-theme `cyber-pulse-exec` / `surv-pulse-exec` / `ren-pulse-exec`) and the per-theme `--pulse-keyframe` / `--pulse-duration` / `--pulse-timing` tokens live in [animations.css](../client/src/themes/animations.css); base.css's `.sq-node[data-executing] .sq-node-box` / `.react-flow__node.executing .{sq-node-box,node}` rule reads `var(--pulse-keyframe, node-pulse)`, so a theme picks its pulse by overriding ONE token (Cyber → `cyber-pulse-exec`, Surveillance → `surv-pulse-exec`, Renaissance → `ren-pulse-exec`) — no high-specificity per-theme animation rule. `node-pulse` is a triple-layer expanding box-shadow (12px ring + 32px mid halo + 56px outer halo) consuming `var(--node-pulse-color)` + the shared `--tint-pulse-*` alphas. Trigger nodes use a separate continuous `.machina-trigger-armed` "listening" heartbeat (`trigger-listening` keyframe) while waiting, plus `.machina-bolt` on the ⚡ badge — distinct from the one-shot execution pulse. **Never animate `opacity` on whole-node selectors** — it fades the node icon + content, not just the glow.- **`data-page-hidden` animation pause** — [Dashboard.tsx](../client/src/Dashboard.tsx) mounts a `visibilitychange` listener that toggles `data-page-hidden` on `<html>`; [base.css](../client/src/themes/base.css) declares `html[data-page-hidden] *, *::before, *::after { animation-play-state: paused !important; }`. Without this, paused CSS keyframes accumulate frames in the compositor queue while the tab is hidden; on tab return all 50+ executing nodes' triple-layer pulses + Cyber's full-viewport `cyber-flicker` / `cyber-roll` resume simultaneously, stalling the GPU compositor 100-200ms and blocking input dispatch (first-click-feels-frozen pattern). The unpause is deferred via `requestAnimationFrame` so input dispatch wins the frame before composite resumes.-- **Parameter panel theme contract** — MasterSkillEditor, OutputPanel, MiddleSection, ToolSchemaEditor, ParameterPanel removed `useAppTheme()`. Every surface renders against Tailwind tokens + new-contract CSS custom props. Section headers carry the display-typography triplet (`font-display tracking-[var(--type-tracking-display)] [text-transform:var(--type-uppercase)] text-fg-default`); action buttons use `<ActionButton intent="run|stop|save|config|tools">`; backgrounds use `bg-bg-elevated` / `bg-bg-panel` / `bg-bg-input` from the surface tier table above. EditableNodeLabel emits both `sq-node-label` and `node-label` classNames so per-theme typography rules fire under either topology. Per-theme scrollbar webkit rules (`::-webkit-scrollbar-thumb` etc.) declared in all 10 themes — gold (Renaissance), square-cornered (Atomic / Edo / Greek / Wasteland / Plague), metallic (Steampunk), phosphor (Rot), neon (Cyber), REC-red (Surveillance), shadcn neutral (Light / Dark).+- **Parameter panel theme contract** — MasterSkillEditor, OutputPanel, MiddleSection, ToolSchemaEditor, ParameterPanel removed `useAppTheme()`. Every surface renders against Tailwind tokens + new-contract CSS custom props. Section headers carry the display-typography triplet (`font-display tracking-[var(--type-tracking-display)] [text-transform:var(--type-uppercase)] text-fg-default`); action buttons use `<ActionButton intent="run|stop|save|config|tools">`; backgrounds use `bg-bg-elevated` / `bg-bg-panel` / `bg-bg-input` from the surface tier table above. EditableNodeLabel emits both `sq-node-label` and `node-label` classNames so per-theme typography rules fire under either topology. Per-theme scrollbar webkit rules (`::-webkit-scrollbar-thumb` etc.) declared in all 12 themes — gold (Renaissance), square-cornered (Atomic / Edo / Greek / Wasteland / Plague), metallic (Steampunk), phosphor (Rot), neon (Cyber), REC-red (Surveillance), shadcn neutral (Light / Dark).- **Canvas-node class topology alignment** (W26) — TriggerNode, StartNode, ToolkitNode migrated from `.node` (rectangular spec-card) topology to `.sq-node` / `.sq-node-box` (square-icon node) topology so per-theme decorations (Steampunk brass rivets, Edo hanko seal, Surveillance REC LED, Renaissance gold emblem) reach them. Status pips, gear buttons, and React Flow handles on every node component now carry `.sq-node-pip` / `.sq-node-gear` / `.sq-node-handle.in/.out` (square nodes) or `.node-pip` / `.node-gear` / `.node-handle.in/.out` (rectangular nodes — AIAgentNode, GenericNode, TeamMonitorNode). Pip background is data-driven via `data-status="idle | executing | waiting | success | error"` — base.css picks the colour from shadcn semantic tokens; per-theme files override. Inline JS-computed `backgroundColor` / `border` / `animation` on these elements stripped — CSS owns visuals. Cyber's `.node-trigger` rule extended to dual-target both rectangular and square topologies (`.sq-node.node-trigger .sq-node-box`). Orphan `.node-output` rule removed. EditableNodeLabel now emits both `sq-node-label` and `node-label` classNames so per-theme typography rules fire. StartNode reads `nodeColor` from `definition?.defaults?.color` instead of hardcoded `theme.dracula.cyan`.- **Audio fix** (W20) — `--sound-pack: chime` typo in `light.css` + `dark.css` was silently falling through to `'none'` (`chime` not in the engine's pack registry), making sound silent for the most-used themes. Renamed `light` → `parchment`, `dark` → `terminal`. Added `Sounds.unlock()` exported from [lib/sound.ts](../client/src/lib/sound.ts) plus a one-shot `pointerdown / keydown / touchstart` capture-phase listener in `useSoundSync()` ([hooks/useSound.ts](../client/src/hooks/useSound.ts)) so the AudioContext resumes on the user's first gesture (Chrome / Safari autoplay policy compliance).- **Canvas-node visual contract** (W21) — every node component (`AIAgentNode`, `SquareNode`, `GenericNode`, `TriggerNode`, `StartNode`, `ToolkitNode`, `TeamMonitorNode`) stripped its inline `background` / `border` / `borderRadius` / `boxShadow` / `animation` props. Each now exposes the per-definition accent hex via `style={{ '--node-color': accentColor }}` on its outer wrapper. Visual styling lives in [base.css](../client/src/themes/base.css) defaults + per-theme overrides; `var(--node-color)` carries the plugin's `visuals.json` accent through CSS without fighting specificity. New `NodeStyle` helper type in [client/src/types/NodeTypes.ts](../client/src/types/NodeTypes.ts) (`React.CSSProperties & Record<\`--${string}\`, string | number>`) makes the inline custom-prop typecheck-clean.- **Per-theme icons** (W23) — [client/src/assets/icons/themedGlyphs.ts](../client/src/assets/icons/themedGlyphs.ts) ports all 290 SVG glyphs from `design_handoff_machinaos_themes/app/icons.js` (29 keys × 10 themes). [NodeIcon.tsx](../client/src/assets/icons/NodeIcon.tsx) consults `THEMED_GLYPHS[activeTheme][key]` first; falls through to `lucide-react` / `lobehub:` / `asset:` dispatch on miss. Renaissance gets heraldic shields, Cyber wireframe + glow, Plague woodcut hatching, etc. Light + dark themes have no entries — they fall through to existing dispatch.- **Canvas grid + cursors** (W24) — `--canvas-grid` and `--cursor-default` slots declared in [base.css](../client/src/themes/base.css) and bound to `.canvas-host`. Per-theme grids: Cyber 24px magenta+cyan grid, Surveillance CCTV crosshair + REC overlay, Renaissance fleur-de-lis cartography, Greek key meander, Steampunk brass bolt grid, Atomic mid-century starburst, Wasteland cracked-earth fissures, Rot flagstone, Plague broadsheet red-X. Custom cursors: Cyber crosshair reticle, Surveillance snooper reticle, Renaissance gold-leaf quill.- **Decorative HTML primitives** (W25) — [client/src/components/SvgFilterDefs.tsx](../client/src/components/SvgFilterDefs.tsx) mounts a hidden inline `<svg><defs>` at app root exposing `#ink-blot` (Renaissance edge warble), `#noise` (Wasteland turbulence), `#crt` (Cyber chromatic aberration) so per-theme `filter: url(#...)` rules resolve. [client/src/components/ui/DropCap.tsx](../client/src/components/ui/DropCap.tsx) wraps content with `v-display drop-cap` className so the Renaissance `::first-letter` ornament rule fires.-- **Token contract**: all 10 themes ported in [client/src/themes/](../client/src/themes) (5 utopian + 5 dystopian). 13 CSS files, ~3,750 LOC total.-- **ThemeProvider** ([client/src/contexts/ThemeContext.tsx](../client/src/contexts/ThemeContext.tsx)) — 10-way `ThemeName`, `DARK_FAMILY` ⊃ `{dark, cyber, wasteland, rot, surveillance, steampunk}`, legacy `darkMode` localStorage migration on first load-- **ThemeSwitcher** — DropdownMenu grouped into System / Utopian / Dystopian sections, all 10 themes+- **Token contract**: all 12 themes in [client/src/themes/](../client/src/themes) (2 base + 5 utopian + 5 dystopian; the 10 decorative ones ported from the handoff). 13 CSS files, ~3,750 LOC total.+- **ThemeProvider** ([client/src/contexts/ThemeContext.tsx](../client/src/contexts/ThemeContext.tsx)) — 12-way `ThemeName`, `DARK_FAMILY` ⊃ `{dark, cyber, wasteland, rot, surveillance, steampunk}`, legacy `darkMode` localStorage migration on first load+- **ThemeSwitcher** — DropdownMenu grouped into System / Utopian / Dystopian sections, all 12 themes- **Tailwind v4 `@theme inline`** exposing new-contract utilities (`bg-bg-app`, `text-fg-default`, `border-border-default`, `font-display`, `font-body`, `rounded-pill`)- **Decorative wrappers** — Dashboard root carries `app-frame`, the React Flow host carries `canvas-host` (aliased as `canvas` so handoff selectors resolve), every Modal carries `modal-frame`. Per-theme CSS targets these classes for outer ornaments (gilded corners, scanlines, riveted borders, REC dot, etc.)- **Per-component structural classNames** (W15) — every chrome / interactive / canvas component carries the handoff selector co-class so per-theme decorative rules resolve without component-level branching. See [Per-component decorations](#per-component-decorations) for the full list.@@ -247,7 +247,7 @@ Result by theme (display font / tracking / case):- **Sound throttling** (W19) — `type` and `hover` events throttled to a 30 ms last-fire window inside the engine to prevent OscillatorNode flooding. See [Throttling](#throttling).- **Migrated chrome**: TopToolbar, WorkflowSidebar, ComponentPalette + ComponentItem + CollapsibleSection, ConsolePanel chrome, SettingsPanel, Modal, ParameterPanel modal title, AIResultModal title, OutputDisplayPanel title, InputSection title- **New shell components**: StatusBar (fixed-bottom system line with WS connection / workflow / theme / clock), CommandPalette (`⌘K`), CommandPaletteHost (canonical command list with Workflow / Run / Open / View / Theme groups)-- **Google Fonts** — deferred-load `<link>` covers all 10 themes' typefaces (Cinzel, Cormorant Garamond, IM Fell English / SC, JetBrains Mono, Major Mono Display, VT323, Shippori Mincho, Sawarabi Mincho, Special Elite, Bevan, Lato, Pirata One, EB Garamond, UnifrakturCook, Anonymous Pro, IBM Plex Mono, Courier Prime, Space Mono)+- **Google Fonts** — deferred-load `<link>` covers all 12 themes' typefaces (Cinzel, Cormorant Garamond, IM Fell English / SC, JetBrains Mono, Major Mono Display, VT323, Shippori Mincho, Sawarabi Mincho, Special Elite, Bevan, Lato, Pirata One, EB Garamond, UnifrakturCook, Anonymous Pro, IBM Plex Mono, Courier Prime, Space Mono)- **Sound contract** — [client/src/lib/sound.ts](../client/src/lib/sound.ts) ports the upstream WebAudio engine with all 10 packs (`parchment`, `marble`, `ink`, `clockwork`, `vibraphone`, `terminal`, `scrap`, `crypt`, `bell`, `telex`). [client/src/hooks/useSound.ts](../client/src/hooks/useSound.ts) reads `--sound-pack` from `:root` on theme change and mirrors the `soundEnabled` Zustand slice into `Sounds.setEnabled()`. Settings panel ships the toggle (Audio section). Persists to `localStorage['machinaos-sound']`; **default ON** (browsers gesture-gate WebAudio without a separate permission API — the AudioContext starts suspended and resumes on the user's first pointerdown / keydown / touchstart via `Sounds.unlock()`, registered as a one-shot capture-phase listener in `useSoundSync()`).- **Canvas overlay packs** — [client/src/hooks/useAppTheme.ts](../client/src/hooks/useAppTheme.ts) extended from 2-way (`{light, dark}`) to 10-way: a `THEME_OVERRIDES` map applies a small overlay (primary, focus, focusRing, action colours, edge palette) on top of `lightColors` / `darkColors`. Existing call sites continue to read `theme.colors.X` and `theme.isDarkMode` unchanged; canvas selection rings, action buttons, and edge strokes pick up the active theme's accents under any of the 10 themes.@@ -519,7 +519,7 @@ cd client && npx eslint <files>| [client/src/themes/animations.css](../client/src/themes/animations.css) | Per-theme `--pulse-keyframe` tokens + executing / trigger keyframes + `.machina-*` helpers || [client/tailwind.config.js](../client/tailwind.config.js) | Colour bridge (`theme.extend.colors` → `var(--X)`, no `hsl()`) || [client/src/index.css](../client/src/index.css) | Plumbing: global resets + `@theme inline` `var()` bridge (no literal colours) |-| [client/src/contexts/ThemeContext.tsx](../client/src/contexts/ThemeContext.tsx) | `<ThemeProvider>` + `useTheme()` (10-way) |+| [client/src/contexts/ThemeContext.tsx](../client/src/contexts/ThemeContext.tsx) | `<ThemeProvider>` + `useTheme()` (12-way) || [client/src/hooks/useAppTheme.ts](../client/src/hooks/useAppTheme.ts) | Canvas overlay packs (10 themes × Colors override) || [client/src/lib/sound.ts](../client/src/lib/sound.ts) | WebAudio engine — 10 sound packs, 9 events || [client/src/hooks/useSound.ts](../client/src/hooks/useSound.ts) | `useSoundSync()` + `useSound()` React glue |@@ -534,7 +534,7 @@ cd client && npx eslint <files>| [client/src/components/SvgFilterDefs.tsx](../client/src/components/SvgFilterDefs.tsx) | Inline `<svg><defs>` carrying `#ink-blot` / `#noise` / `#crt` filter IDs — Wave 25 || [client/src/components/ui/DropCap.tsx](../client/src/components/ui/DropCap.tsx) | `v-display drop-cap` wrapper for Renaissance ornament rule — Wave 25 || [client/src/types/NodeTypes.ts](../client/src/types/NodeTypes.ts) | `NodeStyle` helper type for inline `--*` custom props — Wave 21 |-| [client/index.html](../client/index.html) | Google Fonts loading (all 10 themes' typefaces) |+| [client/index.html](../client/index.html) | Google Fonts loading (all 12 themes' typefaces) || `design_handoff_machinaos_themes/` (external design handoff, not in repo) | Reference HTML mocks + token spec || `design_handoff_machinaos_themes/app/icons.js` (external) | Upstream icon source (ported by Wave 23) || `design_handoff_machinaos_themes/MIGRATION_PLAYBOOK.md` (external) | Upstream recipe — all originally-deferred items now landed (W14–W25) |diff --git a/server/core/config.py b/server/core/config.pyindex fd7a127a..0ba1f9ff 100644--- a/server/core/config.py+++ b/server/core/config.py@@ -101,11 +101,16 @@ class Settings(BaseSettings):# task queue) alongside the TemporalWorkerManager, and# MachinaWorkflow._resolve_activity returns cls.task_queue so# per-type activities land on their specialised pool worker.+ #+ # Wave 16.4: default flipped to True — per-queue routing is the+ # production default (LLM / browser / code-exec / REST workloads+ # get isolated slot pools instead of sharing one 100-slot queue).# Rollback: TEMPORAL_WORKER_POOL_ENABLED=false + restart — the pool# stops and every activity routes back to the single manager worker- # on "machina-tasks". Default off until the 16.4 stability window.+ # on "machina-tasks". Opt out via the env var, never by reverting+ # this default (locked by test_task_queue_coverage.py).temporal_worker_pool_enabled: bool = Field(- default=False,+ default=True,env="TEMPORAL_WORKER_POOL_ENABLED",)# Wave 17.4: deployment topology hint. Drives worker identitydiff --git a/server/main.py b/server/main.pyindex e09b8175..364de950 100644--- a/server/main.py+++ b/server/main.py@@ -351,10 +351,11 @@ async def lifespan(app: FastAPI):await worker_manager.start()app.state.temporal_worker_manager = worker_manager- # Wave 16.2: per-queue activity worker pool. Gated —- # inert until TEMPORAL_WORKER_POOL_ENABLED=true. Starts- # AFTER the manager so workflow registration is in- # place before specialised activity workers poll.+ # Wave 16: per-queue activity worker pool (default-on+ # since 16.4; TEMPORAL_WORKER_POOL_ENABLED=false is+ # the rollback channel). Starts AFTER the manager so+ # workflow registration is in place before+ # specialised activity workers poll.if settings.temporal_worker_pool_enabled:from services.temporal.worker import TemporalWorkerPooldiff --git a/server/services/temporal/worker.py b/server/services/temporal/worker.pyindex 62c23d21..dd6e4c99 100644--- a/server/services/temporal/worker.py+++ b/server/services/temporal/worker.py@@ -466,7 +466,6 @@ class TemporalWorkerPool:worker_kwargs: dict = dict(task_queue=queue,activities=activities,- max_concurrent_workflow_tasks=10,graceful_shutdown_timeout=_graceful_shutdown_timeout(),identity=_worker_identity(queue),interceptors=[ObservabilityWorkerInterceptor()],@@ -477,13 +476,19 @@ class TemporalWorkerPool:# lower task volume than the manager's default queue.activity_task_poller_behavior=PollerBehaviorAutoscaling(initial=1, minimum=1, maximum=5),)- # Wave 18.4: the Worker rejects tuner + max_concurrent_activities- # together — resource-tuned queues get the tuner, the rest keep- # fixed sizing.+ # Wave 18.4: the Worker rejects `tuner` alongside ANY of+ # `max_concurrent_workflow_tasks` / `max_concurrent_activities` /+ # `max_concurrent_local_activities` / `max_concurrent_nexus_tasks`+ # — the SDK enforces "size via tuner OR via max_concurrent_*,+ # never both" (temporalio.worker.Worker.__init__). Resource-tuned+ # queues get the tuner (which already carries a+ # FixedSizeSlotSupplier(10) for workflow slots via+ # `_tuner_for`); the rest keep fixed sizing.if tuner is not None:worker_kwargs["tuner"] = tunerelse:worker_kwargs["max_concurrent_activities"] = concurrency+ worker_kwargs["max_concurrent_workflow_tasks"] = 10worker = Worker(self.client, **worker_kwargs)task = asyncio.create_task(worker.run(), name=f"worker-{queue}")diff --git a/server/tests/test_task_queue_coverage.py b/server/tests/test_task_queue_coverage.pyindex 0dcfaea5..30685d4c 100644--- a/server/tests/test_task_queue_coverage.py+++ b/server/tests/test_task_queue_coverage.py@@ -87,3 +87,41 @@ class TestTaskQueueCoverage:"update docs-internal/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md ""if a queue was deliberately removed.")+++class TestWorkerPoolDefaultOn:+ """Wave 16.4: per-queue routing is the production default.++ Source-introspection rather than ``Settings()`` instantiation because+ conftest stubs ``core.config.Settings`` as a MagicMock for the test+ session — same pattern as+ ``test_event_framework_phase_a.py::test_event_framework_enabled_defaults_true``.+ """++ def test_temporal_worker_pool_enabled_defaults_true(self):+ import re+ from pathlib import Path++ config_src = (Path(__file__).parent.parent / "core" / "config.py").read_text(encoding="utf-8")+ default_true = re.compile(+ r"default=True[\s,]+env=\"TEMPORAL_WORKER_POOL_ENABLED\"",+ re.DOTALL,+ )+ default_false = re.compile(+ r"default=False[\s,]+env=\"TEMPORAL_WORKER_POOL_ENABLED\"",+ re.DOTALL,+ )+ assert default_true.search(config_src) is not None, (+ "temporal_worker_pool_enabled default flipped to True in "+ "Wave 16.4. The declaration "+ '``Field(default=True, env="TEMPORAL_WORKER_POOL_ENABLED")`` '+ "is missing from core/config.py. Opt out via "+ "TEMPORAL_WORKER_POOL_ENABLED=false in .env — not by "+ "reverting the default."+ )+ assert default_false.search(config_src) is None, (+ "temporal_worker_pool_enabled reverted to default=False. "+ "Wave 16.4 flipped the default to True. Use the env var "+ "(TEMPORAL_WORKER_POOL_ENABLED=false) for opt-out, not a "+ "code revert."+ )