Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
docs(temporal): worker tuning recipe + RFC status table (Wave 18.5)
trohitg9bafa3467854
diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.mdindex 86e5140a..59640901 100644--- a/docs-internal/TEMPORAL_ARCHITECTURE.md+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md@@ -272,9 +272,9 @@ Benefits:Add more workers = handle more concurrent nodes```-### Specialized Worker Pools (Future)+### Specialized Worker Pools (Wave 16 — gated by `TEMPORAL_WORKER_POOL_ENABLED`)-F4.A laid the infrastructure (per-type dispatch with `task_queue=cls.task_queue` on `BaseNode`) but the single `TemporalWorkerManager` still polls one queue. Wiring `TemporalWorkerPool` in `main.py` is the deferred follow-up:+`TemporalWorkerPool` is wired into the `main.py` lifespan (starts right after `TemporalWorkerManager`, stops before it). When `TEMPORAL_WORKER_POOL_ENABLED=true`, one activity-only Worker per plugin-declared queue polls its specialised queue, and `MachinaWorkflow._resolve_activity` returns `(activity_name, cls.task_queue)` so per-type activities land there. Flag off (default until the stability window) = queue `None` = everything routes to the single manager worker on `machina-tasks`. **The flag is the rollback channel.**```Queue: rest-api Queue: ai-heavy Queue: code-exec@@ -285,25 +285,40 @@ Queue: rest-api Queue: ai-heavy Queue: code-exec| brave | | chatA | | js exec || twitter | | deepA | | ts exec |+---------+ +---------+ +---------+--Plugin classes already declare cls.task_queue. When the pool wires-in, `MachinaWorkflow._resolve_activity` starts returning-(activity_name, cls.task_queue) instead of (activity_name, None).```-Per-queue defaults (concurrency caps in `services/temporal/worker.py:TemporalWorkerPool.DEFAULT_CONCURRENCY`):+Pre-flight invariants live in `server/tests/test_task_queue_coverage.py` (every queue populated, every plugin queue declared, `DEFAULT_CONCURRENCY` covers every queue — an off-registry queue would hang activities at schedule-to-start).-| Queue | Default concurrency | Use case |-|---|---|---|-| `machina-default` | 20 | Catch-all |-| `rest-api` | 50 | Lightweight HTTP calls |-| `ai-heavy` | 4 | LLM agent loops |-| `code-exec` | 10 | Python / JS / TS sandboxes |-| `triggers-poll` | 100 | Gmail polling, etc. |-| `triggers-event` | 100 | Event-waiter triggers |-| `android` | 10 | ADB / relay ops |-| `browser` | 4 | Playwright / agent-browser |-| `messaging` | 20 | WhatsApp / Telegram |+### Worker Performance Tuning (Waves 17-18)++All knobs live in `services/temporal/worker.py`; every default is env-overridable. `DEPLOYMENT_MODE` (`local` / `cloud` / `self_hosted`) is the topology hint (`core/config.py`).++| Queue | Concurrency (cloud) | Concurrency (local = halved, floor 1) | Rate limit (act/s) | Slot sizing |+|---|---|---|---|---|+| `machina-default` | 20 | 10 | — | fixed |+| `rest-api` | 50 | 25 | 100 | fixed |+| `ai-heavy` | 4 | 2 | 60 | **resource-based** (80% CPU+mem target) |+| `code-exec` | 10 | 5 | — | fixed |+| `triggers-poll` | 100 | 50 | — | fixed |+| `triggers-event` | 100 | 50 | — | fixed |+| `android` | 10 | 5 | — | fixed |+| `browser` | 4 | 2 | 10 | **resource-based** (80% CPU+mem target) |+| `messaging` | 20 | 10 | 20 | fixed |++Env overrides: `TEMPORAL_<QUEUE>_CONCURRENCY` (int) and `TEMPORAL_<QUEUE>_RATE_LIMIT` (float/sec) always win over the mode-scaled defaults.++- **Worker identity** (Wave 17.4): `machina-<queue>-<deployment_mode>` — readable Workers tab in the Temporal Web UI.+- **Sticky workflow cache** (Wave 18.2, manager worker only): `max_cached_workflows` = local 50 / cloud 500 / self_hosted 100. Cached workflows skip Event-History replay; evictions are a latency cost, not an error.+- **Poller autoscaling** (Wave 18.3): `PollerBehaviorAutoscaling` — manager activity pollers 1-10 (initial 2), workflow pollers 1-20 (initial 2); pool workers 1-5 (initial 1). Invariant per the [worker-performance docs](https://docs.temporal.io/develop/worker-performance): pollers stay below executor slots.+- **Resource-based slot supplier** (Wave 18.4): `ai-heavy` + `browser` (unpredictable workloads) use `ResourceBasedSlotSupplier` targeting 80% host CPU + memory, `minimum_slots=1`, `maximum_slots` = the per-queue concurrency. Requires `temporalio>=1.25.0` (pinned).+- **Observability interceptors** (Wave 17.3, `services/temporal/_interceptors.py`): `activity_retry` WARN when `activity.info().attempt > 1` — the "worker died and Temporal re-dispatched" signal; `workflow_start` guarded by `workflow.unsafe.is_replaying()`.+- **Periodic activity heartbeat** (Wave 17.6, `plugin/base.py::as_activity`): 30s background beat during long bodies so a laptop-sleep crash is detected within one `heartbeat_timeout` (2 min) instead of `start_to_close` (10 min).+- **Cron catch-up bound** (Wave 17.1, `services/temporal/schedules.py`): `SchedulePolicy(catchup_window=24h)` + `SKIP` overlap — a laptop offline a week does not replay 168 hourly ticks on wake.+- **One-shot LLM-step retry** (Wave 17.2): `LLM_STEP_RETRY(maximum_attempts=1)` on `agent.execute_llm_step.v1` — LLM calls are not idempotent; the workflow surfaces the failure instead of silently re-billing the prompt. `agent.refresh_tools.v1` deliberately keeps 3 attempts (idempotent canvas rebuild).++**Metric watchlist** (Temporal Web UI / metrics endpoint): `schedule_to_start_latency` (elevated = raise pollers or slots), `worker_task_slots_available` (0 = raise concurrency or add hosts), `poll_success_rate` (target >= 90%), `sticky_cache_evictions` (persistent growth = raise `max_cached_workflows`).++**Tuning order** (per the worker-performance docs): host provisioning -> executor slots -> poller counts -> rate limits.### Agent-as-child-workflow (F4.B)diff --git a/docs-internal/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md b/docs-internal/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.mdindex 184667c4..ab38aa62 100644--- a/docs-internal/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md+++ b/docs-internal/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md@@ -12,8 +12,20 @@| 15.1 — drop `WorkflowEvent.connection_status` | ✅ shipped `8bc96c1` || 15.2 — retire APScheduler cron stack | ✅ shipped `eda3a20` (-272 LOC; Temporal hard-required for cron) || 15.3 — retire Redis-Streams branch (D2b) | ✅ shipped `76576b1` (-347/+57 LOC; `dispatch_async` gone, `set_cache_service` → `capture_main_loop`, cache consumer-group methods gone; `_streams_available` flag KEPT — the surviving `stream_add` guards on it, a detail §4 originally missed) |-| 15.4 — docs sweep | ✅ shipped (this commit) |-| 16.x / 17.x / 18.x | ⏳ in progress |+| 15.4 — docs sweep | ✅ shipped `1af2e00` |+| 16.1 — task-queue coverage invariants | ✅ shipped `3262fb0` |+| 16.2 — pool wired into lifespan (gated off) | ✅ shipped `18df875` |+| 16.3 — `_resolve_activity` returns `cls.task_queue` when pool on | ✅ shipped `d4b55c4` |+| 16.4 — default flip | ⏳ parked for one-release stability window (`TEMPORAL_WORKER_POOL_ENABLED` stays default-false; flip + `test_temporal_worker_pool_enabled_defaults_true` invariant afterwards) |+| 17.1 — cron `catchup_window=24h` | ✅ shipped `5bf9000` (note: `catchup_window` is a `SchedulePolicy` field in the Python SDK, not `ScheduleSpec` as §6 originally said) |+| 17.2 — one-shot LLM-step retry | ✅ shipped `03c2bf6` (`LLM_STEP_RETRY` constant; refresh_tools keeps 3 attempts, documented at callsite) |+| 17.3 — observability interceptors | ✅ shipped `9ecd743` (replay guard via `workflow.unsafe.is_replaying()`) |+| 17.4 — `DEPLOYMENT_MODE` + worker identity | ✅ shipped `a8ab832` |+| 17.5 — mode-aware concurrency defaults | ✅ shipped `9b79e1a` |+| 17.6 — periodic heartbeat during long bodies | ✅ shipped `dd32d50` |+| 18.1-18.3 — rate limits + sticky cache + poller autoscaling | ✅ shipped `7842452` (temporalio floor bumped to >=1.25.0) |+| 18.4 — resource-based slot supplier (ai-heavy + browser) | ✅ shipped `aa094be` |+| 18.5 — tuning-recipe docs | ✅ shipped (this commit) |---