Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
refactor(temporal): wire UI port, terminate-on-startup, Win port-kill graceful
trohitg7836499b26bc
diff --git a/.env.template b/.env.templateindex 81acde84..615410e8 100644--- a/.env.template+++ b/.env.template@@ -64,8 +64,20 @@ TEMPORAL_FRONTEND_GRPC_PORT=7233# Browse to http://<host>:<TEMPORAL_UI_PORT>/.TEMPORAL_UI_PORT=8080# SQLite db file ``temporal server start-dev --db-filename`` opens.-# Relative to DATA_DIR unless absolute.-TEMPORAL_SQLITE_PATH=temporal/temporal.db+# Relative to DATA_DIR (= ~/.machina/) unless absolute. Flat under+# DATA_DIR like ``credentials.db`` / ``workflow.db``. History is+# always preserved across restarts (Temporal UI keeps showing every+# workflow that ever ran); see ``TEMPORAL_TERMINATE_RUNNING_ON_STARTUP``+# below for the resumption knob.+TEMPORAL_SQLITE_PATH=temporal.db+# On server boot, terminate every workflow still in ``Running`` state.+# History is preserved (workflows show as ``Terminated`` in the UI,+# NOT deleted) but resumption is disabled, because MachinaOS's+# ``DeploymentManager`` has no boot-time reconcile against Temporal+# Visibility yet — resumed workflows would otherwise keep executing+# but be invisible to the MachinaOS UI. Flip to ``false`` once the+# reconcile lands to re-enable durable resumption.+TEMPORAL_TERMINATE_RUNNING_ON_STARTUP=true# Graceful shutdown window for the Temporal child process.TEMPORAL_GRACEFUL_SHUTDOWN_SECONDS=30# Supervisor readiness-probe window. The CLI binary may need todiff --git a/CLAUDE.md b/CLAUDE.mdindex 16f4b106..1e8f0208 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -1632,45 +1632,45 @@ Workflows execute via Temporal for durability and horizontal scaling. Three dispFull architecture, dispatch matrix, per-node lifecycle, heartbeat semantics, the 14 migrating agent types + 2 bypass agents (rlm_agent / claude_code_agent), the 6 F4.B agent activities (`prepare_payload.v1` / `execute_llm_step.v1` / `persist_turn.v1` / `compact_memory.v1` / `store_output.v1` / `broadcast_progress.v1`), and the future `TemporalWorkerPool` per-queue routing live in [docs-internal/TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md). Tool-call dispatch under F4.A documented at [docs-internal/tool_building_pipeline.md §9](./docs-internal/tool_building_pipeline.md).-**Configuration** (`.env`):+**Configuration** (`.env.template` — canonical defaults; `.env` layers user overrides):```env+TEMPORAL_ENABLED=trueTEMPORAL_SERVER_ADDRESS=localhost:7233TEMPORAL_NAMESPACE=defaultTEMPORAL_TASK_QUEUE=machina-tasksTEMPORAL_PER_TYPE_DISPATCH=true # F4.A flag — per-type activity dispatchTEMPORAL_AGENT_WORKFLOW_ENABLED=true # F4.B flag — agent-as-child-workflow+TEMPORAL_FRONTEND_GRPC_PORT=7233+TEMPORAL_UI_PORT=8080 # Web UI (override of CLI default 8233)+TEMPORAL_SQLITE_PATH=temporal.db # Flat under DATA_DIR (~/.machina/temporal.db)+TEMPORAL_GRACEFUL_SHUTDOWN_SECONDS=30+TEMPORAL_SERVER_READY_TIMEOUT_SECONDS=120+TEMPORAL_TERMINATE_RUNNING_ON_STARTUP=true # see "Resumption" below```-**Temporal binary + persistence** are managed in-process by the plugin-folder pattern under [`server/services/temporal/`](./server/services/temporal/) — the `temporal-server` npm package is no longer used. Binary distribution + lifecycle = standard libs, zero custom code:+**Temporal binary + persistence** are managed in-process by the plugin-folder pattern under [`server/services/temporal/`](./server/services/temporal/). Single supervised process — the official `temporal` CLI's `server start-dev` mode (per [docs.temporal.io/develop/python/set-up-your-local-python](https://docs.temporal.io/develop/python/set-up-your-local-python)):-- **`_install.py`** — [`pooch`](https://pypi.org/project/pooch/) downloads `temporal-server` + `temporal-sql-tool` from `temporalio/temporal` GitHub releases (SHA-256 verified, XDG-cached cross-platform). Falls back to `shutil.which("temporal-server")` when a system install is on PATH (brew / scoop / cargo).-- **`_runtime.py`** — `PostgresRuntime` (subclasses `BaseSupervisor`; wraps [`pgserver`](https://pypi.org/project/pgserver/) which bundles PostgreSQL 16.2 binaries cross-platform) + `TemporalServerRuntime` (subclasses `BaseProcessSupervisor`; runs the binary with a Python-rendered YAML config). Same singleton + lifecycle surface every other supervised service uses ([`nodes/whatsapp/_runtime.py`](./server/nodes/whatsapp/_runtime.py) is the template).-- **`_config.py`** — `pyyaml.safe_dump` of the Temporal cluster config; `temporal-sql-tool create-database` + `setup-schema` + `update-schema` for both `temporal` and `temporal_visibility` Postgres databases (idempotent).+- **`_install.py`** — [`pooch`](https://pypi.org/project/pooch/) downloads the official `temporal` CLI archive from `https://temporal.download/cli/archive/latest?platform=<os>&arch=<arch>`. Cross-platform (Windows zip / macOS+Linux tar.gz), XDG-cached. Standalone entry: `python -m services.temporal._install` (invoked by `machina build` step [6/6]).+- **`_runtime.py`** — `TemporalServerRuntime(BaseProcessSupervisor)` spawns `temporal server start-dev` with `--port`, `--ui-port`, `--db-filename`, `--namespace`. Same singleton + lifecycle surface every other supervised service uses ([`nodes/whatsapp/_runtime.py`](./server/nodes/whatsapp/_runtime.py) is the template).- **`_handlers.py` + `_refresh.py`** — `temporal_status` / `_start` / `_stop` WS commands and the WS-connect status snapshot, registered through `services.ws_handler_registry` and `services.status_broadcaster.register_service_refresh`.--**Backend selection** via `TEMPORAL_BACKEND` env var:-- `sqlite` (dev default) — single ServiceSpec runs `temporal api` against an in-memory / file-based SQLite. Same dev experience as before.-- `postgres` (prod) — two ServiceSpecs run via the `_supervised_runtime.py` shim: `temporal-postgres` (pgserver) then `temporal-server` (Temporal binary). The supervisor's TCP readiness probe orders postgres → temporal automatically.+- **`client.py`** — `TemporalClientWrapper.connect()` + `terminate_running_workflows()` (resumption toggle, see below).**Ports:**-| Service | Port | URL |-|-----------|------|------------------------|-| gRPC | 7233 | localhost:7233 |-| HTTP API | 8233 | http://localhost:8233 |-| Web UI | 8080 | http://localhost:8080 |-| Metrics | 9090 | http://localhost:9090 |-| Postgres | dyn | postgresql://...:NNNNN (`postgres` backend; pgserver picks a free port — read via `get_postgres_runtime().uri`) |--**CLI commands** (still available when `temporal` is on PATH or pooch-installed):-```bash-temporal-server start --config <yaml> # what TemporalServerRuntime invokes-temporal api # what the sqlite ServiceSpec invokes-temporal --version-```+| Service | Port | URL |+|---------|------|------------------------|+| gRPC | 7233 | localhost:7233 |+| Web UI | 8080 | http://localhost:8080 |++Both ports are bound by the same `temporal.exe` process (per docs.temporal.io/cli/server — "all running in a single process").++**Resumption** — controlled by `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` (default `true`):+- Temporal SQLite is always persisted (history preserved across restarts; UI keeps showing every workflow that ever ran).+- On server boot, after the Temporal client connects but BEFORE the worker starts, `TemporalClientWrapper.terminate_running_workflows()` queries Visibility for every `Running` workflow and calls `handle.terminate(reason="MachinaOS startup: auto-resumption disabled")`. Workflows show as `Terminated` (not deleted) in the Temporal UI.+- Why: `DeploymentManager` has no boot-time reconcile against Temporal Visibility yet — resumed workflows would otherwise keep executing but be invisible to the MachinaOS UI. Flip to `false` once the reconcile lands.-**Start script integration** ([`cli/commands/start.py`](./cli/commands/start.py) + [`cli/commands/dev.py`](./cli/commands/dev.py)): both call `temporal_specs(root, cfg)` ([`cli/commands/_temporal_specs.py`](./cli/commands/_temporal_specs.py)) which returns the ServiceSpec for the supervised Temporal dev server. The supervised-runtime shim lives at [`server/services/temporal/_supervised_runtime.py`](./server/services/temporal/_supervised_runtime.py) (relocated from `cli/commands/` so the spawned `uv run python -m services.temporal._supervised_runtime` resolves out of the workspace `.venv` without any cross-tree path plumbing). The supervisor's `_temporal_running()` probe still short-circuits if Temporal is already running on 7233 (lets you run `temporal-server start` standalone for debugging).+**Start script integration** ([`cli/commands/start.py`](./cli/commands/start.py) + [`cli/commands/dev.py`](./cli/commands/dev.py)): both call `temporal_specs(root, cfg)` ([`cli/commands/_temporal_specs.py`](./cli/commands/_temporal_specs.py)) which returns the single ServiceSpec for the supervised Temporal dev server. The supervised-runtime shim lives at [`server/services/temporal/_supervised_runtime.py`](./server/services/temporal/_supervised_runtime.py) so the spawned `uv run python -m services.temporal._supervised_runtime` resolves out of the workspace `.venv` without any cross-tree path plumbing. The supervisor's `_temporal_running()` probe short-circuits if Temporal is already running on 7233 (lets you run `temporal server start-dev` standalone for debugging).-**Docker**: Optional. The Python supervisor handles cross-platform lifecycle without Docker; production-scale Temporal+Postgres runs identically on Linux / macOS / Windows via pip-installed binaries.+**Docker**: Not used. The Python supervisor handles cross-platform lifecycle via the pooch-downloaded CLI — single binary, single process, single SQLite file.**Execution Routing** (`workflow.py`):1. If `TEMPORAL_ENABLED=true` and Temporal configured → `_execute_temporal()`diff --git a/cli/ports.py b/cli/ports.pyindex 33954dd8..e89497df 100644--- a/cli/ports.py+++ b/cli/ports.py@@ -7,13 +7,22 @@ same battle-tested ``psutil`` paths that are already in production.from __future__ import annotationsimport os+import signalimport sysimport subprocess+import timefrom dataclasses import dataclassimport psutil+# Post-kill grace before re-checking the port. Windows can lag a few+# hundred ms releasing the listener socket after the bound process+# dies — without this, ``kill_port`` reports the port still in use+# even though the kill succeeded.+_POST_KILL_RECHECK_DELAY = 0.5++@dataclassclass KillResult:port: int@@ -50,10 +59,30 @@ def find_pids_by_port(port: int) -> set[int]:def kill_pid(pid: int, *, graceful_timeout: float = 3.0) -> bool:- """Terminate ``pid`` gracefully, then force-kill on timeout."""+ """Terminate ``pid`` gracefully, then force-kill on timeout.++ Windows: send ``CTRL_BREAK_EVENT`` first so daemons spawned with+ ``CREATE_NEW_PROCESS_GROUP`` (the supervisor's children — see+ ``cli/tree.py:new_session_kwargs``) get a real shutdown signal and+ can release listener sockets cleanly. ``proc.terminate()``+ (= ``TerminateProcess``) is the fallback for processes that weren't+ spawned with a process group — equivalent to SIGKILL, leaves the+ OS holding sockets briefly. Same pattern as+ ``cli/supervisor.py:_stop_proc``.++ POSIX: plain ``proc.terminate()`` (SIGTERM).+ """try:proc = psutil.Process(pid)- proc.terminate()+ if sys.platform == "win32":+ try:+ os.kill(pid, signal.CTRL_BREAK_EVENT)+ except (OSError, ProcessLookupError):+ # Target wasn't in its own process group — fall back+ # to TerminateProcess via psutil.terminate.+ proc.terminate()+ else:+ proc.terminate()try:proc.wait(timeout=graceful_timeout)except psutil.TimeoutExpired:@@ -64,7 +93,15 @@ def kill_pid(pid: int, *, graceful_timeout: float = 3.0) -> bool:def kill_port(port: int) -> KillResult:- """Kill anything listening on ``port`` and report whether the port is free."""+ """Kill anything listening on ``port`` and report whether the port is free.++ Post-kill recheck sleeps ``_POST_KILL_RECHECK_DELAY`` because Windows+ can lag a few hundred ms releasing the listener socket after the+ bound process dies — without this, ``temporal server start-dev``'s+ UI port (8080) frequently re-reports as in-use immediately after the+ gRPC port kill, even though the kill on temporal.exe (one process+ binds both ports — per docs.temporal.io/cli/server) succeeded.+ """my_pid = os.getpid()killed: list[int] = []for pid in find_pids_by_port(port):@@ -72,6 +109,8 @@ def kill_port(port: int) -> KillResult:continueif kill_pid(pid):killed.append(pid)+ if killed:+ time.sleep(_POST_KILL_RECHECK_DELAY)port_free = not find_pids_by_port(port)return KillResult(port=port, killed_pids=killed, port_free=port_free)diff --git a/docs-internal/SCRIPTS.md b/docs-internal/SCRIPTS.mdindex 4c6c9a4d..642b1d5f 100644--- a/docs-internal/SCRIPTS.md+++ b/docs-internal/SCRIPTS.md@@ -54,11 +54,10 @@ Run with `npm run <script>` from the project root.|--------|---------|-------------|| `client:start` | `cd client && npm run start` | Start React frontend (Vite) || `python:start` | `cd server && uv run uvicorn main:app ...` | Start Python backend |-| `temporal:start` | `temporal-server api` | Start Temporal server (foreground) |-| `temporal:stop` | `temporal-server stop` | Stop Temporal server |-| `temporal:status` | `temporal-server status` | Check Temporal server status || `temporal:worker` | `cd server && uv run python -m services.temporal.worker` | Start standalone Temporal worker |+Temporal server lifecycle is managed by `machina start` / `machina dev` / `machina stop` directly (see [Temporal Architecture](./TEMPORAL_ARCHITECTURE.md)). The official `temporal` CLI is downloaded by `pooch` to a per-OS cache during `machina build` and spawned as a supervised subprocess.+### Docker Scripts (Development)| Script | Command | Description |@@ -95,17 +94,16 @@ Run with `npm run <script>` from the project root.Cross-platform start script that runs all services concurrently.-**What it does:**+**What it does** (mirrored by the canonical `machina start` / `machina dev` CLI commands):1. Validates build artifacts exist2. Creates `.env` from `.env.template` if missing-3. Checks if Temporal is already running via `temporal-server status` CLI-4. Frees configured app ports (client, backend, WhatsApp, Node.js executor)-5. Starts services via `concurrently` with `--kill-others`:+3. Frees configured app ports (client, backend, WhatsApp, Node.js executor, Temporal gRPC, Temporal UI)+4. Spawns each service in its own process group (Windows: `CREATE_NEW_PROCESS_GROUP` for graceful `CTRL_BREAK_EVENT` shutdown):- Static client server (`serve-client.js`)- Python backend (uvicorn) -- supervises the edgymeow Go binary lazily via [server/nodes/whatsapp/_runtime.py](../server/nodes/whatsapp/_runtime.py)- - Temporal server via `npm:temporal:start` (unless already running)+ - Temporal dev server -- the supervised-runtime shim ([server/services/temporal/_supervised_runtime.py](../server/services/temporal/_supervised_runtime.py)) spawning `temporal server start-dev`-**Temporal handling:** If `temporal-server status` reports the server is already running, Temporal is skipped in the concurrently service list. This prevents `temporal-server api` from exiting immediately with "Already running", which would trigger `--kill-others` and cascade-kill all services.+**Temporal handling:** The supervisor's TCP readiness probe on 7233 short-circuits if Temporal is already running, so launching `machina start` against a pre-existing Temporal works without conflict.**Ports (configurable in .env):**- `VITE_CLIENT_PORT` - Frontend (default: 3000)diff --git a/docs-internal/SETUP.md b/docs-internal/SETUP.mdindex e40b1f1f..7cc9bd99 100644--- a/docs-internal/SETUP.md+++ b/docs-internal/SETUP.md@@ -80,10 +80,11 @@ Services will be available at:### Temporal Server (Distributed Execution)- Provides durable workflow execution with per-node retry and horizontal scaling-- Installed globally: `npm install -g temporal-server`-- Managed via CLI: `temporal-server start/stop/status`-- Ports: gRPC 7233, HTTP API 8233, Web UI 8080, Metrics 9090+- Official `temporal` CLI downloaded by `pooch` from `https://temporal.download/cli/archive/latest` on `machina build` (or first `machina start`)+- Supervised by MachinaOS as `temporal server start-dev` — single process, SQLite at `~/.machina/temporal.db`+- Ports: gRPC 7233, Web UI 8080- Embedded worker runs inside Python backend (`TemporalWorkerManager` in `main.py`)+- Workflow auto-resumption disabled at startup (history preserved); see `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP`- See [Temporal Architecture](./TEMPORAL_ARCHITECTURE.md) and [CLI Services Guide](./cli_services_integration.md)### Database (SQLite)diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.mdindex 6da655a8..851e1866 100644--- a/docs-internal/TEMPORAL_ARCHITECTURE.md+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md@@ -435,79 +435,55 @@ client = await Client.connect(server_address, namespace=namespace, runtime=runti## Server Management-The Temporal binary + persistence layer are managed in-process by the plugin-folder pattern at [`server/services/temporal/`](../server/services/temporal/) — the `temporal-server` npm package has been retired. Six sibling files (`_install.py`, `_runtime.py`, `_config.py`, `_handlers.py`, `_refresh.py`, plus `__init__.py` for registry wiring) match the [Wave 11 plugin-folder pattern](./plugin_system.md#self-contained-plugin-folders) that `server/nodes/whatsapp/` uses for its Go binary.+The Temporal binary + persistence are managed in-process by the plugin-folder pattern at [`server/services/temporal/`](../server/services/temporal/). Single supervised process — the official `temporal` CLI's `server start-dev` mode, per [docs.temporal.io/develop/python/set-up-your-local-python](https://docs.temporal.io/develop/python/set-up-your-local-python). Five sibling files (`_install.py`, `_runtime.py`, `_handlers.py`, `_refresh.py`, `client.py`, plus `__init__.py` for registry wiring) match the [Wave 11 plugin-folder pattern](./plugin_system.md#self-contained-plugin-folders) that `server/nodes/whatsapp/` uses for its Go binary.-**Persistence backends** (selected by `TEMPORAL_BACKEND` env var):--| Backend | When | What runs |-|---|---|---|-| `sqlite` (dev default) | `TEMPORAL_BACKEND=sqlite` | Single supervised process running `temporal api` (the binary's built-in `start-dev` mode). In-memory or file-based SQLite. Zero deps beyond the Temporal binary. |-| `postgres` (prod) | `TEMPORAL_BACKEND=postgres` | Two supervised processes: `temporal-postgres` (pgserver-managed PostgreSQL 16.2) then `temporal-server` (binary with YAML config pointing at the Postgres). Ordered automatically by the supervisor's TCP readiness probe. |+**What runs**: one process — `temporal server start-dev --port 7233 --ui-port 8080 --db-filename ~/.machina/temporal.db --namespace default`. Both gRPC + Web UI bind to the same process (per docs.temporal.io/cli/server — "all running in a single process"). SQLite-backed durability; history persisted across restarts.**Modern libs doing the heavy lifting** (zero custom infrastructure code):-- **[`pooch`](https://pypi.org/project/pooch/)** — `services/temporal/_install.py` downloads `temporal-server` + `temporal-sql-tool` from `temporalio/temporal` GitHub releases. SHA-256 verified (digests pinned in `_install.py:_CHECKSUMS`); XDG-cached via `pooch.os_cache()`; tar.gz / zip extraction handled by `pooch.Untar()` / `pooch.Unzip()`. Falls back to `shutil.which("temporal-server")` for system installs. ~30 LOC total.-- **[`pgserver`](https://pypi.org/project/pgserver/)** — bundles PostgreSQL 16.2 binaries cross-platform (Linux / macOS / Windows, x86_64 + ARM64) via pip. Single API: `pgserver.get_server(data_dir)` returns a managed instance. `PostgresRuntime` wraps its lifecycle in `BaseSupervisor.start()` / `.stop()`.-- **`BaseProcessSupervisor` + `BaseSupervisor`** (`server/services/_supervisor/`) — the existing in-house supervisor base classes that `server/nodes/whatsapp/_runtime.py` already uses. Provides cross-platform signal handling (POSIX setsid + Windows Job Objects + psutil tree-kill), restart policy via tenacity, log draining, status snapshots. We subclass both — zero custom supervisor logic.+- **[`pooch`](https://pypi.org/project/pooch/)** — `services/temporal/_install.py` downloads the official `temporal` CLI archive from `https://temporal.download/cli/archive/latest?platform=<os>&arch=<arch>`. Cross-platform (Windows zip / macOS+Linux tar.gz), XDG-cached via `pooch.os_cache("machinaos-temporal")`. Standalone entry: `python -m services.temporal._install` (invoked by `machina build` step [6/6] so first `machina start` doesn't pay the ~114 MB download).+- **`BaseProcessSupervisor` + `BaseSupervisor`** (`server/services/_supervisor/`) — the in-house supervisor base classes that `server/nodes/whatsapp/_runtime.py` also uses. Provides cross-platform signal handling (POSIX `setsid` + Windows Job Objects + `CREATE_NEW_PROCESS_GROUP` for graceful `CTRL_BREAK_EVENT` shutdown), restart policy via tenacity, log draining, status snapshots. We subclass both — zero custom supervisor logic.-**ServiceSpec wiring**: [`cli/commands/_temporal_specs.py`](../cli/commands/_temporal_specs.py) returns the ServiceSpec for the supervised Temporal dev server (env reads happen inside `temporal_specs()` at call time, not module load -- safe to import before `cli.config.load_config()` runs). Both `start.py` and `dev.py` call it. The generic supervised-runtime shim lives at [`server/services/temporal/_supervised_runtime.py`](../server/services/temporal/_supervised_runtime.py) (~50 LOC) -- colocated with the factory it imports so the spawned `uv run python -m services.temporal._supervised_runtime services.temporal._runtime:get_temporal_server_runtime` resolves out of the workspace `.venv` without any cross-tree path plumbing.+**ServiceSpec wiring**: [`cli/commands/_temporal_specs.py`](../cli/commands/_temporal_specs.py) returns one ServiceSpec for the supervised Temporal dev server (env reads happen inside `temporal_specs()` at call time, safe to import before `cli.config.load_config()` runs). Both `start.py` and `dev.py` call it. The generic supervised-runtime shim lives at [`server/services/temporal/_supervised_runtime.py`](../server/services/temporal/_supervised_runtime.py) so the spawned `uv run python -m services.temporal._supervised_runtime services.temporal._runtime:get_temporal_server_runtime` resolves out of the workspace `.venv` without any cross-tree path plumbing.**WS surface**: `_handlers.py` registers `temporal_status` / `temporal_start` / `temporal_stop` via `services.ws_handler_registry.register_ws_handlers`. `_refresh.py` registers a WS-connect callback via `services.status_broadcaster.register_service_refresh` so the FE health indicator stays current.-**Port management**: Temporal owns ports 7233 / 8233 / 8080 / 9090. Postgres picks a dynamic free port (read via `get_postgres_runtime().uri`). Both are excluded from MachinaOS `allPorts` and from port-freeing in `scripts/start.js`.+**Resumption toggle**: [`TemporalClientWrapper.terminate_running_workflows`](../server/services/temporal/client.py) — runs once at server lifespan startup, between client connect and worker start, gated on `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` (default `true`). Queries Visibility for every `Running` workflow and calls `handle.terminate(reason="MachinaOS startup: auto-resumption disabled")`. **History is preserved** (UI shows workflows as `Terminated`, not deleted); only active execution stops. Disables resumption while `DeploymentManager` has no boot-time reconcile against Temporal Visibility yet — resumed workflows would otherwise keep executing but be invisible to the MachinaOS UI. Flip to `false` once the reconcile lands.-**Direct CLI access** is still possible — the `temporal` binary on PATH (or under `pooch.os_cache("machinaos-temporal")`) supports `temporal server start --config <yaml>`, `temporal-sql-tool` subcommands, and `tctl`. The supervisor invokes these binaries; you can too.+**Port management**: Temporal owns ports 7233 (gRPC) + 8080 (Web UI). Both bound by the same `temporal.exe` process; both listed in `cli.config.Config.all_ports` so `machina stop`'s port-freeing pre-flight covers them.-**Cluster tunables** (all live in [`core/config.py:Settings`](../server/core/config.py); zero magic numbers in the runtime / config / supervisor wiring):+**Direct CLI access**: the pooch-installed `temporal` binary lives under `~/.cache/MachinaOs/machinaos-temporal/` (Linux) / `~/Library/Caches/MachinaOs/machinaos-temporal/` (macOS) / `%LOCALAPPDATA%\MachinaOs\Cache\machinaos-temporal\` (Windows). Run `temporal --version`, `temporal workflow list`, etc. directly from there.-| Setting | Env var | Default | Purpose |+**Cluster tunables** — all sourced from `.env.template` (canonical defaults; no Python-side fallbacks). Settings fields with `Field(env="...")` require the env var to be present, surfaced via `cli.config.load_config()`'s `.env.template` → `.env` → `os.environ` merge.++| Setting | Env var | `.env.template` default | Purpose ||---|---|---|---|-| `temporal_backend` | `TEMPORAL_BACKEND` | `postgres` | Backend selector — `postgres` (default, durable) / `sqlite` (fast-iteration dev). |-| `temporal_frontend_grpc_port` | `TEMPORAL_FRONTEND_GRPC_PORT` | `7233` | gRPC port the SDK client connects to. Drives the readiness probe + cluster `rpcAddress`. |-| `temporal_matching_grpc_port` | `TEMPORAL_MATCHING_GRPC_PORT` | `7235` | Internal matching service. |-| `temporal_history_grpc_port` | `TEMPORAL_HISTORY_GRPC_PORT` | `7234` | Internal history service. |-| `temporal_worker_grpc_port` | `TEMPORAL_WORKER_GRPC_PORT` | `7239` | Internal worker service. |-| `temporal_bind_local_only` | `TEMPORAL_BIND_LOCAL_ONLY` | `true` | When `true`, all gRPC ports bind `127.0.0.1`. Flip to `false` for multi-host deployments (`0.0.0.0`). |-| `temporal_num_history_shards` | `TEMPORAL_NUM_HISTORY_SHARDS` | `4` | History-shard count. Bump for higher write throughput; not online-resizable. |-| `temporal_default_max_conns` | `TEMPORAL_DEFAULT_MAX_CONNS` | `20` | Postgres pool size for the `temporal` (history) datastore. |-| `temporal_visibility_max_conns` | `TEMPORAL_VISIBILITY_MAX_CONNS` | `4` | Postgres pool size for the `temporal_visibility` datastore. |-| `temporal_max_conn_lifetime` | `TEMPORAL_MAX_CONN_LIFETIME` | `5m` | Postgres connection rotation interval. Accepts Go-duration strings (`30m`, `2h`, etc.). Short window forces periodic refresh — Temporal community recommendation. |-| `temporal_binary_version` | `TEMPORAL_BINARY_VERSION` | `1.31.0` | Pinned `temporalio/temporal` release tag. Bumping requires a matching entry in `_install.py:_CHECKSUMS_BY_VERSION`. |-| `temporal_graceful_shutdown_seconds` | `TEMPORAL_GRACEFUL_SHUTDOWN_SECONDS` | `30` | SIGTERM → SIGKILL grace window for the Temporal binary supervisor. Shared with the embedded worker shutdown. |-| `temporal_postgres_dsn` | `TEMPORAL_POSTGRES_DSN` | (unset) | Reserved for multi-host deployments — when set, bypasses pgserver and points Temporal at an external Postgres. Out of scope for current sprint. |--Two supervisor-build-time env vars (read inside `_temporal_specs.py`, not in `Settings`):--| Env var | Default | Purpose |+| `temporal_enabled` | `TEMPORAL_ENABLED` | `true` | Master toggle. When false, `WorkflowService` falls back to parallel/sequential executor. |+| `temporal_server_address` | `TEMPORAL_SERVER_ADDRESS` | `localhost:7233` | Address the Python SDK client connects to. |+| `temporal_namespace` | `TEMPORAL_NAMESPACE` | `default` | Bootstrapped at server start. |+| `temporal_task_queue` | `TEMPORAL_TASK_QUEUE` | `machina-tasks` | Default task queue for the embedded worker. |+| `temporal_per_type_dispatch` | `TEMPORAL_PER_TYPE_DISPATCH` | `true` | F4.A flag — per-type activity dispatch. |+| `temporal_agent_workflow_enabled` | `TEMPORAL_AGENT_WORKFLOW_ENABLED` | `true` | F4.B flag — agent-as-child-workflow. |+| `temporal_frontend_grpc_port` | `TEMPORAL_FRONTEND_GRPC_PORT` | `7233` | gRPC port (`--port`). Drives the readiness probe. |+| `temporal_ui_port` | `TEMPORAL_UI_PORT` | `8080` | Web UI port (`--ui-port`). CLI default is `--port + 1000 = 8233`; we override to 8080 for muscle memory. |+| `temporal_sqlite_path` | `TEMPORAL_SQLITE_PATH` | `temporal.db` | SQLite file (`--db-filename`). Resolved relative to `DATA_DIR` (= `~/.machina/`) unless absolute — flat under `~/.machina/` like `credentials.db` / `workflow.db`. |+| `temporal_graceful_shutdown_seconds` | `TEMPORAL_GRACEFUL_SHUTDOWN_SECONDS` | `30` | `CTRL_BREAK_EVENT` (Windows) / `SIGTERM` (POSIX) → tree-kill grace window. Shared with the embedded worker shutdown. |+| `temporal_terminate_running_on_startup` | `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP` | `true` | Resumption toggle (see above). |++One supervisor-build-time env var (read inside `_temporal_specs.py`, not in `Settings`):++| Env var | `.env.template` default | Purpose ||---|---|---|-| `TEMPORAL_PG_READY_TIMEOUT_SECONDS` | `60` | How long the supervisor waits for pgserver to be ready before marking the spec failed. |-| `TEMPORAL_SERVER_READY_TIMEOUT_SECONDS` | `120` | How long the supervisor waits for Temporal's gRPC port to come up. Includes the first-run binary download (~90 MB). |--### Known benign log noise--Temporal's matching engine periodically writes task-queue metadata for the **internal `temporal-system` namespace** (e.g. `/_sys/default-worker-tq/1`). When two writes contend, the older one's request context gets canceled — Temporal then auto-retries, and workflow execution is unaffected. The cancellation surfaces in the server log as:--```-ERROR msg="Operation failed with internal error." error="UpdateTaskQueue failed. Failed to start transaction. Error: context canceled" operation=UpdateTaskQueue-ERROR msg="Persistent store operation failure" component=matching-engine wf-task-queue-name=/_sys/default-worker-tq/1 wf-namespace=temporal-system-```--These are **not caused by the Postgres backend** — they appeared identically with the previous SQLite path and are reported as benign across the Temporal community ([forum thread](https://community.temporal.io/t/persistent-store-operation-failure-on-updatetaskqueue-operations/11173)). The connection-pool tunables above (`temporal_default_max_conns`, `temporal_max_conn_lifetime`) are sized per Temporal's recommended baseline to minimise the frequency. No further action is required unless you see them in a **user namespace** (e.g. `default` instead of `temporal-system`), which would indicate a real persistence problem.+| `TEMPORAL_SERVER_READY_TIMEOUT_SECONDS` | `120` | How long the supervisor waits for Temporal's gRPC port to come up. Covers the first-run binary download (~114 MB) if `machina build` didn't pre-cache. |## Debugging```bash-# Check task queue pollers-curl http://localhost:8233/api/v1/namespaces/default/task-queues/machina-tasks--# List recent workflows-curl "http://localhost:8233/api/v1/namespaces/default/workflows"--# Get workflow history-curl "http://localhost:8233/api/v1/namespaces/default/workflows/{id}/history"-# Temporal Web UIopen http://localhost:8080-# Temporal HTTP API-open http://localhost:8233+# List workflows via the local CLI binary (under platformdirs cache)+~/.cache/MachinaOs/machinaos-temporal/.../temporal.exe workflow list --address localhost:7233++# Re-fetch the binary at any time+uv run python -m services.temporal._install```diff --git a/docs-internal/cli_services_integration.md b/docs-internal/cli_services_integration.mdindex bec3d9ad..e73458b5 100644--- a/docs-internal/cli_services_integration.md+++ b/docs-internal/cli_services_integration.md@@ -12,67 +12,25 @@ MachinaOS integrates external services that manage their own lifecycle via CLI t## Integrated CLI Services-### Temporal Server (`temporal-server`)+### Temporal Server (official `temporal` CLI)-Bundles the official Temporal CLI binary with SQLite persistence for local development.+Wraps the official `temporal` CLI's `server start-dev` mode (per [docs.temporal.io/develop/python/set-up-your-local-python](https://docs.temporal.io/develop/python/set-up-your-local-python)) — SQLite-backed dev server, single process, gRPC + Web UI both embedded.-**Install:**-```bash-npm install -g temporal-server-```--**CLI:**-```bash-temporal-server start # Daemon (background)-temporal-server api # Foreground (blocks)-temporal-server stop # Stop-temporal-server status # Show status-temporal-server restart # Restart-temporal-server clean # Stop + remove data-```--**Ports (managed by temporal-server, NOT by MachinaOS):**-| Service | Port |-|----------|------|-| gRPC | 7233 |-| HTTP API | 8233 |-| Web UI | 8080 |-| Metrics | 9090 |--**npm scripts:**-```json-{- "temporal:start": "temporal-server api",- "temporal:stop": "temporal-server stop",- "temporal:status": "temporal-server status"-}-```+**Install:** Automated. `machina build` step [6/6] runs `python -m services.temporal._install`, which uses `pooch` to download the official CLI archive from `https://temporal.download/cli/archive/latest?platform=<os>&arch=<arch>` into a per-OS cache (`~/.cache/MachinaOs/machinaos-temporal/` on Linux, `~/Library/Caches/MachinaOs/machinaos-temporal/` on macOS, `%LOCALAPPDATA%\MachinaOs\Cache\machinaos-temporal\` on Windows). No npm package, no system install required.-**Integration in start.js:**-```javascript-// Check CLI status before building service list-let temporalRunning = false;-try {- const status = execSync('temporal-server status', {- encoding: 'utf-8', timeout: 5000, stdio: 'pipe'- });- temporalRunning = /running|UP/i.test(status);-} catch {- temporalRunning = false;-}+**Lifecycle:** Managed entirely by the CLI commands (`machina start` / `machina dev` / `machina stop`) — there's no separate `temporal:start` npm script. The supervisor at [cli/supervisor.py](../cli/supervisor.py) spawns the official CLI via the supervised-runtime shim at [server/services/temporal/_supervised_runtime.py](../server/services/temporal/_supervised_runtime.py).-// Only add to concurrently if NOT already running-if (!temporalRunning) services.push('npm:temporal:start');-```+**Ports (declared in `.env.template`, freed by `machina stop`'s port-kill pre-flight):**+| Service | Port | Env var |+|---------|------|---------|+| gRPC | 7233 | `TEMPORAL_FRONTEND_GRPC_PORT` |+| Web UI | 8080 | `TEMPORAL_UI_PORT` (CLI default is 8233; we override) |-**Integration in dev.js:**-No status check needed -- `dev.js` does not use `--kill-others`, so `temporal-server api` exiting early (because it's already running) is harmless.+Both bound by the same `temporal.exe` process. Killing the process releases both.-**Integration in stop.js:**-```javascript-// Kill temporal processes by name pattern (not by port)-const temporalPids = await killByPattern('temporal');-```+**Persistence + resumption:**+- SQLite db at `~/.machina/temporal.db` (`TEMPORAL_SQLITE_PATH=temporal.db`, resolved under `DATA_DIR`). History is preserved across restarts; the Temporal UI keeps showing every workflow that ever ran.+- Workflow auto-resumption is disabled at boot: [`TemporalClientWrapper.terminate_running_workflows`](../server/services/temporal/client.py) runs once after client connect and terminates every `Running` workflow with `reason="MachinaOS startup: auto-resumption disabled"`. Workflows show as `Terminated` (not deleted) in the UI. Gated by `TEMPORAL_TERMINATE_RUNNING_ON_STARTUP=true`. Flip to `false` once `DeploymentManager` reconcile-against-Visibility lands.**Embedded worker:**The Temporal worker runs inside the Python backend via `TemporalWorkerManager` in `main.py` lifespan. No separate worker process needed for single-server deployments. For horizontal scaling, run standalone workers:diff --git a/docs-internal/errors.md b/docs-internal/errors.mdindex 19840482..ae092e8d 100644--- a/docs-internal/errors.md+++ b/docs-internal/errors.md@@ -77,32 +77,16 @@ component=matching-engine wf-namespace=temporal-system**Root cause**: Temporal's SQLite database runs in DELETE journal mode by default, which allows only one writer at a time. Temporal's internal system workflows (namespace replication, queue metadata maintenance, backlog counters) contend for write access. When multiple internal workflows try to update task queue metadata concurrently, `BeginTx` blocks, the gRPC context deadline elapses, and the transaction is cancelled.-**Fix**: Enable WAL (Write-Ahead Logging) mode via `--sqlite-pragma` flags. This is configured in the `temporal-server` npm package (v0.0.10+) via `configs/server.json`:+**Status**: benign with the current single-process `temporal server start-dev` setup (the official CLI handles its own SQLite pragmas internally). If you see `persistence_error_with_type` errors in a **user namespace** (e.g. `default` instead of `temporal-system`), it's a real problem — file an issue. Errors in `temporal-system` are auto-retried by Temporal and don't affect workflow execution.-```json-{- "sqlitePragma": ["journal_mode=wal", "busy_timeout=5000"]-}-```--- `journal_mode=wal` -- concurrent read/write access instead of single-writer DELETE mode-- `busy_timeout=5000` -- wait up to 5s for a write lock instead of failing immediately--After changing the pragma config, delete the existing `temporal.db` so the new DB is created with WAL mode from the start:--```bash-temporal stop-rm <temporal-server-package>/data/temporal.db-temporal start-```--**Verification**: After restart, check the metrics endpoint. `persistence_error_with_type` counters for `serviceerror_Unavailable` should stop incrementing:+**If you do hit a stuck DB**: delete the db file and let `temporal server start-dev` recreate it on next boot:```bash-curl -s http://localhost:9090/metrics | grep "persistence_error_with_type" | grep "Unavailable"+machina stop+rm ~/.machina/temporal.db # macOS / Linux+rm "$env:USERPROFILE/.machina/temporal.db" # Windows PowerShell+machina start```-WAL mode is confirmed when `.db-wal` and `.db-shm` sidecar files appear next to `temporal.db`.----## 3. Temporal Activity `CancelledError` on Long-Running Nodes@@ -184,32 +168,34 @@ GET http://localhost:3010/api/auth/status net::ERR_CONNECTION_REFUSEDFailed to check auth status (attempt 4/6): TypeError: Failed to fetch```-**Root cause**: The FastAPI lifespan blocks on Temporal client connection (`await temporal_client_wrapper.connect(retries=10, delay=3.0)`) for up to 30 seconds. During this window, uvicorn has not called `yield` yet, so it is not accepting HTTP connections. The frontend retry window (1+2+4+8+16 = 31s) barely overlaps with the blocking window and can exhaust before the backend starts serving.+**Root cause**: An older version of the FastAPI lifespan blocked on Temporal client connection for up to 30 seconds before yielding to uvicorn. During that window, uvicorn was not accepting HTTP connections, so the frontend retry window could exhaust before the backend started serving.-**Fix**: Move the Temporal initialization into a background `asyncio.create_task()` so the lifespan yields immediately. WorkflowService falls back to parallel/sequential execution until Temporal connects in the background. See the stashed refactor in `git stash list` for the implementation.--**Workaround** (if the refactor is not applied): Increase the frontend retry count or delay in `AuthContext.tsx`, or ensure `temporal-server` is already running before starting the Python backend.+**Status**: fixed. Temporal initialization runs in a background `asyncio.create_task()` ([server/main.py:_init_temporal_background](../server/main.py)) so the lifespan yields immediately. WorkflowService falls back to parallel/sequential execution until Temporal connects in the background.----## 6. `temporal-server` Binary Not Found+## 6. `temporal` CLI Binary Not Found-**Symptom**:+**Symptom**: Supervisor logs:```-Binary not found. Run: npm run postinstall+FileNotFoundError: temporal binary not found at ...```+or pooch fails to download during `machina start`.-**Root cause**: The `temporal-server` npm package downloads the Temporal CLI binary (~60MB) during its `postinstall` script. If `pnpm install` was interrupted, if `--ignore-scripts` was used, or if a previous install was corrupted, the binary at `<package>/bin/temporal.exe` is missing.+**Root cause**: The official `temporal` CLI archive is downloaded by `pooch` to a per-OS cache (`~/.cache/MachinaOs/machinaos-temporal/` on Linux, `~/Library/Caches/MachinaOs/machinaos-temporal/` on macOS, `%LOCALAPPDATA%\MachinaOs\Cache\machinaos-temporal\` on Windows) during `machina build` step [6/6]. If the build was skipped or interrupted, or if network access to `temporal.download` was blocked, the binary is missing.**Fix**:```bash-# Re-run the postinstall download-npx --no-install temporal postinstall+# Re-fetch the binary (idempotent — cache hit if already downloaded)+cd server+uv run python -m services.temporal._install-# Or reinstall the package-pnpm install+# Or re-run the build step+machina build```+If pooch keeps failing, check connectivity to `https://temporal.download/cli/archive/latest?platform=<os>&arch=<arch>` (the URL the docs document at https://docs.temporal.io/develop/python/set-up-your-local-python). Behind a corporate proxy, set `HTTPS_PROXY` in your environment before running.+---## 7. Python Version Mismatch Warningdiff --git a/server/core/config.py b/server/core/config.pyindex 0614a543..59983527 100644--- a/server/core/config.py+++ b/server/core/config.py@@ -108,9 +108,18 @@ class Settings(BaseSettings):)# SQLite db path passed to ``temporal server start-dev# --db-filename ...``. Resolved relative to ``DATA_DIR``- # (``Settings._resolve_under_data``) unless absolute. Sourced from- # ``.env.template`` (canonical default lives there).+ # (``Settings._resolve_under_data``) unless absolute. Sourced+ # from ``.env.template`` (canonical default lives there).temporal_sqlite_path: str = Field(env="TEMPORAL_SQLITE_PATH")+ # Terminate every workflow in ``Running`` state on server boot.+ # Preserves history (workflows show as ``Terminated`` in the+ # Temporal UI, not deleted) but prevents auto-resumption, since+ # MachinaOS's ``DeploymentManager`` has no boot-time reconcile+ # against Temporal Visibility yet — resumed workflows would+ # otherwise be invisible to the UI.+ temporal_terminate_running_on_startup: bool = Field(+ env="TEMPORAL_TERMINATE_RUNNING_ON_STARTUP",+ )# API Keys (all optional, injected at runtime)google_maps_api_key: Optional[str] = Field(default=None, env="GOOGLE_MAPS_API_KEY")diff --git a/server/core/paths.py b/server/core/paths.pyindex 42b0c97e..2d5a831b 100644--- a/server/core/paths.py+++ b/server/core/paths.py@@ -17,6 +17,7 @@ State layout under ``~/.machina/`` (flat, no redundant nesting):- ``~/.machina/claude/`` OAuth state for spawned claude subprocesses- ``~/.machina/workspaces/`` per-workflow scratch dirs- ``~/.machina/whatsapp/`` persistent WhatsApp session DB+ - ``~/.machina/temporal/`` temporal-server cwd + SQLite db when persist=true- ``~/.machina/credentials.db`` Fernet-encrypted secrets- ``~/.machina/workflow.db`` SQLite app DB@@ -31,11 +32,11 @@ uses:- macOS: ``~/Library/Caches/MachinaOs/<service>/``- Windows: ``%LOCALAPPDATA%\\MachinaOs\\Cache\\<service>\\``-Out of scope here: ``temporal`` (separate ``pooch`` cache, postgres-backend only — see ``services/temporal/_install.py``), Himalaya-(``cargo``/``brew`` system install), and npm packages declared in-``package.json`` whose binaries pnpm manages directly under-``node_modules/`` (e.g. ``whatsapp-rpc``'s ``edgymeow``).+Out of scope here: the downloaded Temporal CLI binary (pooch cache+under :func:`packages_dir`, see ``services/temporal/_install.py``),+Himalaya (``cargo``/``brew`` system install), and npm packages+declared in ``package.json`` whose binaries pnpm manages directly+under ``node_modules/`` (e.g. ``whatsapp-rpc``'s ``edgymeow``).The split keeps ``machina clean`` semantics clear: wiping:func:`packages_dir` forces a fresh redownload of MachinaOs-manageddiff --git a/server/main.py b/server/main.pyindex 1e72fd96..4b5d2f33 100644--- a/server/main.py+++ b/server/main.py@@ -289,6 +289,26 @@ async def lifespan(app: FastAPI):)else:try:+ # Disable workflow auto-resumption while+ # DeploymentManager has no boot-time reconcile+ # against Temporal Visibility. History stays in+ # the SQLite db; UI shows workflows as+ # ``Terminated`` rather than continuing to run+ # invisibly to MachinaOS. Toggle off via+ # TEMPORAL_TERMINATE_RUNNING_ON_STARTUP=false.+ if settings.temporal_terminate_running_on_startup:+ try:+ terminated = await temporal_client_wrapper.terminate_running_workflows()+ if terminated:+ _startup_log(+ f"[Temporal] Terminated {terminated} running workflow(s) "+ "at startup (history preserved)"+ )+ except Exception as term_exc: # noqa: BLE001 — non-fatal+ logger.warning(+ f"Startup terminate-running sweep failed: {term_exc}",+ )+temporal_executor = TemporalExecutor(client=client,task_queue=settings.temporal_task_queue,diff --git a/server/services/temporal/_runtime.py b/server/services/temporal/_runtime.pyindex 1866659e..bfa57a97 100644--- a/server/services/temporal/_runtime.py+++ b/server/services/temporal/_runtime.py@@ -111,7 +111,12 @@ class TemporalServerRuntime(BaseProcessSupervisor):# https://docs.temporal.io/cli/server (subset we use):# --port frontend gRPC port (gates ready-probe)# --ui-port Web UI port (default ``--port + 1000``)- # --db-filename SQLite file (omit for in-memory)+ # --db-filename SQLite file — preserves history across+ # restarts. Running workflows are not+ # auto-resumed; see ``TemporalClientWrapper.+ # terminate_running_workflows`` (run from+ # ``main.py`` lifespan) for the boot-time+ # terminate-but-preserve-history behaviour.# --metrics-port 0 disables the Prometheus endpoint# --log-level warn keeps the supervisor log readable# --namespace default namespace bootstrapped at start@@ -126,9 +131,9 @@ class TemporalServerRuntime(BaseProcessSupervisor):]def cwd(self) -> Path:- # cwd is the parent of the SQLite file so any default- # output / log files land alongside the db rather than in the- # supervisor's working directory.+ # Land any incidental output / log files next to the SQLite+ # file (when persisting) or under DATA_DIR (when in-memory) —+ # either way, never the supervisor's own working directory.return self._sqlite_path.parentdef env(self) -> dict[str, str]:diff --git a/server/services/temporal/client.py b/server/services/temporal/client.pyindex c83b65b2..ba01444f 100644--- a/server/services/temporal/client.py+++ b/server/services/temporal/client.py@@ -102,3 +102,49 @@ class TemporalClientWrapper:if self._client is not None:self._client = Nonelogger.info("Disconnected from Temporal server")++ async def terminate_running_workflows(+ self, *, reason: str = "MachinaOS startup: auto-resumption disabled",+ ) -> int:+ """Terminate every workflow in ``Running`` state in our namespace.++ Preserves history — terminated workflows remain visible in the+ Temporal UI as ``Terminated``, only the active execution stops.+ Run from ``main.py`` lifespan between client connect and worker+ start so the worker doesn't accept activities from workflows+ that are about to be terminated. Gated by+ ``Settings.temporal_terminate_running_on_startup`` at the call+ site; this method always terminates when invoked.++ Returns the count of workflows terminated. Failures on+ individual workflows are logged but don't abort the sweep —+ terminating a workflow that completed mid-query is a benign+ race that produces ``WorkflowNotFoundError``.+ """+ if self._client is None:+ logger.warning(+ "terminate_running_workflows called before connect; no-op"+ )+ return 0++ count = 0+ async for wf in self._client.list_workflows(+ "ExecutionStatus = 'Running'",+ ):+ try:+ handle = self._client.get_workflow_handle(+ wf.id, run_id=wf.run_id,+ )+ await handle.terminate(reason=reason)+ count += 1+ except Exception as exc: # noqa: BLE001 — best-effort sweep+ logger.debug(+ f"Failed to terminate workflow id={wf.id} "+ f"run_id={wf.run_id}: {exc}"+ )+ if count:+ logger.info(+ f"Terminated {count} running workflow(s) at startup "+ "(history preserved; resumption disabled)"+ )+ return count