Skip to content

Self-improving AI Employees for everyone

docs: refresh auth, temporal install, clean keep-list, and setup claims

trohitgd9f1bc09c42c
diff --git a/CLAUDE.md b/CLAUDE.md
index 1284943c..cff4c761 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -1652,7 +1652,7 @@ TEMPORAL_TERMINATE_RUNNING_ON_STARTUP=true # see "Resumption" below
**Temporal binary + persistence** are managed in-process by the plugin-folder pattern under [`server/services/temporal/`](./server/services/temporal/). Single supervised process — the official `temporal` CLI's `server start-dev` mode (per [docs.temporal.io/develop/python/set-up-your-local-python](https://docs.temporal.io/develop/python/set-up-your-local-python)):
-- **`_install.py`** — [`pooch`](https://pypi.org/project/pooch/) downloads the official `temporal` CLI archive from `https://temporal.download/cli/archive/latest?platform=<os>&arch=<arch>`. Cross-platform (Windows zip / macOS+Linux tar.gz), XDG-cached. Standalone entry: `python -m services.temporal._install` (invoked by `machina build` step [6/6]).
+- **`_install.py`** — [`pooch`](https://pypi.org/project/pooch/) downloads the official `temporal` CLI archive from `https://temporal.download/cli/archive/latest?platform=<os>&arch=<arch>`. Cross-platform (Windows zip / macOS+Linux tar.gz), cached at `<DATA_DIR>/packages/temporal/`. Download uses an explicit `HTTPDownloader(timeout=300)` — the timeout is per-socket-read, not total transfer, so slow links can finish a ~114 MB fetch (pooch's 30 s default aborted them). Standalone entry: `python -m services.temporal._install` — invoked by `machina build` step [6/6] (fatal there, locked by `test_temporal_install_is_fatal_on_failure`) and by npm postinstall (**non-fatal** there: `TemporalServerRuntime._pre_spawn()` re-downloads lazily on first `machina start`, so a failed eager fetch must never fail `npm install -g`). Note dev mode resolves `DATA_DIR` to repo-local `.machina/`, so dev and `~/.machina` installs keep separate binary caches.
- **`_runtime.py`** — `TemporalServerRuntime(BaseProcessSupervisor)` spawns `temporal server start-dev` with `--port`, `--ui-port`, `--db-filename`, `--namespace`. Same singleton + lifecycle surface every other supervised service uses ([`nodes/whatsapp/_runtime.py`](./server/nodes/whatsapp/_runtime.py) is the template).
- **`_handlers.py` + `_refresh.py`** — `temporal_status` / `_start` / `_stop` WS commands and the WS-connect status snapshot, registered through `services.ws_handler_registry` and `services.status_broadcaster.register_service_refresh`.
- **`client.py`** — `TemporalClientWrapper.connect()` + `terminate_running_workflows()` (resumption toggle, see below).
@@ -3054,10 +3054,12 @@ useEffect(() => {
```
# server/pyproject.toml
bcrypt>=4.1.0
-python-jose[cryptography]>=3.3.0
+pyjwt>=2.13.0
email-validator>=2.0.0
```
+JWT handling uses **PyJWT** (`import jwt`, `jwt.encode` / `jwt.decode`, catch `jwt.PyJWTError`) — HS256 with `Settings.jwt_secret_key`. Do not reintroduce `python-jose`: it drags in pure-Python `ecdsa`, which carries an unpatchable Minerva timing-attack advisory (GHSA-wj6h-64fc-37mp).
+
## Encrypted Credentials System
### Overview
@@ -5312,7 +5314,7 @@ This function:
- **Supervisor Job Object loud-failure (cli/tree.py + cli/supervisor.py)** (8a64eb9): the Windows process-tree mechanism depends entirely on a Job Object with `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE` (no fallback — `start_new_session` is POSIX-only). `pywin32>=308` is a hard supervisor dependency on Windows; the declaration in the root `pyproject.toml` (where `machinaos-cli` lives — `cli/pyproject.toml` was removed as a stale duplicate) carries an inline comment so future maintainers don't drop it. `_JobObject.__init__` splits failures into explicit `ImportError` (with reinstall command) and `Exception` (with `repr`) branches — both write to `stderr` instead of swallowing. `_JobObject.add()` verifies enrollment via `win32job.IsProcessInJob(handle, self._handle)` after assignment; on mismatch it queries `IsProcessInJob(handle, None)` so the warning explicitly names the wrapper-job hypothesis (npm / pnpm / conhost wrappers occasionally place us in their own non-nesting Job). `supervisor._spawn_once` checks `add_to_job` return value and emits a yellow `WARN: pid=N not enrolled in Job Object` per child if False. Without these guards, a stale pywin32 install or a wrapping Job Object silently leaks orphan Python processes on every Ctrl-C and they accumulate to the point of holding SQLite locks that block subsequent backend startup.
- **Credentials envelope-shape invariant** (dc94cde): the `useCredentialPanel` query stores `{values: CredentialFormValues, hadStored: boolean}` (an envelope, NOT a flat dict). The earlier `writeValues` called `setQueryData<CredentialFormValues>` with a spread `(prev) => ({...prev, [key]: value})`, which at runtime merged the typed character at the envelope level next to `values` and `hadStored` instead of inside `.values` — the input selector `panel.values[field.key]` re-rendered with the original (server) value on every keystroke and the input felt frozen. Fix: `writeValues` now preserves the envelope shape and updates only the inner `values` dict; `hadStored` is preserved verbatim (it reflects real server state, not local edits). When introducing other queries shaped as `{data, meta}` envelopes, follow the same pattern — never spread `prev` as if it were the inner payload.
- **`core.paths` is the SSOT for on-disk locations**: only generic helpers (`data_path` / `packages_dir` / `package_dir(name)` / `daemons_dir` / `workspaces_dir` / `workspace_dir(workflow_id)` / `example_workflows_dir`). Plugin-specific subpaths are composed inline at the plugin's call site — adding a new package never requires touching `core.paths`. Reference subtree under `machina_root()` (= `~/.machina/` per default `DATA_DIR`): `workspaces/<slug>/` (per-workflow scratch), `daemons/` (supervised event-source daemon cwds — `stripe listen` etc.; shared root by default, no per-namespace subdir), `claude/` (Claude Code's `CLAUDE_CONFIG_DIR` state — composed as `data_path("claude")` in `nodes/agent/claude_code_agent/_oauth.py`), `packages/` (MachinaOs-managed install root). `packages/` holds a **single shared npm tree** — one `package.json` + `package-lock.json` + `node_modules/` covering every MachinaOs-managed npm package (`@anthropic-ai/claude-code`, `edgymeow`, `agent-browser`). Each plugin's `_install.py` runs `npm install <pkg> --prefix <packages_dir>` which extends the shared tree idempotently; npm manages everything else. Non-npm binaries (Stripe CLI, Temporal CLI) sit under sibling subdirs `packages/stripe/`, `packages/temporal/` via `package_dir(name)`. Also under `<DATA_DIR>/`: `whatsapp/` (WhatsApp session DB), `credentials.db` / `workflow.db` / `temporal.db`. Pre-fix `packages_dir()` routed through `platformdirs.user_cache_path("MachinaOs")` (`~/.cache/MachinaOs/` etc.) and the Temporal CLI sat in its own `pooch.os_cache("machinaos-temporal")` namespace — operators reported both as "not local". The WhatsApp Go bridge (`edgymeow`) used to be a top-level pnpm dep at `<repo>/node_modules/edgymeow/`; now MachinaOs-managed via `nodes/whatsapp/_install.py` like the other CLIs. Daemon cwds used to live under `workspaces/_<namespace>/` and polluted per-workflow scratch with framework state. Out of scope: globally-installed binaries (Himalaya — system package manager). Shipped seed workflows are the lone exception — `example_workflows_dir()` is hardcoded to `<repo>/.machina/workflows/` (git-tracked seeds, NOT under `DATA_DIR`) so they survive `machina clean` and stay at the same path across `DATA_DIR=~/.machina` vs `DATA_DIR=.machina` configs.
-- **`machina clean` preserves `.machina/workflows/`**: `cli/commands/clean.py` iterates `<repo>/.machina/` children and skips anything in `_MACHINA_KEEP = frozenset({"workflows"})`. Wipes `claude/`, `workspaces/`, `*.db` as before; preserves the shipped seed JSONs so a fresh clone + clean doesn't lose user-importable examples. Test `cli/tests/test_clean.py::test_machina_keep_preserves_workflows` locks the keep-list.
+- **`machina clean` preserves `.machina/{workflows,deploy,packages}/`**: `cli/commands/clean.py` iterates `<repo>/.machina/` children and skips anything in `_MACHINA_KEEP = frozenset({"workflows", "deploy", "packages"})`. Wipes `claude/`, `workspaces/`, `*.db` as before. `workflows/` holds the shipped seed JSONs (git-tracked); `deploy/` holds Terraform state for LIVE cloud resources (only `machina deploy destroy` removes it); `packages/` holds the MachinaOs-managed binaries (Temporal CLI ~114 MB, Stripe CLI, shared npm tree) — re-fetchable but expensive, so clean+build cycles stay offline-safe cache hits. Test `cli/tests/test_clean.py::test_machina_keep_preserves_workflows_deploy_and_packages` locks the keep-list.
- **No raw `print()` outside three sanctioned helpers**: `main._startup_log` (pre-logger boot markers), `core.container._clog` (DI-bootstrap markers), and `nodes.code.python_executor.captured_print` (the sandbox builtin handed to user code). Everything else goes through `logger = get_logger(__name__)`. The supervisor prefixes every aggregated line with `[HH:MM:SS.fff]` so no inner `TimeStamper` is needed; structlog console mode is deliberately timestamp-less. Test `server/tests/test_no_raw_prints.py` AST-walks the tree and flags any unsanctioned `print(...)` call.
- **`configure_logging(settings)` must run BEFORE plugin self-registration imports**: in `main.py`, `Settings()` + `configure_logging(settings)` + `init_tracing()` + `get_logger(__name__)` happen ahead of `from core.container import container` and `from routers import …`. Otherwise plugin folders that register on import call `logger.debug(...)` while structlog is still on its default processor chain (which includes `TimeStamper` + no `filter_by_level`) — symptom is double timestamps + debug records leaking despite `LOG_LEVEL=INFO`.
- **Plugin-identifier shape validator (`services/plugin/identifiers.py`)** (`7700f87`): `NODE_TYPE_PATTERN = r"^[A-Za-z_][A-Za-z0-9_]*$"` + `is_valid_node_type(value)`. Single source of truth used by FastAPI URL routes (`Path(pattern=NODE_TYPE_PATTERN)` in `routers/schemas.py`) AND internal helpers (`nodes/_visuals.py::get_plugin_meta` / `get_plugin_icon_path`). Closes CodeQL `py/path-injection` alerts on the icon-resolution endpoints: the registry lookup `get_node_class(node_type)` already gates the taint in practice, but CodeQL can't follow registry-mediated sanitization — adding `fullmatch` at the function boundary is the canonical pattern its rule docs prescribe. Contract test (`tests/services/test_identifiers.py`, 47 cases) locks the regex against `../etc/passwd`, `..\windows\system32`, `foo\x00bar`, `%2e%2e%2fetc`, `foo;bar`, `${HOME}`, `` foo`bar ``, CRLF, and non-string input.
diff --git a/docs-internal/SETUP.md b/docs-internal/SETUP.md
index 7cc9bd99..0f626b72 100644
--- a/docs-internal/SETUP.md
+++ b/docs-internal/SETUP.md
@@ -161,13 +161,17 @@ PYTHON_BACKEND_PORT=3011
```
### Python dependencies fail
-Create a virtual environment:
+The server is uv-managed — prefer `uv sync` from `server/` (creates `server/.venv` against `uv.lock`). The pip fallback works too:
```bash
cd server
python -m venv venv
source venv/bin/activate # or venv\Scripts\activate on Windows
pip install -r requirements.txt
```
+`server/requirements.txt` is an exact-pin export of the lock — regenerate after dependency changes with:
+```bash
+uv export --frozen --no-emit-project --no-hashes --no-dev -o requirements.txt
+```
### Database issues
SQLite database is created automatically at `server/workflow.db`.
diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.md
index b192c4b7..2b3089fc 100644
--- a/docs-internal/TEMPORAL_ARCHITECTURE.md
+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md
@@ -471,7 +471,7 @@ The Temporal binary + persistence are managed in-process by the plugin-folder pa
**What runs**: one process — `temporal server start-dev --port 7233 --ui-port 8080 --db-filename ~/.machina/temporal.db --namespace default`. Both gRPC + Web UI bind to the same process (per docs.temporal.io/cli/server — "all running in a single process"). SQLite-backed durability; history persisted across restarts.
**Modern libs doing the heavy lifting** (zero custom infrastructure code):
-- **[`pooch`](https://pypi.org/project/pooch/)** — `services/temporal/_install.py` downloads the official `temporal` CLI archive from `https://temporal.download/cli/archive/latest?platform=<os>&arch=<arch>`. Cross-platform (Windows zip / macOS+Linux tar.gz), cached at `<DATA_DIR>/packages/temporal/` via `_cache_dir() = core.paths.package_dir("temporal")` (pooch's `path=` argument). Standalone entry: `python -m services.temporal._install` (invoked by `machina build` step [6/6] so first `machina start` doesn't pay the ~114 MB download). Pre-fix this used `pooch.os_cache("machinaos-temporal")` (`~/.cache/MachinaOs/machinaos-temporal/` etc.) — a separate OS-cache namespace operators reported as "not local"; it now sits under DATA_DIR alongside the Stripe binary and the shared npm tree.
+- **[`pooch`](https://pypi.org/project/pooch/)** — `services/temporal/_install.py` downloads the official `temporal` CLI archive from `https://temporal.download/cli/archive/latest?platform=<os>&arch=<arch>`. Cross-platform (Windows zip / macOS+Linux tar.gz), cached at `<DATA_DIR>/packages/temporal/` via `_cache_dir() = core.paths.package_dir("temporal")` (pooch's `path=` argument). The retrieve call passes an explicit `downloader=pooch.HTTPDownloader(timeout=300, progressbar=True)`: the timeout is per-socket-read (not total transfer), so arbitrarily slow links can finish the ~114 MB fetch — pooch's 30 s default aborted them — and `progressbar` must live on the downloader because `retrieve()` ignores its own kwarg when `downloader=` is explicit. Failed downloads never poison the cache (pooch writes to a temp file and renames atomically on success). Standalone entry: `python -m services.temporal._install` — invoked **fatally** by `machina build` step [6/6] (so first `machina start` doesn't pay the download; contract locked by `test_temporal_install_is_fatal_on_failure`) and **non-fatally** by npm postinstall (`scripts/install.js` try/catch — `TemporalServerRuntime._pre_spawn()` re-downloads lazily on first `machina start`, so a failed eager fetch never fails `npm install -g`). The binary cache survives `machina clean` (`packages` is in `_MACHINA_KEEP`, `cli/commands/clean.py`). Pre-fix this used `pooch.os_cache("machinaos-temporal")` (`~/.cache/MachinaOs/machinaos-temporal/` etc.) — a separate OS-cache namespace operators reported as "not local"; it now sits under DATA_DIR alongside the Stripe binary and the shared npm tree.
- **`BaseProcessSupervisor` + `BaseSupervisor`** (`server/services/_supervisor/`) — the in-house supervisor base classes that `server/nodes/whatsapp/_runtime.py` also uses. Provides cross-platform signal handling (POSIX `setsid` + Windows Job Objects + `CREATE_NEW_PROCESS_GROUP` for graceful `CTRL_BREAK_EVENT` shutdown), restart policy via tenacity, log draining, status snapshots. We subclass both — zero custom supervisor logic.
**ServiceSpec wiring**: [`cli/commands/_temporal_specs.py`](../cli/commands/_temporal_specs.py) returns one ServiceSpec for the supervised Temporal dev server (env reads happen inside `temporal_specs()` at call time, safe to import before `cli.config.load_config()` runs). Both `start.py` and `dev.py` call it. The generic supervised-runtime shim lives at [`server/services/temporal/_supervised_runtime.py`](../server/services/temporal/_supervised_runtime.py) so the spawned `uv run python -m services.temporal._supervised_runtime services.temporal._runtime:get_temporal_server_runtime` resolves out of the workspace `.venv` without any cross-tree path plumbing.
diff --git a/docs-internal/release_build_pipeline.md b/docs-internal/release_build_pipeline.md
index 708bc690..5e1b8fe9 100644
--- a/docs-internal/release_build_pipeline.md
+++ b/docs-internal/release_build_pipeline.md
@@ -62,7 +62,7 @@ The npm tarball still excludes `__pycache__/` per `package.json` `files` (cross-
### 4b. Temporal binary fetch + DATA_DIR parity
-- Step `[6/6]` runs `uv run python -m services.temporal._install`, which pooch-downloads the official `temporal` CLI into `<DATA_DIR>/packages/temporal/` (= `~/.machina/packages/temporal/` by default). Pre-fetching at build time turns the ~114 MB download into a sub-second cache hit on first `machina start`.
+- Step `[6/6]` runs `uv run python -m services.temporal._install`, which pooch-downloads the official `temporal` CLI into `<DATA_DIR>/packages/temporal/` (= `~/.machina/packages/temporal/` by default). Pre-fetching at build time turns the ~114 MB download into a sub-second cache hit on first `machina start`. The download uses an explicit `HTTPDownloader(timeout=300)` (per-socket-read timeout — slow links can finish; pooch's 30 s default aborted them). Fatality differs by entry point: `machina build` keeps the step **fatal** (locked by `test_temporal_install_is_fatal_on_failure`), while npm postinstall (`scripts/install.js`) wraps it in a **non-fatal** try/catch because `TemporalServerRuntime._pre_spawn()` re-downloads lazily on first `machina start`. The cache survives `machina clean` (`packages` ∈ `_MACHINA_KEEP`), so clean+build cycles don't re-download.
- **`machina build` layers `.env.dev` first.** `build_command()` calls `cli.config.load_dev_overrides(root)` before the install steps, so the build's `DATA_DIR` matches what the runtime sees. Without it, a repo checkout's `machina build` read `DATA_DIR=~/.machina` from `.env.template` and installed Temporal under user home, but `machina dev` then read `DATA_DIR=.machina` from `.env.dev` and re-downloaded into `<repo>/.machina/` — a redundant ~114 MB fetch on every fresh clone.
- **Safe for global installs.** `.env.dev` is git-committed for contributors but is NOT in the npm `files` list, so an npm-distributed copy has no `.env.dev` — `load_dev_overrides` is a no-op and everything falls through to the `.env.template` default (`DATA_DIR=~/.machina`), matching `machina start` / `machina daemon`.