Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
docs(cli_agent): claude_code_interactive_mode.md + CLAUDE.md index update (Step 11)
trohitg3d74c701c1d8
diff --git a/CLAUDE.md b/CLAUDE.mdindex bea7f470..de377406 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -16,10 +16,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire| **[Schema Source of Truth RFC](./docs-internal/schema_source_of_truth_rfc.md)** | Backend is SSOT for node schemas, visual metadata, handlers, palette metadata, icons. Plugin pattern: one `BaseNode` subclass in `server/nodes/<group>/<plugin>/__init__.py`. Wire format: `asset:<key>` / `<lib>:<brand>` / URL / emoji. Endpoint: `/api/schemas/nodes/{type}/spec.json`. Live invariant total via `pytest --collect-only`. || **[Plugin System (Wave 11)](./docs-internal/plugin_system.md)** | Class-based plugin-first architecture. `BaseNode` / `ActionNode` / `TriggerNode` / `ToolNode` + `@Operation` decorator. Pydantic `Params`/`Output`. Declarative `Routing` DSL + `Connection` facade (Nango pattern). 18 `Credential` subclasses live in each node folder's `_credentials.py` (or inline for single-use). `TaskQueue` constants route to Temporal worker pools. Plugins live across 9 queues (live count via `glob server/nodes/**/__init__.py`); handler bodies fully inlined (`services/handlers/` shrank 12.8K → 1.1K LOC across 16 → 4 files; only cross-cutting orchestration remains: `tools.py` AI-tool dispatch + agent delegation, `google_auth.py`, `triggers.py`). **Wave 11.H added "self-contained plugin folders"** — five generic registries (`ws_handler_registry`, `event_waiter.{register_filter_builder,register_trigger_precheck}`, `status_broadcaster.register_service_refresh`, `node_output_schemas.register_output_schema`) so rich plugins like telegram own their entire surface area without core-services edits. || **[Nodes Cookbook](./server/nodes/README.md)** | 5-minute recipe + folder map + shared helpers (`_base.py` / `_inline.py` per domain) + shared credentials + **canonical folder-per-plugin shape (telegram is the reference implementation)** + contract invariants + common pitfalls. Lives next to the plugin files. |-| **[Node Creation Guide](./docs-internal/node_creation.md)** | Canonical plugin recipe — one self-contained folder per plugin under `server/nodes/<group>/<plugin>/`, rooted at `__init__.py`. Multi-file split (`_service.py` / `_handlers.py` / etc.) when the plugin owns long-lived state. Zero frontend edits, zero core-services edits. Auto-registers via `BaseNode.__init_subclass__` + the five `register_*` hooks. |-| **[AI Tool Node Guide](./docs-internal/ai_tool_node_creation.md)** | Detailed guide for creating dedicated AI Agent tool nodes (schemas, handlers, toolkits) |-| **[Specialized Agent Guide](./docs-internal/specialized_agent_node_creation.md)** | Guide for creating specialized AI agents (Android, Coding, Web, Task, Social, Travel, Tool, Productivity, Payments, Consumer) with full AI configuration |-| **[Dual-Purpose Tool Guide](./docs-internal/dual_purpose_tool_node_creation.md)** | Guide for nodes that work as both workflow nodes AND AI Agent tools (e.g., whatsappSend) |+| **[Node Creation Guide](./docs-internal/node_creation.md)** | Canonical plugin recipe — one self-contained folder per plugin under `server/nodes/<group>/<plugin>/`, rooted at `__init__.py`. Multi-file split (`_service.py` / `_handlers.py` / etc.) when the plugin owns long-lived state. Zero frontend edits, zero core-services edits. Auto-registers via `BaseNode.__init_subclass__` + the six `register_*` hooks. Covers tool nodes, dual-purpose nodes (workflow + AI tool), and specialized agents as variations of the same recipe. || **[Agent Architecture](./docs-internal/agent_architecture.md)** | How AI Agent and Chat Agent discover skills/tools, inject them into LLM prompts, and execute via LangGraph || **[Agent Delegation](./docs-internal/agent_delegation.md)** | How memory, parameters, and execution context flow when one AI agent delegates work to another agent connected as a tool || **[Agent Teams](./docs-internal/agent_teams.md)** | Claude SDK Agent Teams pattern - AI Employee and Orchestrator nodes with input-teammates handle for multi-agent coordination |@@ -50,6 +47,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire| **[RLM Service](./docs-internal/rlm_service.md)** | Recursive Language Model agent with REPL-based execution (llm_query, rlm_query, FINAL) || **[Claude Code Agent](./docs-internal/claude_code_agent_architecture.md)** | Claude Code integration internals (LangGraph reference; companion to `cli_agent_framework.md`) || **[CLI Agent Framework](./docs-internal/cli_agent_framework.md)** | Multi-provider CLI runtime (Claude Code / Codex / Gemini): `AICliService.run_batch`, per-task worktree isolation, FastMCP bridge, **memory bridge** (native `--resume` via stable `cwd=repo_root` + UUID5/`last_session_id` round-trip + auto-clear on stale UUID + `node_parameters_updated` broadcast). |+| **[Claude Code Interactive Mode](./docs-internal/claude_code_interactive_mode.md)** | The interactive-TUI cutover: MachinaOs no longer uses `claude -p` headless. Cross-platform PTY (`ptyprocess` POSIX / `pywinpty>=3.0.3` Windows) keeps `claude` alive; events come from the on-disk session JSONL. `ClaudeSessionPool` keyed by `simpleMemory.node_id` reuses warm processes across batches (non-disruptive — preserves memory continuity). Four typed CloudEvents (`claude.session.{spawned,cleared,terminated,usage}`) fire from the pool. `/compact` events forward to `CompactionService`. `/usage` data sourced from `result` events (TUI scraping not parseable per Anthropic). || **[CLI Agent Canonical Patterns RFC](./docs-internal/cli_agent_canonical_patterns_rfc.md)** | Audit of MachinaOs's `services/cli_agent` against the official Claude Code spec — six invariants (skills-as-files, MCP-only transport, `list_changed`, deferral, visible-tool filtering, native-session-continuity-via-stable-cwd) with current implementation status. || **[Claude Code CLI Reference (snapshot)](./docs-internal/claude_code_cli_reference.md)** | Verbatim snapshot of [code.claude.com/docs/en/cli-reference](https://code.claude.com/docs/en/cli-reference) — every CLI subcommand + flag + system-prompt-flag matrix + the subset we emit from `services/cli_agent/providers/anthropic_claude.py`. Fetched 2026-05-11. || **[Claude Code Env Vars (snapshot)](./docs-internal/claude_code_env_vars_reference.md)** | Categorised [code.claude.com/docs/en/env-vars](https://code.claude.com/docs/en/env-vars) snapshot — auth, Bedrock/Vertex, model, bash, MCP, telemetry, session/debug, paths. Documents `CLAUDE_CONFIG_DIR`, `MAX_MCP_OUTPUT_TOKENS`, `ENABLE_TOOL_SEARCH` which we touch directly. |@@ -1049,7 +1047,7 @@ Nodes that work BOTH as standalone workflow nodes AND as AI Agent tools. When co- **gmaps_locations**: Google Maps Geocoding service for address-to-coordinates conversion. Tool name: `geocode`.- **gmaps_nearby_places**: Google Places API nearbySearch. Tool name: `nearby_places`.-See [Dual-Purpose Tool Node Guide](./docs-internal/dual_purpose_tool_node_creation.md) for implementation details.+See [plugin_system.md](./docs-internal/plugin_system.md) for the dual-purpose pattern (mark `group: ['category', 'tool']` and set `usable_as_tool = True` on the plugin class); whatsapp / twitter / google plugin folders are the live examples.### Location Services (3 nodes)- **gmaps_create**: Google Maps creation with customizable center, zoom, and map type (display only, not a tool)diff --git a/docs-internal/claude_code_interactive_mode.md b/docs-internal/claude_code_interactive_mode.mdnew file mode 100644index 00000000..f706e249--- /dev/null+++ b/docs-internal/claude_code_interactive_mode.md@@ -0,0 +1,252 @@+# Claude Code interactive mode (in-process PTY + on-disk JSONL)++> **TL;DR.** MachinaOs drives the same interactive `claude` invocation a+> normal user runs at their terminal — not `claude -p`. A PTY keeps the+> TUI alive in-process; events come off the on-disk session JSONL,+> not stdout. A pool keyed by `simpleMemory.node_id` keeps the warm+> process across batches.++## Why we cut over from `-p`++Pre-cutover MachinaOs spawned `claude -p --output-format stream-json`+and parsed NDJSON events on stdout. Two-direction motivation:++1. **Match how users actually use Claude Code.** The headless path is+ sanctioned for automation, but most Claude Code users run the+ interactive TUI. Driving the same surface lets us inherit fixes /+ behavior changes the team prioritises for the human-facing mode.+2. **Anthropic billing change (2026-06-15).** Per+ [`code.claude.com/docs/en/headless`](https://code.claude.com/docs/en/headless),+ subscription plans now bill `claude -p` and Agent-SDK usage from a+ separate "monthly Agent SDK credit." Interactive usage stays on the+ subscription's interactive bucket. The cutover puts us in the+ user's interactive bucket when they supply their own API key.++The corollary: every protocol detail still works in interactive mode+because the on-disk JSONL is the same format `-p` wrote to stdout+(Claude Code CHANGELOG 2.1.101 / 2.1.126 — shared writer). The argv+flags we care about — `--mcp-config`, `--strict-mcp-config`,+`--allowedTools`, `--permission-mode`, `--append-system-prompt`,+`--resume`, `--model`, `--effort`, `--add-dir`, `--disallowedTools`,+`--agent` — all work in interactive mode.++## What we dropped++| Flag | Why dropped |+|---|---|+| `-p` / `--print` | The headless toggle itself. |+| `--output-format stream-json` | Only meaningful under `-p`. |+| `--verbose` | Only meaningful with `-p --output-format stream-json`. |+| `--include-partial-messages` | Only meaningful with stream-json. |+| `--include-hook-events` | Only meaningful with stream-json. The hooks fire regardless and we can detect them via JSONL. |+| `--max-turns` | `-p`-only. External counter is a Phase 2 follow-up. |+| `--max-budget-usd` | `-p`-only. External monitor on `result.total_cost_usd` is a Phase 2 follow-up. |+| `--session-id <UUID>` | No longer pre-mint UUIDs. First-run claude assigns its own; we discover it via the new JSONL filename. |+| `--fallback-model` | `-p`-only. |++`ClaudeTaskSpec` keeps the corresponding Pydantic fields for+back-compat; they're silently dropped in `interactive_argv`.++## Architecture — transport vs. protocol++Two orthogonal layers:++| Layer | macOS / Linux | Windows |+|---|---|---|+| **Transport** (PTY holds `claude` alive) | `ptyprocess>=0.7.0` | `pywinpty>=3.0.3` (Rust-backed via PyO3 + winpty-rs + Maturin) |+| **Protocol** (read events) | tail `<CLAUDE_CONFIG_DIR>/projects/<project_key>/<session>.jsonl` | identical |+| **Send turn** | `pty.write((text + '\r').encode())` (NOT close stdin — see [#15553](https://github.com/anthropics/claude-code/issues/15553)) | identical |+| **Send slash command** | `pty.write(b"/clear\r")` | identical |+| **Completion detect** | JSONL `result` entry | identical |+| **Resume across spawns** | `claude --resume <UUID>` | identical |++`pywinpty` v3.x is now production-grade (used by Jupyter Lab terminals+and Spyder's IPython console) so the historical reason to route Windows+through a Node.js sidecar (pywinpty flakiness) no longer holds. Both+backends run in-process behind a single `PtyTransport` Protocol; the+factory in [`server/services/cli_agent/transports/__init__.py`](../server/services/cli_agent/transports/__init__.py)+picks per `sys.platform`.++### Code layout (`server/services/cli_agent/`)++```+cli_agent/+├── providers/+│ └── anthropic_claude.py # interactive_argv (replaces headless_argv)+├── transports/+│ ├── __init__.py # get_pty_transport() factory+│ ├── base.py # PtyTransport / PtyHandle Protocols+│ ├── posix.py # PosixPtyTransport (ptyprocess)+│ └── windows.py # WindowsPtyTransport (pywinpty)+├── jsonl_watcher.py # JsonlWatcher (tail-f) + JsonlDirWatcher+├── session.py # AICliSession — one-shot non-pooled path+├── session_pool.py # ClaudeSessionPool — warm-process reuse+└── service.py # AICliService.run_batch + _run_pooled_turn+```++## Session pool — warm-process reuse++[`ClaudeSessionPool`](../server/services/cli_agent/session_pool.py) is+keyed by `simpleMemory.node_id`. Successive batches against the same+memory node reuse the same warm `claude` PTY:++- **Acquire (warm)** — returns the existing live session. **No+ implicit `/clear`** — the next prompt appends to claude's current+ JSONL so the conversation continues. This is the load-bearing+ invariant for `simpleMemory` continuity.+- **Acquire (cold)** — spawn fresh via `PtyTransport`, locate the+ newly-created JSONL by mtime, start a `JsonlWatcher` on it. Stash+ the MCP bearer token on the `PooledClaudeSession` so re-acquires+ use the same token the spawned claude already has in its argv.+- **Explicit `clear(session)`** — sends `/clear` to the PTY, awaits+ the new JSONL filename via the persistent `JsonlDirWatcher`, swaps+ the file watcher. Per+ [`claude-code#32871`](https://github.com/anthropics/claude-code/issues/32871),+ `/clear` mints a **new UUID** with a **new JSONL file** rather than+ clearing in-place. Pre-clear conversation stays resumable by its+ old UUID.++Lifecycle policy:++- **Idle TTL** 30 min (`_DEFAULT_IDLE_TTL`). Background reaper task+ terminates pooled sessions whose `last_used_at` exceeds the cap+ AND aren't currently locked.+- **Max size** 16 (`_DEFAULT_MAX_SIZE`). LRU eviction at cap.+- **Crash recovery** — `acquire` calls `pty_handle.is_alive()` and+ respawns transparently when the pooled PTY has died.+- **Concurrency** — per-key `asyncio.Lock` serialises turns against+ the same pooled session. `lock.locked()` doubles as the reaper's+ in-flight detector.+- **Shutdown** — `ClaudeSessionPool.shutdown_all()` is the target for+ FastAPI's lifespan `shutdown` event (TODO: wire from `main.py`).++## MCP bearer token lifecycle++`claude` handles its own MCP authentication. We embed the bearer in+`--mcp-config` at spawn time; the spawned claude uses it for the+lifetime of its process. The pool does NOT issue or unregister tokens+— the caller (`AICliService.run_batch`) owns the lifecycle, with the+twist that for pooled sessions the token persists with the warm PTY+across batches:++```python+# run_batch — non-pool path (unchanged)+token = issue_token()+register_batch(token, ctx)+try:+ results = await asyncio.gather(*(run_one(t) for t in tasks))+finally:+ unregister_batch(token)++# run_batch — pool path+token = issue_token()+register_batch(token, ctx)+try:+ result = await self._run_pooled_turn(..., mcp_bearer_token=token)+ # token stays embedded in the warm PTY for the next batch.+ # On pool eviction / shutdown the registration orphans (bounded+ # by max pool size, acceptable for Phase 1).+finally:+ if not use_pool:+ unregister_batch(token)+```++## CloudEvents broadcasts++Four typed `WorkflowEvent` factories on+[`server/services/events/envelope.py`](../server/services/events/envelope.py),+fired via the corresponding `broadcaster.broadcast_claude_session_*`+methods on+[`server/services/status_broadcaster.py`](../server/services/status_broadcaster.py):++| Event type | Wire key | When fired |+|---|---|---|+| `com.machinaos.claude.session.spawned` | `claude_session_event` | Cold spawn (new PTY) |+| `com.machinaos.claude.session.cleared` | `claude_session_event` | Explicit `pool.clear` → new UUID |+| `com.machinaos.claude.session.terminated` | `claude_session_event` | Pool terminate (reason: `idle` / `crashed` / `evicted` / `shutdown` / `explicit`) |+| `com.machinaos.claude.session.usage` | `claude_session_usage` | Each `result` event (per-turn cost + tokens) |++Frontend listeners should switch on `envelope.type` and route the+inner `data` payload (memory_node_id, session_uuid, cost, tokens, …)+into whichever store renders the simpleMemory usage panel.++## `/usage` and `/compact` semantics++Per the research:++- **`/usage`** — TUI-only plain text per Anthropic. Not parseable.+ The data it displays (cost, tokens, duration, num_turns) is already+ on every `result` JSONL event under `usage` + `total_cost_usd`. We+ surface it via the `claude.session.usage` CloudEvent instead.+- **`/clear`** — mints a new session UUID + new JSONL file (issue+ [#32871](https://github.com/anthropics/claude-code/issues/32871)).+ Pool exposes this as `pool.clear(session)` for explicit reset; not+ fired on warm acquire.+- **`/compact`** — emits `system/compact_boundary` with+ `compact_metadata.pre_tokens` + `compact_metadata.trigger` (auto vs+ manual). Both `AICliSession._on_jsonl_event` and+ `ClaudeSessionPool._handle_jsonl_event` forward it to+ `CompactionService.record(...)` so the local-threshold path doesn't+ double-fire.++Commands MachinaOs must **avoid** in PTY-driven automation (they open+interactive dialogs, kill the session, or open a browser):+`/feedback`, `/bug`, `/exit`, `/quit`, `/login`, `/logout`,+`/permissions`, `/agents`, `/mcp`, `/model` (with no arg), `/rewind`,+`/config`, `/install-github-app`, `/install-slack-app`, `/teleport`,+`/heapdump`.++## Memory bridge — unchanged++The `simpleMemory.last_session_id` ↔ `--resume <UUID>` contract works+identically in interactive mode because the on-disk JSONL is the+same format. The invariants:++- `memory_bound=True` makes the spawn cwd = `repo_root` so claude's+ `project_key` stays constant across batches.+- First run: claude assigns its own UUID; the pool / `AICliSession`+ discovers it via the new JSONL filename, saves it to+ `simpleMemory.last_session_id`.+- Subsequent runs (non-pooled): argv `--resume <last_session_id>`.+- Subsequent runs (pooled, warm): no respawn — the warm PTY continues+ in its current UUID.+- Subsequent runs (pooled, after `pool.clear`): a new UUID is captured+ on the next `result` event and persisted.+- Auto-clear-stale: if claude reports `No conversation found with+ session ID: <UUID>`, `_persist_memory` wipes the broken+ `last_session_id` so the next run falls into the first-run branch.++## Out-of-process pty-host (deferred upgrade path)++If `pywinpty` / `ptyprocess` stability becomes an issue (crashes+bringing down FastAPI), promote `PtyTransport` to an out-of-process+Python sidecar — VSCode's `ptyHostMain.ts` pattern, in Python.+Supervised by `machina/tree.py`'s Job Object, JSON-RPC over stdio.+The `PtyTransport` Protocol boundary makes the swap cheap (~1 day).+See VSCode issue+[#74620](https://github.com/microsoft/vscode/issues/74620) for the+rationale (heartbeat, flow control, isolation from main app).++## Open follow-ups++- **Cancellation path.** Pool sessions bypass+ `AICliService._active_sessions`, so `cancel_workflow` /+ `cancel_node` don't reach the pooled PTY. Add pool-in-flight+ tracking or hook cancel into `pool.terminate`.+- **Lifespan integration.** Wire `pool.shutdown_all()` to FastAPI's+ shutdown event from `main.py`. Currently the pool dies with the+ process; orphan PTY children are caught by `machina/tree.py` Job+ Object on Windows and `os.setsid` on POSIX.+- **Frontend usage panel.** `claude.session.usage` CloudEvents are+ flowing; the React hook + simpleMemory panel UI are the missing+ half (Step 7 of the implementation plan).+- **`--fork-session` UI.** `--fork-session` is the documented working+ fork primitive; surface a "Branch this conversation" `<ActionButton>`+ on simpleMemory + a `branches: List[{uuid, name, created_at}]`+ param. Avoids `/rewind` which has bug+ [#55347](https://github.com/anthropics/claude-code/issues/55347)+ (documented as fork but implemented as in-place mutation).+- **External cost cap / turn cap.** Replacements for the dropped+ `--max-budget-usd` and `--max-turns` flags. Aggregate+ `result.total_cost_usd` / count `assistant` events with+ `stop_reason: "tool_use"` and kill the PTY at the limit.