Skip to content

Self-improving AI Employees for everyone

docs(cli_agent): claude_code_interactive_mode.md + CLAUDE.md index update (Step 11)

trohitg3d74c701c1d8
diff --git a/CLAUDE.md b/CLAUDE.md
index bea7f470..de377406 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -16,10 +16,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire
| **[Schema Source of Truth RFC](./docs-internal/schema_source_of_truth_rfc.md)** | Backend is SSOT for node schemas, visual metadata, handlers, palette metadata, icons. Plugin pattern: one `BaseNode` subclass in `server/nodes/<group>/<plugin>/__init__.py`. Wire format: `asset:<key>` / `<lib>:<brand>` / URL / emoji. Endpoint: `/api/schemas/nodes/{type}/spec.json`. Live invariant total via `pytest --collect-only`. |
| **[Plugin System (Wave 11)](./docs-internal/plugin_system.md)** | Class-based plugin-first architecture. `BaseNode` / `ActionNode` / `TriggerNode` / `ToolNode` + `@Operation` decorator. Pydantic `Params`/`Output`. Declarative `Routing` DSL + `Connection` facade (Nango pattern). 18 `Credential` subclasses live in each node folder's `_credentials.py` (or inline for single-use). `TaskQueue` constants route to Temporal worker pools. Plugins live across 9 queues (live count via `glob server/nodes/**/__init__.py`); handler bodies fully inlined (`services/handlers/` shrank 12.8K → 1.1K LOC across 16 → 4 files; only cross-cutting orchestration remains: `tools.py` AI-tool dispatch + agent delegation, `google_auth.py`, `triggers.py`). **Wave 11.H added "self-contained plugin folders"** — five generic registries (`ws_handler_registry`, `event_waiter.{register_filter_builder,register_trigger_precheck}`, `status_broadcaster.register_service_refresh`, `node_output_schemas.register_output_schema`) so rich plugins like telegram own their entire surface area without core-services edits. |
| **[Nodes Cookbook](./server/nodes/README.md)** | 5-minute recipe + folder map + shared helpers (`_base.py` / `_inline.py` per domain) + shared credentials + **canonical folder-per-plugin shape (telegram is the reference implementation)** + contract invariants + common pitfalls. Lives next to the plugin files. |
-| **[Node Creation Guide](./docs-internal/node_creation.md)** | Canonical plugin recipe — one self-contained folder per plugin under `server/nodes/<group>/<plugin>/`, rooted at `__init__.py`. Multi-file split (`_service.py` / `_handlers.py` / etc.) when the plugin owns long-lived state. Zero frontend edits, zero core-services edits. Auto-registers via `BaseNode.__init_subclass__` + the five `register_*` hooks. |
-| **[AI Tool Node Guide](./docs-internal/ai_tool_node_creation.md)** | Detailed guide for creating dedicated AI Agent tool nodes (schemas, handlers, toolkits) |
-| **[Specialized Agent Guide](./docs-internal/specialized_agent_node_creation.md)** | Guide for creating specialized AI agents (Android, Coding, Web, Task, Social, Travel, Tool, Productivity, Payments, Consumer) with full AI configuration |
-| **[Dual-Purpose Tool Guide](./docs-internal/dual_purpose_tool_node_creation.md)** | Guide for nodes that work as both workflow nodes AND AI Agent tools (e.g., whatsappSend) |
+| **[Node Creation Guide](./docs-internal/node_creation.md)** | Canonical plugin recipe — one self-contained folder per plugin under `server/nodes/<group>/<plugin>/`, rooted at `__init__.py`. Multi-file split (`_service.py` / `_handlers.py` / etc.) when the plugin owns long-lived state. Zero frontend edits, zero core-services edits. Auto-registers via `BaseNode.__init_subclass__` + the six `register_*` hooks. Covers tool nodes, dual-purpose nodes (workflow + AI tool), and specialized agents as variations of the same recipe. |
| **[Agent Architecture](./docs-internal/agent_architecture.md)** | How AI Agent and Chat Agent discover skills/tools, inject them into LLM prompts, and execute via LangGraph |
| **[Agent Delegation](./docs-internal/agent_delegation.md)** | How memory, parameters, and execution context flow when one AI agent delegates work to another agent connected as a tool |
| **[Agent Teams](./docs-internal/agent_teams.md)** | Claude SDK Agent Teams pattern - AI Employee and Orchestrator nodes with input-teammates handle for multi-agent coordination |
@@ -50,6 +47,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire
| **[RLM Service](./docs-internal/rlm_service.md)** | Recursive Language Model agent with REPL-based execution (llm_query, rlm_query, FINAL) |
| **[Claude Code Agent](./docs-internal/claude_code_agent_architecture.md)** | Claude Code integration internals (LangGraph reference; companion to `cli_agent_framework.md`) |
| **[CLI Agent Framework](./docs-internal/cli_agent_framework.md)** | Multi-provider CLI runtime (Claude Code / Codex / Gemini): `AICliService.run_batch`, per-task worktree isolation, FastMCP bridge, **memory bridge** (native `--resume` via stable `cwd=repo_root` + UUID5/`last_session_id` round-trip + auto-clear on stale UUID + `node_parameters_updated` broadcast). |
+| **[Claude Code Interactive Mode](./docs-internal/claude_code_interactive_mode.md)** | The interactive-TUI cutover: MachinaOs no longer uses `claude -p` headless. Cross-platform PTY (`ptyprocess` POSIX / `pywinpty>=3.0.3` Windows) keeps `claude` alive; events come from the on-disk session JSONL. `ClaudeSessionPool` keyed by `simpleMemory.node_id` reuses warm processes across batches (non-disruptive — preserves memory continuity). Four typed CloudEvents (`claude.session.{spawned,cleared,terminated,usage}`) fire from the pool. `/compact` events forward to `CompactionService`. `/usage` data sourced from `result` events (TUI scraping not parseable per Anthropic). |
| **[CLI Agent Canonical Patterns RFC](./docs-internal/cli_agent_canonical_patterns_rfc.md)** | Audit of MachinaOs's `services/cli_agent` against the official Claude Code spec — six invariants (skills-as-files, MCP-only transport, `list_changed`, deferral, visible-tool filtering, native-session-continuity-via-stable-cwd) with current implementation status. |
| **[Claude Code CLI Reference (snapshot)](./docs-internal/claude_code_cli_reference.md)** | Verbatim snapshot of [code.claude.com/docs/en/cli-reference](https://code.claude.com/docs/en/cli-reference) — every CLI subcommand + flag + system-prompt-flag matrix + the subset we emit from `services/cli_agent/providers/anthropic_claude.py`. Fetched 2026-05-11. |
| **[Claude Code Env Vars (snapshot)](./docs-internal/claude_code_env_vars_reference.md)** | Categorised [code.claude.com/docs/en/env-vars](https://code.claude.com/docs/en/env-vars) snapshot — auth, Bedrock/Vertex, model, bash, MCP, telemetry, session/debug, paths. Documents `CLAUDE_CONFIG_DIR`, `MAX_MCP_OUTPUT_TOKENS`, `ENABLE_TOOL_SEARCH` which we touch directly. |
@@ -1049,7 +1047,7 @@ Nodes that work BOTH as standalone workflow nodes AND as AI Agent tools. When co
- **gmaps_locations**: Google Maps Geocoding service for address-to-coordinates conversion. Tool name: `geocode`.
- **gmaps_nearby_places**: Google Places API nearbySearch. Tool name: `nearby_places`.
-See [Dual-Purpose Tool Node Guide](./docs-internal/dual_purpose_tool_node_creation.md) for implementation details.
+See [plugin_system.md](./docs-internal/plugin_system.md) for the dual-purpose pattern (mark `group: ['category', 'tool']` and set `usable_as_tool = True` on the plugin class); whatsapp / twitter / google plugin folders are the live examples.
### Location Services (3 nodes)
- **gmaps_create**: Google Maps creation with customizable center, zoom, and map type (display only, not a tool)
diff --git a/docs-internal/claude_code_interactive_mode.md b/docs-internal/claude_code_interactive_mode.md
new file mode 100644
index 00000000..f706e249
--- /dev/null
+++ b/docs-internal/claude_code_interactive_mode.md
@@ -0,0 +1,252 @@
+# Claude Code interactive mode (in-process PTY + on-disk JSONL)
+
+> **TL;DR.** MachinaOs drives the same interactive `claude` invocation a
+> normal user runs at their terminal — not `claude -p`. A PTY keeps the
+> TUI alive in-process; events come off the on-disk session JSONL,
+> not stdout. A pool keyed by `simpleMemory.node_id` keeps the warm
+> process across batches.
+
+## Why we cut over from `-p`
+
+Pre-cutover MachinaOs spawned `claude -p --output-format stream-json`
+and parsed NDJSON events on stdout. Two-direction motivation:
+
+1. **Match how users actually use Claude Code.** The headless path is
+ sanctioned for automation, but most Claude Code users run the
+ interactive TUI. Driving the same surface lets us inherit fixes /
+ behavior changes the team prioritises for the human-facing mode.
+2. **Anthropic billing change (2026-06-15).** Per
+ [`code.claude.com/docs/en/headless`](https://code.claude.com/docs/en/headless),
+ subscription plans now bill `claude -p` and Agent-SDK usage from a
+ separate "monthly Agent SDK credit." Interactive usage stays on the
+ subscription's interactive bucket. The cutover puts us in the
+ user's interactive bucket when they supply their own API key.
+
+The corollary: every protocol detail still works in interactive mode
+because the on-disk JSONL is the same format `-p` wrote to stdout
+(Claude Code CHANGELOG 2.1.101 / 2.1.126 — shared writer). The argv
+flags we care about — `--mcp-config`, `--strict-mcp-config`,
+`--allowedTools`, `--permission-mode`, `--append-system-prompt`,
+`--resume`, `--model`, `--effort`, `--add-dir`, `--disallowedTools`,
+`--agent` — all work in interactive mode.
+
+## What we dropped
+
+| Flag | Why dropped |
+|---|---|
+| `-p` / `--print` | The headless toggle itself. |
+| `--output-format stream-json` | Only meaningful under `-p`. |
+| `--verbose` | Only meaningful with `-p --output-format stream-json`. |
+| `--include-partial-messages` | Only meaningful with stream-json. |
+| `--include-hook-events` | Only meaningful with stream-json. The hooks fire regardless and we can detect them via JSONL. |
+| `--max-turns` | `-p`-only. External counter is a Phase 2 follow-up. |
+| `--max-budget-usd` | `-p`-only. External monitor on `result.total_cost_usd` is a Phase 2 follow-up. |
+| `--session-id <UUID>` | No longer pre-mint UUIDs. First-run claude assigns its own; we discover it via the new JSONL filename. |
+| `--fallback-model` | `-p`-only. |
+
+`ClaudeTaskSpec` keeps the corresponding Pydantic fields for
+back-compat; they're silently dropped in `interactive_argv`.
+
+## Architecture — transport vs. protocol
+
+Two orthogonal layers:
+
+| Layer | macOS / Linux | Windows |
+|---|---|---|
+| **Transport** (PTY holds `claude` alive) | `ptyprocess>=0.7.0` | `pywinpty>=3.0.3` (Rust-backed via PyO3 + winpty-rs + Maturin) |
+| **Protocol** (read events) | tail `<CLAUDE_CONFIG_DIR>/projects/<project_key>/<session>.jsonl` | identical |
+| **Send turn** | `pty.write((text + '\r').encode())` (NOT close stdin — see [#15553](https://github.com/anthropics/claude-code/issues/15553)) | identical |
+| **Send slash command** | `pty.write(b"/clear\r")` | identical |
+| **Completion detect** | JSONL `result` entry | identical |
+| **Resume across spawns** | `claude --resume <UUID>` | identical |
+
+`pywinpty` v3.x is now production-grade (used by Jupyter Lab terminals
+and Spyder's IPython console) so the historical reason to route Windows
+through a Node.js sidecar (pywinpty flakiness) no longer holds. Both
+backends run in-process behind a single `PtyTransport` Protocol; the
+factory in [`server/services/cli_agent/transports/__init__.py`](../server/services/cli_agent/transports/__init__.py)
+picks per `sys.platform`.
+
+### Code layout (`server/services/cli_agent/`)
+
+```
+cli_agent/
+├── providers/
+│ └── anthropic_claude.py # interactive_argv (replaces headless_argv)
+├── transports/
+│ ├── __init__.py # get_pty_transport() factory
+│ ├── base.py # PtyTransport / PtyHandle Protocols
+│ ├── posix.py # PosixPtyTransport (ptyprocess)
+│ └── windows.py # WindowsPtyTransport (pywinpty)
+├── jsonl_watcher.py # JsonlWatcher (tail-f) + JsonlDirWatcher
+├── session.py # AICliSession — one-shot non-pooled path
+├── session_pool.py # ClaudeSessionPool — warm-process reuse
+└── service.py # AICliService.run_batch + _run_pooled_turn
+```
+
+## Session pool — warm-process reuse
+
+[`ClaudeSessionPool`](../server/services/cli_agent/session_pool.py) is
+keyed by `simpleMemory.node_id`. Successive batches against the same
+memory node reuse the same warm `claude` PTY:
+
+- **Acquire (warm)** — returns the existing live session. **No
+ implicit `/clear`** — the next prompt appends to claude's current
+ JSONL so the conversation continues. This is the load-bearing
+ invariant for `simpleMemory` continuity.
+- **Acquire (cold)** — spawn fresh via `PtyTransport`, locate the
+ newly-created JSONL by mtime, start a `JsonlWatcher` on it. Stash
+ the MCP bearer token on the `PooledClaudeSession` so re-acquires
+ use the same token the spawned claude already has in its argv.
+- **Explicit `clear(session)`** — sends `/clear` to the PTY, awaits
+ the new JSONL filename via the persistent `JsonlDirWatcher`, swaps
+ the file watcher. Per
+ [`claude-code#32871`](https://github.com/anthropics/claude-code/issues/32871),
+ `/clear` mints a **new UUID** with a **new JSONL file** rather than
+ clearing in-place. Pre-clear conversation stays resumable by its
+ old UUID.
+
+Lifecycle policy:
+
+- **Idle TTL** 30 min (`_DEFAULT_IDLE_TTL`). Background reaper task
+ terminates pooled sessions whose `last_used_at` exceeds the cap
+ AND aren't currently locked.
+- **Max size** 16 (`_DEFAULT_MAX_SIZE`). LRU eviction at cap.
+- **Crash recovery** — `acquire` calls `pty_handle.is_alive()` and
+ respawns transparently when the pooled PTY has died.
+- **Concurrency** — per-key `asyncio.Lock` serialises turns against
+ the same pooled session. `lock.locked()` doubles as the reaper's
+ in-flight detector.
+- **Shutdown** — `ClaudeSessionPool.shutdown_all()` is the target for
+ FastAPI's lifespan `shutdown` event (TODO: wire from `main.py`).
+
+## MCP bearer token lifecycle
+
+`claude` handles its own MCP authentication. We embed the bearer in
+`--mcp-config` at spawn time; the spawned claude uses it for the
+lifetime of its process. The pool does NOT issue or unregister tokens
+— the caller (`AICliService.run_batch`) owns the lifecycle, with the
+twist that for pooled sessions the token persists with the warm PTY
+across batches:
+
+```python
+# run_batch — non-pool path (unchanged)
+token = issue_token()
+register_batch(token, ctx)
+try:
+ results = await asyncio.gather(*(run_one(t) for t in tasks))
+finally:
+ unregister_batch(token)
+
+# run_batch — pool path
+token = issue_token()
+register_batch(token, ctx)
+try:
+ result = await self._run_pooled_turn(..., mcp_bearer_token=token)
+ # token stays embedded in the warm PTY for the next batch.
+ # On pool eviction / shutdown the registration orphans (bounded
+ # by max pool size, acceptable for Phase 1).
+finally:
+ if not use_pool:
+ unregister_batch(token)
+```
+
+## CloudEvents broadcasts
+
+Four typed `WorkflowEvent` factories on
+[`server/services/events/envelope.py`](../server/services/events/envelope.py),
+fired via the corresponding `broadcaster.broadcast_claude_session_*`
+methods on
+[`server/services/status_broadcaster.py`](../server/services/status_broadcaster.py):
+
+| Event type | Wire key | When fired |
+|---|---|---|
+| `com.machinaos.claude.session.spawned` | `claude_session_event` | Cold spawn (new PTY) |
+| `com.machinaos.claude.session.cleared` | `claude_session_event` | Explicit `pool.clear` → new UUID |
+| `com.machinaos.claude.session.terminated` | `claude_session_event` | Pool terminate (reason: `idle` / `crashed` / `evicted` / `shutdown` / `explicit`) |
+| `com.machinaos.claude.session.usage` | `claude_session_usage` | Each `result` event (per-turn cost + tokens) |
+
+Frontend listeners should switch on `envelope.type` and route the
+inner `data` payload (memory_node_id, session_uuid, cost, tokens, …)
+into whichever store renders the simpleMemory usage panel.
+
+## `/usage` and `/compact` semantics
+
+Per the research:
+
+- **`/usage`** — TUI-only plain text per Anthropic. Not parseable.
+ The data it displays (cost, tokens, duration, num_turns) is already
+ on every `result` JSONL event under `usage` + `total_cost_usd`. We
+ surface it via the `claude.session.usage` CloudEvent instead.
+- **`/clear`** — mints a new session UUID + new JSONL file (issue
+ [#32871](https://github.com/anthropics/claude-code/issues/32871)).
+ Pool exposes this as `pool.clear(session)` for explicit reset; not
+ fired on warm acquire.
+- **`/compact`** — emits `system/compact_boundary` with
+ `compact_metadata.pre_tokens` + `compact_metadata.trigger` (auto vs
+ manual). Both `AICliSession._on_jsonl_event` and
+ `ClaudeSessionPool._handle_jsonl_event` forward it to
+ `CompactionService.record(...)` so the local-threshold path doesn't
+ double-fire.
+
+Commands MachinaOs must **avoid** in PTY-driven automation (they open
+interactive dialogs, kill the session, or open a browser):
+`/feedback`, `/bug`, `/exit`, `/quit`, `/login`, `/logout`,
+`/permissions`, `/agents`, `/mcp`, `/model` (with no arg), `/rewind`,
+`/config`, `/install-github-app`, `/install-slack-app`, `/teleport`,
+`/heapdump`.
+
+## Memory bridge — unchanged
+
+The `simpleMemory.last_session_id` ↔ `--resume <UUID>` contract works
+identically in interactive mode because the on-disk JSONL is the
+same format. The invariants:
+
+- `memory_bound=True` makes the spawn cwd = `repo_root` so claude's
+ `project_key` stays constant across batches.
+- First run: claude assigns its own UUID; the pool / `AICliSession`
+ discovers it via the new JSONL filename, saves it to
+ `simpleMemory.last_session_id`.
+- Subsequent runs (non-pooled): argv `--resume <last_session_id>`.
+- Subsequent runs (pooled, warm): no respawn — the warm PTY continues
+ in its current UUID.
+- Subsequent runs (pooled, after `pool.clear`): a new UUID is captured
+ on the next `result` event and persisted.
+- Auto-clear-stale: if claude reports `No conversation found with
+ session ID: <UUID>`, `_persist_memory` wipes the broken
+ `last_session_id` so the next run falls into the first-run branch.
+
+## Out-of-process pty-host (deferred upgrade path)
+
+If `pywinpty` / `ptyprocess` stability becomes an issue (crashes
+bringing down FastAPI), promote `PtyTransport` to an out-of-process
+Python sidecar — VSCode's `ptyHostMain.ts` pattern, in Python.
+Supervised by `machina/tree.py`'s Job Object, JSON-RPC over stdio.
+The `PtyTransport` Protocol boundary makes the swap cheap (~1 day).
+See VSCode issue
+[#74620](https://github.com/microsoft/vscode/issues/74620) for the
+rationale (heartbeat, flow control, isolation from main app).
+
+## Open follow-ups
+
+- **Cancellation path.** Pool sessions bypass
+ `AICliService._active_sessions`, so `cancel_workflow` /
+ `cancel_node` don't reach the pooled PTY. Add pool-in-flight
+ tracking or hook cancel into `pool.terminate`.
+- **Lifespan integration.** Wire `pool.shutdown_all()` to FastAPI's
+ shutdown event from `main.py`. Currently the pool dies with the
+ process; orphan PTY children are caught by `machina/tree.py` Job
+ Object on Windows and `os.setsid` on POSIX.
+- **Frontend usage panel.** `claude.session.usage` CloudEvents are
+ flowing; the React hook + simpleMemory panel UI are the missing
+ half (Step 7 of the implementation plan).
+- **`--fork-session` UI.** `--fork-session` is the documented working
+ fork primitive; surface a "Branch this conversation" `<ActionButton>`
+ on simpleMemory + a `branches: List[{uuid, name, created_at}]`
+ param. Avoids `/rewind` which has bug
+ [#55347](https://github.com/anthropics/claude-code/issues/55347)
+ (documented as fork but implemented as in-place mutation).
+- **External cost cap / turn cap.** Replacements for the dropped
+ `--max-budget-usd` and `--max-turns` flags. Aggregate
+ `result.total_cost_usd` / count `assistant` events with
+ `stop_reason: "tool_use"` and kill the PTY at the limit.