Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
docs: add Nerdy Docs section to README, sync counts, add 4 new architecture docs
trohitg2bc11f3fba49
diff --git a/CLAUDE.md b/CLAUDE.mdindex 2bbda0ec..94211bca 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -31,6 +31,13 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire| **[Onboarding Service](./docs-internal/onboarding.md)** | First-launch welcome wizard with 5 steps, database persistence, and replay from Settings || **[CLI Services Integration](./docs-internal/cli_services_integration.md)** | Guide for integrating CLI-based services (Temporal, etc.) with proper lifecycle management || **[Temporal Architecture](./docs-internal/TEMPORAL_ARCHITECTURE.md)** | Distributed workflow execution: activities, FIRST_COMPLETED scheduling, horizontal scaling |+| **[Native LLM SDK](./docs-internal/native_llm_sdk.md)** | Native SDK layer in services/llm/: Protocol-based providers, config-driven base URLs, 10 providers, native vs LangChain path routing |+| **[Event Waiter System](./docs-internal/event_waiter_system.md)** | Generic asyncio.Future/Redis-Streams waiter for push-based trigger nodes (WhatsApp, Telegram, Webhook, Chat, Task completion) |+| **[Credentials Encryption](./docs-internal/credentials_encryption.md)** | Fernet + PBKDF2 encryption pipeline, separate credentials.db, two credential systems (OAuth vs API keys), multi-backend abstraction |+| **[Status Broadcaster](./docs-internal/status_broadcaster.md)** | WebSocket-first communication: StatusBroadcaster singleton, 89 handlers, broadcast message types, Android two-state model |+| **[RLM Service](./docs-internal/rlm_service.md)** | Recursive Language Model agent with REPL-based execution (llm_query, rlm_query, FINAL) |+| **[Claude Code Agent](./docs-internal/claude_code_agent_architecture.md)** | Claude Code SDK integration as a specialized agent node |+| **[Autonomous Agent Creation](./docs-internal/autonomous_agent_creation.md)** | Creating autonomous agents with Code Mode patterns and agentic loops || **[Polyglot Server](../polyglot-server/ARCHITECTURE.md)** | Plugin registry microservice with MCP gateway (optional integration) |## Design Principles & Standards@@ -388,8 +395,8 @@ class CacheEntry(SQLModel, table=True):## Codebase Summary- **Hybrid architecture**: Node.js + Python + React TypeScript-- **96 implemented workflow nodes** with clean service separation (9 AI models + 3 AI agents/memory + 13 specialized agents + 1 skill + 4 dedicated tools + 3 search + 16 Android + 3 WhatsApp + 4 Twitter + 2 Telegram + 2 Social + 3 Location + 3 Code + 6 Utility + 6 Document + 2 Chat + 2 Scheduler + 2 Workflow + 7 Google Workspace + 1 Apify + 1 Crawlee + 3 Proxy)-- **WebSocket-First Architecture**: WebSocket as primary frontend-backend communication (125 message handlers)+- **96 implemented workflow nodes** with clean service separation (9 AI models + 3 AI agents/memory + 15 specialized agents + 1 skill + 4 dedicated tools + 3 search + 16 Android + 3 WhatsApp + 4 Twitter + 2 Telegram + 2 Social + 3 Location + 3 Code + 6 Utility + 6 Document + 2 Chat + 2 Scheduler + 2 Workflow + 7 Google Workspace + 1 Apify + 1 Crawlee + 3 Proxy)+- **WebSocket-First Architecture**: WebSocket as primary frontend-backend communication (89 message handlers)- **Recent optimizations**: REST APIs replaced with WebSocket, AI endpoints migrated to Python, Android automation integrated## Architecture Refactoring@@ -413,7 +420,7 @@ The project was completely refactored from schema-based node definitions to expl- `src/nodeDefinitions/aiModelNodes.ts` - AI chat model definitions using factory pattern- `src/nodeDefinitions/aiAgentNodes.ts` - AI agent and processing components- `src/nodeDefinitions/skillNodes.ts` - Skill node definition (masterSkill aggregator)-- `src/nodeDefinitions/specializedAgentNodes.ts` - Specialized AI agent definitions (13 nodes) with shared AI_AGENT_PROPERTIES and centralized dracula theming+- `src/nodeDefinitions/specializedAgentNodes.ts` - Specialized AI agent definitions (15 nodes) with shared AI_AGENT_PROPERTIES and centralized dracula theming- `src/nodeDefinitions/toolNodes.ts` - AI Agent tool nodes (calculatorTool, currentTimeTool, duckduckgoSearch)- `src/nodeDefinitions/searchNodes.ts` - Search API nodes (braveSearch, serperSearch, perplexitySearch)- `src/nodeDefinitions/androidServiceNodes.ts` - 16 Android service nodes (monitoring, apps, automation, sensors, media)@@ -544,7 +551,7 @@ React Flow edges use dynamic Dracula colors via `getEdgeStyles(theme.dracula)`:### WebSocket-First ArchitectureThe project uses WebSocket as the primary communication method between frontend and backend, replacing most REST API calls:- `src/contexts/WebSocketContext.tsx` - Central WebSocket context with request/response pattern-- `server/routers/websocket.py` - WebSocket endpoint with 125 message handlers+- `server/routers/websocket.py` - WebSocket endpoint with 89 message handlers- `server/services/status_broadcaster.py` - Connection management and broadcasting## Implemented Node Types@@ -1438,7 +1445,7 @@ See **[Scripts Reference](./docs-internal/SCRIPTS.md)** for full documentation.## Current Status✅ **INodeProperties System**: Fully implemented with 93 functional node components-✅ **WebSocket-First Architecture**: 125 message handlers replacing REST APIs+✅ **WebSocket-First Architecture**: 89 message handlers replacing REST APIs✅ **Code Editor**: Python, JavaScript, and TypeScript executors with syntax-highlighted editor (react-simple-code-editor + prismjs) and console output✅ **Node.js Executor**: Persistent Node.js server (Express + tsx) for fast JS/TS execution, replacing subprocess spawning✅ **Component Palette**: Emoji icons with distinct dracula-themed category colors, localStorage persistence for collapsed sections@@ -5005,7 +5012,7 @@ This function:- **Performance**: Fast HMR updates and clean TypeScript compilation- **AI Architecture**: 5-layer system with factory pattern and secure credential management- **Android Architecture**: Factory-based node creation with ADB integration for device automation-- **WebSocket-First Architecture**: 125 message handlers replace REST APIs for parameters, execution, API keys, Android, WhatsApp, and skill operations+- **WebSocket-First Architecture**: 89 message handlers replace REST APIs for parameters, execution, API keys, Android, WhatsApp, and skill operations- **WebSocket Hooks**: Dedicated React hooks (useWhatsApp, useExecution, useApiKeys, useAndroidOperations, useParameterPanel) for clean component integration- **WebSocket Support**: Persistent remote Android device connections via WebSocket proxy with background tasks- Connection stays alive across multiple API requests until switched to local ADBdiff --git a/README.md b/README.mdindex 64d10123..623836ba 100644--- a/README.md+++ b/README.md@@ -3,6 +3,7 @@<a href="https://www.npmjs.com/package/machinaos" target="_blank"><img src="https://img.shields.io/npm/v/machinaos.svg" alt="npm version"></a><a href="https://opensource.org/licenses/MIT" target="_blank"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT"></a><a href="https://discord.gg/c9pCJ7d8Ce" target="_blank"><img src="https://img.shields.io/discord/1455977012308086895?logo=discord&logoColor=white&label=Discord" alt="Discord"></a>+<a href="https://deepwiki.com/trohitg/MachinaOS" target="_blank"><img src="https://deepwiki.com/badge.svg" alt="Ask DeepWiki"></a>Personal AI Assistant/Co-Employees that Perform Human Like Tasks. Connect AI Assistant to your email, calendar, messages, phone, and more.@@ -134,3 +135,538 @@ Join our [Discord](https://discord.gg/NHUEQVSC) for help, feedback, and updates.## LicenseMIT++---++## Nerdy Docs++A detailed look at how MachinaOS works under the hood. For the TL;DR just use the product; this section is for contributors, system designers, and the curious. For a deep, navigable tour the [DeepWiki badge](https://deepwiki.com/trohitg/MachinaOS) at the top of this README covers the same ground with hyperlinks. In-repo docs live under [docs-internal/](docs-internal/).++### System Overview++MachinaOS is three loosely-coupled tiers talking over a single persistent WebSocket connection:++```++------------------------------------------------------------------------++| Frontend tier (client/) |+| React + TypeScript + React Flow + Zustand + Ant Design |+| Dracula-themed canvas, parameter panel, credentials modal |++------------------------------------------------------------------------++ |+ WebSocket | /ws/status (single long-lived connection)+ v++------------------------------------------------------------------------++| Backend tier (server/) |+| FastAPI + 89 WebSocket handlers + dependency injection container |+| WorkflowService (facade) -> NodeExecutor -> per-node handlers |+| ParameterResolver -> template {{node.field}} substitution |+| AuthService -> encrypted credentials (Fernet + PBKDF2) |++------------------------------------------------------------------------++ |+ v++------------------------------------------------------------------------++| Execution tier |+| Temporal workers (distributed) or local asyncio decide loop |+| Redis (cache, locks, streams) SQLite (machina.db, credentials.db) |+| Node.js server (JS/TS code exec) WhatsApp RPC (port 9400) |+| Android relay WebSocket Temporal server (ports 7233/8080) |++------------------------------------------------------------------------++ |+ v+ External services: OpenAI, Anthropic, Gemini, OpenRouter, xAI,+ DeepSeek, Kimi, Mistral, Groq, Cerebras, Google Workspace, WhatsApp,+ Telegram, Twitter/X, Brave, Serper, Perplexity, Apify, residential+ proxies, webhooks.+```++Notable characteristics at a glance:++- **96 workflow nodes** across AI models, agents, social, Android, document processing, Google Workspace, code executors, proxies, and utilities+- **10 LLM providers**: OpenAI, Anthropic, Gemini, OpenRouter, xAI, DeepSeek, Kimi, Mistral, Groq, Cerebras+- **15 specialized agents**: android, coding, web, task, social, travel, tool, productivity, payments, consumer, autonomous, orchestrator, ai_employee, rlm, claude_code+- **89 WebSocket handlers** replacing most REST endpoints+- **49 built-in skills** across 10 categories (assistant, android_agent, autonomous, coding_agent, productivity_agent, rlm_agent, social_agent, task_agent, travel_agent, web_agent)+- **Three execution modes** with automatic fallback: Temporal distributed -> Redis parallel -> sequential+- **Encrypted credentials** stored in a separate SQLite database using Fernet + PBKDF2 (600K iterations)++<details>+<summary><b>Execution Engine</b></summary>++### Conductor's Decide Pattern++Workflow orchestration is a single function with fork/join parallelism:++```+_workflow_decide(ctx):+ 1. Find ready nodes (all dependencies satisfied)+ 2. asyncio.gather() the ready layer -> run in parallel+ 3. Checkpoint state to Redis/SQLite+ 4. Recurse until every node terminal (completed / skipped / failed)+```++Each workflow run has its own isolated `ExecutionContext` (`server/services/execution/models.py`). No shared global state between concurrent runs. Decide loops serialize per execution via Redis `SETNX` distributed locks.++### Three Execution Modes++```+workflow.execute(workflow_id, workflow_data)+ |+ v+ TEMPORAL_ENABLED and server reachable?+ |+ yes -> _execute_temporal() -- per-node activities,+ retries, horizontal scaling+ (primary production mode)+ no -> Redis available?+ |+ yes -> _execute_parallel() -- decide loop + Kahn layers+ + Prefect-style input hash cache+ + DLQ + heartbeat recovery+ no -> _execute_sequential() -- topological walk+ (fallback for minimal env)+```++### Layer Computation via Kahn's Algorithm++Before execution, the DAG is sorted into layers. Layer 0 is the set of nodes with no dependencies; each subsequent layer depends only on earlier layers. Parallel execution runs each layer with `asyncio.gather()`:++```+Layer 0: [start, cronScheduler]+Layer 1: [httpRequest, whatsappReceive]+Layer 2: [aiAgent]+Layer 3: [whatsappSend, console]+```++Toolkit sub-nodes (e.g., Android service nodes connected to `androidTool`) are detected and excluded from execution layers -- they run only when the parent toolkit invokes them via tool calling. See the Toolkit Sub-Node Execution Pattern in [CLAUDE.md](CLAUDE.md).++### Result Caching, Recovery, and DLQ++- **Prefect-style caching**: every node result is stored in Redis/SQLite keyed by `hash_inputs(inputs)`. Re-running an identical node returns the cached result with status `TaskStatus.CACHED`.+- **Heartbeat recovery**: `RecoverySweeper` scans `executions:active` every 60s; nodes with stale heartbeats (>5 min) are marked stuck and recovered on next startup.+- **Dead Letter Queue**: failed nodes (after retry exhaustion) are quarantined in `dlq:entries:*` with full input snapshot. Inspect, replay, or purge via the `get_dlq_*` / `replay_dlq_entry` / `purge_dlq` WebSocket handlers.++### Edge Conditions++Edges carry optional conditions for runtime branching with 20+ operators (`eq`, `neq`, `gt`, `lt`, `contains`, `exists`, `matches`, `in`, ...). Unmatched branches are marked `TaskStatus.SKIPPED`.++**Read more**: [DESIGN.md](docs-internal/DESIGN.md) - [TEMPORAL_ARCHITECTURE.md](docs-internal/TEMPORAL_ARCHITECTURE.md) - [workflow-schema.md](docs-internal/workflow-schema.md)++</details>++<details>+<summary><b>Event-Driven Deployment</b></summary>++Deployments are event-driven: each trigger event spawns an **independent concurrent execution run**. There is no iteration loop.++```+deploy_workflow(workflow_id)+ |+ v+ Set up triggers, return+ |+ +-> cronScheduler fires -> ExecutionRun 1 (isolated context)+ +-> cronScheduler fires -> ExecutionRun 2 (isolated context)+ +-> whatsappReceive fires -> ExecutionRun 3 (isolated context)+ +-> webhookTrigger fires -> ExecutionRun 4 (isolated context)+ +-> telegramReceive fires -> ExecutionRun 5 (isolated context)+ +-> taskTrigger fires -> ExecutionRun 6 (from delegated agent)+```++Multiple runs execute simultaneously with no interference. The firing trigger is marked complete before downstream execution starts; every other trigger node in the same run is auto-marked `_pre_executed` with `{not_triggered: True}` so they never block as event waiters.++### Push vs Polling Triggers++```+Push triggers (asyncio.Future + dispatch):+ whatsappReceive, webhookTrigger, chatTrigger, taskTrigger,+ telegramReceive, start++Polling triggers (asyncio.Queue + poll coroutine):+ twitterReceive (X API has no webhook on free tier)+ gmailReceive (Gmail push requires paid Google Cloud setup)++Scheduler:+ cronScheduler (APScheduler directly, not through event waiter)+```++Every push trigger goes through the generic `event_waiter` module: register a `Waiter` with a filter closure, suspend on `wait_for_event()`, resume when an external service dispatches a matching event. Backend supports both in-memory (`asyncio.Future`) and Redis Streams modes for Temporal multi-worker deployments.++**Read more**: [event_waiter_system.md](docs-internal/event_waiter_system.md) - [workflow-schema.md](docs-internal/workflow-schema.md)++</details>++<details>+<summary><b>Node System (96 Nodes)</b></summary>++### INodeProperties Interface++Node definitions follow an n8n-inspired TypeScript interface system. Every node has a strongly-typed `INodeTypeDescription` with `displayName`, `group`, `inputs`, `outputs`, and `properties`. Parameters support conditional display via `displayOptions.show`, type-specific editors (string, number, boolean, options, collection, file, array), drag-and-drop template variable mapping, and live validation.++Source of truth: `client/src/nodeDefinitions/` (23 files).++### Category Breakdown++| Category | Count | File |+|---|---|---|+| AI Chat Models | 9 | `aiModelNodes.ts` |+| AI Agents + Memory | 3 | `aiAgentNodes.ts` |+| Specialized Agents | 15 | `specializedAgentNodes.ts` |+| AI Tools (dedicated) | 4 | `toolNodes.ts` |+| Skills (masterSkill) | 1 | `skillNodes.ts` |+| Search (dual-purpose) | 3 | `searchNodes.ts` |+| Location / Maps | 3 | `locationNodes.ts` |+| Google Workspace | 7 | `googleWorkspaceNodes.ts` |+| WhatsApp | 3 | `whatsappNodes.ts` |+| Twitter/X | 4 | `twitterNodes.ts` |+| Telegram | 2 | `telegramNodes.ts` |+| Social (unified) | 2 | `socialNodes.ts` |+| Android services | 16 | `androidServiceNodes.ts` |+| Code executors | 3 | `codeNodes.ts` |+| Document processing | 6 | `documentNodes.ts` |+| Utilities | 6 | `utilityNodes.ts` |+| Proxy | 3 | `proxyNodes.ts` |+| Web scraping | 2 | `apifyNodes.ts` + `crawleeNodes.ts` |+| Chat | 2 | `chatNodes.ts` |+| Schedulers | 2 | `schedulerNodes.ts` |+| Workflow triggers | 2 | `workflowNodes.ts` |++### Dispatch Registry (backend)++`NodeExecutor` in `server/services/node_executor.py` uses a registry built with `functools.partial` for dependency injection:++```python+registry = {+ 'start': handle_start,+ 'aiAgent': partial(handle_ai_agent, ai_service=self.ai_service, database=self.database),+ 'chatAgent': partial(handle_chat_agent, ai_service=self.ai_service, database=self.database),+ 'rlm_agent': partial(handle_rlm_agent, ai_service=self.ai_service, database=self.database),+ 'gmail': partial(handle_google_gmail, database=self.database),+ 'whatsappSend': partial(handle_whatsapp_send, database=self.database),+ # ... one entry per node type+}+```++No if-else chains, no global singletons.++### Dual-Purpose Nodes++Many nodes work as **both** workflow nodes and AI Agent tools. Examples: `whatsappSend`, `whatsappDb`, `twitterSend`, `twitterSearch`, `twitterUser`, `telegramSend`, `socialSend`, `httpRequest`, `pythonExecutor`, `javascriptExecutor`, `typescriptExecutor`, `braveSearch`, `serperSearch`, `perplexitySearch`, `apifyActor`, `proxyConfig`, `gmaps_locations`, `gmaps_nearby_places`, and all 7 Google Workspace nodes. When connected to an AI agent's `input-tools` handle, the LLM fills the node's parameter schema via structured tool calls.++### Config Node Pattern++Four special handles on agent nodes accept **config nodes** instead of main data flow:++| Handle | Purpose |+|---|---|+| `input-memory` | Connect a `simpleMemory` node for conversation history |+| `input-tools` | Connect tool nodes (dedicated tools or dual-purpose nodes) |+| `input-skill` | Connect skill nodes (`masterSkill` aggregator) |+| `input-teammates` | Connect other agents (team lead pattern only) |++Config nodes are automatically excluded from parallel execution layers and inherit their parent agent's main-input connections for template variable mapping.++**Read more**: [node_creation.md](docs-internal/node_creation.md) - [dual_purpose_tool_node_creation.md](docs-internal/dual_purpose_tool_node_creation.md) - [ai_tool_node_creation.md](docs-internal/ai_tool_node_creation.md)++</details>++<details>+<summary><b>AI Agent System</b></summary>++### 10 LLM Providers++| Provider | Native SDK Path | LangChain Path | Notes |+|---|---|---|---|+| OpenAI | yes (`providers/openai.py`) | yes | GPT 4.x/5.x + o-series reasoning |+| Anthropic | yes (`providers/anthropic.py`) | yes | Extended thinking via `budget_tokens` |+| Gemini | yes (`providers/gemini.py`) | yes (fallback) | Native bypasses LangChain Windows hang |+| OpenRouter | yes (`providers/openrouter.py`) | yes | 200+ models through one API |+| xAI | yes (shared OpenAIProvider) | yes | OpenAI-compatible |+| DeepSeek | yes (shared OpenAIProvider) | yes | Chat + Reasoner (always-on CoT) |+| Kimi | yes (shared OpenAIProvider) | yes | Moonshot K2.5 / K2-thinking |+| Mistral | yes (shared OpenAIProvider) | yes | Large / Small / Codestral |+| Groq | no (LangChain only) | yes | Llama 4, Qwen3, GPT-OSS |+| Cerebras | no (LangChain only) | yes | Llama, Qwen |++### Dual-Path Architecture++```+execute_chat(parameters) execute_agent(parameters)+ direct chat completions LangGraph tool-calling loop+ | |+ v v+is_native_provider(provider)? create_model(provider, ...)+ | |+ yes -+-- create_provider() -> LLMResponse v+ | LangChain ChatOpenAI /+ no -+-- create_model() -> chat_model ChatAnthropic /+ .invoke() -> LLMResponse ChatGoogleGenerativeAI+ |+ v+ bind_tools(...)+ |+ v+ LangGraph StateGraph+ (agent node <-> tools node loop)+```++The native path returns a normalized `LLMResponse` dataclass across all providers. The LangChain path is used for agent tool-calling because LangGraph's checkpointer, state graph, and tool-execution callback layer have no native equivalent today.++### 15 Specialized Agents++All specialized agents share the same handle architecture (`input-main`, `input-memory`, `input-skill`, `input-tools`, `input-task`) and inherit `AI_AGENT_PROPERTIES`. They only differ in icon, title, theme color, and default prompt.++```+android_agent coding_agent web_agent task_agent+social_agent travel_agent tool_agent productivity_agent+payments_agent consumer_agent autonomous_agent orchestrator_agent+ai_employee rlm_agent claude_code_agent+```++Routing:++- **13 agents** route to `handle_chat_agent` (LangGraph loop, shared code path)+- **`rlm_agent`** routes to `handle_rlm_agent` -> `RLMService` (REPL-based recursive LM)+- **`claude_code_agent`** routes to `handle_claude_code_agent` -> Claude Code SDK++### Agent Teams Topology++`orchestrator_agent` and `ai_employee` have an extra `input-teammates` handle. Connected agents become `delegate_to_<agent_type>` tools automatically:++```+ +-------------------------++ | AI Employee |+ | (orchestrator_agent) |+ +-----+------+------+-----++ | | | input-teammates+ +-------------+ | +------------++ | | |+ +------v------+ +-------v-----+ +-------v-----++ | coding_agent| | web_agent | | task_agent |+ +-------------+ +-------------+ +-------------++ delegate_ delegate_ delegate_+ to_coding_ to_web_ to_task_+ agent tool agent tool agent tool+```++The team lead's LLM decides when to delegate based on task context. Delegation is **fire-and-forget**: the child spawns as `asyncio.create_task()`, the parent continues, and the child broadcasts its own status updates independently. Results can be retrieved via the auto-injected `check_delegated_tasks` tool or consumed by `taskTrigger` nodes elsewhere in the workflow.++### LangGraph StateGraph Flow++```+ +---------++ START ---> | agent |+ | (LLM) |+ +----+----++ |+ should_continue()?+ / \+ tools / \ end+ v v+ +---------+ END+ | tools |+ | (exec) |+ +----+----++ |+ +---------> loop back to agent+```++Max iterations guard against infinite tool loops. Tools are built as Pydantic-schema `StructuredTool` instances with the node's parameter schema.++**Read more**: [agent_architecture.md](docs-internal/agent_architecture.md) - [agent_teams.md](docs-internal/agent_teams.md) - [agent_delegation.md](docs-internal/agent_delegation.md) - [native_llm_sdk.md](docs-internal/native_llm_sdk.md) - [rlm_service.md](docs-internal/rlm_service.md)++</details>++<details>+<summary><b>Memory, Skills, Tokens, and Cost</b></summary>++### Markdown-Based Memory++The `simpleMemory` node stores conversation history as editable Markdown in the parameter panel. The AI Agent handler reads, parses to LangChain messages, executes, appends the new exchange, trims to a window, and archives removed messages to an `InMemoryVectorStore` (optional) using HuggingFace `BAAI/bge-small-en-v1.5` embeddings.++```+AI Agent reads memoryContent (Markdown)+ |+ v+_parse_memory_markdown() -> LangChain Messages+ |+ v+(optional) vector store similarity_search(prompt, k=retrievalCount)+ |+ v+Execute LLM with full history + retrieved context+ |+ v+Append human + ai messages to Markdown+ |+ v+_trim_markdown_window(windowSize) -> (kept, removed)+ |+ v+If longTermEnabled: store.add_texts(removed)+ |+ v+Save updated Markdown back to node parameters+```++### Skill System++49 built-in skills across 10 folders under `server/skills/`:++```+assistant/ (5) android_agent/ (12) autonomous/ (5)+coding_agent/ (2) productivity_agent/ (6) rlm_agent/ (1)+social_agent/ (5) task_agent/ (3) travel_agent/ (2)+web_agent/ (8)+```++Each skill is a folder with a `SKILL.md` file containing YAML frontmatter (name, description, allowed-tools, metadata) and Markdown instructions. First load seeds the database; after that the database is source of truth so users can edit skill instructions in the UI. "Reset to Default" reloads the original `SKILL.md`.++The `masterSkill` node aggregates multiple skills with enable/disable toggles. A split-panel editor shows the skill list on the left and the selected skill's Markdown on the right. When connected to an agent, the backend expands the `skillsConfig` parameter into individual skill entries injected into the agent's system message.++### Token Tracking and Compaction++Every AI execution stores a `TokenUsageMetric` row with input/output/cache/reasoning token counts and calculated costs (USD) based on `server/config/pricing.json`. Cumulative state per session lives in `SessionTokenState`.++Compaction threshold priority:++1. Per-session `custom_threshold` (user-set)+2. **Model-aware**: 50% of the model's context window (e.g., 500K for Claude Opus 4.6 with 1M context)+3. Global `COMPACTION_THRESHOLD` fallback from `.env`++When the cumulative session tokens cross the threshold, `CompactionService.compact_context()` generates a 5-section summary (Task Overview, Current State, Important Discoveries, Next Steps, Context to Preserve) following the Claude Code pattern and replaces the memory content. Anthropic and OpenAI also have native compaction APIs (`context_management` edits, `compact_threshold`) that are configured transparently via `svc.anthropic_api_config()` and `svc.openai_config()`.++**Read more**: [memory_compaction.md](docs-internal/memory_compaction.md) - [pricing_service.md](docs-internal/pricing_service.md) - [Skill Creation Guide](server/skills/GUIDE.md)++</details>++<details>+<summary><b>Communication Layer</b></summary>++A single persistent WebSocket at `/ws/status` handles all frontend-backend communication. There are **89 WebSocket handlers** registered with the `@ws_handler` decorator in `server/routers/websocket.py`, covering: node parameters, tool schemas, node execution, triggers, dead letter queue, deployment, AI operations, API keys, OAuth flows (Claude, Twitter, Google), Android, WhatsApp, Telegram, workflow storage, chat messages, console logs, skills, memory, user settings, pricing, agent teams, and model registry.++### Request/Response Pattern++```+Frontend Backend+ | |+ |-- {type: "execute_node", data: ...}-->|+ | |-- dispatch to handle_execute_node()+ | |-- run node handler+ | |+ |<-- {type: "node_status", ... }---------| (broadcast)+ |<-- {type: "node_output", ... }---------| (broadcast)+ |<-- {type: "execute_node_response", ...}| (direct reply, request_id matched)+```++### Broadcast Message Types++```+node_status workflow_status api_key_status+node_output android_status token_usage_update+variable_update compaction_starting node_parameters_updated+variables_update compaction_completed+```++### Auto-Reconnect and Keepalive++Frontend sends `{"type": "ping"}` every 30 seconds. On disconnect, the `WebSocketContext` schedules a reconnect after 3 seconds with a 100ms mount delay to avoid React Strict Mode double-connect in development. Connection is gated on `isAuthenticated` so logged-out users never connect.++**Read more**: [status_broadcaster.md](docs-internal/status_broadcaster.md)++</details>++<details>+<summary><b>Cache, Persistence, and Security</b></summary>++### Cache Fallback Hierarchy++MachinaOS follows the n8n cache pattern with automatic environment-based fallback:++```+Production (Docker): Redis -> SQLite (cache_entries) -> in-memory dict+Local development: SQLite (cache_entries) -> in-memory dict+ (Redis disabled via REDIS_ENABLED=false)+```++`CacheService` in `server/core/cache.py` checks each backend in order. TTL expiration is supported across all three. A background `cleanup_expired_cache()` task removes expired SQLite rows.++### Encrypted Credentials++API keys and OAuth tokens live in a **separate SQLite database** (`credentials.db`) isolated from the main `machina.db`. Encryption uses Fernet (AES-128-CBC + HMAC-SHA256) with keys derived from a server-scoped config key via PBKDF2HMAC (600K iterations, OWASP 2024 recommendation).++```+API_KEY_ENCRYPTION_KEY (.env) + salt (from credentials.db, 256 bits)+ |+ v+ PBKDF2HMAC-SHA256+ (600,000 iterations)+ |+ v+ urlsafe_b64encode -> Fernet key+ |+ v+ held in memory only+ |+ v+ encrypt() / decrypt() inside EncryptionService+```++Two separate credential systems inside `credentials.db`:++- **API key system** (`EncryptedAPIKey` table): provider keys the user enters manually+- **OAuth token system** (`EncryptedOAuthToken` table): tokens from OAuth flows (Google, Twitter, Claude.ai)++All routers access credentials through `AuthService` only. Direct database access is forbidden. Decrypted values are cached in `AuthService` memory dicts and never written to disk or Redis.++For cloud deployments, `CREDENTIAL_BACKEND` can be switched to `keyring` (OS-native) or `aws` (Secrets Manager) via `CredentialBackend` abstraction.++### Authentication++JWT in HttpOnly cookies following the n8n pattern. Two modes:++- `AUTH_MODE=single` - first user becomes owner, registration disabled after+- `AUTH_MODE=multi` - open registration for cloud deployments+- `VITE_AUTH_ENABLED=false` - bypass login entirely for local development++WebSocket connections check the JWT cookie before accepting. Auth context has exponential-backoff retry (5 attempts) to survive race conditions where the frontend starts before the backend is ready.++**Read more**: [credentials_encryption.md](docs-internal/credentials_encryption.md)++</details>++<details>+<summary><b>Full Documentation Index</b></summary>++| Document | Description |+|---|---|+| [DESIGN.md](docs-internal/DESIGN.md) | Execution engine architecture, design patterns, execution modes |+| [TEMPORAL_ARCHITECTURE.md](docs-internal/TEMPORAL_ARCHITECTURE.md) | Distributed execution via Temporal activities |+| [workflow-schema.md](docs-internal/workflow-schema.md) | Workflow JSON schema and full node catalog (96 nodes) |+| [ROADMAP.md](docs-internal/ROADMAP.md) | Implementation status and completed phases |+| [SETUP.md](docs-internal/SETUP.md) | Development environment setup |+| [SCRIPTS.md](docs-internal/SCRIPTS.md) | npm/shell scripts reference |+| [server-readme.md](docs-internal/server-readme.md) | Python backend architecture and API |+| [agent_architecture.md](docs-internal/agent_architecture.md) | AI Agent / Chat Agent skill and tool discovery |+| [agent_delegation.md](docs-internal/agent_delegation.md) | How delegated agents share context and memory |+| [agent_teams.md](docs-internal/agent_teams.md) | Agent Teams pattern with `input-teammates` handle |+| [native_llm_sdk.md](docs-internal/native_llm_sdk.md) | Native LLM SDK layer and provider protocol |+| [rlm_service.md](docs-internal/rlm_service.md) | Recursive Language Model agent via REPL |+| [claude_code_agent_architecture.md](docs-internal/claude_code_agent_architecture.md) | Claude Code SDK integration as a specialized agent |+| [autonomous_agent_creation.md](docs-internal/autonomous_agent_creation.md) | Autonomous agents with Code Mode patterns |+| [event_waiter_system.md](docs-internal/event_waiter_system.md) | Push-based trigger waiters |+| [status_broadcaster.md](docs-internal/status_broadcaster.md) | WebSocket broadcaster and 89 handlers |+| [credentials_encryption.md](docs-internal/credentials_encryption.md) | Fernet + PBKDF2 credentials system |+| [memory_compaction.md](docs-internal/memory_compaction.md) | Token tracking and model-aware compaction |+| [pricing_service.md](docs-internal/pricing_service.md) | LLM and API cost tracking |+| [proxy_service.md](docs-internal/proxy_service.md) | Residential proxy provider management |+| [ci_cd.md](docs-internal/ci_cd.md) | GitHub Actions workflows |+| [node_creation.md](docs-internal/node_creation.md) | How to create new nodes |+| [ai_tool_node_creation.md](docs-internal/ai_tool_node_creation.md) | Creating dedicated AI Agent tool nodes |+| [specialized_agent_node_creation.md](docs-internal/specialized_agent_node_creation.md) | Creating specialized AI agent nodes |+| [dual_purpose_tool_node_creation.md](docs-internal/dual_purpose_tool_node_creation.md) | Dual-purpose workflow + tool nodes |+| [new_service_integration.md](docs-internal/new_service_integration.md) | External service integration guide |+| [cli_services_integration.md](docs-internal/cli_services_integration.md) | CLI service lifecycle management |+| [onboarding.md](docs-internal/onboarding.md) | Welcome wizard and replay |+| [Skill Creation Guide](server/skills/GUIDE.md) | How to create new skills |++</details>diff --git a/docs-internal/DESIGN.md b/docs-internal/DESIGN.mdindex 5d66270b..054423cb 100644--- a/docs-internal/DESIGN.md+++ b/docs-internal/DESIGN.md@@ -2,7 +2,91 @@## Overview-The MachinaOs execution engine implements a robust workflow orchestration system combining industry-standard patterns from **Netflix Conductor**, **Prefect 3.0**, and **Redis Streams**. This document details the architectural decisions, design patterns, and standards used.+The MachinaOs execution engine implements a robust workflow orchestration system combining industry-standard patterns from **Netflix Conductor**, **Prefect 3.0**, **Temporal**, and **Redis Streams**. This document details the architectural decisions, design patterns, and standards used.++Related docs:+- [TEMPORAL_ARCHITECTURE.md](TEMPORAL_ARCHITECTURE.md) - distributed execution via Temporal activities+- [event_waiter_system.md](event_waiter_system.md) - push-based trigger waiters+- [native_llm_sdk.md](native_llm_sdk.md) - LLM provider layer+- [rlm_service.md](rlm_service.md) - Recursive Language Model agent execution+- [proxy_service.md](proxy_service.md) - residential proxy provider management+- [workflow-schema.md](workflow-schema.md) - workflow JSON schema and node catalog++---++## Execution Modes++`WorkflowService` in `server/services/workflow.py` is a thin facade (~460 lines) that routes every workflow run through one of three execution modes. Selection is based on available infrastructure:++```+workflow.execute(workflow_id, workflow_data)+ |+ v+ +---------------------++ | TEMPORAL_ENABLED | yes --> _execute_temporal()+ | and Temporal up? | |+ +---------------------+ v+ | no TemporalExecutor+ v (per-node activities,+ +---------------------+ retries, horizontal scaling)+ | Redis available? |+ +---------------------++ | yes --> _execute_parallel()+ | |+ v v+ | WorkflowExecutor+ | (decide loop, Kahn layers,+ | distributed locking)+ |+ | no --> _execute_sequential()+ v+ Sequential fallback+ (simple topological walk)+```++### 1. Temporal Distributed (primary production mode)++When `TEMPORAL_ENABLED=true` and the Temporal server is reachable, every workflow node executes as an independent Temporal activity on the `machina-tasks` task queue.++- Per-node retries (3 attempts by default) and per-node timeouts (10 min default)+- Horizontal scaling via worker pool+- Connection pooling via shared `aiohttp` session for WebSocket-based node execution+- Pure orchestrator in `services/temporal/workflow.py` using `asyncio.wait(FIRST_COMPLETED)` so dependent nodes start immediately when their specific dependency completes+- Class-based activities in `services/temporal/activities.py` with lifecycle hooks++See [TEMPORAL_ARCHITECTURE.md](TEMPORAL_ARCHITECTURE.md) for full details.++### 2. Parallel Local (Redis)++When Temporal is not enabled but Redis is available, workflows use `WorkflowExecutor` in `services/execution/executor.py`:++- Conductor's decide pattern in `_workflow_decide()` under distributed Redis SETNX locks+- Fork/join via `asyncio.gather()` for concurrent node execution+- Prefect-style result caching keyed by `hash_inputs()` (see `services/execution/cache.py`)+- Crash recovery via `RecoverySweeper` with heartbeats+- Dead Letter Queue for failed nodes (`services/execution/dlq.py`)++This is the local-development default when `REDIS_ENABLED=true` is set in `server/.env`.++### 3. Sequential Fallback++When neither Temporal nor Redis is available, workflow execution falls back to a simple topological walk with no parallelism, no caching, and no retry. Intended only for minimal environments and CI smoke tests.++---++## RLM Agent Execution++`rlm_agent` is the only agent node type that does not go through `handle_chat_agent`. It has a dedicated handler `handle_rlm_agent` that routes to `RLMService` in `services/rlm_service.py`.++The Recursive Language Model pattern replaces LangGraph's tool-calling loop with a Python REPL (`exec()`) where the agent writes code that can call:++- `llm_query(prompt)` - invoke the small model connected to `input-model`+- `rlm_query(prompt)` - recursively invoke the agent itself+- `FINAL(answer)` - signal completion++This yields 81-98% token savings on tool-heavy workflows because the agent can orchestrate multiple LLM calls in a single code block instead of paying the round-trip cost of tool calling.++See [rlm_service.md](rlm_service.md) for the full design.---diff --git a/docs-internal/agent_architecture.md b/docs-internal/agent_architecture.mdindex dbb3cf55..33fbc436 100644--- a/docs-internal/agent_architecture.md+++ b/docs-internal/agent_architecture.md@@ -690,20 +690,56 @@ Both methods live in `server/services/ai.py` and follow the same general pattern### Specialized Agent Routing-All 10 specialized agents route to `handle_chat_agent` via the NodeExecutor registry:+There are **15 specialized agents**. Most route to `handle_chat_agent`; `rlm_agent` and `claude_code_agent` have dedicated handlers:```python# server/services/node_executor.pySPECIALIZED_AGENT_TYPES = {'android_agent', 'coding_agent', 'web_agent', 'task_agent', 'social_agent',- 'travel_agent', 'tool_agent', 'productivity_agent', 'payments_agent', 'consumer_agent'+ 'travel_agent', 'tool_agent', 'productivity_agent', 'payments_agent', 'consumer_agent',+ 'autonomous_agent', 'orchestrator_agent', 'ai_employee',+ # rlm_agent and claude_code_agent are handled by dedicated handlers, not handle_chat_agent}-# All specialized agents map to handle_chat_agent+# Most specialized agents map to handle_chat_agentfor agent_type in SPECIALIZED_AGENT_TYPES:registry[agent_type] = partial(handle_chat_agent, ai_service=self.ai_service, database=self.database)++# Dedicated handlers for agents that do not use LangGraph tool-calling+registry['rlm_agent'] = partial(handle_rlm_agent, ai_service=self.ai_service, database=self.database)+registry['claude_code_agent'] = partial(handle_claude_code_agent, ...)+```++**Team leads** (`orchestrator_agent`, `ai_employee`) use the same `handle_chat_agent` routing but add an `input-teammates` handle. Connected agents become `delegate_to_<type>` tools automatically via `_collect_teammate_connections()`. See [agent_teams.md](agent_teams.md).++### RLM Agent Pattern++`rlm_agent` replaces LangGraph tool-calling with a Python REPL executing LM calls recursively. Instead of paying one network round-trip per tool call, the RLM agent writes a code block that orchestrates many model invocations at once:++```python+# The LLM generates code like this, executed by RLMService+results = [llm_query(f"summarize: {url}") for url in urls]+best = rlm_query(f"pick the most relevant: {results}")+FINAL(best)```+Exposed helpers inside the REPL:++| Helper | Purpose |+|---|---|+| `llm_query(prompt)` | Call the small model connected to `input-model` |+| `rlm_query(prompt)` | Recursively invoke the same RLM agent |+| `FINAL(answer)` | Signal completion and return the final answer |++Routing:++```python+# server/services/node_executor.py+registry['rlm_agent'] = partial(handle_rlm_agent, ai_service=self.ai_service, database=self.database)+```++`handle_rlm_agent` delegates to `RLMService` in `server/services/rlm_service.py`. See [rlm_service.md](rlm_service.md) for full details.+### Chat Agent Conditional Graph```pythondiff --git a/docs-internal/credentials_encryption.md b/docs-internal/credentials_encryption.mdnew file mode 100644index 00000000..28fbb871--- /dev/null+++ b/docs-internal/credentials_encryption.md@@ -0,0 +1,190 @@+# Encrypted Credentials System++API keys, OAuth tokens, and other secrets in MachinaOS are stored in a separate encrypted SQLite database (`credentials.db`) using Fernet (AES-128-CBC + HMAC-SHA256). The encryption key is derived from a server-scoped config key using PBKDF2HMAC with 600,000 iterations, following the n8n pattern.++This document covers the encryption pipeline, the two separate credential systems (OAuth vs API keys), the single-point-of-access rule, and the multi-backend abstraction.++## Why a Separate Database++Credentials are isolated from the main `machina.db` for three reasons:++1. **Blast radius**: a dump of `machina.db` for debugging never contains secrets.+2. **Independent backups**: `credentials.db` can be excluded from snapshots and SQLite dumps.+3. **Backend pluggability**: the file-backed SQLite can be swapped for OS keyring or AWS Secrets Manager without touching workflow storage.++## Files++```+server/core/+|-- encryption.py EncryptionService (Fernet + PBKDF2)+|-- credentials_database.py CredentialsDatabase (async SQLite, salt storage)+|-- credential_backends.py Multi-backend abstraction (Fernet / Keyring / AWS)+`-- config.py credential_backend, aws_secret_arn settings++server/services/+`-- auth.py AuthService (single access point, caching)+```++## Cryptographic Pipeline++```+API_KEY_ENCRYPTION_KEY (from .env)+ ++ salt (random 32 bytes, stored in credentials.db)+ |+ v+ PBKDF2HMAC-SHA256+ (600,000 iterations, OWASP 2024)+ |+ v+ urlsafe_b64encode -> 32-byte Fernet key+ |+ v+ Fernet cipher (held in memory for process lifetime)+ |+ v+ encrypt(plaintext) / decrypt(ciphertext)+```++- **AES-128-CBC** for confidentiality (Fernet's block cipher).+- **HMAC-SHA256** for authenticity (Fernet appends a MAC).+- **PKCS7 padding** (handled by Fernet).+- **600,000 iterations** is the OWASP 2024 recommendation for PBKDF2-SHA256.+- **Salt is 256 bits**, generated once on first startup, stored in `credentials.db`.++The derived Fernet key lives only in `EncryptionService._fernet` in process memory. It is never written to disk or to Redis.++## Lifecycle++```+Server startup (main.py lifespan)+ |+ v+CredentialsDatabase.initialize() -> creates tables, returns existing or new salt+ |+ v+EncryptionService.initialize(password=API_KEY_ENCRYPTION_KEY, salt=<bytes>)+ |+ v+AuthService caches decrypted credentials in memory-only dicts+ |+ v+Routers call AuthService.get_api_key() / get_oauth_tokens() ...+ |+ v+(on shutdown) EncryptionService.clear() wipes the in-memory key+```++`EncryptionService.is_initialized()` is checked before any encrypt/decrypt call. If the server key is misconfigured, the service raises at startup rather than returning unusable ciphertext later.++## Two Separate Credential Systems++There are **two distinct storage paths** inside `credentials.db`, and they are not interchangeable. This is the most common source of bugs in this area.++### 1. API Key System++For secrets the user enters manually in the Credentials modal (OpenAI API key, Anthropic key, Google client ID, Google client secret, Twitter client secret, Brave Search key, etc.).++- Table: `EncryptedAPIKey`+- Access: `AuthService.store_api_key(provider, key, models=[...], session_id=...)` and `AuthService.get_api_key(provider)`+- Cache: `AuthService._api_key_cache: Dict[str, str]`++### 2. OAuth Token System++For tokens obtained via OAuth 2.0 flows (Google Workspace, Twitter/X, Claude.ai).++- Table: `EncryptedOAuthToken`+- Access: `AuthService.store_oauth_tokens(provider, access_token, refresh_token, ...)` and `AuthService.get_oauth_tokens(provider, customer_id="owner")`+- Cache: `AuthService._oauth_cache: Dict[str, Dict[str, Any]]`++### The Mistake to Avoid++Google access tokens live in the OAuth system, not the API key system. Reading them via `get_api_key("google_access_token")` returns None even if the user is fully logged in. All Google Workspace handlers must use `get_google_credentials()` from `server/services/handlers/google_auth.py`, which calls `get_oauth_tokens("google")` internally.++Twitter has the same split: `twitter_client_id` and `twitter_client_secret` are in the API key system, but `twitter_access_token` is in the OAuth system.++## Single Point of Access++**All credential operations must go through `AuthService`. Routers must never touch `CredentialsDatabase` directly.**++```python+# Correct:+from core.container import container+auth = container.auth_service()+tokens = await auth.get_oauth_tokens("google")++# Wrong (will not go through cache, will not respect backend abstraction):+credentials_db = get_credentials_db()+row = await credentials_db.query(...)+```++Enforcement:++- `AuthService` owns the Fernet cipher initialization.+- `AuthService` maintains the memory-only decryption cache.+- `CredentialsDatabase` is injected into `AuthService` and not exposed via DI to other services.++The in-memory cache is important: decrypting on every request would be slow, and writing decrypted values to Redis would defeat the encryption. Each `AuthService` instance caches decrypted credentials in process memory only, and `AuthService.clear_cache()` flushes them on demand (used by the logout handler).++## Multi-Backend Abstraction++For deployment flexibility, `credential_backends.py` defines an abstract interface that can be swapped via the `CREDENTIAL_BACKEND` env var.++```python+class CredentialBackend(ABC):+ async def store(self, key: str, value: str, metadata: Dict = None) -> bool+ async def retrieve(self, key: str) -> Optional[str]+ async def delete(self, key: str) -> bool+ def is_available(self) -> bool+```++| Backend | Use Case | Env var value |+|---|---|---|+| `FernetBackend` | Default; Fernet-encrypted SQLite | `fernet` |+| `KeyringBackend` | Desktop apps; Windows Credential Locker, macOS Keychain, Linux Secret Service | `keyring` |+| `AWSSecretsBackend` | Cloud deployments; AWS Secrets Manager | `aws` |++The factory `create_backend(settings, credentials_db)` returns the selected backend with automatic fallback to Fernet if the requested backend is unavailable (e.g. `boto3` not installed for AWS).++Optional dependencies for alternate backends:++```toml+[project.optional-dependencies]+keyring = ["keyring>=25.0.0"]+aws = ["boto3>=1.34.0"]+```++## Configuration++```env+# server/.env++# Required for Fernet backend+API_KEY_ENCRYPTION_KEY=<any string, at least 32 chars for good entropy>++# Which backend to use+CREDENTIAL_BACKEND=fernet # fernet | keyring | aws++# Path to credentials SQLite file+CREDENTIALS_DB_PATH=credentials.db++# AWS backend only+AWS_SECRET_ARN=arn:aws:secretsmanager:...+AWS_REGION=us-east-1+```++If `API_KEY_ENCRYPTION_KEY` is missing or changed, existing ciphertext becomes undecryptable. There is no key-rotation mechanism today: users re-enter their keys after a key change. This is a deliberate simplification inherited from the n8n pattern.++## Security Properties++- **Server-scoped key**: not tied to user login sessions. JWT cookies expire, but the encryption key survives across restarts.+- **No plaintext on disk**: credentials are only decrypted in memory.+- **No plaintext in Redis**: even in Redis mode, only encrypted envelopes cross the wire (the cache layer never stores decrypted credentials).+- **Salt per install**: different MachinaOS installs have different salts, so ciphertext is not portable across installs even with the same server key.+- **Wipes on shutdown**: `EncryptionService.clear()` zeroes the Fernet reference, preventing cold-boot recovery of the derived key.++## Related Docs++- [DESIGN.md](DESIGN.md) - overall security posture+- [new_service_integration.md](new_service_integration.md) - where to put credentials for new service integrations+- [status_broadcaster.md](status_broadcaster.md) - WebSocket handlers for credentials (get/save/delete)diff --git a/docs-internal/event_waiter_system.md b/docs-internal/event_waiter_system.mdnew file mode 100644index 00000000..2880abae--- /dev/null+++ b/docs-internal/event_waiter_system.md@@ -0,0 +1,215 @@+# Event Waiter System++Trigger nodes in MachinaOS suspend workflow execution until an external event arrives (WhatsApp message, webhook request, Telegram message, chat input, delegated task completion, etc.). The Event Waiter system is the generic primitive that backs all push-based triggers. It has two backends: in-memory (`asyncio.Future`) and Redis Streams (for multi-worker durability).++Source file: `server/services/event_waiter.py`++## What It Solves++Before this system, each trigger had its own ad-hoc waiting code. That meant:++- No common cancel path+- No common filter semantics+- No way to persist waits across Temporal worker restarts+- No way to debug which triggers were active++A single waiter module replaces all of that. Adding a new trigger now means adding one entry to the registry and one filter builder.++## Core Concepts++### Waiter++A `Waiter` is a single subscription:++```python+@dataclass+class Waiter:+ id: str # UUID+ node_id: str # workflow node waiting+ node_type: str # 'whatsappReceive', 'webhookTrigger', ...+ event_type: str # 'whatsapp_message_received', ...+ params: Dict # node parameters (used to rebuild filter in Redis mode)+ filter_fn: Callable[[Dict], bool]+ future: Optional[asyncio.Future] # only used in memory mode+ cancelled: bool+ created_at: float+```++### Trigger Registry++Every event-based trigger node type is registered with the event name it listens for:++```python+TRIGGER_REGISTRY: Dict[str, TriggerConfig] = {+ 'start': TriggerConfig('start', 'deploy_triggered', 'Deploy Start'),+ 'whatsappReceive': TriggerConfig('whatsappReceive', 'whatsapp_message_received','WhatsApp Message'),+ 'webhookTrigger': TriggerConfig('webhookTrigger', 'webhook_received', 'Webhook Request'),+ 'chatTrigger': TriggerConfig('chatTrigger', 'chat_message_received', 'Chat Message'),+ 'taskTrigger': TriggerConfig('taskTrigger', 'task_completed', 'Task Completed'),+ 'twitterReceive': TriggerConfig('twitterReceive', 'twitter_event_received', 'Twitter Event'),+ 'gmailReceive': TriggerConfig('gmailReceive', 'gmail_email_received', 'Gmail Email'),+ 'telegramReceive': TriggerConfig('telegramReceive', 'telegram_message_received','Telegram Message'),+}+```++`cronScheduler` is **not** in this registry: it uses APScheduler directly and does not wait for events.++### Filter Builders++Each trigger type has a filter builder that reads the node's parameters once and returns a closure evaluated per event:++```python+FILTER_BUILDERS = {+ 'whatsappReceive': build_whatsapp_filter,+ 'webhookTrigger': build_webhook_filter,+ 'chatTrigger': build_chat_filter,+ 'taskTrigger': build_task_completed_filter,+ 'twitterReceive': build_twitter_filter,+ 'gmailReceive': build_gmail_filter,+ 'telegramReceive': build_telegram_filter,+}+```++Filter closures capture parameter values at registration time. For example, `build_whatsapp_filter` captures `messageTypeFilter`, `sender_filter`, `contact_phone`, `group_id`, `keywords`, `ignore_own`, and `forwarded_filter`, then returns a function that checks each incoming WhatsApp message against those constraints.++## Execution Flow++```+Deployment starts+ |+ v+Trigger node encountered in execution layer+ |+ v+event_waiter.register(node_type, node_id, params) -> Waiter+ |+ v+event_waiter.wait_for_event(waiter) (suspends)+ |+ External event arrives (WhatsApp RPC, Telegram long-polling, webhook HTTP request, ...)+ |+ v+ event_waiter.dispatch(event_type, data) (sync, thread-safe)+ or event_waiter.dispatch_async(event_type, data) (Redis mode)+ |+ v+ For each Waiter with matching event_type:+ if waiter.filter_fn(data):+ waiter.future.set_result(data) (memory mode)+ or stream_ack(...) (Redis mode)+ |+ v+Workflow resumes, trigger node completes, downstream nodes execute+```++## Two Backends++### Memory Mode++Default for local development. Uses `asyncio.Future` with a module-level `_waiters` dict.++- `register()` creates an `asyncio.Future` and stores it in `_waiters[waiter.id]`.+- `wait_for_event()` awaits the future.+- `dispatch()` iterates `_waiters`, runs `filter_fn(data)` on each, calls `future.set_result(data)` on matches.++Thread-safe: `dispatch()` detects whether it is running in an async loop or a thread (e.g. APScheduler callback) and uses `asyncio.run_coroutine_threadsafe(..., _main_loop)` when needed.++### Redis Streams Mode++Activated automatically when `CacheService.is_streams_available()` returns True. Used in Temporal multi-worker deployments where waiters may live on different processes than dispatchers.++- Events are appended to `events:{event_type}` streams via `XADD`.+- Each waiter creates its own consumer group `waiter_group_{waiter_id}` for broadcast semantics (every waiter sees every event, not load-balanced).+- `_wait_redis()` polls with blocking `XREADGROUP` in 5-second chunks so cancellation is responsive.+- On match, the stream entry is `XACK`ed and the waiter is cleaned up.+- Waiter metadata is stored in `waiters:{waiter_id}` with 24-hour TTL for visibility in other workers.++The Redis mode is initialized by `set_cache_service(cache)` during app startup in `main.py`.++## Polling Triggers vs Event Triggers++Some triggers do not fit the push model because the upstream service has no webhook or long-polling API. These use a different primitive in `TriggerManager`:++| Trigger Type | Mechanism | Location |+|---|---|---|+| `whatsappReceive` | Event (push) | `event_waiter.py` |+| `webhookTrigger` | Event (push via FastAPI router) | `event_waiter.py` |+| `chatTrigger` | Event (push via WebSocket) | `event_waiter.py` |+| `taskTrigger` | Event (push via delegation) | `event_waiter.py` |+| `telegramReceive` | Event (push via long-polling) | `event_waiter.py` + `TelegramService` |+| `twitterReceive` | **Polling** | `deployment/triggers.py` + `asyncio.Queue` |+| `gmailReceive` | **Polling** | `deployment/triggers.py` + `asyncio.Queue` |+| `cronScheduler` | APScheduler | `deployment/triggers.py` |++Polling triggers are routed in `DeploymentManager._create_poll_coroutine()` and use `TriggerManager.setup_polling_trigger()` with a custom poll function and an `asyncio.Queue`. See [workflow-schema.md](workflow-schema.md) for the full trigger list.++## Cancellation++Users can cancel a waiting trigger from the UI (Cancel button on the trigger node). The path:++1. Frontend sends `cancel_event_wait` WebSocket message with `waiter_id` or `node_id`.+2. `handle_cancel_event_wait()` in `server/routers/websocket.py` calls either `event_waiter.cancel(waiter_id)` or `event_waiter.cancel_for_node(node_id)`.+3. `cancel()` sets `w.cancelled = True`, calls `future.cancel()` in memory mode, and deletes the Redis metadata key in Redis mode.+4. The suspended `wait_for_event()` raises `asyncio.CancelledError`, which bubbles up through the node executor.++## Debugging++`event_waiter.get_active_waiters()` returns a list of all currently active waiters with age, backend mode, and done/cancelled status. The `get_active_waiters` WebSocket handler exposes this to the frontend for the active-triggers debug view.++```python+{+ "id": "uuid",+ "node_id": "whatsappReceive-1",+ "node_type": "whatsappReceive",+ "event_type": "whatsapp_message_received",+ "done": False,+ "cancelled": False,+ "age_seconds": 142.3,+ "mode": "redis" # or "memory"+}+```++## Adding a New Trigger Type++1. **Register it** in `TRIGGER_REGISTRY`:++ ```python+ 'mqttTrigger': TriggerConfig('mqttTrigger', 'mqtt_message_received', 'MQTT Message'),+ ```++2. **Build a filter** closure from node params and add it to `FILTER_BUILDERS`:++ ```python+ def build_mqtt_filter(params: Dict) -> Callable[[Dict], bool]:+ topic = params.get('topic', '')+ qos = params.get('qos', 0)++ def matches(m: Dict) -> bool:+ if topic and m.get('topic') != topic:+ return False+ if qos and m.get('qos') != qos:+ return False+ return True+ return matches++ FILTER_BUILDERS['mqttTrigger'] = build_mqtt_filter+ ```++3. **Dispatch events** from the external service:++ ```python+ from services import event_waiter+ event_waiter.dispatch('mqtt_message_received', {'topic': ..., 'payload': ...})+ ```++4. **Add the node definition** in `client/src/nodeDefinitions/` with `group: ['..., trigger']`.++5. **Add output schema** in `client/src/components/parameterPanel/InputSection.tsx` so downstream nodes can drag fields.++No changes are needed in the execution engine, cancel path, or deployment manager.++## Related Docs++- [DESIGN.md](DESIGN.md) - how triggers fit into execution+- [workflow-schema.md](workflow-schema.md) - trigger node catalog+- [TEMPORAL_ARCHITECTURE.md](TEMPORAL_ARCHITECTURE.md) - why Redis mode exists for Temporal workersdiff --git a/docs-internal/native_llm_sdk.md b/docs-internal/native_llm_sdk.mdnew file mode 100644index 00000000..206c50d9--- /dev/null+++ b/docs-internal/native_llm_sdk.md@@ -0,0 +1,254 @@+# Native LLM SDK Architecture++MachinaOS uses a hybrid LLM architecture: a native SDK layer in `server/services/llm/` for direct chat completions and model fetching, and a LangChain + LangGraph path in `server/services/ai.py` for agent tool-calling. This document describes the native layer, its design, and how both paths coexist.++## Why a Native Layer++LangChain added a useful abstraction early on, but it introduced three recurring problems:++1. **Windows/Python 3.13 hangs**: `langchain_google_genai` deadlocks on gRPC init. The native Gemini path bypasses LangChain entirely and uses `google.genai.Client` directly.+2. **Parameter translation loss**: LangChain's `max_completion_tokens` rewrite breaks OpenAI-compatible providers (DeepSeek, Kimi, Mistral) that expect `max_tokens`.+3. **Hard-coded URLs**: Each provider class in LangChain hard-codes its base URL. The native layer reads `base_url` from `server/config/llm_defaults.json`, so adding an OpenAI-compatible provider is a config change.++The native layer also gives a single normalized response shape (`LLMResponse`) across providers, which simplifies downstream code for token tracking, cost calculation, and thinking extraction.++## Layer Overview++```+server/services/llm/+|-- __init__.py Public API exports+|-- protocol.py Message, ToolDef, ToolCall, Usage, LLMResponse, LLMProvider (Protocol)+|-- config.py ProviderConfig, PROVIDER_CONFIGS (built from llm_defaults.json)+|-- factory.py create_provider() lazy-import factory, NATIVE_PROVIDERS set+|-- messages.py filter_empty_messages, is_valid_message_content+`-- providers/+ |-- __init__.py+ |-- anthropic.py AnthropicProvider (anthropic SDK)+ |-- openai.py OpenAIProvider (openai SDK)+ |-- gemini.py GeminiProvider (google-genai SDK)+ `-- openrouter.py OpenRouterProvider (extends OpenAIProvider with headers)+```++## Supported Providers++The native layer currently supports **10 providers**, grouped by implementation:++| Provider | Implementation | SDK | Notes |+|---|---|---|---|+| `anthropic` | `providers/anthropic.py` | `anthropic` | Extended thinking via `budget_tokens` |+| `openai` | `providers/openai.py` | `openai` | Reasoning models (o1/o3) and GPT-5 hybrid thinking |+| `gemini` | `providers/gemini.py` | `google-genai` | Bypasses LangChain (Windows hang fix) |+| `openrouter` | `providers/openrouter.py` | `openai` | Sets `HTTP-Referer` + `X-Title` headers |+| `xai` | `providers/openai.py` + base_url | `openai` | OpenAI-compatible at `api.x.ai/v1` |+| `deepseek` | `providers/openai.py` + base_url | `openai` | OpenAI-compatible at `api.deepseek.com` |+| `kimi` | `providers/openai.py` + base_url | `openai` | Moonshot AI, OpenAI-compatible |+| `mistral` | `providers/openai.py` + base_url | `openai` | OpenAI-compatible |+| `groq` | LangChain fallback | `langchain-groq` | Not yet on native path |+| `cerebras` | LangChain fallback | `langchain-cerebras` | Not yet on native path |++Source of truth for this list: `server/config/llm_defaults.json` (the `providers` dict) and `server/services/llm/factory.py` (`NATIVE_PROVIDERS` constant).++## Provider Protocol++Every native provider implements a structural Protocol with two methods:++```python+@runtime_checkable+class LLMProvider(Protocol):+ provider_name: str++ async def chat(+ self,+ messages: List[Message],+ *,+ model: str,+ temperature: float = 0.7,+ max_tokens: int = 4096,+ thinking: Optional[ThinkingConfig] = None,+ tools: Optional[List[ToolDef]] = None,+ ) -> LLMResponse: ...++ async def fetch_models(self, api_key: str) -> List[str]: ...+```++All providers return the same `LLMResponse` dataclass, regardless of SDK:++```python+@dataclass+class LLMResponse:+ content: str = ""+ thinking: Optional[str] = None+ tool_calls: List[ToolCall] = field(default_factory=list)+ usage: Usage = field(default_factory=Usage)+ model: str = ""+ finish_reason: str = "stop"+ raw: Any = None+```++Downstream code never special-cases a provider SDK shape.++## Factory and Lazy Imports++Each provider is imported only when first used to avoid loading large SDKs at startup:++```python+# server/services/llm/factory.py+def create_provider(provider: str, api_key: str, *, proxy_url: Optional[str] = None) -> LLMProvider:+ if provider == "anthropic":+ from services.llm.providers.anthropic import AnthropicProvider+ return AnthropicProvider(api_key, proxy_url=proxy_url)++ if provider == "openai":+ from services.llm.providers.openai import OpenAIProvider+ return OpenAIProvider(api_key, proxy_url=proxy_url)++ if provider == "gemini":+ from services.llm.providers.gemini import GeminiProvider+ return GeminiProvider(api_key, proxy_url=proxy_url)++ if provider == "openrouter":+ from services.llm.providers.openrouter import OpenRouterProvider+ return OpenRouterProvider(api_key, proxy_url=proxy_url)++ # OpenAI-compatible providers: reuse OpenAIProvider with base_url from config+ from services.llm.config import get_provider_config+ config = get_provider_config(provider)+ if config and config.base_url:+ from services.llm.providers.openai import OpenAIProvider+ return OpenAIProvider(api_key, base_url=config.base_url, proxy_url=proxy_url)++ raise ValueError(f"Unknown provider: {provider}")+```++`is_native_provider(name)` is the gate used by `AIService.execute_chat()` to choose the native path vs LangChain fallback.++## Config-Driven Base URLs++`ProviderConfig` is built at import time from `llm_defaults.json`:++```python+@dataclass+class ProviderConfig:+ name: str+ default_model: str+ detection_patterns: Tuple[str, ...]+ models_endpoint: str+ api_key_header: str # "Authorization", "x-api-key"+ api_key_format: str = "Bearer {key}"+ extra_headers: Dict[str, str] = field(default_factory=dict)+ base_url: str = "" # OpenAI-compatible base URL+```++Adding a new OpenAI-compatible provider is a pure config change:++```json+"deepseek": {+ "default_model": "deepseek-chat",+ "detection_patterns": ["deepseek"],+ "models_endpoint": "https://api.deepseek.com/v1/models",+ "base_url": "https://api.deepseek.com/v1",+ "max_output_tokens": { "_default": 8192 },+ "context_length": { "_default": 128000 },+ "temperature_range": [0.0, 2.0]+}+```++No new provider class, no new import. The factory's fallback branch picks up `base_url` and reuses `OpenAIProvider`.++## Native Path vs LangChain Path++`AIService` in `server/services/ai.py` has two top-level execution methods. They take different routes:++```+execute_chat(parameters, node_id, node_type) -> native SDK where possible+execute_agent(parameters, node_id) -> LangChain + LangGraph+execute_chat_agent(parameters, node_id) -> LangChain + LangGraph+```++**`execute_chat()` routing:**++```+node_type / model selects provider+ |+ v+is_native_provider(provider)?+ |+ +-- yes --> create_provider(...) -> provider.chat(...) -> LLMResponse+ |+ +-- no --> self.create_model(...) -> chat_model.invoke(...) -> extract to LLMResponse shape+```++**`execute_agent()` and `execute_chat_agent()` routing:**++All providers go through `create_model()` which returns a LangChain `ChatOpenAI` / `ChatAnthropic` / `ChatGoogleGenerativeAI` / `ChatGroq` / `ChatCerebras` instance. The agent then builds a LangGraph `StateGraph` with `bind_tools()` for tool calling. This path is used because LangGraph's agent loop, checkpointer, and tool-execution callback layer do not have a native-layer equivalent today.++OpenAI-compatible providers like DeepSeek/Kimi/Mistral still use `ChatOpenAI` with `base_url` from `llm_defaults.json` in the agent path, and pass `max_tokens` via `extra_body` to bypass LangChain's `max_completion_tokens` conversion.++## Thinking and Reasoning++The unified `ThinkingConfig` dataclass is translated per provider:++```python+@dataclass+class ThinkingConfig:+ enabled: bool = False+ budget: int = 2048 # Anthropic budget_tokens, Gemini 2.5 thinking_budget+ effort: str = "medium" # OpenAI reasoning_effort (low/medium/high)+ level: str = "medium" # Gemini 3+ thinking_level+ format: str = "parsed" # Groq Qwen3 reasoning_format (parsed/hidden)+```++Each provider's `chat()` method reads only the fields it supports. The extracted thinking text is returned in `LLMResponse.thinking` so downstream nodes can display or drag it into parameters.++See [memory_compaction.md](memory_compaction.md) for how thinking token counts are tracked separately from output tokens.++## Model Max Tokens Resolution++`resolve_max_tokens()` in `services/llm/config.py` implements the clamp/default logic:++```python+def resolve_max_tokens(params: dict, model: str, provider: str) -> int:+ registry = get_model_registry()+ model_max = registry.get_max_output_tokens(model, provider)+ user_val = params.get("max_tokens") or params.get("maxTokens")+ if user_val:+ user_int = int(user_val)+ if user_int > model_max:+ return model_max # clamp user value to model hard limit+ return user_int+ return model_max+```++Paired with `ModelRegistryService` (`server/services/model_registry.py`), which loads `model_registry.json` (cached from OpenRouter's `/api/v1/models` endpoint) and falls back to `llm_defaults.json` for unknown models. The registry is the single source for:++- `max_output_tokens`+- `context_length`+- `temperature_range`+- `is_reasoning_model` (fixes temperature to 1.0)+- `supports_thinking`+- `thinking_type`++## Adding a New Provider++1. **OpenAI-compatible provider** (DeepSeek, Kimi, Mistral pattern):+ - Add an entry to `llm_defaults.json` with `base_url`, `default_model`, `detection_patterns`, `max_output_tokens._default`, `context_length._default`, `temperature_range`.+ - Add the provider name to `NATIVE_PROVIDERS` in `factory.py` if you want `execute_chat()` to route natively.+ - Add to `PROVIDER_CONFIGS` in `services/ai.py` for LangChain agent path (reuse `ChatOpenAI`, set `base_url` in `create_model()`).++2. **Custom-SDK provider** (Anthropic, Gemini pattern):+ - Create `services/llm/providers/<name>.py` implementing `LLMProvider` protocol.+ - Add a branch in `create_provider()` in `factory.py`.+ - Add the provider name to `_DEDICATED_PROVIDERS` and `NATIVE_PROVIDERS`.+ - Add config entry in `llm_defaults.json` (no `base_url` needed if the SDK handles URLs itself).++3. **Frontend wiring** (in either case):+ - Add node definition in `client/src/nodeDefinitions/aiModelNodes.ts` using `createBaseChatModel()`.+ - Add to `CREDENTIAL_TO_PROVIDER` map in `ModelNode.tsx`.+ - Add credential entry in `CredentialsModal.tsx`.++## Related Docs++- [DESIGN.md](DESIGN.md) - overall backend architecture+- [memory_compaction.md](memory_compaction.md) - token tracking and compaction using this layer+- [pricing_service.md](pricing_service.md) - cost calculation from `LLMResponse.usage`+- [agent_architecture.md](agent_architecture.md) - how agents use LangChain path on top of this layerdiff --git a/docs-internal/status_broadcaster.md b/docs-internal/status_broadcaster.mdnew file mode 100644index 00000000..dda952fe--- /dev/null+++ b/docs-internal/status_broadcaster.md@@ -0,0 +1,216 @@+# WebSocket Status Broadcaster++MachinaOS uses WebSocket as the primary communication channel between the React frontend and the FastAPI backend. A single `StatusBroadcaster` service manages all active WebSocket connections and broadcasts real-time state updates (node status, workflow progress, Android device status, variable changes, etc.). This replaces REST polling with push-based updates and is the reason the UI can animate node execution, show live tool calls, and update Android status without reloading.++This document covers the broadcaster architecture, the 89 WebSocket message handlers, the broadcast message types, and the Android two-state connection model.++Source files:+- `server/services/status_broadcaster.py` - `StatusBroadcaster` singleton+- `server/routers/websocket.py` - WebSocket endpoint and 89 message handlers+- `client/src/contexts/WebSocketContext.tsx` - frontend WebSocket provider with hooks++## Why WebSocket-First++Early iterations of MachinaOS used REST endpoints for most frontend-backend operations. That caused three recurring problems:++1. **Polling noise**: the frontend had to re-fetch parameters, workflows, Android status, and node outputs on an interval to look "live".+2. **No push**: node execution status changes (running, completed, errored) could not reach the UI without polling; tool calls during agent execution were invisible until the whole agent finished.+3. **Latency**: fetch -> parse -> re-render cycles added 100-200ms to every user action.++Moving all frontend-backend traffic to a single WebSocket connection eliminated polling, enabled real-time status animations, and simplified authentication (one handshake, one cookie check).++## Architecture++```+React UI (WebSocketContext.tsx)+ |+ | ws://host/ws/status (single persistent connection)+ v+FastAPI /ws/status endpoint (server/routers/websocket.py)+ |+ v+StatusBroadcaster (server/services/status_broadcaster.py)+ |+ +-- connection set: Set[WebSocket]+ +-- current status: Dict[str, Any]+ | |-- android: {connected, paired, device_id, ...}+ | |-- nodes: {node_id: {status, output, error, ...}}+ | |-- variables: {name: value}+ | `-- workflow: {executing, current_node, progress}+ |+ +-- connect(ws) -> accept + send initial_status+ +-- disconnect(ws) -> remove from set+ +-- update_*(...) -> mutate state + _broadcast()+ +-- _broadcast(msg) -> fan out to all connected clients+```++## Connection Lifecycle++```+Frontend mounts <WebSocketProvider>+ |+ v+WebSocket connects to /ws/status+ |+ v+Backend middleware checks JWT cookie (if auth enabled)+ |+ +-- invalid --> close(4001, "Not authenticated")+ |+ v+StatusBroadcaster.connect(ws)+ |+ +-- accept+ +-- add to _connections+ `-- send {"type": "initial_status", "data": current_status}+ |+ v+Message loop:+ receive_json -> dispatch to handler in _HANDLERS registry+ send_json -> pushed by broadcaster on state change+ ping/pong -> keepalive every 30s from frontend+ |+ v+Frontend unmounts or logs out+ |+ v+StatusBroadcaster.disconnect(ws) -> remove from _connections+```++Auto-reconnect is handled by `WebSocketContext.tsx`: on disconnect, it schedules a reconnect after 3 seconds with a 100ms mount delay to avoid React Strict Mode double-connect in dev.++## Handler Registry++Handlers are registered via the `@ws_handler` decorator:++```python+@ws_handler()+async def handle_get_node_parameters(data: Dict, websocket: WebSocket) -> Dict:+ node_id = data.get("node_id")+ params = await database.get_node_parameters(node_id)+ return {"success": True, "parameters": params}+```++The decorator populates a module-level `_HANDLERS: Dict[str, Callable]` map from the function name (`handle_get_node_parameters` -> `"get_node_parameters"`). The dispatcher reads the incoming message's `type` field and looks up the handler.++Current total: **89 WebSocket handlers** in `server/routers/websocket.py`.++### Handler Categories++| Category | Example handlers |+|---|---|+| Status / ping | `ping`, `get_status`, `get_android_status`, `get_node_status`, `get_variable` |+| Node parameters | `get_node_parameters`, `save_node_parameters`, `delete_node_parameters`, `get_all_node_parameters` |+| Tool schemas | `get_tool_schema`, `save_tool_schema`, `delete_tool_schema`, `get_all_tool_schemas` |+| Node execution | `execute_node`, `execute_workflow`, `cancel_execution`, `get_node_output`, `clear_node_output` |+| Triggers / events | `cancel_event_wait`, `get_active_waiters` |+| Dead letter queue | `get_dlq_entries`, `replay_dlq_entry`, `remove_dlq_entry`, `purge_dlq`, `get_dlq_stats` |+| Deployment | `deploy_workflow`, `cancel_deployment`, `get_deployment_status`, `update_deployment_settings` |+| AI operations | `execute_ai_node`, `get_ai_models`, `test_ai_proxy` |+| API keys | `validate_api_key`, `get_stored_api_key`, `save_api_key`, `delete_api_key` |+| OAuth flows | `claude_oauth_login`, `twitter_oauth_login`, `twitter_logout`, `google_oauth_login`, `google_logout` |+| Android | `get_android_devices`, `execute_android_action`, `android_relay_connect`, `android_relay_disconnect`, `android_relay_reconnect` |+| WhatsApp | `whatsapp_status`, `whatsapp_qr`, `whatsapp_send`, `whatsapp_chat_history`, `whatsapp_newsletters`, `whatsapp_diagnostics`, ... |+| Telegram | `telegram_connect`, `telegram_disconnect`, `telegram_status`, `telegram_send`, `telegram_get_me`, `telegram_get_chat` |+| Workflow storage | `save_workflow`, `get_workflow`, `get_all_workflows`, `delete_workflow` |+| Chat messages | `send_chat_message`, `get_chat_messages`, `clear_chat_messages`, `get_chat_sessions` |+| Console / terminal | `get_console_logs`, `clear_console_logs`, `get_terminal_logs`, `clear_terminal_logs` |+| Skills | `get_skill_content`, `save_skill_content`, `get_user_skills`, `create_user_skill`, `scan_skill_folder` |+| Memory | `clear_memory`, `reset_skill`, `configure_compaction`, `get_compaction_stats` |+| User settings | `get_user_settings`, `save_user_settings`, `get_provider_defaults`, `save_provider_defaults` |+| Pricing / usage | `get_pricing_config`, `save_pricing_config`, `get_api_usage_summary`, `get_provider_usage_summary` |+| Agent teams | `create_team`, `add_team_task`, `claim_team_task`, `complete_team_task`, `get_team_messages` |+| Model registry | `get_model_constraints`, `refresh_model_registry` |++The exact set drifts over time. The canonical count comes from counting `@ws_handler(` occurrences in `server/routers/websocket.py`.++## Broadcast Messages (Server -> Clients)++Broadcasts are sent to all connected clients without a request-response correlation. They fire on state changes in the backend:++| Message Type | Trigger | Payload |+|---|---|---|+| `android_status` | Android relay connect/disconnect/pair | `{connected, paired, device_id, device_name, connection_type, qr_data, ...}` |+| `node_status` | Node enters executing / waiting / success / error | `{node_id, status, data, workflow_id, timestamp}` |+| `node_output` | Node produces output | `{node_id, output, workflow_id}` |+| `variable_update` | Single variable changes | `{name, value}` |+| `variables_update` | Batch variable update | `{variables: Dict}` |+| `workflow_status` | Workflow start / progress / complete | `{executing, current_node, progress}` |+| `api_key_status` | Credential validation result | `{provider, valid, models}` |+| `node_parameters_updated` | Parameters saved by another client | `{node_id, parameters}` |+| `token_usage_update` | AI execution updates token counters | `{session_id, data: {total, threshold, needs_compaction}}` |+| `compaction_starting` | Memory compaction begins | `{session_id, node_id}` |+| `compaction_completed` | Memory compaction ends | `{session_id, success, tokens_before, tokens_after}` |++## Android Two-State Connection Model++Android support uses a two-state model because a relay WebSocket can be connected without a device being paired. The UI needs both signals.++| State | Meaning | Frontend behavior |+|---|---|---|+| `connected` | Relay WebSocket to `wss://relay.zeenie.xyz/ws` is active | Not shown directly |+| `paired` | Android app has scanned QR and established session | Green dot on Android nodes |++```+User clicks Connect+ |+ v+Relay WebSocket opens --> broadcast_connected({connected: true, paired: false})+ | QR code shown+ v+User scans QR with Android app+ |+ v+Device pairs --> broadcast_connected({connected: true, paired: true, device_id, device_name})+ | Green dot lights up+ v+App disconnects --> broadcast_device_disconnected({connected: true, paired: false, qr_data, session_token})+ | QR shown again for re-pairing+ v+Relay WebSocket closes --> broadcast_relay_disconnected({connected: false, paired: false})+```++Android service nodes (`batteryMonitor`, `wifiAutomation`, etc.) check `androidStatus.paired` (not `connected`) before allowing execution. See `client/src/components/SquareNode.tsx`.++## Auto-Reconnect and Ping Keepalive++**Frontend** (`WebSocketContext.tsx`):++- 30-second `setInterval` sends `{"type": "ping"}`.+- On disconnect: schedule reconnect after 3 seconds.+- 100ms mount delay avoids React Strict Mode double-connect in development.+- `isMountedRef` prevents connections after unmount.+- WebSocket is gated on `isAuthenticated` from `AuthContext`: if auth is disabled or not yet loaded, the provider defers connection.++**Backend** (`websocket.py`):++- Responds to `ping` with `{"type": "pong"}`.+- On `get_status`, returns the full current status snapshot.+- On disconnect, removes the WebSocket from `StatusBroadcaster._connections`.++## Error Handling++All handlers use a try/except wrapper that returns a structured error response instead of killing the WebSocket:++```python+try:+ result = await handler(data, websocket)+ await websocket.send_json({"type": response_type, **result})+except Exception as e:+ logger.error(f"Handler {handler_name} failed: {e}")+ await websocket.send_json({+ "type": "error",+ "handler": handler_name,+ "code": type(e).__name__,+ "message": str(e)+ })+```++The frontend matches responses by a `request_id` field (set by the frontend, echoed by the backend) to resolve pending promises in `WebSocketContext`.++## Related Docs++- [DESIGN.md](DESIGN.md) - overall backend architecture+- [credentials_encryption.md](credentials_encryption.md) - credential handlers that go through this layer+- [event_waiter_system.md](event_waiter_system.md) - `cancel_event_wait` and `get_active_waiters` handlers+- [memory_compaction.md](memory_compaction.md) - token usage broadcast messagesdiff --git a/docs-internal/workflow-schema.md b/docs-internal/workflow-schema.mdindex 9eb82835..8767e28e 100644--- a/docs-internal/workflow-schema.md+++ b/docs-internal/workflow-schema.md@@ -59,68 +59,104 @@ A workflow JSON document contains:}```-## Supported Node Types (67 Total)+## Supported Node Types (96 Total)-### Workflow Nodes (1 node)+> Counts are grouped by source file under `client/src/nodeDefinitions/`. The canonical list lives there; this section is a human-readable index.++### Workflow Nodes (2 nodes)- `start` - Workflow entry point with initial data+- `taskTrigger` - Event-driven trigger for delegated child agent completion### Scheduler Nodes (2 nodes)- `timer` - Delay/wait before continuing (seconds, minutes, hours)- `cronScheduler` - Recurring scheduled execution (seconds to months, timezone support)-### AI Chat Model Nodes (6 nodes)-- `openaiChatModel` - OpenAI GPT models (GPT-4o, GPT-4, o1/o3/o4 series with reasoning)-- `anthropicChatModel` - Anthropic Claude models (Claude 3.5/3 with extended thinking)-- `geminiChatModel` - Google Gemini models (2.5/Flash with thinking support)-- `openrouterChatModel` - OpenRouter unified API (200+ models from multiple providers)-- `groqChatModel` - Groq ultra-fast inference (Llama, Mixtral, Qwen3/QwQ with reasoning)+### AI Chat Model Nodes (9 nodes)+- `openaiChatModel` - OpenAI GPT 4.x/5.x + reasoning models (o1/o3/o4 series)+- `anthropicChatModel` - Anthropic Claude 4.x with extended thinking+- `geminiChatModel` - Google Gemini 2.5/3 with thinking support+- `openrouterChatModel` - OpenRouter unified API (200+ models)+- `groqChatModel` - Groq ultra-fast inference (Llama, Qwen3, GPT-OSS)- `cerebrasChatModel` - Cerebras custom AI hardware (Llama, Qwen)--### AI Agent Nodes (3 nodes)-- `aiAgent` - AI agent with tool calling, memory support (LangGraph)-- `chatAgent` - Conversational agent with skill support-- `simpleMemory` - Conversation history storage (buffer/window modes)--### AI Skill Nodes (9 nodes)-Connect to Chat Agent's `input-skill` handle:-- `claudeSkill` - Default Claude assistant personality-- `whatsappSkill` - WhatsApp messaging via Chat Agent-- `memorySkill` - Long-term memory management-- `mapsSkill` - Location services (geocoding, places, directions)-- `httpSkill` - HTTP requests to external APIs-- `schedulerSkill` - Task scheduling (timers, cron)-- `androidSkill` - Android device control-- `codeSkill` - Python/JavaScript code execution-- `customSkill` - User-created custom skills (database-stored)--### AI Tool Nodes (4 nodes)+- `deepseekChatModel` - DeepSeek V3 (chat + reasoner with Chain-of-Thought)+- `kimiChatModel` - Moonshot Kimi K2.5 / K2-thinking+- `mistralChatModel` - Mistral Large / Small / Codestral++### AI Agents and Memory (3 nodes)+- `aiAgent` - LangGraph-based agent with tool calling+- `chatAgent` - Conversational agent (Zeenie) with skill support+- `simpleMemory` - Markdown-based conversation memory with optional vector DB++### Specialized AI Agents (15 nodes)+Pre-configured agents for specific domains. All inherit `AI_AGENT_PROPERTIES`. See [agent_architecture.md](agent_architecture.md):+- `android_agent` - Android device control+- `coding_agent` - Code execution (Python, JavaScript, TypeScript)+- `web_agent` - Web automation and scraping+- `task_agent` - Task management and scheduling+- `social_agent` - Social messaging (WhatsApp, Telegram)+- `travel_agent` - Location and maps+- `tool_agent` - Generic tool orchestration+- `productivity_agent` - Google Workspace workflows+- `payments_agent` - Payment processing+- `consumer_agent` - Customer support+- `autonomous_agent` - Code Mode with agentic loops+- `orchestrator_agent` - Team lead for multi-agent coordination+- `ai_employee` - Team lead (alternate UI/branding)+- `rlm_agent` - Recursive Language Model (REPL-based, dedicated handler)+- `claude_code_agent` - Claude Code SDK integration++### AI Tool Nodes (4 dedicated)Connect to AI Agent's `input-tools` handle:-- `calculatorTool` - Mathematical operations+- `masterSkill` - Aggregates multiple skills with enable/disable toggles+- `calculatorTool` - Math operations- `currentTimeTool` - Current date/time with timezone- `duckduckgoSearch` - DuckDuckGo web search (free, no API key)-- `androidTool` - Android toolkit aggregator (sub-node pattern)--### Search Nodes (3 nodes)-Dual-purpose nodes (workflow + AI tool). Defined in `searchNodes.ts`:-- `braveSearch` - Brave Search API web results-- `serperSearch` - Google SERP via Serper API (web, news, images, places)-- `perplexitySearch` - Perplexity Sonar AI-powered search with markdown answer and citations--### Location Nodes (3 nodes)-- `createMap` - Google Maps creation with center, zoom, map type-- `addLocations` - Google Maps Geocoding (address to coordinates)-- `showNearbyPlaces` - Google Places nearbySearch--### WhatsApp Nodes (4 nodes)-- `whatsappSend` - Send messages (text, media, location, contact)-- `whatsappConnect` - Connection status and QR code-- `whatsappReceive` - Event-driven trigger with filters (type, sender, group, keywords)-- `whatsappChatHistory` - Retrieve message history--### Android Nodes (17 nodes)--#### Device Setup (1 node)-- `androidDeviceSetup` - Connect via local ADB or remote WebSocket relay+- `taskManager` - Task creation and tracking tool++### Search Nodes (3 dual-purpose)+Work as both workflow nodes and AI tools. Defined in `searchNodes.ts`:+- `braveSearch` - Brave Search API+- `serperSearch` - Google SERP via Serper API+- `perplexitySearch` - Perplexity Sonar with citations++### Location Nodes (3 nodes, 2 dual-purpose)+- `gmaps_create` - Google Maps creation with center, zoom, map type+- `gmaps_locations` - Geocoding (address to coordinates)+- `gmaps_nearby_places` - Google Places nearbySearch++### Google Workspace Nodes (7 nodes)+Consolidated operation-based nodes sharing one OAuth connection. See [new_service_integration.md](new_service_integration.md):+- `gmail` - send / search / read (dual-purpose)+- `gmailReceive` - Polling trigger for incoming emails+- `calendar` - create / list / update / delete+- `drive` - upload / download / list / share+- `sheets` - read / write / append+- `tasks` - create / list / complete / update / delete+- `contacts` - create / list / search / get / update / delete++### WhatsApp Nodes (3 nodes)+- `whatsappSend` - Dual-purpose: send text/media/location/contact to contacts, groups, channels+- `whatsappDb` - Dual-purpose: 18 operations covering chat history, contacts, groups, newsletters+- `whatsappReceive` - Event-driven trigger with filters (type, sender, group, keywords, channel)++### Twitter/X Nodes (4 nodes)+- `twitterSend` - Dual-purpose: tweet, reply, retweet, like, delete+- `twitterSearch` - Dual-purpose: rich search with expansions and citations+- `twitterUser` - Dual-purpose: profiles, followers, following+- `twitterReceive` - Polling trigger for mentions, DMs, timeline++### Telegram Nodes (2 nodes)+- `telegramSend` - Dual-purpose: text, photo, document, location, contact+- `telegramReceive` - Event-driven trigger via long-polling++### Social Nodes (2 nodes)+Unified multi-platform messaging (WhatsApp, Telegram, Discord, Slack, SMS, Email, Matrix, Teams):+- `socialReceive` - Normalize messages from platform triggers+- `socialSend` - Dual-purpose unified send++### Android Nodes (16 nodes)++> Android device connection is now configured via the Credentials modal (Android panel), not a workflow node. Service nodes can be connected directly to any agent's `input-tools` handle.#### System Monitoring (4 nodes)- `batteryMonitor` - Battery status, level, charging, temperature@@ -148,21 +184,24 @@ Dual-purpose nodes (workflow + AI tool). Defined in `searchNodes.ts`:- `cameraControl` - Camera info, take photos- `mediaControl` - Media playback, volume-### Utility Nodes (5 nodes)+### Utility Nodes (6 nodes)- `httpRequest` - HTTP requests (GET, POST, PUT, DELETE, PATCH) with optional proxy support (`useProxy: true`)- `webhookTrigger` - Incoming HTTP webhook trigger at `/webhook/{path}`- `webhookResponse` - Custom response to webhook caller- `chatTrigger` - Console message input trigger- `console` - Debug logging output+- `teamMonitor` - Real-time monitoring of Agent Team operations### Proxy Nodes (3 nodes)- `proxyRequest` - HTTP requests through residential proxy providers with geo-targeting and failover-- `proxyConfig` - Configure proxy providers, credentials, and routing rules+- `proxyConfig` - Dual-purpose: configure providers, credentials, routing rules- `proxyStatus` - View proxy provider health, scores, and usage statistics-### Code Nodes (2 nodes)-- `pythonExecutor` - Python code execution with input_data access-- `javascriptExecutor` - JavaScript code execution+### Code Nodes (3 nodes)+All dual-purpose (workflow node + AI tool):+- `pythonExecutor` - Python code execution (in-process)+- `javascriptExecutor` - JavaScript execution via persistent Node.js server+- `typescriptExecutor` - TypeScript execution via persistent Node.js server (tsx)### Chat Nodes (2 nodes)- `chatSend` - Send via JSON-RPC 2.0 WebSocket@@ -177,6 +216,10 @@ RAG pipeline nodes for document ingestion, processing, and vector storage:- `embeddingGenerator` - Generate vector embeddings (HuggingFace, OpenAI, Ollama providers)- `vectorStore` - Store/query vectors (ChromaDB, Qdrant, Pinecone backends)+### Web Scraping Nodes (2 nodes)+- `apifyActor` - Dual-purpose: run Apify pre-built actors (Instagram, TikTok, Twitter, LinkedIn, Google Search, Maps, etc.)+- `crawleeScraper` - Crawlee web scraping with static and Playwright modes+## Example Workflow JSON### Basic Workflow (Start -> AI Agent)