Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
docs(plugin-extraction): reflect Wave 11.I shipped state
trohitg7d8cfce81bc2
diff --git a/CLAUDE.md b/CLAUDE.mdindex be1e1481..d7c20c3d 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -5167,4 +5167,5 @@ This function:- **Credentials: DB as single source of truth + symmetric broadcasts + cache dedup**: `CredentialsDatabase` (encrypted SQLite) is canonical; every other layer is a derived cache with explicit invalidation. Backend `AuthService._memory_cache + _models_cache` collapsed into one `_api_key_cache: Dict[str, ApiKeyCacheEntry]` dataclass — single write/evict site. Per RFC 9700 (OAuth 2.0 BCP, 2024) the `_oauth_cache` no longer carries `refresh_token`; new `get_oauth_refresh_token(provider, customer)` reads from the encrypted DB on every call. `validate_api_key`, `save_api_key`, `delete_api_key`, `twitter_logout`, `google_logout` now broadcast symmetrically: `update_api_key_status` (in-memory map) + `broadcast_credential_event(...)` (refetch signal) wrapping `WorkflowEvent` (CloudEvents v1.0 from `services/events/envelope.py`, the same envelope the Wave 12 EventSource framework uses). The dead-letter `credential_catalogue_updated` event is finally emitted by the backend. Frontend retired the 200-LOC `client/src/components/credentials/providers.tsx` static fallback — `useCatalogueQuery` is the only source; cold-boot renders `<Skeleton>`, server-unreachable shows an explicit error state. `ApiKeyStatus.hasKey` mirror dropped (duplicated catalogue's `provider.stored`); two new selector hooks (`useProviderStored`, `useStoredProviderCount`) read the catalogue. Pytest invariant `test_credential_broadcasts.py` (14 tests) locks the broadcast contract via `inspect.getsource` introspection + the CloudEvents v1.0 envelope shape + AuthService DB-write-then-cache-update ordering + the no-refresh-token-in-cache rule.- **Local-LLM provider routing + per-model context**: Ollama and LM Studio are first-class providers (9 in total for agents now: cloud 7 + 2 local). Provider detection is driven by `detect_ai_provider` in `server/constants.py` plus the `provider` Literal in `nodes/agent/{ai_agent,chat_agent,_specialized}.py` — both MUST list `ollama` / `lmstudio` or chat-model nodes / agent dropdowns silently fall through to `'openai'` and `execute_chat` calls api.openai.com with the placeholder key. The validator at `nodes/model/_local_validator.py` probes via the official SDKs (`ollama.AsyncClient.ps()` for typed `ProcessResponse.Model`, `lmstudio.AsyncClient.llm.list_loaded()` for typed `LlmInstanceInfo`) — reads only typed fields (`context_length`, `max_context_length`, `vision`, `trained_for_tool_use`, `architecture`, `params_string`, `format`, plus Ollama's `details.{family,parameter_size,quantization_level}`). No regex, no Modelfile-parameters parsing, no `/api/show` modelinfo dict-key hunting. Per-model params persist in `EncryptedAPIKey.models["model_params"]` (via `save_api_key(model_params=...)`) AND in `model_registry.json` via `register_local_model()` — sync `get_context_length()` / `get_max_output_tokens()` find real values without async DB lookups, and entries survive process restart. `is_model_valid_for_provider` returns `True` for open-world providers (`openrouter` / `ollama` / `lmstudio`) so local model names like `qwen/qwen3.6-27b` aren't rejected by the cloud-style pattern check. Runtime path uses `OpenAIProvider(base_url={user_proxy_url}, api_key="ollama")` — traffic stays on `localhost`.- **Typed openai exception → NodeUserError**: `execute_chat` / `execute_agent` / `execute_chat_agent` in `services/ai.py` each have **one** `except openai.OpenAIError as e` block that re-raises `NodeUserError(str(e)) from e`. The openai SDK's typed exception hierarchy — `BadRequestError` (400, e.g. `n_keep > n_ctx` overflow), `AuthenticationError` (401), `PermissionDeniedError` (403), `NotFoundError` (404), `RateLimitError` (429), `APIConnectionError` (server unreachable), `APITimeoutError` — IS the user-correctable contract. `BaseNode.execute()` catches `NodeUserError` and logs at WARN with one line, no stacktrace. The plain `except Exception` block keeps the full-traceback path for real bugs. Zero string matching, zero flag plumbing, zero classification helpers. The `frontend toast` reads `response.message ?? response.error` so both clean envelopes (validator-style) and uncaught-error envelopes (handler-wrapper-style) surface a useful message.+- **Plugin extraction (Wave 11.I)**: every plugin's WS handlers, OAuth client, FastAPI router, and lifecycle service live entirely under `nodes/<plugin>/`. Eight plugin domains migrated this round (whatsapp / twitter / google-workspace / android / browser / email / code / credential-validation-scaffold for maps+apify+ollama+lmstudio) follow the telegram pattern. `routers/websocket.py` shrunk from ~3,785 to ~2,977 LOC (-808). Three plugin routers (`routers/twitter.py`, `routers/google.py`, `routers/android.py`) moved into `nodes/<plugin>/_router.py` and mount via the plugin-router loop in `main.py`; the explicit `app.include_router(<plugin>.router)` calls and `from routers import <plugin>` imports in `main.py` are gone. **`register_router(router, name=...)` is the new sibling helper to `register_ws_handlers` (same file: `services/ws_handler_registry.py`)** — six generic registries total now (ws_handlers, router, filter_builder, trigger_precheck, service_refresh, output_schema). **`Credential.validate(data) -> dict` + `Credential._probe(api_key) -> ProbeResult` is the new shared validator scaffold** in `services/plugin/credential.py` — replaces the per-router `_SPECIAL_PROVIDER_VALIDATORS` dict; one `_LLMApiKey._probe` method serves all 9 cloud LLM providers, dedicated `_probe` overrides handle Maps + Apify, `_LocalLLM.validate` overrides for Ollama / LM Studio's 2-storage edge case. **`tests/test_plugin_self_containment.py` locks the contract** with 7 invariant classes (forbidden-imports / no-router-outside-nodes / per-plugin self-registration / registry-API sanity / stale-paths-absent / main.py-does-not-mount / WS_HANDLERS-non-empty); same `inspect.getsource` introspection style as `test_credential_broadcasts.py`. Wire format unchanged — frontend WS message-type strings (`whatsapp_status`, `twitter_oauth_login`, `google_oauth_status`, `android_relay_connect`, etc.) and HTTP route paths (`/api/twitter/callback`, `/api/google/callback`, `/api/android/*`) are byte-identical post-migration.- never use emojis in prints\ No newline at end of filediff --git a/docs-internal/plugin_system.md b/docs-internal/plugin_system.mdindex 3840a1e6..d69ff64d 100644--- a/docs-internal/plugin_system.md+++ b/docs-internal/plugin_system.md@@ -761,14 +761,29 @@ every CI run. Examples:LOC.- Wave 11.G — Nodes cookbook (`server/nodes/README.md`) + CLAUDE.mdplugin section + this file refreshed to match shipped state.-- Wave 11.H — Self-contained plugin folders. Five new generic- registries replace per-plugin hardcoding in core services:- `services.ws_handler_registry` (WebSocket commands),+- Wave 11.H — Self-contained plugin folders. Six generic registries+ replace per-plugin hardcoding in core services:+ `services.ws_handler_registry.register_ws_handlers` (WebSocket+ commands), `services.ws_handler_registry.register_router` (FastAPI+ routers — sibling concern in the same file as `register_ws_handlers`,+ added Wave 11.I),`event_waiter.register_filter_builder` (event filters),`event_waiter.register_trigger_precheck` (trigger pre-executionchecks), `status_broadcaster.register_service_refresh` (WS-connectrefresh callbacks), `node_output_schemas.register_output_schema`(output schemas).+- Wave 11.I — Eight more plugin domains migrated to the+ self-contained pattern: WhatsApp, Twitter, Google Workspace,+ Android, Browser, Email, Code (Claude Code), and the credential+ validation scaffold (`Credential.validate` + `Credential._probe`)+ for Maps / Apify / Ollama / LM Studio. `routers/websocket.py`+ shrunk by ~808 LOC; three plugin routers (twitter / google /+ android) moved into `nodes/<plugin>/_router.py` and mount via the+ plugin-router loop in `main.py`. `tests/test_plugin_self_containment.py`+ locks the contract with 7 invariant classes (forbidden-imports /+ no-router-outside-nodes / per-plugin self-registration /+ registry-API sanity / stale-paths-absent / main.py-does-not-mount /+ WS_HANDLERS-non-empty).- Wave 12 — Generalized event framework([`services/events/`](../server/services/events/)). Adds`WorkflowEvent` (CloudEvents v1.0 envelope, in-house Pydantic),@@ -794,14 +809,15 @@ every CI run. Examples:changes. Frontend identifies plugin commands by WebSocket messagetype strings, not Python module paths.-`services/handlers/` is now **4 files / 1,112 LOC** (down from 16-files / 12,800 LOC):+`services/handlers/` is now **4 files / ~970 LOC** (down from 16+files / 12,800 LOC; `google_auth.py` moved to `nodes/google/_auth_helper.py`+in Wave 11.I commit D):| File | LOC | Purpose ||---|---|---|| `tools.py` | 821 | AI-tool dispatcher, plugin fast-path, agent delegation infrastructure (shared `_delegated_tasks` / `_delegation_results` state). |-| `google_auth.py` | 142 | Shared OAuth helper for the 7 Google plugins. |-| `triggers.py` | 126 | Generic event-trigger handler for `twitterReceive` and other polling triggers. |+| `triggers.py` | 126 | Generic event-trigger handler for polling triggers (gmailReceive, twitterReceive, etc.). |+| `todo.py` | 65 | TaskManager / writeTodos invocation surface for AI tool nodes. || `__init__.py` | 23 | Package docstring; nothing imports from `services.handlers` at package level. |Every domain owns its own code under `nodes/<group>/` — plugin file +diff --git a/server/nodes/README.md b/server/nodes/README.mdindex b3149938..b04b547b 100644--- a/server/nodes/README.md+++ b/server/nodes/README.md@@ -234,7 +234,7 @@ server/nodes/telegram/└── telegram_receive.py # TriggerNode```-### Five generic registries to plug into+### Six generic registries to plug intoTelegram's `__init__.py` is the canonical wiring example. Adding anyof these concerns to your plugin is one `register_*` call from your@@ -243,14 +243,26 @@ package's `__init__.py` — the consumer never imports your folder.| Concern | Where to register | What it does ||---|---|---|| Credentials-modal WebSocket commands | `services.ws_handler_registry.register_ws_handlers({"<type>": handler})` | Adds `<type>` to the central WS dispatcher (no router edits) |+| FastAPI router (Wave 11.I) | `services.ws_handler_registry.register_router(router, name="<name>")` | Plugin's HTTP router mounts via the plugin loop in `main.py`; sibling concern in the same file as `register_ws_handlers` || Event-trigger filter | `services.event_waiter.register_filter_builder(node_type, fn)` | Plugs into `FILTER_BUILDERS` for `event_waiter.build_filter()` || Trigger pre-execution check | `services.event_waiter.register_trigger_precheck(node_type, async_fn)` | Generic `triggers.py` handler runs `run_trigger_precheck` before entering the wait loop || Service-status refresh on WS connect | `services.status_broadcaster.register_service_refresh(async_callback)` | Callback runs once per `_refresh_all_services` cycle || Output schema | `services.node_output_schemas.register_output_schema(node_type, ModelClass)` | Avoids declaring a duplicate `Output` class in the central schema file |-All five are idempotent (same callable / class for the same key is a+All six are idempotent (same callable / class for the same key is ano-op; conflicts raise `ValueError`).+### Credential validation (Wave 11.I)++The credential-validator dispatch is a sibling concern, handled by the+existing `services/plugin/credential.py:Credential` base class. Your+`Credential` subclass overrides `_probe(api_key) -> ProbeResult` (or,+in rare cases like local-LLM 2-storage, the whole `validate(data)+-> dict` classmethod). Maps, Apify, all 9 cloud LLM providers, and+both local-LLM providers (Ollama / LM Studio) all dispatch through the+same scaffold — no `_SPECIAL_PROVIDER_VALIDATORS` dict in+`routers/websocket.py`.+### When NOT to use this shapeDon't create `_service.py` / `_handlers.py` siblings unless the plugin