Skip to content

Self-improving AI Employees for everyone

docs(plugin-extraction): reflect Wave 11.I shipped state

trohitg7d8cfce81bc2
diff --git a/CLAUDE.md b/CLAUDE.md
index be1e1481..d7c20c3d 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -5167,4 +5167,5 @@ This function:
- **Credentials: DB as single source of truth + symmetric broadcasts + cache dedup**: `CredentialsDatabase` (encrypted SQLite) is canonical; every other layer is a derived cache with explicit invalidation. Backend `AuthService._memory_cache + _models_cache` collapsed into one `_api_key_cache: Dict[str, ApiKeyCacheEntry]` dataclass — single write/evict site. Per RFC 9700 (OAuth 2.0 BCP, 2024) the `_oauth_cache` no longer carries `refresh_token`; new `get_oauth_refresh_token(provider, customer)` reads from the encrypted DB on every call. `validate_api_key`, `save_api_key`, `delete_api_key`, `twitter_logout`, `google_logout` now broadcast symmetrically: `update_api_key_status` (in-memory map) + `broadcast_credential_event(...)` (refetch signal) wrapping `WorkflowEvent` (CloudEvents v1.0 from `services/events/envelope.py`, the same envelope the Wave 12 EventSource framework uses). The dead-letter `credential_catalogue_updated` event is finally emitted by the backend. Frontend retired the 200-LOC `client/src/components/credentials/providers.tsx` static fallback — `useCatalogueQuery` is the only source; cold-boot renders `<Skeleton>`, server-unreachable shows an explicit error state. `ApiKeyStatus.hasKey` mirror dropped (duplicated catalogue's `provider.stored`); two new selector hooks (`useProviderStored`, `useStoredProviderCount`) read the catalogue. Pytest invariant `test_credential_broadcasts.py` (14 tests) locks the broadcast contract via `inspect.getsource` introspection + the CloudEvents v1.0 envelope shape + AuthService DB-write-then-cache-update ordering + the no-refresh-token-in-cache rule.
- **Local-LLM provider routing + per-model context**: Ollama and LM Studio are first-class providers (9 in total for agents now: cloud 7 + 2 local). Provider detection is driven by `detect_ai_provider` in `server/constants.py` plus the `provider` Literal in `nodes/agent/{ai_agent,chat_agent,_specialized}.py` — both MUST list `ollama` / `lmstudio` or chat-model nodes / agent dropdowns silently fall through to `'openai'` and `execute_chat` calls api.openai.com with the placeholder key. The validator at `nodes/model/_local_validator.py` probes via the official SDKs (`ollama.AsyncClient.ps()` for typed `ProcessResponse.Model`, `lmstudio.AsyncClient.llm.list_loaded()` for typed `LlmInstanceInfo`) — reads only typed fields (`context_length`, `max_context_length`, `vision`, `trained_for_tool_use`, `architecture`, `params_string`, `format`, plus Ollama's `details.{family,parameter_size,quantization_level}`). No regex, no Modelfile-parameters parsing, no `/api/show` modelinfo dict-key hunting. Per-model params persist in `EncryptedAPIKey.models["model_params"]` (via `save_api_key(model_params=...)`) AND in `model_registry.json` via `register_local_model()` — sync `get_context_length()` / `get_max_output_tokens()` find real values without async DB lookups, and entries survive process restart. `is_model_valid_for_provider` returns `True` for open-world providers (`openrouter` / `ollama` / `lmstudio`) so local model names like `qwen/qwen3.6-27b` aren't rejected by the cloud-style pattern check. Runtime path uses `OpenAIProvider(base_url={user_proxy_url}, api_key="ollama")` — traffic stays on `localhost`.
- **Typed openai exception → NodeUserError**: `execute_chat` / `execute_agent` / `execute_chat_agent` in `services/ai.py` each have **one** `except openai.OpenAIError as e` block that re-raises `NodeUserError(str(e)) from e`. The openai SDK's typed exception hierarchy — `BadRequestError` (400, e.g. `n_keep > n_ctx` overflow), `AuthenticationError` (401), `PermissionDeniedError` (403), `NotFoundError` (404), `RateLimitError` (429), `APIConnectionError` (server unreachable), `APITimeoutError` — IS the user-correctable contract. `BaseNode.execute()` catches `NodeUserError` and logs at WARN with one line, no stacktrace. The plain `except Exception` block keeps the full-traceback path for real bugs. Zero string matching, zero flag plumbing, zero classification helpers. The `frontend toast` reads `response.message ?? response.error` so both clean envelopes (validator-style) and uncaught-error envelopes (handler-wrapper-style) surface a useful message.
+- **Plugin extraction (Wave 11.I)**: every plugin's WS handlers, OAuth client, FastAPI router, and lifecycle service live entirely under `nodes/<plugin>/`. Eight plugin domains migrated this round (whatsapp / twitter / google-workspace / android / browser / email / code / credential-validation-scaffold for maps+apify+ollama+lmstudio) follow the telegram pattern. `routers/websocket.py` shrunk from ~3,785 to ~2,977 LOC (-808). Three plugin routers (`routers/twitter.py`, `routers/google.py`, `routers/android.py`) moved into `nodes/<plugin>/_router.py` and mount via the plugin-router loop in `main.py`; the explicit `app.include_router(<plugin>.router)` calls and `from routers import <plugin>` imports in `main.py` are gone. **`register_router(router, name=...)` is the new sibling helper to `register_ws_handlers` (same file: `services/ws_handler_registry.py`)** — six generic registries total now (ws_handlers, router, filter_builder, trigger_precheck, service_refresh, output_schema). **`Credential.validate(data) -> dict` + `Credential._probe(api_key) -> ProbeResult` is the new shared validator scaffold** in `services/plugin/credential.py` — replaces the per-router `_SPECIAL_PROVIDER_VALIDATORS` dict; one `_LLMApiKey._probe` method serves all 9 cloud LLM providers, dedicated `_probe` overrides handle Maps + Apify, `_LocalLLM.validate` overrides for Ollama / LM Studio's 2-storage edge case. **`tests/test_plugin_self_containment.py` locks the contract** with 7 invariant classes (forbidden-imports / no-router-outside-nodes / per-plugin self-registration / registry-API sanity / stale-paths-absent / main.py-does-not-mount / WS_HANDLERS-non-empty); same `inspect.getsource` introspection style as `test_credential_broadcasts.py`. Wire format unchanged — frontend WS message-type strings (`whatsapp_status`, `twitter_oauth_login`, `google_oauth_status`, `android_relay_connect`, etc.) and HTTP route paths (`/api/twitter/callback`, `/api/google/callback`, `/api/android/*`) are byte-identical post-migration.
- never use emojis in prints
\ No newline at end of file
diff --git a/docs-internal/plugin_system.md b/docs-internal/plugin_system.md
index 3840a1e6..d69ff64d 100644
--- a/docs-internal/plugin_system.md
+++ b/docs-internal/plugin_system.md
@@ -761,14 +761,29 @@ every CI run. Examples:
LOC.
- Wave 11.G — Nodes cookbook (`server/nodes/README.md`) + CLAUDE.md
plugin section + this file refreshed to match shipped state.
-- Wave 11.H — Self-contained plugin folders. Five new generic
- registries replace per-plugin hardcoding in core services:
- `services.ws_handler_registry` (WebSocket commands),
+- Wave 11.H — Self-contained plugin folders. Six generic registries
+ replace per-plugin hardcoding in core services:
+ `services.ws_handler_registry.register_ws_handlers` (WebSocket
+ commands), `services.ws_handler_registry.register_router` (FastAPI
+ routers — sibling concern in the same file as `register_ws_handlers`,
+ added Wave 11.I),
`event_waiter.register_filter_builder` (event filters),
`event_waiter.register_trigger_precheck` (trigger pre-execution
checks), `status_broadcaster.register_service_refresh` (WS-connect
refresh callbacks), `node_output_schemas.register_output_schema`
(output schemas).
+- Wave 11.I — Eight more plugin domains migrated to the
+ self-contained pattern: WhatsApp, Twitter, Google Workspace,
+ Android, Browser, Email, Code (Claude Code), and the credential
+ validation scaffold (`Credential.validate` + `Credential._probe`)
+ for Maps / Apify / Ollama / LM Studio. `routers/websocket.py`
+ shrunk by ~808 LOC; three plugin routers (twitter / google /
+ android) moved into `nodes/<plugin>/_router.py` and mount via the
+ plugin-router loop in `main.py`. `tests/test_plugin_self_containment.py`
+ locks the contract with 7 invariant classes (forbidden-imports /
+ no-router-outside-nodes / per-plugin self-registration /
+ registry-API sanity / stale-paths-absent / main.py-does-not-mount /
+ WS_HANDLERS-non-empty).
- Wave 12 — Generalized event framework
([`services/events/`](../server/services/events/)). Adds
`WorkflowEvent` (CloudEvents v1.0 envelope, in-house Pydantic),
@@ -794,14 +809,15 @@ every CI run. Examples:
changes. Frontend identifies plugin commands by WebSocket message
type strings, not Python module paths.
-`services/handlers/` is now **4 files / 1,112 LOC** (down from 16
-files / 12,800 LOC):
+`services/handlers/` is now **4 files / ~970 LOC** (down from 16
+files / 12,800 LOC; `google_auth.py` moved to `nodes/google/_auth_helper.py`
+in Wave 11.I commit D):
| File | LOC | Purpose |
|---|---|---|
| `tools.py` | 821 | AI-tool dispatcher, plugin fast-path, agent delegation infrastructure (shared `_delegated_tasks` / `_delegation_results` state). |
-| `google_auth.py` | 142 | Shared OAuth helper for the 7 Google plugins. |
-| `triggers.py` | 126 | Generic event-trigger handler for `twitterReceive` and other polling triggers. |
+| `triggers.py` | 126 | Generic event-trigger handler for polling triggers (gmailReceive, twitterReceive, etc.). |
+| `todo.py` | 65 | TaskManager / writeTodos invocation surface for AI tool nodes. |
| `__init__.py` | 23 | Package docstring; nothing imports from `services.handlers` at package level. |
Every domain owns its own code under `nodes/<group>/` — plugin file +
diff --git a/server/nodes/README.md b/server/nodes/README.md
index b3149938..b04b547b 100644
--- a/server/nodes/README.md
+++ b/server/nodes/README.md
@@ -234,7 +234,7 @@ server/nodes/telegram/
└── telegram_receive.py # TriggerNode
```
-### Five generic registries to plug into
+### Six generic registries to plug into
Telegram's `__init__.py` is the canonical wiring example. Adding any
of these concerns to your plugin is one `register_*` call from your
@@ -243,14 +243,26 @@ package's `__init__.py` — the consumer never imports your folder.
| Concern | Where to register | What it does |
|---|---|---|
| Credentials-modal WebSocket commands | `services.ws_handler_registry.register_ws_handlers({"<type>": handler})` | Adds `<type>` to the central WS dispatcher (no router edits) |
+| FastAPI router (Wave 11.I) | `services.ws_handler_registry.register_router(router, name="<name>")` | Plugin's HTTP router mounts via the plugin loop in `main.py`; sibling concern in the same file as `register_ws_handlers` |
| Event-trigger filter | `services.event_waiter.register_filter_builder(node_type, fn)` | Plugs into `FILTER_BUILDERS` for `event_waiter.build_filter()` |
| Trigger pre-execution check | `services.event_waiter.register_trigger_precheck(node_type, async_fn)` | Generic `triggers.py` handler runs `run_trigger_precheck` before entering the wait loop |
| Service-status refresh on WS connect | `services.status_broadcaster.register_service_refresh(async_callback)` | Callback runs once per `_refresh_all_services` cycle |
| Output schema | `services.node_output_schemas.register_output_schema(node_type, ModelClass)` | Avoids declaring a duplicate `Output` class in the central schema file |
-All five are idempotent (same callable / class for the same key is a
+All six are idempotent (same callable / class for the same key is a
no-op; conflicts raise `ValueError`).
+### Credential validation (Wave 11.I)
+
+The credential-validator dispatch is a sibling concern, handled by the
+existing `services/plugin/credential.py:Credential` base class. Your
+`Credential` subclass overrides `_probe(api_key) -> ProbeResult` (or,
+in rare cases like local-LLM 2-storage, the whole `validate(data)
+-> dict` classmethod). Maps, Apify, all 9 cloud LLM providers, and
+both local-LLM providers (Ollama / LM Studio) all dispatch through the
+same scaffold — no `_SPECIAL_PROVIDER_VALIDATORS` dict in
+`routers/websocket.py`.
+
### When NOT to use this shape
Don't create `_service.py` / `_handlers.py` siblings unless the plugin