Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
docs: replace hand-maintained counts with derivation methods
trohitgb0364eb2c605
diff --git a/CLAUDE.md b/CLAUDE.mdindex ff54f4e4..f1223389 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -13,8 +13,8 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire| **[UI Migration Plan](./docs-internal/ui_migration_plan.md)** | antd → shadcn/ui migration plan + completion log. Waves 1–10 done. Full frontend is schema-driven (backend SSOT); remaining DIY widget registry (ex-Phase 6) is the one deferred item. || **[Node Allowlist](./docs-internal/node_allowlist.md)** | Single-config UI visibility — `server/config/node_allowlist.json` controls which nodes / credential categories / skill folders show in the UI. Five lists with two enforcement tiers (mode-gated allowlist + absolute blocklist). `useNodeAllowlist` hook exposes `isVisible` / `isBlocked` / `isAllowed` / `isCredentialCategoryDisabled` / `isSkillFolderDisabled`. Adding a new disabled domain = single JSON edit, no code change. || **[Theme System](./docs-internal/theme_system.md)** | 10-way visual theme system — 5 utopian (light, dark, renaissance, greek, edo, steampunk, atomic) + 5 dystopian (cyber, wasteland, rot, plague, surveillance) — driven by `<html data-theme>` + per-theme CSS files in `client/src/themes/`. Token taxonomy (surface / fg / border / accent / typography / motion), shadcn HSL-triplet bridge, action role tokens, decorative-layer wrappers (`.app-frame` / `.canvas-host` / `.modal-frame`), per-component decorative ornaments (panel textures + canvas decorations + node pseudo-element overlays + theme-specific keyframes), **canvas-node visual contract via `--node-color` CSS custom property** (no inline `background` / `border` on node components — base.css + per-theme CSS owns visuals; `NodeStyle` helper type at [types/NodeTypes.ts](./client/src/types/NodeTypes.ts) makes the inline custom-prop typecheck-clean), **`--node-pulse-color` separate from `--node-color`** so executing-node glow uses each theme's highest-contrast accent regardless of plugin accent (Cyber neon cyan, Surveillance REC red, Renaissance ultramarine, etc.), **`data-page-hidden` animation pause** (toggled by Dashboard's `visibilitychange` listener; base.css declares `html[data-page-hidden] *, *::before, *::after { animation-play-state: paused !important }` to prevent compositor stall on tab return), **per-theme icon glyph system** (290 SVGs across 29 keys × 10 themes via [themedGlyphs.ts](./client/src/assets/icons/themedGlyphs.ts) + theme-aware [NodeIcon.tsx](./client/src/assets/icons/NodeIcon.tsx)), **per-theme canvas-grid + custom cursors** via `--canvas-grid` / `--cursor-default` slots, **decorative HTML primitives** (`<SvgFilterDefs>` mounting `#ink-blot` / `#noise` / `#crt` filter IDs at app root, `<DropCap>` wrapper for Renaissance ornament rule), **parameter panel migrated to Tailwind tokens** (no `useAppTheme()` reads; section headers carry the display-typography triplet; raw `<Button>` swapped to `<ActionButton intent>`), **per-theme scrollbar webkit rules** in all 10 themes, 9-event WebAudio sound system (10 packs via `--sound-pack` token + `useSound()` hook + global hover delegate + sonner toast monkey-patch + `withSound()` HOC + `Sounds.unlock()` gesture-unlock for AudioContext autoplay-policy compliance), `@media (prefers-reduced-motion: reduce)` accessibility, 30 ms throttle on `type` / `hover`, migration recipe, anti-patterns. Read this before adding a new theme, migrating a component to the new contract, or adding a canvas-node component. |-| **[Schema Source of Truth RFC](./docs-internal/schema_source_of_truth_rfc.md)** | Backend is SSOT for node schemas, visual metadata, handlers, palette metadata, icons. Plugin pattern: one `BaseNode` subclass in `server/nodes/<group>/<name>.py`. Wire format: `asset:<key>` / `<lib>:<brand>` / URL / emoji. Endpoint: `/api/schemas/nodes/{type}/spec.json`. 124 pytest invariants lock the contract. |-| **[Plugin System (Wave 11)](./docs-internal/plugin_system.md)** | Class-based plugin-first architecture. `BaseNode` / `ActionNode` / `TriggerNode` / `ToolNode` + `@Operation` decorator. Pydantic `Params`/`Output`. Declarative `Routing` DSL + `Connection` facade (Nango pattern). 18 `Credential` subclasses live in each node folder's `_credentials.py` (or inline for single-use). `TaskQueue` constants route to Temporal worker pools. 111 plugins live across 9 queues; handler bodies fully inlined (`services/handlers/` shrank 12.8K → 1.1K LOC across 16 → 4 files; only cross-cutting orchestration remains: `tools.py` AI-tool dispatch + agent delegation, `google_auth.py`, `triggers.py`). **Wave 11.H added "self-contained plugin folders"** — five generic registries (`ws_handler_registry`, `event_waiter.{register_filter_builder,register_trigger_precheck}`, `status_broadcaster.register_service_refresh`, `node_output_schemas.register_output_schema`) so rich plugins like telegram own their entire surface area without core-services edits. |+| **[Schema Source of Truth RFC](./docs-internal/schema_source_of_truth_rfc.md)** | Backend is SSOT for node schemas, visual metadata, handlers, palette metadata, icons. Plugin pattern: one `BaseNode` subclass in `server/nodes/<group>/<name>.py`. Wire format: `asset:<key>` / `<lib>:<brand>` / URL / emoji. Endpoint: `/api/schemas/nodes/{type}/spec.json`. Live invariant total via `pytest --collect-only`. |+| **[Plugin System (Wave 11)](./docs-internal/plugin_system.md)** | Class-based plugin-first architecture. `BaseNode` / `ActionNode` / `TriggerNode` / `ToolNode` + `@Operation` decorator. Pydantic `Params`/`Output`. Declarative `Routing` DSL + `Connection` facade (Nango pattern). 18 `Credential` subclasses live in each node folder's `_credentials.py` (or inline for single-use). `TaskQueue` constants route to Temporal worker pools. Plugins live across 9 queues (live count via `glob server/nodes/**/__init__.py`); handler bodies fully inlined (`services/handlers/` shrank 12.8K → 1.1K LOC across 16 → 4 files; only cross-cutting orchestration remains: `tools.py` AI-tool dispatch + agent delegation, `google_auth.py`, `triggers.py`). **Wave 11.H added "self-contained plugin folders"** — five generic registries (`ws_handler_registry`, `event_waiter.{register_filter_builder,register_trigger_precheck}`, `status_broadcaster.register_service_refresh`, `node_output_schemas.register_output_schema`) so rich plugins like telegram own their entire surface area without core-services edits. || **[Nodes Cookbook](./server/nodes/README.md)** | 5-minute recipe + folder map + shared helpers (`_base.py` / `_inline.py` per domain) + shared credentials + **self-contained plugin folder pattern (telegram reference)** + contract invariants + common pitfalls. Lives next to the plugin files. || **[Node Creation Guide](./docs-internal/node_creation.md)** | Canonical plugin recipe — one Python file (default), or a self-contained folder for richer plugins (telegram-style). Zero frontend edits, zero core-services edits. Auto-registers via `BaseNode.__init_subclass__` + the five `register_*` hooks. || **[AI Tool Node Guide](./docs-internal/ai_tool_node_creation.md)** | Detailed guide for creating dedicated AI Agent tool nodes (schemas, handlers, toolkits) |@@ -46,7 +46,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire| **[Native LLM SDK](./docs-internal/native_llm_sdk.md)** | Native SDK layer in services/llm/: Protocol-based providers, config-driven base URLs, 10 providers, native vs LangChain path routing || **[Event Waiter System](./docs-internal/event_waiter_system.md)** | Generic asyncio.Future/Redis-Streams waiter for push-based trigger nodes (WhatsApp, Telegram, Webhook, Chat, Task completion) || **[Credentials Encryption](./docs-internal/credentials_encryption.md)** | Fernet + PBKDF2 encryption pipeline, separate credentials.db, two credential systems (OAuth vs API keys), multi-backend abstraction |-| **[Status Broadcaster](./docs-internal/status_broadcaster.md)** | WebSocket-first communication: StatusBroadcaster singleton, 89 handlers, broadcast message types, Android two-state model |+| **[Status Broadcaster](./docs-internal/status_broadcaster.md)** | WebSocket-first communication: StatusBroadcaster singleton, live count via `len(MESSAGE_HANDLERS) + len(get_ws_handlers())`, broadcast message types, Android two-state model || **[RLM Service](./docs-internal/rlm_service.md)** | Recursive Language Model agent with REPL-based execution (llm_query, rlm_query, FINAL) || **[Claude Code Agent](./docs-internal/claude_code_agent_architecture.md)** | Claude Code integration internals (LangGraph reference; companion to `cli_agent_framework.md`) || **[CLI Agent Framework](./docs-internal/cli_agent_framework.md)** | Multi-provider CLI runtime (Claude Code / Codex / Gemini): `AICliService.run_batch`, per-task worktree isolation, FastMCP bridge, **memory bridge** (native `--resume` via stable `cwd=repo_root` + UUID5/`last_session_id` round-trip + auto-clear on stale UUID + `node_parameters_updated` broadcast). |@@ -541,7 +541,7 @@ class CacheEntry(SQLModel, table=True):- **Hybrid architecture**: Python (FastAPI + Pydantic plugins) + React/TypeScript frontend + Node.js subprocess for JS/TS code execution.- **Backend NodeSpec is the single source of truth.** Plugins live in [`server/nodes/<group>/<name>.py`](./server/nodes/) and auto-register via `BaseNode.__init_subclass__`. Authoritative node count is whatever globs out of `server/nodes/**/*.py` (excluding `_*.py` helpers and `__init__.py`); folders cover agent / model / android / google / whatsapp / twitter / telegram / social / email / search / scraper / document / code / filesystem / proxy / location / chat / text / scheduler / trigger / tool / utility / workflow / skill / browser / stripe.- **WebSocket-first frontend-backend communication.** Authoritative handler count is the size of the `MESSAGE_HANDLERS` dict in [`server/routers/websocket.py`](./server/routers/websocket.py) plus plugin-registered handlers via `services.ws_handler_registry`. Don't hand-maintain the count in this doc — it drifts on every plugin add.-- **Plugin-first architecture (Wave 11).** One file = one node. `services/handlers/` shrank from 12.8K → 1.1K LOC across 16 → 4 files. 124 pytest invariants lock the contract.+- **Plugin-first architecture (Wave 11).** One file = one node. `services/handlers/` shrank from 12.8K → 1.1K LOC across 16 → 4 files. Live invariant total via `pytest --collect-only`.## Frontend Performance Architecturediff --git a/docs-internal/frontend_architecture.md b/docs-internal/frontend_architecture.mdindex c7583e4b..52f18a2c 100644--- a/docs-internal/frontend_architecture.md+++ b/docs-internal/frontend_architecture.md@@ -459,7 +459,7 @@ Adding new panel behaviour: add a flag to `INodeUIHints`, annotate the relevant### Node output shape — backend as single source of truth-Frontend does **not** declare output shapes anymore. The backend owns them exclusively via Pydantic models in [server/services/node_output_schemas.py](../server/services/node_output_schemas.py) (98 entries today). JSON Schema is emitted via Pydantic's `model_json_schema()` and exposed two ways:+Frontend does **not** declare output shapes anymore. The backend owns them exclusively via Pydantic models in [server/services/node_output_schemas.py](../server/services/node_output_schemas.py) — live size via `len(NODE_OUTPUT_SCHEMAS)`. JSON Schema is emitted via Pydantic's `model_json_schema()` and exposed two ways:- `GET /api/schemas/nodes/{node_type}.json` — static, long-cache (`Cache-Control: public, max-age=86400`), no auth. n8n-style static-asset pattern.- `get_node_output_schema` WebSocket handler — authenticated editor path.diff --git a/docs-internal/node_creation.md b/docs-internal/node_creation.mdindex 9a7ea7df..722f7951 100644--- a/docs-internal/node_creation.md+++ b/docs-internal/node_creation.md@@ -242,9 +242,10 @@ What you **do** still write:## Wave summary (current state)-- **Wave 11** — Class-based plugin system. 111 plugins / 9 Temporal- worker pools / 124 contract invariants. `services/handlers/` shrank- from 12.8K → 1.1K LOC.+- **Wave 11** — Class-based plugin system. 9 Temporal worker pools;+ plugin count via `glob server/nodes/**/__init__.py`; invariant total+ via `pytest --collect-only`. `services/handlers/` shrank from+ 12.8K → 1.1K LOC.- **Wave 11.H** — Self-contained plugin folders. Five genericregistries replace per-plugin hardcoding in core. Telegram is thereference.diff --git a/docs-internal/plugin_system.md b/docs-internal/plugin_system.mdindex 532cfcce..ef82df70 100644--- a/docs-internal/plugin_system.md+++ b/docs-internal/plugin_system.md@@ -6,9 +6,11 @@ authoring model inspired by n8n's `INodeType`, Nango'sactivity pattern. One Python file under `server/nodes/<group>/<name>.py`= one node. No cross-cutting edits.-**Status: shipped.** 111 plugin classes cover every node type in the-product across 9 Temporal worker queues; 124 pytest contract-invariants lock the architecture. `services/handlers/` shrank from+**Status: shipped.** Plugin classes cover every node type in the+product — live count via `glob server/nodes/**/__init__.py`. Pytest+contract invariants (live count = test fns under+`server/tests/test_node_spec.py` + `test_plugin_self_containment.py`)+lock the architecture. `services/handlers/` shrank from12.8K → 1.1K LOC across 16 → 4 files. See[`server/nodes/README.md`](../server/nodes/README.md) for theauthoring cookbook (5-minute recipe + shared helpers + common@@ -677,7 +679,7 @@ every CI run. Examples:dependency).- Trigger registry auto-populates for every event-mode plugin.-108 Wave 10 invariants in `test_node_spec.py` still run — 124 total.+All Wave 10 invariants in `test_node_spec.py` still run; Wave 11 invariants in `test_plugin_self_containment.py`. Live total via `pytest --collect-only`.## Canonical principlesdiff --git a/docs-internal/schema_source_of_truth_rfc.md b/docs-internal/schema_source_of_truth_rfc.mdindex 70a2546d..0ba98bf8 100644--- a/docs-internal/schema_source_of_truth_rfc.md+++ b/docs-internal/schema_source_of_truth_rfc.md@@ -4,7 +4,7 @@## Landed outcome (Wave 3 — output schemas)-- Backend registry seeded with **98 Pydantic models** in [server/services/node_output_schemas.py](../server/services/node_output_schemas.py) covering every node type the deleted `sampleSchemas` map covered, plus all 18 specialized agents and 9 chat models.+- Backend registry seeded with Pydantic models in [server/services/node_output_schemas.py](../server/services/node_output_schemas.py) — live size via `len(NODE_OUTPUT_SCHEMAS)` — covering every node type the deleted `sampleSchemas` map covered, plus all specialized agents and chat models.- `GET /api/schemas/nodes/{node_type}.json` serves static JSON Schema (Cache-Control `public, max-age=86400`). 404 for unknown types.- `get_node_output_schema` WebSocket handler for authenticated editor fetches.- [InputSection.tsx](../client/src/components/parameterPanel/InputSection.tsx) consumes schemas lazy via `queryClient.fetchQuery(['nodeOutputSchema', nodeType])`, in-memory cache only. **1,673 → 972 LOC** (−701). sampleSchemas map, 20 `isXxxNode` detection constants, and 60-line pattern-match chain all gone.@@ -17,12 +17,12 @@ Wave 3 proved the pattern for output schemas. Wave 6 generalises it to the **ful**Shipped contract:**-- **Input schemas** — [server/services/node_input_schemas.py](../server/services/node_input_schemas.py) registers **105 Pydantic input models** across every node type, auto-expanding `SpecializedAgentParams` (16 variants) and `AndroidServiceParams` (16 variants) via Literal union introspection.+- **Input schemas** — [server/services/node_input_schemas.py](../server/services/node_input_schemas.py) registers Pydantic input models across every node type — live count via `len(_DIRECT_MODELS)` — auto-expanding `SpecializedAgentParams` and `AndroidServiceParams` via Literal union introspection.- **NodeSpec envelope** — [server/services/node_spec.py](../server/services/node_spec.py) assembles `{type, displayName, icon, group, description, version, inputs (JSON Schema), outputs (JSON Schema, reuses Wave 3), credentials, uiHints}` fusing three sources of truth.-- **Display metadata** — [server/models/node_metadata.py](../server/models/node_metadata.py) carries **105 entries** (one per input-modeled type). The only new server-side data file.+- **Display metadata** — [server/models/node_metadata.py](../server/models/node_metadata.py) carries one entry per input-modeled type — live size via `len(NODE_METADATA)`. The only new server-side data file.- **Endpoints** — alongside the Wave 3 `/nodes/{type}.json` output endpoint: `GET /api/schemas/nodes/{type}/spec.json`, `/nodes/{type}/input.json`, `/nodes/specs` (list), all with 24h `Cache-Control`. WS mirrors: `get_node_spec`, `list_node_specs`.- **loadOptionsMethod dispatch** — [server/services/node_option_loaders/](../server/services/node_option_loaders/) generalises the WhatsApp-only pattern to a registry. Adding a new dynamic-option loader = one-line registration. `POST /api/schemas/nodes/options/{method}` + WS `load_options`.-- **Node-groups index** — `GET /api/schemas/nodes/groups` returns `{group: [node_type, ...]}` derived from every NodeSpec's `group` array. **25 groups with 110 entries** replacing the **34 hand-rolled `*_NODE_TYPES` arrays** scattered across 6 frontend files.+- **Node-groups index** — `GET /api/schemas/nodes/groups` returns `{group: [node_type, ...]}` derived from every NodeSpec's `group` array — live count via `len(NODE_GROUPS)` — replacing the hand-rolled `*_NODE_TYPES` arrays scattered across 6 frontend files.- **Pydantic Field hints** — authors encode `displayOptions.show`/`.hide`, `loadOptionsMethod`, `placeholder`, `validation`, `typeOptions` via `Field(json_schema_extra={...})`. Currently ~36 rules encoded across TwitterSend/User, TelegramSend, HttpRequest, WhatsAppSend, SocialSend, Gmail, Calendar.**Frontend consumer (feature-flagged):**@@ -357,7 +357,7 @@ Adds 16 invariants to the 108 Wave 10 suite → **124 total**:### Status-111 plugins live across 9 Temporal task queues (`rest-api`,+Plugins (live count via `glob server/nodes/**/__init__.py`) live across 9 Temporal task queues (`rest-api`,`ai-heavy`, `code-exec`, `triggers-poll`, `triggers-event`, `android`,`browser`, `messaging`, `machina-default`). Handler bodies are fullyinlined into plugin files — `services/handlers/` shrank from