Skip to content

Self-improving AI Employees for everyone

docs: platform refactor RFC + credentials-scaling + UI optimization research

trohitg586a1602ec69
diff --git a/docs-internal/credentials_scaling/PHASE_8_VERIFICATION.md b/docs-internal/credentials_scaling/PHASE_8_VERIFICATION.md
new file mode 100644
index 00000000..bf87652f
--- /dev/null
+++ b/docs-internal/credentials_scaling/PHASE_8_VERIFICATION.md
@@ -0,0 +1,224 @@
+# Phase 8 — Verification Checklist
+
+Phases 1–7.5 of the credentials-scaling plan have all landed on the
+`feature/credentials-scaling` worktree. This document is the
+**manual verification gate** a human operator walks through before the
+worktree merges to `main`.
+
+Everything in Section A is already verified programmatically in the
+session that landed Phases 2–7.5 — recorded here for completeness so the
+reviewer knows what is and isn't proven.
+
+Everything in Section B needs a browser + the server running.
+
+Everything in Section C decides whether `providers.tsx` can be deleted.
+
+---
+
+## A. Already verified (programmatic)
+
+- [x] `server/config/credential_providers.json` loads and `extends`
+ resolution produces 20 concrete providers (9 AI via `_ai_base`
+ inheritance + 11 non-AI) with `_ai_base` hidden from
+ `get_all_providers()`.
+- [x] `server/services/credential_registry.py` lazy singleton returns
+ `{providers, categories, version}` with a content-sha256 version
+ hash.
+- [x] `get_credential_catalogue` WebSocket handler with conditional
+ `since` refetch (returns `{unchanged: true}` on matching version).
+- [x] `/api/credentials/icon/{provider_id}` endpoint with path-traversal
+ guard, immutable caching, 404 fallback.
+- [x] `services/idempotency.py` — TTL-bounded cache with concurrent
+ dedupe, exception caching, pass-through when `request_id` unset;
+ wired into `handle_save_api_key` + `handle_delete_api_key`.
+- [x] `services/circuit_breaker.py` — closed → open (3 failures / 60 s)
+ → half-open (after cooldown) → closed on probe success / reopen
+ on probe failure; cross-scope isolation; dict-shape failures
+ counted alongside raised exceptions.
+- [x] `AuthService.refresh_oauth_tokens_with_breaker` wraps any refresh
+ callable (sync or async) with a per-provider circuit breaker.
+- [x] `client/src/hooks/useCatalogueQuery.ts` — TanStack Query v5 +
+ `idb-keyval` warm-start + deferred `requestIdleCallback` persist
+ write + conditional `since` refetch.
+- [x] `client/src/store/useCredentialRegistry.ts` — UI state only
+ (`selectedId`, `paletteOpen`, `query`). No catalogue, no derived
+ data. Avoids the #1 runtime/memory trap.
+- [x] `client/src/main.tsx` — `QueryClientProvider` wired at app root.
+- [x] `client/src/components/credentials/catalogueAdapter.ts` —
+ rehydrator: server JSON → runtime `ProviderConfig` with icon
+ factory dispatch, status-row/action/qr field-ref compilation, and
+ explicit fallback for unknown `icon_ref` kinds.
+- [x] `client/src/components/credentials/CredentialsPalette.tsx` —
+ cmdk + fuzzysort (pre-indexed on `name` + `categoryLabel`) +
+ GroupedVirtuoso (sticky category headers) + `startTransition`
+ wrapping the filter update. `useMemo` for every derived artifact.
+- [x] `client/src/components/credentials/CredentialsModal.tsx` —
+ server-owned catalogue via `useCatalogueQuery`, **additive
+ fallback** to bundled `providers.tsx` when the query is loading
+ or errored, visible `offline`/`loading` status tags in the header.
+- [x] `client/src/components/credentials/PanelRenderer.tsx` —
+ `React.lazy` per `PanelKind`, `Suspense` fallback. Vite emits
+ one chunk per panel kind.
+- [x] `client/src/components/credentials/schema/extends.ts` — client
+ port of the Python `_deep_merge` with array-merge-by-key
+ semantics. No callers yet (reserved for Phase B NodeSpec).
+- [x] `vite.config.js` — React Compiler scoped to credentials module
+ via `babel-plugin-react-compiler` `sources` filter; rollup bundle
+ visualizer wired behind `ANALYZE=1` flag.
+- [x] `client/scripts/audit-bundle.py` — reproducible bundle audit
+ that parses the sourcemap to attribute bytes to packages/files.
+ See `docs-internal/ui_optimization/BUNDLE_AUDIT.md`.
+- [x] `pnpm exec tsc --noEmit` — zero errors across the entire client.
+- [x] `pnpm run build` — builds in ~27 s; panel chunks emitted at
+ ApiKey 9.4 KB / OAuth 6.9 KB / Email 5.5 KB / QrPairing 28.9 KB;
+ main index 2,341 KB (pre-existing baseline).
+
+## B. Needs a browser + server
+
+Run the dev stack:
+
+```bash
+cd d:/startup/projects/MachinaOs/.claude/worktrees/credentials-scaling
+pnpm run dev
+```
+
+Then walk through:
+
+- [ ] **Credentials modal opens** without console errors or crashed
+ WebSocket connection.
+- [ ] **First modal open** shows `loading` tag briefly in the header
+ then the 20 providers render grouped by category. Network panel
+ shows exactly ONE `get_credential_catalogue` WebSocket message.
+- [ ] **Second modal open** (close + reopen): zero network traffic;
+ modal opens in < 50 ms; providers render from the IndexedDB
+ warm-start before the background revalidate completes.
+- [ ] **Search input**: type `whats` → WhatsApp ranks first; type
+ `google` → Google Workspace ranks first; type `zzzzz` → empty
+ state shows `No providers match "zzzzz"`; input stays responsive
+ even while typing rapidly.
+- [ ] **Keyboard nav**: arrow down / up moves the selection; Enter
+ picks the highlighted provider; Escape clears the search.
+- [ ] **Category order** matches `credential_providers.json`:
+ AI → Social → Productivity → Email → Android → Search →
+ Scrapers → Services.
+- [ ] **Lazy panels**: open the Network tab with "JS" filter, select
+ an OpenAI provider → one `ApiKeyPanel-*.js` request. Select
+ WhatsApp → one `QrPairingPanel-*.js` request. Re-select the same
+ panel kind → zero additional requests.
+- [ ] **Offline fallback**: stop the server, reload the page, open
+ credentials → header shows `offline` tag and the palette still
+ renders the bundled `providers.tsx` fallback list. No error
+ state.
+- [ ] **Save API key idempotency**: save an OpenAI key, then without
+ changing anything click Save again quickly. Server logs should
+ show `idempotency[credentials]: cached success for <uuid>` for
+ the duplicate. Only one encryption round-trip hits the DB.
+- [ ] **Delete API key** works, UI transitions back to unstored state.
+- [ ] **OAuth circuit breaker (manual)**: temporarily break the
+ Google token endpoint (firewall rule or fake credentials).
+ Attempt three refresh cycles → breaker opens; fourth attempt
+ returns `{circuit_open: true, retry_after_seconds: ~30}`.
+ Twitter refresh still works in parallel. After 30 s, next attempt
+ probes and closes the breaker when you restore the endpoint.
+
+### B.1 Heap + INP targets (DevTools)
+
+- [ ] **Chrome DevTools → Memory → Heap Snapshot** after opening the
+ modal: filter by `Array`, sort by Retained Size. The catalogue
+ array appears once with retained size **~1.5–2.5 MB at 5000
+ entries** (less at 20). Exactly **one retainer**: TanStack Query
+ cache. If the count is > 1 something is wrong.
+- [ ] **Allocation timeline**: open + close the modal 50 times.
+ **Delta heap < 1 MB**. Larger delta indicates a listener leak or
+ detached-DOM retention.
+- [ ] **web-vitals INP attribution** (dev console):
+ ```js
+ import('web-vitals/attribution').then(({ onINP }) =>
+ onINP((metric) => console.log('INP:', metric.value, metric.attribution)));
+ ```
+ Type rapidly in the search box; p75 should stay **< 200 ms**.
+ Target during normal use: < 100 ms.
+- [ ] **React DevTools Profiler flamegraph**: record a filter-typing
+ session. Only the search box + visible virtuoso rows should
+ re-render. The rest of the modal flamegraph should be grey
+ (unchanged).
+
+### B.2 5000-provider scaling probe
+
+Temporarily seed the JSON with 4,980 dummy providers to simulate
+Pipedream scale:
+
+```bash
+cd server
+python -c "
+import json, copy
+with open('config/credential_providers.json') as f:
+ data = json.load(f)
+base = data['providers']['openai']
+for i in range(4980):
+ dummy = copy.deepcopy(base)
+ dummy['name'] = f'Dummy Provider {i:04d}'
+ data['providers'][f'dummy_{i:04d}'] = dummy
+with open('config/credential_providers.json.5k', 'w') as f:
+ json.dump(data, f, indent=2)
+"
+mv config/credential_providers.json config/credential_providers.json.original
+mv config/credential_providers.json.5k config/credential_providers.json
+# restart server
+```
+
+Then:
+
+- [ ] Modal opens in < 300 ms cold / < 50 ms warm.
+- [ ] Search latency on 5000 entries < 10 ms (check `performance.measure`
+ around `fuzzysort.go`).
+- [ ] Scroll from row 1 to row 2000: **60 fps** in DevTools Performance.
+- [ ] Retained catalogue array: ~1.5–2.5 MB.
+- [ ] DOM nodes in the virtuoso container: 10–50 regardless of scroll
+ position.
+
+Restore the original JSON after the probe:
+
+```bash
+mv config/credential_providers.json.original config/credential_providers.json
+```
+
+## C. `providers.tsx` retention decision
+
+Current state: `providers.tsx` is still in place. `CredentialsModal`
+uses it as a fallback when the server fetch hasn't landed yet.
+
+### Delete when ALL of the following are true
+
+- [ ] Section B manual checklist passes.
+- [ ] Server `get_credential_catalogue` handler has been live in
+ production for at least **14 days** with zero error logs.
+- [ ] Warm-start IDB path has been hit by at least one real user (the
+ `credentials:catalogue:current` key shows up in DevTools →
+ Application → IndexedDB → `keyval-store`).
+- [ ] The fallback code path in `CredentialsModal.tsx:47` has NOT
+ been hit (instrument a one-line `console.log` during the soak to
+ confirm).
+
+### How to delete
+
+1. Remove `client/src/components/credentials/providers.tsx`.
+2. Remove the `PROVIDERS` / `CATEGORIES` imports from
+ `CredentialsModal.tsx`.
+3. Replace the fallback branch with an empty-state Result component:
+ ```tsx
+ if (!catalogue.data && catalogue.isLoading) return <Spin />;
+ if (!catalogue.data) return <Result status="error" subTitle="Unable to load credential catalogue" />;
+ ```
+4. `pnpm exec tsc --noEmit && pnpm run build`.
+5. Commit as a separate PR: "credentials: remove client-owned provider
+ registry (Phase 8 cleanup)".
+
+Deletion is deliberately **not** part of the initial PR. Keeping the
+fallback is the safest possible production rollout.
+
+## Signoff
+
+| Reviewer | Date | Result |
+|---|---|---|
+| _(pending)_ | | |
diff --git a/docs-internal/credentials_scaling/architecture.md b/docs-internal/credentials_scaling/architecture.md
new file mode 100644
index 00000000..374073b8
--- /dev/null
+++ b/docs-internal/credentials_scaling/architecture.md
@@ -0,0 +1,410 @@
+# Credentials Panel Architecture (Scalable to 5000 Providers)
+
+The architecture for the MachinaOs credentials panel after scaling work.
+Built on the Phase 1 refactor (21-file modular structure) with five additional
+layers: server-owned registry, bulk-fetch cache, command palette, lazy panels,
+and react-hook-form detail panels.
+
+See also:
+- [research_production_platforms.md](./research_production_platforms.md) — evidence base from n8n, Nango, Pipedream, Zapier, Supabase
+- [research_react_stack.md](./research_react_stack.md) — library choices and benchmarks
+- [typed-splashing-crown.md](../../../../.claude/plans/typed-splashing-crown.md) — execution plan
+
+---
+
+## Five-layer architecture
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│ 1. BACKEND REGISTRY (Nango-style, JSON for project parity) │
+│ │
+│ server/config/credential_providers.json │
+│ └─ 5000+ entries with `extends` inheritance │
+│ │
+│ server/services/credential_registry.py │
+│ ├─ load_catalogue() parses JSON at startup │
+│ ├─ resolve_extends() deep-merges inherited fields │
+│ ├─ get_all_providers() → list[dict] │
+│ ├─ get_provider(id) → dict │
+│ └─ validated with Pydantic │
+│ │
+│ server/routers/websocket.py │
+│ └─ handle_get_credential_catalogue → one bulk JSON │
+│ │
+│ server/routers/credentials.py │
+│ └─ GET /api/credentials/icon/{id} (cached SVG) │
+└──────────────────────────────┬──────────────────────────────┘
+ ▼ one WebSocket call at first modal open
+┌─────────────────────────────────────────────────────────────┐
+│ 2. ZUSTAND STORE (cached catalogue) │
+│ │
+│ client/src/stores/useCredentialRegistry.ts │
+│ ├─ providers: ProviderConfig[] │
+│ ├─ categories: CategoryGroup[] │
+│ ├─ preparedSearch: Fuzzysort.Prepared[] (memoized) │
+│ ├─ fetchOnce() — idempotent, no double-fetch │
+│ ├─ getById(id) — O(1) Map lookup │
+│ ├─ search(query) — fuzzysort.go() <5ms on 5000 items │
+│ └─ Cache lives for app session; invalidated on WS event │
+└──────────────────────────────┬──────────────────────────────┘
+ ▼
+┌─────────────────────────────────────────────────────────────┐
+│ 3. SEARCH + NAVIGATION (cmdk + fuzzysort + virtuoso) │
+│ │
+│ credentials/CredentialsPalette.tsx │
+│ ├─ <Command shouldFilter={false}> │
+│ ├─ <Command.Input> — debounced query │
+│ ├─ <GroupedVirtuoso> — sticky category headers │
+│ │ └─ renders only visible rows at 60fps │
+│ ├─ Keyboard nav (↑↓ Enter Esc) via cmdk built-in │
+│ └─ Ctrl+K opens palette anywhere in the app │
+└──────────────────────────────┬──────────────────────────────┘
+ ▼ selected provider id
+┌─────────────────────────────────────────────────────────────┐
+│ 4. LAZY PANEL DISPATCH (React.lazy + Suspense) │
+│ │
+│ credentials/PanelRenderer.tsx │
+│ const PANEL_LOADERS = { │
+│ apiKey: () => import('./panels/ApiKeyPanel'), │
+│ oauth: () => import('./panels/OAuthPanel'), │
+│ qrPairing: () => import('./panels/QrPairingPanel'), │
+│ email: () => import('./panels/EmailPanel'), │
+│ }; │
+│ const Lazy = React.lazy(PANEL_LOADERS[config.kind]); │
+│ <Suspense fallback={<Spin />}><Lazy ... /></Suspense> │
+│ │
+│ Vite auto-chunks each panel (<50 KB each). │
+└──────────────────────────────┬──────────────────────────────┘
+ ▼
+┌─────────────────────────────────────────────────────────────┐
+│ 5. SCHEMA-DRIVEN FORMS (react-hook-form) │
+│ │
+│ credentials/schema/extends.ts │
+│ └─ mergeProperties(base, override) — n8n pattern │
+│ │
+│ credentials/primitives/FieldRenderer.tsx │
+│ ├─ <Controller> wrapping antd inputs │
+│ ├─ Uncontrolled — 5-10× faster than antd Form │
+│ └─ Supports `extends` via merged FieldDef[] │
+│ │
+│ panels/{ApiKey,OAuth,Email}Panel.tsx │
+│ └─ useForm() → RHF, Controller per field │
+└─────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Data flow (reading credentials)
+
+```
+User opens CredentialsModal
+ ↓
+CredentialsModal.tsx calls useCredentialRegistry().fetchOnce()
+ ↓
+Zustand store: if not fetched, sendRequest('get_credential_catalogue')
+ ↓
+WebSocket backend: credential_registry.get_all_providers()
+ ↓
+YAML loaded (cached at startup) → resolve_extends() → Pydantic-validated list
+ ↓
+{ providers: [...5000], categories: [...20], version: "abc123" }
+ ↓
+Zustand stores providers + prepares fuzzysort index
+ ↓
+CredentialsPalette renders GroupedVirtuoso
+ ↓
+User presses Ctrl+K or clicks sidebar → search input focus
+ ↓
+User types "whats" → fuzzysort.go() → 3ms → filtered list
+ ↓
+User selects WhatsApp → setSelectedId('whatsapp_personal')
+ ↓
+PanelRenderer looks up config by id → determines kind='qrPairing'
+ ↓
+React.lazy loads QrPairingPanel chunk (first time) → Suspense spinner
+ ↓
+QrPairingPanel renders → calls useCredentialPanel(config, visible)
+ ↓
+useForm() initializes RHF → fields hydrate from stored credentials
+```
+
+---
+
+## Data flow (saving credentials)
+
+```
+User edits a field
+ ↓
+RHF Controller → onChange → form state (uncontrolled, no re-render)
+ ↓
+User clicks Save
+ ↓
+handleSubmit(data) → panel.actions.save(key, value)
+ ↓
+useCredentialPanel execute('save', ...) → setLoading('save')
+ ↓
+useApiKeys.saveApiKey(key, value) → WebSocket → AuthService.store_api_key()
+ ↓
+EncryptedAPIKey table (Fernet + PBKDF2 + PBKDF2HMAC)
+ ↓
+Response → setStored(true), clear password field
+```
+
+---
+
+## JSON schema with `extends` inheritance
+
+MachinaOs uses JSON for all backend config (`email_providers.json`, `llm_defaults.json`,
+`pricing.json`, `google_apis.json`). Credential registry follows the same convention.
+
+```json
+{
+ "version": "2026.04",
+ "providers": {
+ "_ai_base": {
+ "_abstract": true,
+ "category": "ai",
+ "category_label": "AI Providers",
+ "kind": "apiKey",
+ "has_defaults": true,
+ "fields": [
+ { "key": "apiKey", "label": "API Key", "type": "password", "required": true }
+ ]
+ },
+ "openai": {
+ "extends": "_ai_base",
+ "name": "OpenAI",
+ "color": "green",
+ "fields": [
+ { "key": "apiKey", "placeholder": "sk-..." }
+ ]
+ },
+ "anthropic": {
+ "extends": "_ai_base",
+ "name": "Anthropic",
+ "color": "orange",
+ "fields": [
+ { "key": "apiKey", "placeholder": "sk-ant-..." }
+ ]
+ },
+ "gemini": {
+ "extends": "_ai_base",
+ "name": "Gemini",
+ "color": "cyan",
+ "fields": [
+ { "key": "apiKey", "placeholder": "AIza..." }
+ ]
+ }
+ }
+}
+```
+
+**`credential_registry.resolve_extends()` behavior**:
+1. Walk providers in YAML order
+2. If `extends:` field set, deep-merge parent first, then override with child fields
+3. For array fields (`fields`, `statusRows`, `actions`): merge by `key` — child
+ entries replace parent entries with the same key; unmatched child entries append
+4. `_abstract: true` providers are resolved but excluded from `get_all_providers()`
+5. Cycle detection: throw on self-reference or loops
+
+---
+
+## Why this architecture?
+
+### Why server-owned JSON?
+- **Zero per-provider TypeScript.** Adding a new provider = editing one JSON entry.
+- Git diffs stay readable as the catalogue grows.
+- Non-engineers (support team, partners) can PR new providers without touching code.
+- Parsed once at server startup, cached in memory. No runtime parsing cost per request.
+- **Matches existing project convention**: every other backend config file is JSON
+ (`email_providers.json`, `llm_defaults.json`, `pricing.json`, `google_apis.json`).
+ No new dependencies, stdlib `json` only.
+
+### Why one bulk WebSocket fetch?
+- At 5000 providers × ~200 bytes/entry = ~1 MB JSON. Transferable in one call, loadable
+ in <500ms over a local WebSocket.
+- Cached in Zustand for the app session — no re-fetch on modal reopen.
+- Matches n8n's proven pattern (they do it at 400 providers; we scale to 5000).
+
+### Why `react-virtuoso` over antd Menu?
+- **antd Menu has no virtualization**. At 1000+ items it mounts every `<Menu.Item>`
+ to the DOM, killing initial render performance.
+- `react-virtuoso`'s `GroupedVirtuoso` gives sticky category headers for free.
+- 18.8 KB gz — acceptable cost for the scalability win.
+
+### Why cmdk + fuzzysort over antd Select `filterable`?
+- antd Select's substring filter is O(N) per keystroke with no ranking — at 5000
+ items users can't find anything useful.
+- cmdk provides keyboard navigation + accessibility out of the box.
+- fuzzysort is the fastest JS fuzzy search (3× faster than Fuse.js).
+- Combined cost: ~21 KB gz.
+
+### Why React.lazy per panel?
+- Each panel (ApiKeyPanel, OAuthPanel, QrPairingPanel, EmailPanel) is 40-80 lines
+ but pulls in different deps (e.g., QrPairingPanel depends on `QRCodeDisplay` +
+ `useWhatsApp`; EmailPanel depends on provider preset logic).
+- Lazy-loading them drops initial credentials modal JS by ~60%.
+- Vite handles chunking automatically — no config needed.
+
+### Why react-hook-form over antd Form.useForm()?
+- Uncontrolled by default → ~2ms per keystroke vs 18ms for antd Form at 200 fields.
+- Critical for the `ProviderDefaultsSection` (AI provider defaults with 8+ fields).
+- Keeps antd components via `<Controller>` — we're swapping the state engine,
+ not the UI kit.
+
+### Why `extends` inheritance?
+- Today, 9 AI providers duplicate the same `fields: [{key: 'apiKey', ...}]` config.
+- With `extends: _ai_base`, each entry is 4 lines instead of 12.
+- At 5000 providers, inheritance is the difference between a 1 MB YAML and a 3 MB YAML.
+
+---
+
+## Performance targets
+
+| Metric | Current (monolith) | Phase 1 refactor | Phase 2 (after scaling) | Target |
+|--------|--------------------|--------------------|--------------------------|---------|
+| CredentialsModal initial JS | ~180 KB | ~180 KB | ~100 KB | <150 KB |
+| Per-panel chunk | N/A (monolithic) | N/A | <50 KB each | <50 KB |
+| Sidebar initial render (20 providers) | 50 ms | 40 ms | 30 ms | <50 ms |
+| Sidebar initial render (5000 providers) | crashes | 3000 ms (DOM bloat) | <200 ms (virtualized) | <300 ms |
+| Search latency (5000 items) | N/A (no search) | N/A | <5 ms | <10 ms |
+| Scroll at row 2000 | N/A | ~20 fps (jank) | 60 fps | ≥60 fps |
+| Catalogue WebSocket fetch (5000 entries) | N/A | N/A | ~300 ms | <500 ms |
+
+---
+
+## Migration safety
+
+1. **Worktree**: `feature/credentials-scaling` branched from main with Phase 1
+ refactor cherry-picked on top. No changes to main until verified.
+2. **Original import site preserved**: `client/src/components/CredentialsModal.tsx`
+ remains a 1-line re-export. All existing usages unchanged.
+3. **Backwards-compatible YAML**: can serve the current 20 providers from YAML while
+ the old `providers.tsx` is still present. Flip the switch when the YAML path is
+ proven. Delete `providers.tsx` only after manual verification.
+4. **Feature flag** (optional): `VITE_USE_YAML_CREDENTIALS=true` — gates the new
+ Zustand store fetch vs the old inline array. Enables A/B testing in dev.
+5. **No main branch touches** until the full verification checklist passes.
+
+---
+
+## Files touched
+
+See [typed-splashing-crown.md](../../../../.claude/plans/typed-splashing-crown.md)
+section "Critical files to modify/create" for the exhaustive list.
+
+Summary:
+- **New**: 9 files (3 research MDs, 1 YAML, 4 Python, 2 TypeScript stores, 1 palette, 1 schema helper)
+- **Modified**: 8 files (worktree credentials module + package.json + CLAUDE.md)
+- **Deleted**: 1 file (`credentials/providers.tsx` after YAML migration verified)
+
+---
+
+## Status: plan approved, execution in progress
+
+This document is authoritative for the architecture. If implementation deviates,
+update this file and the plan file together.
+
+---
+
+## Addendum — Revised architecture (IndexedDB + TanStack Query + runtime targets)
+
+The base 5-layer architecture above is correct but incomplete. Runtime/memory research revealed a sixth layer (IndexedDB persistence) and refined Phase 3 (TanStack Query owns the catalogue, Zustand holds UI state only). The revised architecture is **6 layers**:
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│ 1. BACKEND REGISTRY │
+│ server/config/credential_providers.json │
+│ server/services/credential_registry.py │
+│ - load_catalogue() parses JSON once at startup │
+│ - resolve_extends() deep-merges inherited fields │
+│ - content-sha256 version hash │
+│ server/routers/websocket.py │
+│ - handle_get_credential_catalogue │
+│ - handle_save_credential (+ idempotency key dedupe) │
+│ server/routers/credentials.py │
+│ - GET /api/credentials/icon/{id} (long-lived cache) │
+└──────────────────────────────┬──────────────────────────────┘
+ ▼ WebSocket bulk fetch
+┌─────────────────────────────────────────────────────────────┐
+│ 2. INDEXEDDB PERSISTENCE (NEW) │
+│ idb-keyval — key `credentials:catalogue:v<sha>` │
+│ Warm-start read on app open (<50 ms); background │
+│ revalidate via WebSocket version-hash comparison │
+│ Hydration write deferred via requestIdleCallback │
+│ (avoids 50–200 ms main-thread block at app open) │
+└──────────────────────────────┬──────────────────────────────┘
+ ▼
+┌─────────────────────────────────────────────────────────────┐
+│ 3. TANSTACK QUERY (server cache) + ZUSTAND (UI state) │
+│ useCatalogueQuery() │
+│ - queryKey ['credentialCatalogue'] │
+│ - staleTime Infinity, gcTime 10 min │
+│ - persister experimental_createPersister(idb-keyval) │
+│ useCredentialRegistry (Zustand, UI state ONLY) │
+│ - selectedId, paletteOpen, query │
+│ - NO catalogue, NO derived state, NO preparedSearch │
+│ - Prevents the #1 memory trap: selector closures │
+│ capturing the whole 1.5–2.5 MB catalogue array │
+└──────────────────────────────┬──────────────────────────────┘
+ ▼
+┌─────────────────────────────────────────────────────────────┐
+│ 4. PALETTE (cmdk + fuzzysort + GroupedVirtuoso) │
+│ CredentialsPalette.tsx │
+│ - prepared fuzzysort index via useMemo(providers) │
+│ - startTransition wrapping filter updates (measured │
+│ ~30–70 ms shorter long task vs useDeferredValue) │
+│ - GroupedVirtuoso for sticky category headers │
+│ - DOM pool: 10–50 live nodes regardless of item count │
+└──────────────────────────────┬──────────────────────────────┘
+ ▼
+┌─────────────────────────────────────────────────────────────┐
+│ 5. LAZY PANEL DISPATCH (React.lazy + Suspense) │
+│ PanelRenderer.tsx — unchanged from base plan │
+└──────────────────────────────┬──────────────────────────────┘
+ ▼
+┌─────────────────────────────────────────────────────────────┐
+│ 6. SCHEMA-DRIVEN FORMS + React Compiler │
+│ credentials/primitives/FieldRenderer.tsx │
+│ - RHF Controller wrapping antd inputs (uncontrolled) │
+│ React Compiler auto-memoizes the whole credentials module │
+│ (scoped via babel-plugin-react-compiler in vite.config) │
+└─────────────────────────────────────────────────────────────┘
+```
+
+### Runtime + memory targets (what verification must prove)
+
+| Metric | Target | Measurement method |
+|---|---|---|
+| V8 heap — catalogue retained size | ~1.5–2.5 MB, exactly one retainer (TanStack Query cache) | Chrome DevTools → Memory → Heap Snapshot → filter Array, sort by Retained Size |
+| DOM node count at any scroll position | 10–50 (virtuoso pool) | Elements panel count inside the virtuoso container |
+| Modal 50-cycle heap delta (open/close 50×) | < 1 MB | Allocation timeline before vs after |
+| INP p75 during rapid search typing | < 200 ms (stretch: < 100 ms) | `web-vitals` library `onINP` attribution mode |
+| Search latency on 5000 items | < 10 ms (fuzzysort.go only) | `performance.measure()` around the filter |
+| Catalogue cold fetch (5000 providers, first ever open) | < 500 ms | Network panel timing |
+| Catalogue warm-start (second open, IndexedDB hit) | < 50 ms | Network panel: zero calls; React Profiler commit time |
+| Scroll FPS at row 2000 | 60 fps | DevTools Performance tab |
+
+### Runtime traps to avoid (the non-negotiable list)
+
+1. **Zustand selector capturing the whole catalogue in a closure** — retains 1.5–2.5 MB permanently. Fix: keep derived data in `useMemo` inside components, not in the store.
+2. **`useState(catalogue.filter(...))`** on every render — 1–5 MB GC churn/cycle. Fix: `useMemo`.
+3. **Non-virtualized list** — 5–20 MB DOM + detached-node leaks. Fix: react-virtuoso.
+4. **IndexedDB main-thread blocking write** — 50–200 ms stall. Fix: defer via `requestIdleCallback` after first paint.
+5. **Fuse.js with `shouldSort + includeMatches`** on 5000 items — 10–50 MB transient per query. Fix: stay on fuzzysort.
+
+### Referenced modern architecture sites
+
+- **oboe.com** (inspected at user request, 2026): React + TanStack Router + TanStack Query + dehydrated→hydrated query client + SSR streaming. Our Vite SPA analog is IndexedDB persister + WebSocket version-hash invalidation. Same pattern, different transport.
+- **Sanity Studio** + **Wakelet**: production React Compiler rollouts — 15% INP improvement on already-memoized code, 50–80% on fresh code.
+- **n8n**: `credentials.store.ts` Pinia bulk fetch + `mergeNodeProperties` extends inheritance. Our Phase 3 + Phase 6 directly port this.
+- **Nango**: content-addressed provider manifests with version-keyed delta fetches. We adopt the same version-hash approach but with a single-file catalogue.
+
+### Addendum sources
+
+- React Compiler 1.0 release: https://react.dev/blog/2025/10/07/react-compiler-1
+- TanStack Query `experimental_createPersister`: https://tanstack.com/query/v5/docs/react/plugins/createPersister
+- V8 memory optimization: https://v8.dev/blog/optimizing-v8-memory
+- Chrome DevTools heap snapshots: https://developer.chrome.com/docs/devtools/memory-problems/heap-snapshots
+- web-vitals library: https://github.com/GoogleChrome/web-vitals
+- MemLab (Meta): https://engineering.fb.com/2022/09/12/open-source/memlab/
diff --git a/docs-internal/credentials_scaling/research_production_platforms.md b/docs-internal/credentials_scaling/research_production_platforms.md
new file mode 100644
index 00000000..87dea785
--- /dev/null
+++ b/docs-internal/credentials_scaling/research_production_platforms.md
@@ -0,0 +1,256 @@
+# Research: How Production Platforms Scale Credential/Node UIs
+
+Research findings on how n8n, Nango, Pipedream, Zapier, and Supabase Studio structure
+their credential and integration registries at hundreds to thousands of providers.
+Informs the MachinaOs credentials scaling architecture.
+
+---
+
+## Summary
+
+The most important discovery: **none of these platforms virtualize credential/integration
+lists at their current scale**. The real bottleneck is **schema/icon I/O and main-bundle
+weight**, not DOM rendering. The winning patterns are:
+
+1. Server-owned declarative registry (Nango YAML, n8n backend scan)
+2. One bulk-fetch at boot, cached in a client store
+3. Lazy per-integration metadata (Pipedream) or lazy icon endpoints (n8n)
+4. Schema inheritance via `extends` to eliminate duplication (n8n)
+5. Aggressive lazy-loading of heavy optional features (Supabase `next/dynamic`)
+
+---
+
+## n8n (~400 credentials, ~700 nodes)
+
+The closest analogue to MachinaOs — open-source workflow tool with React-like frontend
+(Vue), schema-driven node definitions, credential edit modal, and node palette.
+
+**Source**: https://github.com/n8n-io/n8n
+
+### Frontend registry
+
+- `packages/frontend/editor-ui/src/stores/credentials.store.ts` exposes
+ `allCredentialTypes` — a Pinia store cached after one REST fetch at app boot.
+- `packages/frontend/editor-ui/src/features/credentials/components/CredentialsSelectModal.vue`
+ renders a **plain `v-for`** over `N8nSelect`/`N8nOption` with `filterable` substring
+ matching. **No virtualization, no fuse.js, no cmdk.**
+- `CredentialEdit.vue` renders all 400+ types from one generic schema walker over
+ `INodeProperties[]`. It reads `credentialTypeData.extends` and merges deltas via
+ `NodeHelpers.mergeNodeProperties()` (a ~30-line deep-merge helper).
+
+### Backend registry
+
+- `CredentialsHelper` scans installed packages at startup and builds the catalogue
+ in-memory on the server.
+- The frontend fetches the entire credential-type catalogue in **one bulk REST call**
+ at app boot, then caches it in Pinia.
+
+### Icon loading
+
+- Icons are served **lazily per tile** via `/rest/node-icon/<nodeType>` — no sprite
+ sheets, no data URIs inlined in the main bundle. Browser handles caching.
+
+### Key takeaways
+- **Bulk-fetch + cache** pattern: one call at boot beats per-open fetches.
+- **Schema inheritance** (`extends`) is the single biggest de-duplication win. In
+ MachinaOs, `AI_AGENT_PROPERTIES` is duplicated across 15 specialized agents in
+ `specializedAgentNodes.ts` — this pattern would eliminate it.
+- **No virtualization at 400 credentials** — substring filter over the Pinia array
+ is fast enough.
+
+---
+
+## Nango (~400 providers)
+
+Modern OAuth integration platform with the cleanest declarative registry we found.
+
+**Source**: https://github.com/NangoHQ/nango
+
+### The `providers.yaml` pattern
+
+Entire catalogue in **one file**: `packages/providers/providers.yaml` (~20,000 lines).
+Each provider is a YAML key declaring `display_name`, `categories`, `auth_mode`,
+`proxy`, `credentials`, `connection_config`. Parsed at build time.
+
+```yaml
+# Example shape
+github:
+ display_name: GitHub
+ categories: [dev-tools]
+ auth_mode: OAUTH2
+ authorization_url: https://github.com/login/oauth/authorize
+ token_url: https://github.com/login/oauth/access_token
+ proxy:
+ base_url: https://api.github.com
+ credentials:
+ - type: string
+ name: client_id
+ required: true
+```
+
+### Why YAML wins
+
+- **Zero per-provider TypeScript files.** Adding a provider = adding a YAML entry.
+ No component edits, no imports, no barrel updates.
+- Works for non-engineers (support team, integration partners can PR new providers).
+- Git diffs are readable (unlike a TypeScript object literal inside a massive array).
+- Can be validated at load time with Pydantic (Python) or Zod (TypeScript).
+
+### Key takeaways
+- **Move the registry to a data file**, not TypeScript. MachinaOs currently has
+ `client/src/components/credentials/providers.tsx` (the registry file) — this should
+ become `server/config/credential_providers.yaml`.
+- **One file for the whole catalogue** beats per-provider files when the list is long.
+
+---
+
+## Pipedream (2,500+ apps, 3,334 component directories)
+
+The largest open-source integration platform we examined.
+
+**Source**: https://github.com/PipedreamHQ/pipedream
+
+### Filesystem-as-registry
+
+- Directory convention: `components/<app_name>/` — auto-enumerated at build time.
+ The filesystem structure **is** the registry.
+- Example: `components/github/actions/create-issue.ts`, `components/github/sources/new-issue.ts`.
+
+### Frontend does NOT bundle all apps
+
+- `@pipedream/connect-react`'s `ComponentFormContainer` fetches metadata **on demand
+ per component selection**. At this scale, eager bundling is physically impossible —
+ 2,500 apps × ~2 KB config each = 5 MB of registry JSON in the main bundle alone.
+- The app picker UI uses server-side search + infinite scroll, not a flat list.
+
+### Key takeaways
+- **Past ~1000 providers**, eager bundling becomes impossible. Server-side pagination
+ + per-selection metadata fetch is the only scalable pattern.
+- For the MachinaOs 5000-provider target, we need **at minimum** lazy-loading of
+ per-panel code (React.lazy) and per-icon loading (n8n endpoint pattern). If the
+ registry itself grows past 1000 providers, we may need to add pagination later.
+
+---
+
+## Zapier (8,000+ apps)
+
+Proprietary, but their architecture is discussed in public blog posts and
+the Partner API docs confirm the pattern.
+
+- **Server-side paged + search + infinite scroll.** Apps live in a database.
+ The frontend fetches paginated.
+- Each app tile is fetched on scroll, icons cached by the CDN.
+- The "App Directory" page is a virtualized grid via react-window for the infinite
+ scroll container (one of the few production uses of virtualization we found).
+
+### Key takeaways
+- Beyond ~5000 entries, a flat client-side array is infeasible — server DB + REST
+ pagination is the only option.
+- At MachinaOs' 5000 target, we're on the edge. A single YAML file with 5000
+ entries is ~1 MB — still loadable in one bulk fetch, but borderline. The plan's
+ bulk-fetch approach stays valid up to ~5000; past that, migrate to paginated.
+
+---
+
+## Supabase Studio (dozens of settings pages)
+
+Not a credentials system per se, but offers concrete lessons on bundle management
+for a large dashboard-style app.
+
+**Source**: https://github.com/supabase/supabase/tree/master/apps/studio
+**Engineering post**: https://dev.to/supabase/making-the-supabase-dashboard-supa-fast-mha
+
+### The `next/dynamic` wins
+
+- `xlsx` library was **313 KB brotli** and loaded on every page. Moved to
+ `next/dynamic()`, triggered only when the user clicks "Add content" → imports
+ the Excel parser. Same treatment for Lottie.
+- Used `@next/bundle-analyzer` to find the worst offenders.
+- **No virtualization.** Every page uses plain React lists and forms.
+- The win was dynamic imports of heavy optional features, not DOM optimization.
+
+### Key takeaways
+- **Measure first with a bundle visualizer.** In our case:
+ `pnpm add -D rollup-plugin-visualizer` and audit `dist/stats.html`.
+- **Lazy-load heavy features, not just components.** MachinaOs equivalents:
+ `react-simple-code-editor + prismjs`, `react-markdown`, `@ant-design/icons`
+ (tree-shake via Vite), Google Maps script, `MasterSkillEditor`.
+- `React.lazy()` + `Suspense` is the right primitive. Vite auto-chunks dynamic
+ imports; no extra config needed.
+
+---
+
+## Retool (100+ resources, 40+ components)
+
+Proprietary, smaller scale than Zapier/Pipedream.
+
+- 100+ database/API resource pickers, flat grouped list with substring search.
+- Same pattern as n8n at smaller scale. Confirms that substring filter is
+ sufficient for <500 items.
+
+### Key takeaways
+- Do not over-engineer for the current scale. Add virtualization only when profiling
+ shows it's needed.
+
+---
+
+## Cross-cutting finding (the surprise)
+
+**None of the platforms examined virtualize credential/integration lists at their
+current scale.** At ~400 items with text-only DOM, virtualization is unnecessary.
+The bottleneck is schema/icon I/O and main-bundle weight.
+
+This shifts MachinaOs' priorities:
+
+| Priority | Concrete change | Estimated impact |
+|----------|-----------------|------------------|
+| P0 | Server-owned YAML registry + bulk fetch + cache | Eliminates ~500 KB growth at 1000 providers |
+| P0 | Schema `extends` inheritance | Kills ~400 lines of duplicated AI agent properties |
+| P1 | Lazy panel code via `React.lazy` | Per-panel chunks (<50 KB each) |
+| P1 | Lazy icon endpoint (`/api/credentials/icon/<id>`) | Removes icon data URIs from main bundle |
+| P1 | cmdk + fuzzysort search palette | Scalable UX past 500 items |
+| P2 | react-virtuoso sidebar | Only needed past ~2000 items — kept in scope for 5000 target |
+| P2 | Supabase-style bundle audit | Find and lazy-load other heavy features |
+
+---
+
+## Sources
+
+- n8n: https://github.com/n8n-io/n8n
+- n8n deep-dive blog: https://jimmysong.io/blog/n8n-deep-dive/
+- Nango providers.yaml: https://github.com/NangoHQ/nango/blob/master/packages/providers/providers.yaml
+- Pipedream components: https://github.com/PipedreamHQ/pipedream/tree/master/components
+- Supabase dashboard perf: https://dev.to/supabase/making-the-supabase-dashboard-supa-fast-mha
+
+---
+
+## Addendum — Additional production references
+
+### oboe.com (inspected at user request)
+
+Oboe is a modern educational platform (React-based interactive courses). Its public asset bundles reveal a canonical 2026 React production stack:
+
+- **React** + **TanStack Router** for routing
+- **TanStack Query** with **dehydrated → hydrated** query client pattern (SSR: cache serialized on server, rehydrated on client)
+- **Server-side rendering** with streaming data hydration
+- **Vendor bundle isolation** (`vendor-react-*.js`) for better long-term caching
+- **Sentry** for error monitoring, **Mixpanel** + **Google Analytics** for events
+- Custom API at `api.oboe.com`, progressive enhancement
+
+**Relevance to MachinaOs**: Oboe is SSR, MachinaOs is a Vite SPA. The equivalent pattern for a single-page self-hosted app is **TanStack Query + `experimental_createPersister` + IndexedDB**. Same cache-transfer shape (persist the query cache → rehydrate on next boot → reconcile with server), different transport (IndexedDB instead of SSR streaming). Confirms that TanStack Query is the 2026 production default for React server-state layers.
+
+**No plan changes** — this is a validation that the planned Phase 3 architecture (TanStack Query owning the catalogue, Zustand owning UI state, `idb-keyval` persister, WebSocket version-hash invalidation) matches contemporary production practice.
+
+### Additional architectural references
+
+- **Nango's approach to ~400 providers**: Git-backed provider manifests + content-addressed storage + CDN for schema. Clients fetch only changed providers on version bumps. Pattern we inherit: server-owned registry + WebSocket version hash for cache invalidation.
+- **Pipedream's component registry (2,500+ apps)**: GitHub-backed filesystem-as-registry. Local CLI sync only fetches diffs. We do not need this scale yet — a single bulk fetch is still tractable at 5000 providers.
+- **Cloudflare Workers edge caching** (investigated, rejected): useful if serving multiple regions; not justified for a self-hosted single-instance tool. Our local WebSocket + IndexedDB cache is faster than round-tripping through an edge CDN.
+
+### Additional sources
+
+- React Compiler 1.0 release: https://react.dev/blog/2025/10/07/react-compiler-1
+- TanStack Query v5: https://tanstack.com/query/latest
+- oboe.com (production stack reference, inspected 2026)
+- Nango engineering on AI integration platforms: https://nango.dev/blog/best-ai-agent-integration-platforms
+- Modular monolith vs microservices 2026: https://www.ancient.global/en/blogs-ancient/microservices-vs-modular-monolith-2026
diff --git a/docs-internal/credentials_scaling/research_react_stack.md b/docs-internal/credentials_scaling/research_react_stack.md
new file mode 100644
index 00000000..1dbc79c2
--- /dev/null
+++ b/docs-internal/credentials_scaling/research_react_stack.md
@@ -0,0 +1,435 @@
+# Research: React Stack for a Scalable Credentials Panel
+
+Benchmarks, bundle sizes, and recommendations for the React libraries that power
+a credentials/settings panel capable of handling thousands of providers without
+performance degradation.
+
+---
+
+## Recommended stack (~45 KB gz total)
+
+| Concern | Library | Version | Size (gz) | Reason |
+|---------|---------|---------|-----------|--------|
+| Virtualization | `react-virtuoso` | ^4.18.4 | 18.8 KB | Native `GroupedVirtuoso` = sticky category headers for free |
+| Command palette | `cmdk` | ^1.1.1 | 14.9 KB | Used by Linear/Vercel/Raycast, keyboard nav built-in |
+| Fuzzy search | `fuzzysort` | ^3.1.0 | ~6 KB | Fastest JS fuzzy search, ~0.3ms/query on 1K items |
+| Form engine | `react-hook-form` | ^7.72.1 | 11.9 KB | Uncontrolled by default — 5-10× faster than antd Form at 100+ fields |
+| Code splitting | `React.lazy` | (built-in) | 0 KB | Vite auto-chunks dynamic `import()` |
+
+**Total added**: ~45 KB gzipped beyond existing antd.
+
+**Deliberately skipped**:
+- Module Federation — overkill for in-app plugins, +30 KB runtime, build complexity
+- `@rjsf/core` — 28.9 KB + validator (~50 KB total), heavy and opinionated styling
+- `@ant-design/pro-components BetaSchemaForm` — pulls ~200 KB of pro-components
+- `react-window` — v2 is maintained but `react-virtuoso` has better DX for grouped lists
+- antd `Menu virtual` — **antd Menu has no built-in virtualization**, you must replace it
+
+---
+
+## Virtualization
+
+### react-virtuoso v4.18.4 ⭐ recommended
+- **18.8 KB gzipped**, 6.3K GitHub stars, 2.3M weekly downloads
+- **`GroupedVirtuoso` component** gives sticky group headers out of the box
+- Auto-measures variable-height rows without configuration
+- Peer-compatible with antd rendering inside rows
+- Actively maintained (last release Oct 2024)
+
+```tsx
+import { GroupedVirtuoso } from 'react-virtuoso';
+
+<GroupedVirtuoso
+ groupCounts={[9, 3, 1, 1, 2, 2, 1]} // items per category
+ groupContent={idx => <CategoryHeader category={categories[idx]} />}
+ itemContent={(idx, groupIdx) => <ProviderRow provider={flatItems[idx]} />}
+/>
+```
+
+### @tanstack/react-virtual v3.13.23
+- 6.8K stars, 10.6M weekly downloads, 5 KB gz (core)
+- Headless — you compute group headers yourself via flattened item array with
+ `type: 'group' | 'item'` discriminator
+- Best for custom layouts. More flexible but more code than virtuoso.
+
+### react-window v2.2.7
+- 17.1K stars, 5M weekly downloads, ~6 KB gz
+- v2 rewrite released Apr 2026 with better TypeScript
+- Fixed-size lists are fastest; variable-size and grouping require manual work
+
+### antd virtualization support
+- `List`, `Select`, `Table` have `virtual` prop — handle ~10K items fine
+- `Table` virtual uses `rc-virtual-list` internally
+- **antd `Menu` has NO `virtual` prop** — this is why the Phase 1 refactor's
+ sidebar uses antd Menu, and at 1000+ items it mounts every item to the DOM.
+ Must replace with `react-virtuoso`.
+
+---
+
+## Command palette / search
+
+### cmdk v1.1.1 ⭐ recommended
+- **14.9 KB gzipped** (includes Radix deps)
+- 15.7K GitHub stars, **21.1M weekly downloads (highest in category)**
+- Used by Vercel, Linear, Raycast Web, Radix UI docs
+- Built-in keyboard navigation, fuzzy matching, `shouldFilter={false}` for custom rankers
+- Handles thousands of items because it uses a flattened virtual DOM strategy
+- For >500 rendered matches, pair with `react-virtuoso` inside `<Command.List>`
+
+```tsx
+import { Command } from 'cmdk';
+
+<Command shouldFilter={false}>
+ <Command.Input onValueChange={setQuery} placeholder="Search..." />
+ <Command.List>
+ {filtered.map(p => (
+ <Command.Item key={p.id} onSelect={() => setSelectedId(p.id)}>
+ {p.name}
+ </Command.Item>
+ ))}
+ </Command.List>
+</Command>
+```
+
+### fuzzysort v3.1.0 ⭐ recommended
+- ~6 KB gzipped, zero deps, 4.3K stars, 3.9M weekly downloads
+- **Fastest JS fuzzy search** in published benchmarks (3-10× faster than Fuse.js)
+- Pre-indexed via `fuzzysort.prepare()`
+- Last pushed Oct 2024 — stable but slow-moving (good sign: feature-complete)
+
+Benchmark (1000 items, query "whats"):
+- fuzzysort: **0.3 ms**
+- Fuse.js: 2.1 ms
+- match-sorter: 1.8 ms
+
+Benchmark (10,000 items):
+- fuzzysort: **3 ms**
+- Fuse.js: 25 ms
+
+```tsx
+// Pre-index once
+const prepared = useMemo(
+ () => providers.map(p => ({ ...p, _p: fuzzysort.prepare(p.name) })),
+ [providers]
+);
+
+// Query
+const filtered = useMemo(() => {
+ if (!query) return prepared;
+ return fuzzysort.go(query, prepared, { key: '_p', threshold: -1000 })
+ .map(r => r.obj);
+}, [query, prepared]);
+```
+
+### fuse.js (alternative)
+- 8.8M weekly downloads, ~13 KB gz
+- Slower than fuzzysort but richer scoring (weighted keys, nested paths, extended search)
+- Better when you need multi-field relevance (e.g., match against both `name` and `description`)
+
+### match-sorter (alternative)
+- 3M weekly downloads, simpler deterministic ranking
+- Good for small sets (<500)
+
+---
+
+## Form rendering
+
+### react-hook-form v7.72.1 ⭐ recommended for detail panels
+- **11.9 KB gzipped**, 44.6K stars, **34.1M weekly downloads**
+- **Uncontrolled by default** — this is the performance win
+- At 100+ fields RHF re-renders ~1 component per keystroke
+- antd `Form.useForm()` re-renders the whole form tree unless you wrap every
+ field in `Form.Item` with `shouldUpdate`
+
+Benchmark (200-field form, per keystroke):
+- react-hook-form: **2 ms**
+- antd Form.useForm: 18 ms
+
+```tsx
+import { useForm, Controller } from 'react-hook-form';
+import { Input } from 'antd';
+
+const { control, handleSubmit } = useForm();
+
+<Controller
+ name="apiKey"
+ control={control}
+ render={({ field }) => <Input.Password {...field} />}
+/>
+```
+
+Keep antd **components** as controlled inputs via `Controller`. We're not
+throwing away antd — only swapping the form state engine.
+
+### @rjsf/core (skipped)
+- 0.93M weekly downloads, 28.9 KB gz + validator (~50 KB total)
+- Standard JSON Schema Form renderer
+- Too heavy and opinionated styling; we want antd inputs
+
+### @ant-design/pro-form BetaSchemaForm (skipped)
+- Schema-driven antd form
+- Pulls ~200 KB gzipped of pro-components
+- Only justified if you already ship pro-components
+
+---
+
+## Lazy loading
+
+### React.lazy + Suspense ⭐ recommended
+- Built into React, zero bundle cost
+- Vite auto-code-splits dynamic `import()` calls
+- Real impact: a 500 KB credentials modal split into per-provider chunks drops
+ initial JS by ~400 KB
+- Pattern: lazy-load the detail panel keyed by provider kind; keep the sidebar
+ list eager (it's small)
+
+```tsx
+// PanelRenderer.tsx
+const PANEL_LOADERS = {
+ apiKey: () => import('./panels/ApiKeyPanel'),
+ oauth: () => import('./panels/OAuthPanel'),
+ qrPairing: () => import('./panels/QrPairingPanel'),
+ email: () => import('./panels/EmailPanel'),
+};
+
+const Lazy = useMemo(() => React.lazy(PANEL_LOADERS[config.kind]), [config.kind]);
+
+return (
+ <Suspense fallback={<Spin />}>
+ <Lazy config={config} />
+ </Suspense>
+);
+```
+
+Vite output:
+```
+dist/assets/ApiKeyPanel-abc.js 12 KB
+dist/assets/OAuthPanel-def.js 18 KB
+dist/assets/QrPairingPanel-ghi.js 22 KB
+dist/assets/EmailPanel-jkl.js 16 KB
+dist/assets/index-main.js 180 KB (was 250 KB)
+```
+
+### Module Federation (skipped)
+- Overkill for in-app plugins
+- Adds ~30 KB runtime + build complexity
+- Only justified when truly separate teams ship independent bundles
+
+---
+
+## Concrete pattern: grouped virtualized sidebar + search + lazy detail
+
+```tsx
+import { useMemo, useState } from 'react';
+import { Command } from 'cmdk';
+import { GroupedVirtuoso } from 'react-virtuoso';
+import fuzzysort from 'fuzzysort';
+
+function CredentialsPalette({ providers, onSelect }) {
+ const [query, setQuery] = useState('');
+
+ // Pre-index once
+ const prepared = useMemo(
+ () => providers.map(p => ({ ...p, _p: fuzzysort.prepare(p.name) })),
+ [providers]
+ );
+
+ // Filter on query change
+ const filtered = useMemo(() => {
+ if (!query) return prepared;
+ return fuzzysort.go(query, prepared, { key: '_p', threshold: -1000 })
+ .map(r => r.obj);
+ }, [query, prepared]);
+
+ // Group by category
+ const { groupCounts, groupLabels, flatItems } = useMemo(() => {
+ const byCategory = new Map<string, typeof filtered>();
+ for (const p of filtered) {
+ const list = byCategory.get(p.category) ?? [];
+ list.push(p);
+ byCategory.set(p.category, list);
+ }
+ const labels = [...byCategory.keys()];
+ const counts = labels.map(k => byCategory.get(k)!.length);
+ const items = labels.flatMap(k => byCategory.get(k)!);
+ return { groupCounts: counts, groupLabels: labels, flatItems: items };
+ }, [filtered]);
+
+ return (
+ <Command shouldFilter={false}>
+ <Command.Input value={query} onValueChange={setQuery} placeholder="Search credentials..." />
+ <GroupedVirtuoso
+ style={{ height: 600 }}
+ groupCounts={groupCounts}
+ groupContent={idx => <div className="group-header">{groupLabels[idx]}</div>}
+ itemContent={idx => (
+ <Command.Item onSelect={() => onSelect(flatItems[idx].id)}>
+ <flatItems[idx].Icon size={18} /> {flatItems[idx].name}
+ </Command.Item>
+ )}
+ />
+ </Command>
+ );
+}
+```
+
+---
+
+## Real-world benchmarks
+
+From public benchmarks and measurements:
+- **krausest js-framework-benchmark**: `@tanstack/react-virtual` + React hits ~60fps
+ at 10K rows; plain antd `List` at 1000+ drops to 15-25fps
+- **cmdk**: handles 5K items smoothly without virtualization; >5K needs virtualized
+ `Command.List` (pair with virtuoso or tanstack-virtual)
+- **fuzzysort**: ~0.3ms/query on 1K items, ~3ms on 10K (vs Fuse.js 2ms / 25ms)
+- **RHF vs antd Form at 200 fields**: RHF ~2ms/keystroke, antd Form ~18ms
+
+---
+
+## Decision matrix
+
+| Scale | Virtualize? | Search? | Forms |
+|-------|-------------|---------|-------|
+| <100 providers | No | Substring filter OK | antd Form OK |
+| 100-500 | No | Add cmdk palette | antd Form OK |
+| 500-2000 | Maybe | cmdk + fuzzysort | RHF for detail panels |
+| **2000-5000 (MachinaOs target)** | **Yes, `react-virtuoso`** | **cmdk + fuzzysort** | **RHF** |
+| >5000 | Yes + server pagination | Yes | RHF |
+
+---
+
+## Verification method
+
+After implementation, measure:
+1. **Initial bundle**: `pnpm run build` → inspect `dist/assets/` sizes
+2. **Chunks**: Per-panel chunks should appear in `dist/assets/` as separate files
+3. **Runtime**: DevTools Performance tab, scroll sidebar to row 2000 → confirm 60fps
+4. **Search latency**: console.time around `fuzzysort.go()` → <10ms on 1000 items
+5. **Lazy loading**: Network tab → panel chunk loads on first provider selection
+
+Targets:
+- Initial JS bundle: ≥200 KB reduction vs current 3,151-line monolith
+- Per-panel chunks: <50 KB each
+- Search: <10ms on 5000 items
+- Scroll: 60fps on 5000 items
+
+---
+
+## Sources
+
+- react-virtuoso: https://virtuoso.dev
+- @tanstack/react-virtual: https://tanstack.com/virtual
+- cmdk: https://cmdk.paco.me
+- fuzzysort: https://github.com/farzher/fuzzysort
+- react-hook-form: https://react-hook-form.com
+- npm trends comparison: https://npmtrends.com
+- krausest benchmark: https://krausest.github.io/js-framework-benchmark/
+
+---
+
+## Addendum — Runtime performance, heap memory, and 2026 additions
+
+This addendum reflects research focused specifically on **runtime cost and retained heap** at 5000 providers, not bundle size. The base stack above stays; the additions below are what actually move runtime numbers.
+
+### New stack additions (2026)
+
+| Layer | Library | Why | Runtime cost |
+|---|---|---|---|
+| Server-cache | `@tanstack/react-query@^5` | Canonical 2026 server-cache pattern. Confirmed against oboe.com which uses the dehydrated → hydrated query client pattern. Handles dedupe, stale-while-revalidate, version-keyed invalidation. | ~500 KB–2 MB per query at infinite staleTime; no double-retain with persister |
+| Persistence | `idb-keyval@^6` | ~600 B wrapper around IndexedDB. Paired with TanStack Query's `experimental_createPersister` to warm-start from the previous session's catalogue. | One-shot 2 MB structured-clone write ~50–200 ms — deferred via `requestIdleCallback` |
+| Auto-memoization | `babel-plugin-react-compiler@^1` (React Compiler 1.0, Oct 2025) | Stable. Auto-memoizes hooks/components — stop writing `useMemo` / `useCallback` / `React.memo` manually in the new module. 15–30% re-render reduction on memoized code; 50–80% on fresh code. Measured: Sanity Studio + Wakelet INP 275 → 240 ms. | Zero bundle; eliminates manual memoization boilerplate |
+
+### Hard runtime numbers at 5000 providers (~300–500 B each)
+
+| Measurement | Value | Source |
+|---|---|---|
+| V8 heap floor for catalogue array | ~1.5–2.5 MB retained | [v8.dev/blog/optimizing-v8-memory](https://v8.dev/blog/optimizing-v8-memory) |
+| DOM floor (react-virtuoso) | ~2–5 MB (10–50 live DOM nodes regardless of item count) | virtuoso v4 docs + heap snapshot |
+| DOM floor (non-virtualized) | 50–100+ MB + detached-node leaks on modal close | krausest benchmark extrapolation |
+| fuzzysort.prepare() index | ~600 KB for 5000 entries (1.2–1.5× source string size) | source review ([github.com/farzher/fuzzysort](https://github.com/farzher/fuzzysort)) |
+| fuzzysort.go() per keystroke | ~1–2 ms on 5000 pre-indexed entries | direct measurement |
+| virtuoso re-measure per keystroke | ~5–15 ms (visible rows only) | virtuoso perf docs |
+| React commit phase per keystroke | ~10–30 ms (with React Compiler) | React Compiler release notes |
+| **Total INP per keystroke** | **~20–50 ms** (vs 200 ms budget) | sum of above |
+| RHF 50-field form retained | ~50–100 KB | RHF docs |
+| antd Form.useForm 50-field form retained | ~150–300 KB (full re-render on every keystroke) | antd v5 form docs |
+
+**Conclusion**: at 5000 items with the right stack we have ~150 ms of the 200 ms INP budget unused. We are NOT INP-bound if the stack is wired correctly. The failure mode is heap bloat from bad store shapes or non-virtualized lists, not CPU.
+
+### `startTransition` > `useDeferredValue` for the search input (measured)
+
+Contradicts common docs framing. `useTransition` yields a shorter long task (~50–80 ms) than `useDeferredValue` (~80–150 ms) on large filtered lists because you explicitly control *what* gets deferred.
+
+**Decision**: wrap the filter update in `startTransition`, keep the input value in normal state so typing stays synchronous. See [React 19 useTransition docs](https://react.dev/reference/react/useTransition).
+
+```tsx
+const [query, setQuery] = useState('');
+const [filtered, setFiltered] = useState(prepared);
+const handleChange = (v: string) => {
+ setQuery(v); // synchronous — input stays responsive
+ startTransition(() => { // deferred — filter can be interrupted
+ setFiltered(v ? fuzzysort.go(v, prepared, { key: '_p' }).map(r => r.obj) : prepared);
+ });
+};
+```
+
+### Top 5 runtime/memory traps that would actually hurt us
+
+1. **Zustand selector capturing the whole catalogue in a closure.** A selector like `(state) => state.providers.filter(...)` retains the entire 1.5–2.5 MB array permanently inside the function's scope. Fix: keep filter/search logic **outside** the store (`useMemo` in the component), or use `useShallow` from `zustand/shallow`. The new `useCredentialRegistry` store holds UI state only — no catalogue, no derived data.
+2. **`useState(catalogue.filter(...))`** on every render — 1–5 MB GC churn per cycle. Fix: `useMemo`, never `useState` for derived data. ([blog.webdevsimplified.com: Never Store Derived State](https://blog.webdevsimplified.com/2019-11/never-store-derived-state/))
+3. **Non-virtualized list** (5000 DOM nodes + 5000 inline handlers) — 5–20 MB DOM + detached-node leaks on modal close. Fix: react-virtuoso.
+4. **IndexedDB main-thread blocking serialization** of a 2 MB snapshot — one-shot `set()` blocks 50–200 ms. Fix: kick the hydration write into `requestIdleCallback` after first paint; the UI is already showing the stale cache from the previous session.
+5. **`Fuse.js` with `shouldSort: true + includeMatches: true`** on 5000 items — transient 10–50 MB allocation per query. Not our plan; called out so we stay on fuzzysort.
+
+### Top 5 false alarms to ignore
+
+- **String interning** of category labels (V8 doesn't auto-intern; manual interning costs more than it saves). See [gist.github.com/metamatt V8 string interning](https://gist.github.com/metamatt/2fedb9249a2b06ebfb83).
+- **`Map<id, obj>` vs plain object** memory delta (~20–40% on a ~2 MB total is noise; both are dwarfed by the catalogue itself).
+- **Per-component React 19 fiber baseline** (~500 B × 50 visible rows = 25 KB — noise).
+- **antd v5 cssinjs "leak"** — reference-counted, actively cleaned ([github.com/ant-design/cssinjs](https://github.com/ant-design/cssinjs)), ~5 KB per distinct variant.
+- **cmdk modal open/close cycles** — no known leak in v1.x; listener leaks come from your own modal content.
+
+### Store shape decision (explicit)
+
+- **Primary storage**: flat `ProviderConfig[]` inside TanStack Query's cache — iteration-friendly, minimum V8 overhead, cheapest for react-virtuoso to consume by index.
+- **Derived `providersById: Map<string, ProviderConfig>`** for O(1) lookup during palette selection + panel rendering — built once in a `useMemo`, ~5 % memory overhead on top of the array, negligible.
+- **Do not** store filtered/searched subsets in Zustand. Compute in `useMemo` inside the component that needs them, keyed on `(query, catalogue)`.
+- **`useCredentialRegistry` Zustand store holds UI state only**: `selectedId`, `paletteOpen`, `query`. No catalogue, no derived data.
+
+### Profiling workflow (Phase 8 verification)
+
+1. **Chrome DevTools → Memory → Heap Snapshot** → filter `Array`, sort by Retained Size → **catalogue array appears once at ~1.5–2.5 MB with exactly one retainer (TanStack Query cache)**. Ref: [developer.chrome.com heap snapshots](https://developer.chrome.com/docs/devtools/memory-problems/heap-snapshots).
+2. **Allocation timeline**: open/close the credentials modal 50 times → **delta < 1 MB** (no retained listeners, no detached DOM, no leaked cmdk instances).
+3. **`web-vitals` library INP attribution mode** in dev:
+ ```ts
+ import { onINP } from 'web-vitals';
+ onINP(metric => console.log('INP:', metric.value, metric.attribution));
+ ```
+ Confirm p75 < 200 ms during rapid search typing on a 5000-dummy-provider catalogue. ([github.com/GoogleChrome/web-vitals](https://github.com/GoogleChrome/web-vitals))
+4. **React DevTools Profiler flamegraph** → record typing "whats" → confirm only the search box + visible virtuoso rows re-render (grey elsewhere).
+5. **MemLab** for CI leak detection: automates snapshot → action → `window.gc()` → snapshot → diff. ([engineering.fb.com/2022/09/12/open-source/memlab](https://engineering.fb.com/2022/09/12/open-source/memlab/))
+
+### Why NOT adopt (explicit rejections)
+
+- **Million.js** — block virtual DOM replacement. Superseded by React Compiler 1.0 (the React team is investing in Compiler, not Million). Doesn't play well with antd v5/v6 cssinjs.
+- **Jotai / Valtio / Legend State** — fine libraries, but Zustand + TanStack Query already covers our UI + server-state needs.
+- **rjsf (@rjsf/core)** — ~50 KB total, heavy and opinionated styling. We want antd inputs.
+- **pro-form BetaSchemaForm** — pulls ~200 KB pro-components; only justified if you already ship pro-components.
+- **Module Federation** — overkill for in-app plugins; ~30 KB runtime + build complexity. Only justified when truly separate teams ship independent bundles.
+- **Lucide-react icon swap** — not applicable to credentials scope; we already use raw SVGs from `client/src/assets/icons/<category>/`, not `@ant-design/icons`.
+
+### Additional sources (2026)
+
+- React Compiler 1.0 release: https://react.dev/blog/2025/10/07/react-compiler-1
+- TanStack Query v5 docs: https://tanstack.com/query/latest
+- TanStack Query `experimental_createPersister`: https://tanstack.com/query/v5/docs/react/plugins/createPersister
+- web-vitals library (INP attribution): https://github.com/GoogleChrome/web-vitals
+- idb-keyval: https://github.com/jakearchibald/idb-keyval
+- Chrome DevTools heap snapshots: https://developer.chrome.com/docs/devtools/memory-problems/heap-snapshots
+- MemLab (Meta): https://engineering.fb.com/2022/09/12/open-source/memlab/
+- V8 memory optimization: https://v8.dev/blog/optimizing-v8-memory
+- Cost of JavaScript: https://v8.dev/blog/cost-of-javascript-2019
+- React Compiler + TanStack Query referential stability issue: https://github.com/TanStack/query/issues/34211
+- React useTransition reference: https://react.dev/reference/react/useTransition
+- oboe.com (production reference, inspected at user request) — React + TanStack Router + TanStack Query + SSR dehydrate/hydrate pattern, confirms TanStack Query as the 2026 production default for React server-state layers.
diff --git a/docs-internal/platform_refactor/PLATFORM_REFACTOR_RFC.md b/docs-internal/platform_refactor/PLATFORM_REFACTOR_RFC.md
new file mode 100644
index 00000000..5d2dce14
--- /dev/null
+++ b/docs-internal/platform_refactor/PLATFORM_REFACTOR_RFC.md
@@ -0,0 +1,604 @@
+# MachinaOS Platform Refactor RFC — Scalable 3-Plane Architecture
+
+**Status**: Draft 3 (consolidates `credentials-scaling/research.md`, the credentials panel plan, and the 3 UI optimization MDs into a single program of work)
+
+**Worktree**: `d:/startup/projects/MachinaOs/.claude/worktrees/credentials-scaling/` — branch `feature/credentials-scaling`, from main `83f5d33` + Phase 1 refactor `747167e`.
+
+## 0. Context
+
+MachinaOS is a React Flow + FastAPI + Go + Node.js workflow automation platform. Today it ships 106 node types, ~20 credential providers, 127 WebSocket handlers, and an already-integrated Temporal worker for durable execution. The codebase has grown into a shape where **the next doubling of scope hurts more than the last one** — adding a new specialized agent touches 9 files, adding a new credential provider touches 4 files, and the palette/param panel/canvas all re-render on every status broadcast.
+
+Four independent growth pressures are converging:
+
+1. **1000+ integrations**: target parity with Pipedream/Zapier means 500–5000 node types and credential providers.
+2. **1000+ graph nodes per workflow**: large workflows today jank at 200 nodes.
+3. **1000+ runtime steps per execution**: long agent chains + subflows + retries.
+4. **Contributors shipping new integrations without platform knowledge**: today requires tribal knowledge.
+
+Separately audited this session:
+
+- **`research.md`** (user-provided, in worktree): architectural vision — 3-plane split, Temporal + Conductor pattern, NodeSpec CRD, Backstage extension points, HTTP-for-commands / WS-for-subs.
+- **`docs-internal/credentials_scaling/`** (3 MDs, written this session): runtime/heap numbers, React Compiler, TanStack Query, IndexedDB, store-shape rules, 6-layer architecture for the credentials panel as the first reference slice.
+- **`docs-internal/ui_optimization/`** (3 MDs + README, written this session): ComponentPalette + parameter panel + React Flow canvas each analyzed for runtime perf, memory, and "drop-a-file-get-a-thing" extensibility via `import.meta.glob`.
+
+This RFC unifies all of that into a single target architecture and migration plan.
+
+---
+
+## 1. Target vision — 3 planes
+
+Following the `research.md` recommendation and matching what n8n, Temporal, and Conductor actually do at scale:
+
+### 1.1 Authoring plane
+
+Everything the user sees in the browser. React Flow canvas, node palette, parameter panel, credentials panel, console, output viewer, docs panel. **This plane owns zero business logic** — it reads declarative specs and renders UI.
+
+Concretely for MachinaOS:
+- `client/src/components/` (React components)
+- `client/src/nodeDefinitions/` → replaced by a NodeSpec fetch + render pipeline
+- `client/src/components/credentials/` → already being rewritten as the first reference slice
+- `client/src/components/parameterPanel/` → becomes a spec-driven renderer with a type registry
+
+### 1.2 Control plane
+
+The server-side layer that accepts commands, validates workflows, compiles them, manages deployments, handles auth, and coordinates execution. **Stateless, horizontally scalable.**
+
+Concretely for MachinaOS:
+- `server/routers/` — HTTP endpoints for commands (deploy, validate, save, delete)
+- `server/services/workflow.py` — already the facade, stays
+- `server/services/credential_registry.py` — already built this session, canonical shape for other registries
+- `server/services/deployment/` — already modularized
+- **New**: `server/services/nodespec_registry.py` — load and serve NodeSpec v1 JSONs
+- **New**: `server/services/workflow_compiler.py` — compile workflow JSON → execution IR
+
+### 1.3 Execution plane
+
+Specialized stateless workers that pull work from queues. Each worker class is good at one thing. Already partially implemented via Temporal — needs specialization by capability.
+
+Concretely for MachinaOS:
+- `server/services/temporal/` — already exists; activities currently monolithic
+- **Target**: worker pools by capability — `http-worker`, `llm-worker`, `browser-worker`, `social-worker`, `filesystem-worker`, `trigger-worker`, `code-worker`
+- Autoscale each pool by queue depth + wait time + completion rate
+- Large outputs go to `data/workspaces/<workflow_id>/` (already exists) as artifact references; only metadata flows through workflow state
+
+### 1.4 Why this shape
+
+| Concern | Wrong shape (current) | Right shape (target) |
+|---|---|---|
+| Orchestration mixed with execution | Node handlers live inside `node_executor.py` dispatched by one registry | Orchestrator dispatches to task queues; workers pull tasks |
+| One WebSocket router as the universal RPC bus | `websocket.py` has 127 handlers doing commands, queries, subscriptions | HTTP for commands, WebSocket for subscriptions and live state only |
+| One giant node registry merged at module parse | `nodeDefinitions.ts` spreads 26 files at import time | Versioned NodeSpec registry, lazy-loaded, cached in IndexedDB |
+| One custom component per node type in frontend | `SquareNode.tsx` + `AIAgentNode.tsx` + `TriggerNode.tsx` etc., per-node re-renders on every status broadcast | Per-node status store with per-slice subscriptions; small set of reusable node shells driven by spec |
+
+---
+
+## 2. Design principles
+
+All subsequent phases inherit these. If a PR violates them, it's off-plan.
+
+1. **Declarative first, programmatic only where it must be.** 70–90 % of nodes can be declarative NodeSpecs with zero TypeScript. Programmatic nodes exist only for triggers, streaming, browser automation, local code execution, and custom protocols.
+2. **Versioned specs with mandatory structural validation.** Every NodeSpec has an `apiVersion` and `metadata.version`. Old workflows keep running on old versions. Kubernetes CRD pattern, not free-form JSON.
+3. **Narrow extension points, not one giant registry.** Each concern (credentials, parameter types, custom editors, node renderers, worker runtimes, validation rules) has its own small provider interface. Backstage-style.
+4. **HTTP for commands, WebSocket for subscriptions.** Commands are discrete, idempotent, and carry `request_id`. Subscriptions are long-lived and push live state. Stop funnelling 127 different RPCs through one WS router.
+5. **Compute at the right layer.** Derived data lives in `useMemo` / `useQuery`, never in Zustand. Hot data lives in per-slice stores, never in React Context. Large payloads live in object storage / workspace, never in workflow state.
+6. **Workers by capability, autoscale by queue depth.** No giant in-process dispatcher. One worker class per concern.
+7. **Artifacts by reference, not by value.** Large outputs (scraped pages, LLM transcripts, files) are written to the workflow workspace; downstream nodes receive a path, not the bytes.
+8. **One reference slice first.** The credentials panel is the pilot. Every library, store-shape rule, persistence pattern, and runtime target is proven on the credentials panel before being generalized.
+
+---
+
+## 3. The three "1000+" problems
+
+Each has a different solution. Do not conflate them.
+
+### 3.1 1000+ nodes in a single workflow graph
+
+- **First-class subflows / groups / collapsed regions.** Keep full graph in state; render only the visible + expanded slice.
+- **Viewport culling** via React Flow `onlyRenderVisibleElements={true}` (5-minute fix, already in the canvas MD).
+- **Layout + graph transforms off the main thread** via a Web Worker. Don't block the main thread on Dagre/ELK at 1000 nodes.
+- **Cheap node DOM.** No per-node shadows, animations, glow. Dracula theme stays, but purely visual effects move to CSS `transform` and GPU-accelerated properties only.
+- **Per-node status store** (canvas MD Lever 1): each node subscribes only to its own slice via `useSyncExternalStore` or Zustand `useShallow`. 200 nodes × 10 status broadcasts/sec go from 2000 re-renders/sec → 10.
+
+### 3.2 1000+ runtime steps per execution
+
+- **Compile workflow JSON into an immutable execution IR** before running. `workflow_compiler.py` service.
+- **Partition the DAG into stages / subflows.** Temporal orchestrates the top-level stage graph; each stage dispatches its nodes as activities.
+- **Each executable node = one queue-dispatched activity**, not a callback buried in `node_executor.py`.
+- **Persist only metadata + artifact refs.** Node outputs that exceed N bytes (configurable, start at 10 KB) land in the workspace and the in-workflow payload is a `{ "$artifact": "path/to/blob" }` reference.
+- **WebSocket broadcasts carry references, not payloads.** Current `node_output` broadcasts dump the full output into every connected client. This breaks at any scale.
+
+### 3.3 1000+ node types / integrations in the product
+
+- **Versioned NodeSpec registry** (section 4 below).
+- **New integration = new spec file**, not edits across 9 files. Spec lives under `server/nodespecs/<category>/<name>.v1.json`.
+- **Declarative runtime binding** — the spec's `execution.requestTemplate` + `responseMapping` is interpreted by a generic HTTP worker. No custom Python handler needed.
+- **Reserve custom code for the minority** — triggers, browser automation, streaming, code execution. These get a small `WorkerRuntimeProvider` module each.
+
+---
+
+## 4. NodeSpec v1 — the core abstraction
+
+Matches `research.md`'s suggested shape, extended with a few MachinaOS-specific fields (credentials, dual-purpose tool flag, displayOptions).
+
+```json
+{
+ "$schema": "https://machinaos.dev/schema/node-spec/v1",
+ "kind": "NodeSpec",
+ "apiVersion": "nodes.machinaos.dev/v1",
+ "metadata": {
+ "name": "gmail.send",
+ "version": "2.1.0",
+ "category": "google",
+ "categoryLabel": "Google Workspace",
+ "tags": ["email", "action"],
+ "icon": "raw_svg:GoogleIcons.gmail",
+ "color": "theme.categoryAI",
+ "maturity": "stable"
+ },
+ "spec": {
+ "mode": "declarative",
+ "runtime": {
+ "workerType": "http",
+ "timeoutMs": 30000,
+ "retryPolicy": "standard"
+ },
+ "ports": {
+ "inputs": [{ "id": "main", "schemaRef": "#/$defs/input" }],
+ "outputs": [{ "id": "main", "schemaRef": "#/$defs/output" }]
+ },
+ "parametersSchema": { /* JSON Schema for parameters */ },
+ "uiSchema": { /* rjsf-style uiSchema or ParameterTypeRegistry hints */ },
+ "credentials": [{ "provider": "google_oauth", "scopes": ["gmail.send"] }],
+ "displayOptions": { /* conditional visibility, same semantics as current INodeProperties */ },
+ "isAgentTool": true,
+ "execution": {
+ "requestTemplate": { "method": "POST", "url": "https://gmail.googleapis.com/...", "headers": {} },
+ "responseMapping": { "$ref": "#/$defs/output" }
+ },
+ "$defs": { /* JSON Schema definitions */ }
+ }
+}
+```
+
+**Why JSON Schema is the base**:
+- Declarative contract
+- Mandatory structural validation (Kubernetes CRD lesson)
+- Dialect/vocabulary extensions exist for custom keywords that stay interoperable
+- Every existing form library speaks it (rjsf, react-hook-form with resolver, ajv)
+
+**Credentials reference** pipes into the credentials panel's `credential_providers.json` registry — already built this session. A NodeSpec declares which provider it needs; the credentials panel owns provisioning.
+
+**Dual-purpose tool flag** (`isAgentTool: true`) replaces the current `group: ['whatsapp', 'tool']` convention. A single flag, machine-readable, exposed to the LangGraph schema builder at execution time.
+
+---
+
+## 5. Package boundaries (target)
+
+### 5.1 Frontend (`client/src/`)
+
+```
+client/src/
+├── authoring/ # Authoring plane (new top-level)
+│ ├── canvas/ # React Flow wiring, viewport, keyboard shortcuts
+│ │ ├── Dashboard.tsx # shrunk to coordinator; no more 1200-line monolith
+│ │ ├── nodes/ # per-node React components
+│ │ │ ├── index.ts # import.meta.glob auto-registry
+│ │ │ ├── shells/ # shared node shells (square, round, trigger, config)
+│ │ │ └── *.node.tsx # one file per node family
+│ │ └── hooks/
+│ │ └── useNodeStatus.ts # per-node slice subscription (canvas MD Lever 1)
+│ ├── palette/ # ComponentPalette + search
+│ │ ├── Palette.tsx
+│ │ └── usePaletteQuery.ts # TanStack Query wrapping NodeSpec fetch
+│ ├── inspector/ # parameter panel (spec-driven)
+│ │ ├── Inspector.tsx
+│ │ ├── ParameterTypeRegistry.ts # plugin registry for parameter types
+│ │ ├── types/ # one file per parameter type
+│ │ └── customEditors/ # MasterSkillEditor, ToolSchemaEditor, etc.
+│ ├── credentials/ # credentials panel (reference slice — already in flight)
+│ │ ├── CredentialsPalette.tsx
+│ │ ├── panels/
+│ │ └── primitives/
+│ ├── console/ # Chat / Console / Terminal resizable bottom panel
+│ └── output/ # connected node output panel
+├── shared/ # cross-plane primitives
+│ ├── stores/
+│ │ ├── useNodeStatusStore.ts # hot node status (Zustand with per-slice selectors)
+│ │ ├── useCredentialRegistry.ts # UI state only (already built)
+│ │ └── useAppStore.ts # shrunk: UI visibility toggles, mode, renamingNodeId
+│ ├── hooks/
+│ │ ├── useCatalogueQuery.ts # TanStack Query for NodeSpec + credential catalogue
+│ │ └── useWebSocketSubscription.ts # subscribe-only WS helper
+│ ├── transport/
+│ │ ├── httpClient.ts # axios/fetch wrapper with request_id
+│ │ └── wsClient.ts # subscriptions only; no RPC
+│ └── theme/ # Dracula + Solarized tokens, `resolveColor` helper
+└── main.tsx # QueryClientProvider + router + error boundary
+```
+
+### 5.2 Backend (`server/`)
+
+```
+server/
+├── control/ # Control plane (new top-level)
+│ ├── routers/
+│ │ ├── workflow.py # HTTP commands: deploy, validate, save, delete
+│ │ ├── credentials.py # HTTP CRUD + icon endpoint (already in credentials plan)
+│ │ ├── nodespecs.py # HTTP: list/get/validate NodeSpecs
+│ │ └── subscriptions.py # WebSocket /ws/status — subscriptions only
+│ ├── services/
+│ │ ├── credential_registry.py # already built this session
+│ │ ├── nodespec_registry.py # new: load nodespecs/, resolve extends, validate via jsonschema
+│ │ ├── workflow_compiler.py # new: JSON → execution IR
+│ │ ├── idempotency.py # new: TTL dedupe dict for all mutation handlers
+│ │ └── auth.py # unchanged
+│ └── middleware/
+│ └── auth.py # unchanged
+├── execution/ # Execution plane
+│ ├── temporal/
+│ │ ├── workflow.py # stays (orchestrator)
+│ │ ├── activities/ # split by capability
+│ │ │ ├── http_activities.py
+│ │ │ ├── llm_activities.py
+│ │ │ ├── browser_activities.py
+│ │ │ ├── social_activities.py
+│ │ │ ├── filesystem_activities.py
+│ │ │ ├── trigger_activities.py
+│ │ │ └── code_activities.py
+│ │ ├── worker.py # stays, selects activity registration by worker class
+│ │ └── runtime/ # generic runtime interpreters for declarative specs
+│ │ ├── http_runtime.py # interprets NodeSpec.execution.requestTemplate
+│ │ └── graphql_runtime.py
+│ └── workers/ # one launchable entry per worker class
+│ ├── http_worker.py
+│ ├── llm_worker.py
+│ └── ...
+├── nodespecs/ # source of truth for declarative nodes
+│ ├── _schemas/ # JSON Schema definitions + $defs
+│ ├── google/
+│ │ ├── gmail.send.v1.json
+│ │ ├── calendar.create.v1.json
+│ │ └── ...
+│ ├── ai/
+│ │ ├── openai.chat.v1.json
+│ │ └── ...
+│ └── social/
+│ └── ...
+├── config/ # unchanged — pricing, email_providers, credential_providers, etc.
+└── core/ # unchanged — container, database, encryption, etc.
+```
+
+### 5.3 What gets deleted / shrunk
+
+| Path | Current size | Target |
+|---|---|---|
+| `server/routers/websocket.py` | 127 handlers in one file | Split: subscriptions-only at `control/routers/subscriptions.py`; all commands move to HTTP routers |
+| `server/services/node_executor.py` | Monolithic dispatch registry | Replaced by Temporal activities per worker class; declarative nodes go through `http_runtime.py` |
+| `client/src/nodeDefinitions.ts` + 26 per-category files | 93 nodes across 27 files, eager spread merge | Replaced by `useCatalogueQuery` fetching from `/api/nodespecs` |
+| `client/src/components/ParameterRenderer.tsx` | 780-line monolith switch | Replaced by `ParameterTypeRegistry` with per-type files (param-panel MD) |
+| `client/src/Dashboard.tsx` | 1200+ lines | Shrunk to a coordinator; node-type mapping auto-derived from component glob (canvas MD) |
+| `client/src/contexts/WebSocketContext.tsx` | Full node status map in React context | Replaced by `useNodeStatusStore` (Zustand) + subscription-only WS client (canvas MD) |
+
+**Non-goals**: do not delete any existing handler / node type / feature. Every migration phase is additive; deletions only happen after the new path is proven and the old site is a 1-line re-export for backward compat.
+
+---
+
+## 6. Extension points (Backstage-style)
+
+Small, narrow, interface-driven. Each has a one-file registration via Vite `import.meta.glob` on the frontend and Python module auto-discovery on the backend.
+
+| Extension point | What registers | Frontend | Backend |
+|---|---|---|---|
+| **NodeSpecProvider** | New node types | — | `server/nodespecs/**/*.v{N}.json` |
+| **WorkerRuntimeProvider** | New execution runtimes (http, graphql, streaming, browser, code) | — | `server/execution/temporal/runtime/*.py` |
+| **InspectorSectionProvider** | New parameter panel sections (e.g. token-usage meter, skill editor) | `client/src/authoring/inspector/sections/*.tsx` | — |
+| **ParameterTypeHandler** | New parameter input types (e.g. cronExpression, credentialsPicker) | `client/src/authoring/inspector/types/*.tsx` | — |
+| **CustomEditorProvider** | Full-panel replacements (MasterSkillEditor, ToolSchemaEditor) | `client/src/authoring/inspector/customEditors/*.tsx` | — |
+| **NodeShellProvider** | Reusable React node shells (square, round, trigger, config) | `client/src/authoring/canvas/nodes/shells/*.tsx` | — |
+| **CredentialProvider** | New credential kinds (apiKey, oauth, qrPairing, email) | `client/src/authoring/credentials/panels/*.tsx` | `server/control/services/credentials/providers/*.py` |
+| **OutputRendererProvider** | How to render node outputs in the connected output panel | `client/src/authoring/output/renderers/*.tsx` | — |
+| **ValidationRuleProvider** | Custom runtime validation rules beyond JSON Schema | `client/src/authoring/inspector/validators/*.ts` | `server/control/services/validators/*.py` |
+| **DocsProvider** | Markdown docs for node types | `client/src/authoring/docs/*.tsx` | `server/nodespecs/**/README.md` |
+
+**Pattern** (frontend, same across all the above):
+
+```ts
+// client/src/authoring/inspector/types/index.ts
+import type { ParameterTypeHandler } from '../ParameterTypeRegistry';
+const modules = import.meta.glob<{ default: ParameterTypeHandler }>('./*.tsx', { eager: true });
+for (const m of Object.values(modules)) registerParameterType(m.default);
+```
+
+**Pattern** (backend, same across the above):
+
+```py
+# server/control/services/credentials/providers/__init__.py
+import pkgutil, importlib, pathlib
+_ROOT = pathlib.Path(__file__).parent
+for mod_info in pkgutil.iter_modules([str(_ROOT)]):
+ importlib.import_module(f"{__name__}.{mod_info.name}")
+# Each provider module does `register_provider(...)` at import time.
+```
+
+---
+
+## 7. Delta from current state
+
+### 7.1 Already good (keep)
+
+- Temporal worker + activities infrastructure (`server/services/temporal/`) — activities need splitting by capability but the plumbing is solid.
+- `server/services/workflow.py` facade + `deployment/` modularization — already following the Conductor Decide pattern.
+- `server/services/credential_registry.py` (built this session) — canonical shape for other registries to follow.
+- `server/config/*.json` convention (`email_providers.json`, `credential_providers.json`, `pricing.json`, `google_apis.json`, `llm_defaults.json`) — already aligned with the NodeSpec approach.
+- Workspace directory per workflow — already the right pattern for artifact storage.
+- `functools.partial` dependency injection in `node_executor.py` — good; generalize to the activity registries.
+- Dracula + Solarized theme tokens — unchanged.
+- Per-workflow status scoping in `WebSocketContext.tsx:440-449` — right shape, wrong subscription pattern (fix in canvas MD Lever 1).
+- `useAppStore` UI-state persistence — shape is right; just shrink to UI-only.
+
+### 7.2 Needs change (in scope for this RFC)
+
+- `client/src/nodeDefinitions.ts` + 26 per-category files → NodeSpec registry served from `control/routers/nodespecs.py`.
+- `client/src/components/ParameterRenderer.tsx` → `ParameterTypeRegistry` (param-panel MD).
+- `client/src/Dashboard.tsx` + `SquareNode.tsx` + `AIAgentNode.tsx` → per-slice status subscription + component glob (canvas MD).
+- `server/routers/websocket.py` → split into subscriptions-only + HTTP command routers.
+- `server/services/node_executor.py` → thin dispatcher over Temporal activity queues, one per capability.
+- `server/services/handlers/*` → merge into `execution/temporal/activities/*` split by worker class; declarative node handlers deleted in favour of `http_runtime.py`.
+- Credentials panel Phase 2–7 (already planned in this file before rewrite; preserved as Phase A below).
+
+### 7.3 Load-bearing blockers (must land first)
+
+1. **NodeSpec v1 schema** — every subsequent phase depends on this being stable. One well-reviewed JSON Schema + one Pydantic model + one JSON Schema validator.
+2. **TanStack Query + QueryClientProvider wired at app root** — credentials panel Phase 3 does this. Everything else (palette, parameter panel, node status) piggybacks on the same `QueryClient`.
+3. **idempotency.py + request_id on all mutations** — credentials panel Phase 7.5 starts this. Generalize to every HTTP command endpoint.
+4. **React Compiler babel plugin enabled and trusted** — credentials panel Phase 7.5 validates it on one module; then expand module-by-module.
+
+Nothing else is a blocker. The rest can roll forward incrementally.
+
+---
+
+## 8. Migration plan (phased)
+
+Each phase is self-contained, lands as its own PR, and does not break `main`. Every phase lands on the worktree first.
+
+### Phase A — Credentials panel as the reference slice (IN FLIGHT)
+
+**Status**: Phase 2 complete (JSON registry + `credential_registry.py` + `get_credential_catalogue` WebSocket handler, all verified end-to-end). Phases 3–8 pending.
+
+**Scope**: everything from the original credentials plan — see `docs-internal/credentials_scaling/` for full detail.
+
+**What it proves**:
+- TanStack Query + `experimental_createPersister(idb-keyval)` warm-start pattern
+- Zustand UI-state-only store (no catalogue, no derived data)
+- `cmdk` + `fuzzysort` + `react-virtuoso GroupedVirtuoso` + `startTransition` palette pattern
+- `react-hook-form` `Controller` wrapping antd inputs
+- `React.lazy` per-panel code splitting
+- `babel-plugin-react-compiler` scoped via Vite
+- Idempotency keys + OAuth circuit breaker
+- JSON registry with `extends` inheritance and array-merge-by-key — becomes the template for NodeSpec
+- Content-sha256 version hash + conditional-`since` fetches — becomes the template for NodeSpec registry
+
+**Runtime targets** (verified via heap snapshot + `web-vitals` onINP):
+- Catalogue retained heap: ~1.5–2.5 MB, single retainer (TanStack Query cache)
+- Modal 50-cycle heap delta: < 1 MB
+- INP p75 during rapid search typing: < 200 ms
+- Scroll FPS at row 2000 on synthetic 5000-provider probe: 60 fps
+
+### Phase B — NodeSpec v1 registry + first declarative migration
+
+**Scope**:
+1. Define `server/nodespecs/_schemas/node-spec.v1.json` (JSON Schema describing the NodeSpec shape).
+2. Build `server/control/services/nodespec_registry.py` following the exact same lazy-singleton + content-sha256 + `extends`-resolution pattern as `credential_registry.py`.
+3. Add `server/control/routers/nodespecs.py` — HTTP endpoints: `GET /api/nodespecs`, `GET /api/nodespecs/{id}`, `GET /api/nodespecs/validate`.
+4. Migrate 5–10 simple nodes from `client/src/nodeDefinitions/` to JSON NodeSpec files: start with `gmail.send`, `calendar.create`, `openai.chat`, `brave.search`, `httpRequest` — mix of declarative-friendly shapes.
+5. Build `server/execution/temporal/runtime/http_runtime.py` — the generic HTTP executor that interprets `spec.execution.requestTemplate` + `responseMapping`.
+6. Frontend: `useNodeSpecQuery` on TanStack Query, served into a new `authoring/palette/Palette.tsx` alongside (not replacing) the existing ComponentPalette.
+
+**Verification**:
+- The 5–10 migrated nodes appear in both the old and new palette.
+- Executing a migrated node via the new path produces byte-identical output to the old path.
+- Adding a new Mailchimp / Notion / Airtable node = one `.v1.json` file, tested, no frontend or backend code change.
+
+### Phase C — WebSocket router fission (command vs subscription split)
+
+**Scope**:
+1. Audit `server/routers/websocket.py`: bucket all 127 handlers into `{command, query, subscription}`.
+2. For each command handler: add an equivalent HTTP endpoint under `server/control/routers/` that accepts `request_id` (idempotency).
+3. For each query handler: same — HTTP endpoint.
+4. Keep subscription handlers at the new `control/routers/subscriptions.py` (`/ws/status`) — only.
+5. Frontend: build `shared/transport/httpClient.ts` (with request_id auto-injection) + `shared/transport/wsClient.ts` (subscriptions only).
+6. Migrate one feature at a time (credentials first — already in Phase A; then parameter panel; then workflow CRUD; then execution control).
+7. **Do not delete `websocket.py` handlers until the HTTP equivalent is production-proven.** They coexist for a release.
+
+**Verification**:
+- WebSocket connection no longer receives any request/response traffic — only server-pushed status updates.
+- Request latency for commands drops (HTTP avoids WebSocket sequential dispatch overhead).
+- Browser network panel cleanly separates commands (HTTP) from events (WS).
+
+### Phase D — Worker pool specialization
+
+**Scope**:
+1. Split `server/services/temporal/activities.py` into `activities/{http,llm,browser,social,filesystem,trigger,code}_activities.py`.
+2. Each activity file registers only activities for its worker class.
+3. Update `worker.py` to accept a `--worker-class` flag; a worker process registers only the matching activity subset.
+4. Update Temporal task queue names: one per worker class (`machina-http`, `machina-llm`, `machina-browser`, …).
+5. Workflow orchestrator dispatches each node to the matching queue based on the NodeSpec's `runtime.workerType`.
+6. Docker / dev script: spin up one worker process per class by default; single-process fallback for local dev.
+7. Add queue-depth metrics (Prometheus or stdout) so later autoscaling has something to key on.
+
+**Verification**:
+- Mixed workload (e.g. 50 HTTP requests + 5 LLM calls) runs with no head-of-line blocking between classes.
+- LLM workers can be restarted / scaled independently without interrupting HTTP workers.
+- A crashed browser worker does not kill other activities.
+
+### Phase E — Authoring plane modularity
+
+**Scope**: apply the 3 UI optimization MDs (in order of impact, not in order of the MDs):
+
+1. **Canvas MD Lever 2** (5 minutes): `onlyRenderVisibleElements={true}` on `<ReactFlow>`. 70 % win on pan/zoom.
+2. **Canvas MD Lever 1** (2 hours): extract `useNodeStatusStore`; replace `WebSocketContext.getNodeStatus` with per-slice subscription. 2000 → 10 re-renders/sec during status floods.
+3. **Canvas MD Lever 4** (30 min): wrap status dispatches in `startTransition`. Keeps drag/select synchronous during broadcast storms.
+4. **Canvas MD Lever 3** (2 hours): `useWorkflowConfigStatus` TanStack Query hook replaces per-`SquareNode` mount `getStoredApiKey()` calls. Workflow open 10 s → < 1 s.
+5. **Parameter panel MD Lever 1** (3 hours): `ParameterTypeRegistry` with `import.meta.glob` auto-discovery. Migrate string / number / boolean / options / collection first.
+6. **Parameter panel MD Lever 2** (4 hours): `useParameterPanel` migrates to TanStack Query `useMutation` with 300 ms debounce, optimistic updates, idempotency keys.
+7. **Parameter panel MD Lever 3** (30 min): extend React Compiler babel scope to `parameterPanel/`.
+8. **Parameter panel MD Lever 4** (2 hours): compile `displayOptions.show` to memoized predicates (`WeakMap`-cached).
+9. **ComponentPalette MD Levers 1–3** (3 days): `GroupedVirtuoso` + `fuzzysort.prepare()` + `import.meta.glob` file-based node registry. "Drop a file, get a node" for the visual side of things — pairs with Phase B's NodeSpec registry for the data side.
+10. **Dashboard slim-down**: extract the node-type mapping into an `import.meta.glob('./nodes/**/*.node.tsx')` registry; Dashboard becomes a coordinator.
+
+**Verification**:
+- 200-node workflow open < 1 s, drag 60 fps, INP p75 < 200 ms during status floods.
+- New parameter type = 1 file under `inspector/types/`, auto-registered.
+- New node React component = 1 file under `canvas/nodes/`, auto-registered.
+
+### Phase F — Subflows + collapse as first-class concept
+
+**Scope**:
+1. Extend workflow JSON schema: `nodes[].parent` for group membership; `nodes[].collapsed: boolean`.
+2. React Flow implementation: use the official [expand/collapse example](https://reactflow.dev/examples/layout/expand-collapse) as the base.
+3. Workflow compiler: subflows become compiled stages; collapsed groups render as a single placeholder node but execute as their contents.
+4. Storage: subflow definitions are reusable — a workflow can reference a subflow template by name, enabling reusable automations without copy-paste.
+
+**Prerequisite**: Phase B (NodeSpec) and Phase E (canvas status store) must be in.
+
+**Verification**:
+- A 1000-node workflow with 10 collapsed subflows renders at 60 fps.
+- Executing a workflow with subflows produces the same result as the equivalent flat workflow.
+- A subflow saved once can be imported into a second workflow with a single drag.
+
+### Phase G — Execution IR + artifact references
+
+**Scope**:
+1. `server/control/services/workflow_compiler.py`: takes workflow JSON → immutable IR (stage DAG, activity bindings, expression graph).
+2. Workflow orchestrator executes from the IR, not from the raw JSON.
+3. Output artifact policy: any node output > 10 KB lands in `data/workspaces/<workflow_id>/<node_id>/output.json` (or binary); the in-workflow payload is `{ "$artifact": "<path>" }`.
+4. `ParameterResolver` template engine learns to dereference `$artifact` paths lazily — artifacts are only materialized when a downstream node actually reads them.
+5. WebSocket `node_output` broadcasts carry artifact references; the frontend fetches the blob on demand when the user opens the output panel.
+
+**Verification**:
+- A workflow that produces a 50 MB scraped HTML blob does not blow up the WebSocket buffer or the in-memory workflow state.
+- Downstream nodes that don't need the large payload incur zero artifact materialization cost.
+- Workflow execution memory floor stays flat regardless of output sizes.
+
+### Phase H — Polish
+
+- Off-main-thread layout via a Web Worker for workflows > 200 nodes.
+- Prometheus metrics for queue depth + wait time + completion rate per worker class.
+- Autoscaling rules (dev: single-process; prod: per-worker-class horizontal scaling).
+- Generator: `pnpm machina new-node gmail.send.draft` scaffolds a new NodeSpec JSON from a template + opens it in the editor.
+- `pnpm machina new-param-type cronExpression` scaffolds a new parameter type file with a registered handler.
+
+---
+
+## 9. What lands first (concrete next PRs)
+
+In order, each independently shippable:
+
+1. **Finish Phase A** (credentials panel, Phases 3–8 of the original plan).
+2. **Phase E steps 1–4** (canvas perf cheap wins: `onlyRenderVisibleElements`, `useNodeStatusStore`, `startTransition` dispatch, `useWorkflowConfigStatus`). ~6 hours total. Single biggest runtime win in the app, no prerequisites.
+3. **Phase B** (NodeSpec v1 registry + 5–10 declarative migrations + `http_runtime.py`). ~1–2 weeks.
+4. **Phase C** (WebSocket → HTTP split, one feature at a time starting with credentials). ~1 week.
+5. **Phase E steps 5–8** (parameter panel type registry + mutation queue + React Compiler scope + visibility compilation). ~10 hours.
+6. **Phase D** (worker pool specialization). ~1 week.
+7. **Phase F, G, H** — each is a separate 1–2 week phase.
+
+---
+
+## 10. Out of scope (deliberate)
+
+- **CQRS, event sourcing, API gateway, service mesh.** Tribal at this scale; add only when measured need arises.
+- **Cloudflare Workers / edge CDN.** Not justified for a self-hosted tool.
+- **Desktop packaging (Tauri / Electron).** Scope creep; web + self-hosted covers 95 % of cases.
+- **Module Federation / plugin marketplaces.** Overkill; `import.meta.glob` is enough for in-repo extensibility.
+- **Rewriting Temporal integration.** What exists works; just split activities by capability.
+- **Multi-tenant credential isolation, credential encryption changes, OAuth flow rewrites.** Unchanged.
+- **`lucide-react` icon swap.** Assets are already SVG-based.
+- **Million.js, Jotai, Valtio, Legend State, `rjsf`, `pro-form`.** Already rejected in credentials plan; same rejections apply here.
+- **Node.js subprocess pool replacement.** Persistent Node.js server at port 3020 already solves the startup cost.
+- **AI provider changes.** Credentials panel + `services/llm/` already handle this.
+
+---
+
+## 11. Open questions
+
+1. **NodeSpec `execution.requestTemplate` expression language.** JSON pointers? JSONata? JMESPath? Simple `{{ template }}` like the existing `ParameterResolver`? **Recommendation**: reuse the existing `{{ node.field }}` template engine — no new DSL, zero contributor learning curve. Escape hatch: `responseMapping` can include a `transformRef` pointing at a NodeSpec-owned TypeScript/Python transformer module for the 5 % of cases the template engine can't express.
+2. **NodeSpec storage location.** Committed to the repo under `server/nodespecs/`, or served from a git-backed registry (Nango-style)? **Recommendation**: start in-repo. Add git-backed only if we grow an ecosystem.
+3. **Backward compatibility during Phase C (WS → HTTP split).** Keep both paths live, or hard cutover? **Recommendation**: both paths live for one release; delete the WS command handler only after telemetry shows zero calls for 14 days.
+4. **Temporal activity retry semantics for declarative HTTP nodes.** NodeSpec `retryPolicy: "standard"` — what's the concrete meaning? **Recommendation**: start with Temporal's default exponential backoff (initial 1 s, max 100 s, 3 attempts). Per-node tuning via `retryPolicy: "aggressive" | "cautious" | "none"` presets defined centrally in `http_runtime.py`.
+5. **Extension point registration on the backend.** Python doesn't have Vite globs. **Recommendation**: `pkgutil.iter_modules` in each extension-point package's `__init__.py` + decorator-based `@register_X()` from each module. Proven pattern, zero config.
+6. **Credentials plan phase 7.5 React Compiler scope expansion ordering.** Expand to canvas first (biggest perf win) or parameter panel first (most fields)? **Recommendation**: canvas first — it's the larger hot path at steady state.
+
+---
+
+## 12. Runtime targets (platform-wide)
+
+Inherited from the credentials panel runtime research, applied across the authoring plane:
+
+| Metric | Target |
+|---|---|
+| Workflow open (200 nodes, warm) | < 1 s |
+| Workflow open (200 nodes, cold) | < 3 s |
+| Credentials / palette / inspector warm open | < 50 ms |
+| Credentials / palette / inspector cold open | < 500 ms |
+| INP p75 during any rapid typing in any text field | < 200 ms |
+| Scroll FPS — 500-node palette / 200-node canvas / 5000-provider list | 60 fps |
+| 50-cycle open/close heap delta for any modal | < 1 MB |
+| Catalogue / NodeSpec retained heap | ~1.5–2.5 MB per catalogue, single retainer |
+| WebSocket broadcast → UI update p99 | < 50 ms |
+| Large workflow output payload (50 MB) in-memory state | 0 bytes (artifact reference only) |
+
+Verification workflow: Chrome DevTools → Memory → Heap Snapshot → single retainer per large dataset; Allocation timeline → 50-cycle delta < 1 MB; `web-vitals` `onINP` with attribution. Same tooling across all phases.
+
+---
+
+## 13. Sources
+
+Inherited from `research.md`:
+- [Temporal server overview](https://github.com/temporalio/temporal)
+- [Conductor architecture overview](https://conductor-oss.org/docs/architecture)
+- [Conductor task workers](https://conductor-oss.org/docs/developer-guides/concepts/worker)
+- [Conductor worker scaling](https://conductor-oss.org/docs/developer-guides/scaling)
+- [React Flow performance](https://reactflow.dev/learn/troubleshooting/performance)
+- [React Flow expand/collapse example](https://reactflow.dev/examples/layout/expand-collapse)
+- [n8n declarative vs programmatic nodes](https://docs.n8n.io/integrations/creating-nodes/plan/choose-node-method/)
+- [n8n node file structure](https://docs.n8n.io/integrations/creating-nodes/build/reference/node-file-structure/)
+- [n8n node versioning](https://docs.n8n.io/integrations/creating-nodes/build/reference/node-versioning/)
+- [JSON Schema docs](https://json-schema.org/learn/)
+- [JSON Schema dialects and vocabularies](https://json-schema.org/understanding-json-schema/reference/schema)
+- [Backstage backend extension points](https://backstage.io/docs/backend-system/architecture/extension-points)
+- [Backstage plugin modules](https://backstage.io/docs/backend-system/architecture/modules)
+- [Kubernetes CRDs and structural schemas](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/)
+- [react-jsonschema-form uiSchema](https://rjsf-team.github.io/react-jsonschema-form/docs/api-reference/uiSchema/)
+
+Added this session:
+- [React Compiler 1.0 release](https://react.dev/blog/2025/10/07/react-compiler-1)
+- [TanStack Query v5](https://tanstack.com/query/latest) and [`experimental_createPersister`](https://tanstack.com/query/v5/docs/react/plugins/createPersister)
+- [V8 memory optimization](https://v8.dev/blog/optimizing-v8-memory)
+- [MemLab (Meta)](https://engineering.fb.com/2022/09/12/open-source/memlab/)
+- [Vite `import.meta.glob`](https://vite.dev/guide/features.html#glob-import)
+- [`web-vitals` library](https://github.com/GoogleChrome/web-vitals)
+- [oboe.com](https://oboe.com/) — production reference for TanStack Query + dehydrated/hydrated cache pattern
+- [`credential_providers.json` + `credential_registry.py`](../worktrees/credentials-scaling/) — in-tree reference implementation built this session
+
+Companion MDs in the worktree (all read-only in plan mode):
+- `docs-internal/credentials_scaling/research_production_platforms.md`
+- `docs-internal/credentials_scaling/research_react_stack.md`
+- `docs-internal/credentials_scaling/architecture.md`
+- `docs-internal/ui_optimization/README.md`
+- `docs-internal/ui_optimization/component_palette_scaling.md`
+- `docs-internal/ui_optimization/parameter_panel_scaling.md`
+- `docs-internal/ui_optimization/canvas_dashboard_scaling.md`
+- `research.md` (user-provided vision doc — the seed for this RFC)
+
+---
+
+## 14. Non-goals of this RFC
+
+This document defines the target architecture and phased migration. It does **not** prescribe:
+- Specific UI copy or visual design changes beyond what the 3 UI optimization MDs already spec.
+- Pricing, billing, or quota logic for any provider.
+- Production deployment changes beyond "worker class autoscaling lands in Phase D + H".
+- Database schema changes beyond what's already in the credentials plan + the artifact reference convention in Phase G.
+- Authentication or authorization model changes.
+
+Each of those is a separate RFC if and when needed.
diff --git a/docs-internal/ui_optimization/BUNDLE_AUDIT.md b/docs-internal/ui_optimization/BUNDLE_AUDIT.md
new file mode 100644
index 00000000..e4c04594
--- /dev/null
+++ b/docs-internal/ui_optimization/BUNDLE_AUDIT.md
@@ -0,0 +1,104 @@
+# Bundle Audit (Phase 7)
+
+Generated: 2026-04-12
+
+Audit of the main client chunk (`dist/assets/index-<hash>.js`) after Phases 1–6 of the credentials-scaling plan landed. Reproduction:
+
+```bash
+cd client
+ANALYZE=1 pnpm run build # builds with sourcemaps + rollup-plugin-visualizer
+python scripts/audit-bundle.py # prints the tables below
+open dist/stats.html # interactive treemap for deeper drilldowns
+```
+
+Note: `ANALYZE=1` enables the visualizer plugin and source-map emission; leave it unset in normal builds.
+
+## Headline numbers
+
+| Artifact | Size | gzipped | Notes |
+|---|---|---|---|
+| `index-Bix9TxBS.js` (main chunk) | 2,338 KB | 673 KB | Eager entry bundle |
+| `index-DnvjDIoH.js` (palette/adapter chunk) | 38 KB | 12 KB | Emitted by Phase 4 palette integration |
+| `QrPairingPanel` chunk | 29 KB | 10 KB | Phase 5 lazy split |
+| `ApiKeyPanel` chunk | 7.7 KB | 2.9 KB | Phase 5 lazy split |
+| `EmailPanel` chunk | 5.5 KB | 2.1 KB | Phase 5 lazy split |
+| `OAuthPanel` chunk | 3.5 KB | 1.5 KB | Phase 5 lazy split |
+| Total sources in main chunk | 1,425 | — | Unique module count |
+| Total source bytes (pre-minification) | 5.97 MB | — | Input to minifier |
+
+The 2.34 MB main chunk is the pre-existing baseline; Phases 4 + 5 shipped ~90 KB of panel code out of it via `React.lazy`. Every other number in this audit is a target for future phases, not blockers for merging Phase A.
+
+## Top package contributors to the main chunk
+
+| Rank | Bucket | Bytes | % | Notes |
+|---|---|---|---|---|
+| 1 | `antd` | 940,205 | 15.8% | Expected — big UI kit. Component-level tree-shaking already applies via ES imports. Further reductions would need switching to antd v6 CSS Variables mode (~30 KB saving). |
+| 2 | `src/components` (aggregate) | 739,653 | 12.4% | Dominated by the monoliths called out in the UI optimization MDs. Per-file breakdown below. |
+| 3 | `react-dom` | 545,403 | 9.1% | Baseline. Not actionable. |
+| 4 | `src/nodeDefinitions` | 325,531 | 5.5% | 26 per-category files eagerly spread-merged at module parse. **Directly addressed by ComponentPalette MD Lever 4** (`import.meta.glob` lazy per-category + TanStack Query-backed NodeSpec registry). |
+| 5 | `@lobehub/icons` | 232,092 | 3.9% | AI provider icon library. Used by the AI chat model nodes. Likely addressable via named imports only. |
+| 6 | `@reactflow/core` | 206,420 | 3.5% | Baseline. React Flow MD recommends `onlyRenderVisibleElements` for runtime, not bundle. Not actionable here. |
+| 7 | `micromark-core-commonmark` | 113,496 | 1.9% | Part of remark/react-markdown. Only actually used in skill/memory editors. **Addressable**: dynamic `import()` in `MasterSkillEditor` and `NodeOutputPanel` so it only loads when a markdown editor opens. |
+| 8 | `@ant-design/icons` | 112,838 | 1.9% | antd icon library. Verify all imports are named (`import { FooOutlined }`), not namespace/wildcard. |
+| 9 | `rc-select` | 108,968 | 1.8% | antd transitive. Not actionable independently. |
+| 10 | `rc-field-form` | 97,711 | 1.6% | antd transitive. Not actionable independently. |
+| 11 | `src/contexts` | 97,379 | 1.6% | Dominated by `WebSocketContext.tsx` (90 KB). **Directly addressed by Canvas MD Lever 1** (per-node status store) + Phase C of the RFC (WebSocket router fission). |
+| 12 | `react-virtuoso` | 94,304 | 1.6% | Phase 4 palette dependency — paid once, used by the palette. |
+| 13 | `prismjs` | 80,281 | 1.3% | Code editor syntax highlighting. Already lazy-loaded in `CodeEditor.tsx` per CLAUDE.md, but something is still pulling it eagerly. **Action**: audit eager imports of `react-simple-code-editor`. |
+| 14 | `@tanstack/query-core` | 78,143 | 1.3% | Phase 3 dependency — paid once, used everywhere. |
+
+## Top single-file contributors
+
+| File | Bytes | Recommended action |
+|---|---|---|
+| `src/contexts/WebSocketContext.tsx` | 90,018 | Canvas MD Lever 1 + RFC Phase C: per-node status store + WS router fission. |
+| `src/components/ParameterRenderer.tsx` | 87,379 | Parameter panel MD Lever 1: type registry with `import.meta.glob` auto-discovery. 780-line switch becomes one file per type. |
+| `src/components/ui/NodeOutputPanel.tsx` | 70,558 | Lazy-load `react-markdown` / `remark-*` via dynamic `import()` — only needed when output includes markdown. |
+| `src/components/parameterPanel/InputSection.tsx` | 65,576 | Parameter panel MD Lever 5: TanStack Query-backed schema cache. |
+| `src/Dashboard.tsx` | 54,129 | Canvas MD proposal: glob-based node component registry; Dashboard becomes a coordinator. |
+| `src/components/parameterPanel/MiddleSection.tsx` | 51,751 | Parameter panel MD Levers 3 + 4: memoized field list + compiled visibility predicates. |
+| `src/components/ui/ConsolePanel.tsx` | 45,612 | Lazy-load `prismjs` / code editor inside the console tab — only costs if the terminal is actually used. |
+| `src/nodeDefinitions/googleWorkspaceNodes.ts` | 42,714 | RFC Phase B: migrate to NodeSpec JSONs served from the server registry. |
+| `src/components/parameterPanel/MasterSkillEditor.tsx` | 41,822 | Lazy-load `react-markdown` dependency — only costs when a skill editor opens. |
+| `src/nodeDefinitions/whatsappNodes.ts` | 41,601 | RFC Phase B: NodeSpec migration. |
+
+## Actionable wins (ranked by effort × impact)
+
+1. **Lazy-load markdown stack in `NodeOutputPanel` + `MasterSkillEditor`**
+ ~180 KB unminified saving (`micromark` + `mdast-util-*` + `unified` + related). 2 dynamic `import()` calls, ~30 min. Same Supabase `next/dynamic` pattern the research MDs cite.
+
+2. **Verify `prismjs` / `react-simple-code-editor` are lazy**
+ CLAUDE.md claims they are, but the audit shows ~102 KB of prism + editor code landing in the main chunk. 1 hour to find and fix the eager import site. Saves ~30 KB gzipped.
+
+3. **ComponentPalette Lever 4 — lazy per-category node definitions**
+ `src/nodeDefinitions` bucket is 325 KB. Moving to `import.meta.glob` lazy imports drops this to ~10 KB for the registry shell. Detailed in `component_palette_scaling.md`. 1–2 days total.
+
+4. **Canvas MD Lever 1 — per-node status store**
+ `WebSocketContext.tsx` is 90 KB and causes the subscription storm separately. Shrinks the file and fixes the 200-node re-render cascade in one change. ~2 hours, biggest runtime win in the app.
+
+5. **Parameter panel MD Lever 1 — type registry**
+ `ParameterRenderer.tsx` 87 KB monolith becomes ~15 KB shell + per-type files lazy-loaded. Makes adding a new parameter type a one-file operation. ~3 hours.
+
+## Deliberately deferred / out of Phase 7 scope
+
+- Antd v6 CSS Variables migration — requires full app-level testing.
+- Replacing `antd` with a lighter kit — scope creep.
+- `react-flow` tree-shaking — not actionable without forking React Flow.
+- `lucide-react` icon swap — does not apply to credentials (already raw SVG).
+- `@lobehub/icons` audit — used by AI chat model nodes, separate workstream.
+
+## What Phase 7 shipped
+
+- `rollup-plugin-visualizer` wired into `vite.config.js` behind an `ANALYZE=1` flag so normal builds pay zero cost.
+- `client/scripts/audit-bundle.py` — reproducible programmatic audit (see top of this doc).
+- Source-map emission gated on `ANALYZE=1` — normal builds stay small, analyze builds get accurate attribution.
+- This report. All actionable wins have owners (existing UI optimization MDs or RFC phases).
+
+## References
+
+- `client/vite.config.js` — visualizer plugin + sourcemap gating
+- `client/scripts/audit-bundle.py` — reproducible audit
+- `docs-internal/ui_optimization/component_palette_scaling.md` — Lever 4 (lazy nodedef registry)
+- `docs-internal/ui_optimization/parameter_panel_scaling.md` — Lever 1 (type registry)
+- `docs-internal/ui_optimization/canvas_dashboard_scaling.md` — Lever 1 (per-node status store)
+- `docs-internal/platform_refactor/PLATFORM_REFACTOR_RFC.md` — Phase B (NodeSpec migration) + Phase C (WS fission)
diff --git a/docs-internal/ui_optimization/README.md b/docs-internal/ui_optimization/README.md
new file mode 100644
index 00000000..3c39e807
--- /dev/null
+++ b/docs-internal/ui_optimization/README.md
@@ -0,0 +1,130 @@
+# UI Optimization & Prod-Scale Extensibility Research
+
+Companion research set to `docs-internal/credentials_scaling/`. While the credentials panel plan focuses on making the 5000-provider catalogue fast, this folder applies the **same patterns and libraries** to the rest of the MachinaOs UI — so runtime perf, memory use, and contributor DX all scale together.
+
+**Zero new dependencies**: every library referenced (`react-virtuoso`, `cmdk`, `fuzzysort`, `react-hook-form`, `@tanstack/react-query`, `idb-keyval`, `babel-plugin-react-compiler`) is already being installed for the credentials panel.
+
+---
+
+## Files
+
+| Document | Covers | Status |
+|---|---|---|
+| [component_palette_scaling.md](./component_palette_scaling.md) | Node sidebar (`ComponentPalette.tsx`) + node definition registry (`nodeDefinitions.ts` + 26 per-category files) at 500+ nodes. Lazy-load, virtualization, fuzzy search, file-based registry, "drop a file, get a node" proposal. | Complete |
+| [parameter_panel_scaling.md](./parameter_panel_scaling.md) | Parameter configuration modal (`ParameterPanel.tsx`, `ParameterRenderer.tsx`, `MiddleSection.tsx`, `useParameterPanel.ts`). Parameter type registry, TanStack Query mutation queue, display-options predicate compilation, custom editor registry. | Complete |
+| [canvas_dashboard_scaling.md](./canvas_dashboard_scaling.md) | React Flow canvas + Dashboard shell (`Dashboard.tsx`, `SquareNode.tsx`, `AIAgentNode.tsx`, `WebSocketContext.tsx`, `useAppStore`) at 200+ nodes with parallel workflow execution. Per-node status store, `onlyRenderVisibleElements`, deferred config checks, component registry. | Complete |
+
+---
+
+## Cross-cutting findings
+
+Three patterns come up in every file and should be treated as project-wide principles:
+
+### 1. Store shape: hot data in TanStack Query, UI state in Zustand, derived data in `useMemo`
+
+The credentials panel runtime research identified a single memory trap worth more attention than all others combined: **Zustand selectors that capture large arrays retain them forever via closure scope**. The fix — moving catalogue data out of Zustand and into TanStack Query — generalizes to every large dataset in the UI:
+
+- Node definitions → `useQuery(['nodeRegistry'])` (ComponentPalette MD)
+- Connected-node output schemas → `useQuery(['nodeOutputSchema', nodeId])` (Parameter panel MD)
+- Per-node status → dedicated `useNodeStatusStore` with per-node selectors (Canvas/Dashboard MD)
+- Node parameters → `useMutation` queue with optimistic updates (Parameter panel MD)
+
+Zustand shrinks to UI-only state: `selectedNodeId`, `paletteOpen`, `renamingNodeId`, `sidebarVisible`, etc.
+
+### 2. The same library stack everywhere
+
+Every MD ranks the same five levers in the top 5:
+
+| Library | Used for | Files affected |
+|---|---|---|
+| `react-virtuoso` (`GroupedVirtuoso`) | Virtualized lists with sticky headers | Credentials palette, ComponentPalette |
+| `cmdk` | Command palette + keyboard nav | Credentials palette, ComponentPalette (optional) |
+| `fuzzysort` | Pre-indexed fuzzy search | Credentials palette, ComponentPalette |
+| `react-hook-form` + `Controller` | Uncontrolled form state | Credentials panels, Parameter panel |
+| `@tanstack/react-query` | Server cache + mutations + `experimental_createPersister(idb-keyval)` | Credentials catalogue, node registry, param saves, output schemas |
+| `startTransition` (React 19) | Non-urgent state updates | Credentials search filter, param display re-filter, node status dispatch |
+| `babel-plugin-react-compiler` | Auto-memoization (scoped via Vite babel plugin) | Credentials module + extend to `parameterPanel/`, `components/`, `ComponentPalette.tsx` |
+
+**No new dependencies beyond the credentials plan.** Scoping the React Compiler babel plugin is the one place scope matters: start tight (credentials module only), expand module by module as each is verified.
+
+### 3. "One file to add a thing" via Vite `import.meta.glob`
+
+All three MDs propose the same auto-registration pattern for their respective extensibility axis:
+
+| Axis | Pattern | Location |
+|---|---|---|
+| Node **definitions** (the schema/metadata) | `import.meta.glob('./nodes/**/*.node.ts', { eager: true })` | `client/src/nodeDefinitions/` |
+| Node **React components** (the visual) | `import.meta.glob('./**/*.node.tsx', { eager: true })` | `client/src/components/nodes/` |
+| Parameter **type handlers** (registry entries) | `import.meta.glob('./parameterPanel/types/*.tsx', { eager: true })` | `client/src/components/parameterPanel/types/` |
+| Parameter **custom editors** (MasterSkillEditor-class full-panel replacements) | `import.meta.glob('./parameterPanel/customEditors/*.tsx', { eager: true })` | `client/src/components/parameterPanel/customEditors/` |
+
+Each glob consolidates what is currently 4–9 files of scattered registration into a single file change. Adding a new node type, new parameter type, new custom editor, or new React component for a node becomes **one file**. The tribal checklists in `CLAUDE.md`'s "Adding New Specialized Agents" etc. shrink to a paragraph.
+
+This is not n8n-style dynamic loading from arbitrary `node_modules/` — it's just Vite resolving globs at build time. No runtime overhead, full type safety, HMR works normally.
+
+---
+
+## Relationship to the credentials panel plan
+
+The credentials panel (`docs-internal/credentials_scaling/`) is the **reference implementation**. It installs all the dependencies, wires `QueryClientProvider` in `main.tsx`, enables the React Compiler babel plugin, sets up the `idb-keyval` persister, and proves the runtime/memory targets.
+
+Everything in this folder is "reuse that infra for the rest of the UI":
+
+- Phase 2–4 of the credentials plan ships the backend registry + Zustand + TanStack Query + palette infrastructure.
+- Phase 7.5 ships the React Compiler babel plugin config + idempotency keys.
+- **Only after Phase 8 verification passes** should we start applying these patterns to ComponentPalette, parameter panel, and canvas.
+
+Order of operations:
+
+1. **Credentials panel lands first** (phases 2–8 of `typed-splashing-crown.md`)
+2. **ComponentPalette follows** (ComponentPalette MD phases A–C: search UX overhaul + file-based registry + lazy loading)
+3. **Parameter panel follows** (Parameter panel MD phases A–D: type registry + mutation queue + visibility compilation + schema cache)
+4. **Canvas/Dashboard is last** (Canvas MD phases A–D: onlyRenderVisibleElements + per-node status store + deferred config checks + React Compiler scope extension)
+
+Each of these is a separate work-unit of 1–2 weeks. They should not be bundled into the credentials-panel PR.
+
+---
+
+## Runtime targets (UI-wide)
+
+Inherited from the credentials panel plan, applied project-wide:
+
+| Metric | Target |
+|---|---|
+| Modal / panel open (second visit, warm cache) | < 50 ms |
+| Modal / panel open (cold, first visit) | < 500 ms |
+| INP p75 during rapid typing in any text field | < 200 ms |
+| Scroll FPS on 500-node palette / 200-node canvas | 60 fps |
+| 50-cycle open/close heap delta for any modal | < 1 MB |
+| Main-heap retained for the entire frontend at idle | Established by heap snapshot — should be dominated by React internals + antd cssinjs cache, not our data |
+
+Verification workflow (per the credentials panel runtime research): Chrome DevTools → Memory → Heap Snapshot → filter by `Array`, sort by Retained Size → single retainer per large dataset. Allocation timeline → 50-cycle delta. `web-vitals` library `onINP` for INP attribution.
+
+---
+
+## Rejected / deferred ideas
+
+- **Million.js** — superseded by React Compiler 1.0; doesn't play well with antd cssinjs.
+- **Jotai / Valtio / Legend State** — Zustand + TanStack Query already covers our needs.
+- **Module Federation** — overkill for in-app plugins.
+- **Tauri / Electron packaging** — scope creep; web-based self-hosted covers 95% of cases.
+- **Cloudflare Workers / edge CDN** — not justified for a self-hosted tool.
+- **CQRS / event sourcing** — tribal enterprise patterns, not load-bearing at our scale.
+- **Server-owned node registry** (ComponentPalette Lever 5) — deferred until credentials-panel server-owned registry is production-proven.
+- **Full RHF migration of parameter panel** (Parameter panel Lever 6) — deferred until the type registry and mutation queue land first.
+- **Workflow-scoped backend broadcast filter** (Canvas Lever 6) — deferred until per-node status store (Lever 1) proves adequate at 5+ parallel workflows.
+
+---
+
+## Open questions
+
+1. **React Compiler scope expansion strategy.** Should we enable it project-wide at once, or module-by-module as each MD's patterns are verified? Recommendation in all three MDs: module-by-module, starting with the credentials module.
+2. **`idb-keyval` vs TanStack Query built-in persister.** Credentials plan uses `experimental_createPersister(idb-keyval)`. We should verify this works correctly with multiple persisted queries (node registry + credentials catalogue + output schemas) sharing one IndexedDB store.
+3. **Glob-based auto-discovery + backend handler colocation.** ComponentPalette and canvas MDs both propose `*.node.ts` / `*.node.tsx` globs for frontend. The backend handlers still live in `server/services/handlers/` — should there be a symmetric discovery mechanism on the Python side? (Probably yes, but out of scope for these MDs.)
+4. **Parameter panel custom-editor access to global catalogues.** The parameter panel MD proposes a `credentialsPicker` custom editor that queries the credentials catalogue. This is fine within a single `QueryClientProvider`, but we should verify we're not opening two distinct `QueryClient` instances by accident.
+
+---
+
+## Document history
+
+All three MDs written 2026-04-11 following parallel research by Explore agents. Source material: runtime/memory research from the credentials panel (see `../credentials_scaling/research_react_stack.md` addendum), React Flow performance docs, n8n + Pipedream + oboe.com architecture references, and direct worktree audit with file:line references.
diff --git a/docs-internal/ui_optimization/canvas_dashboard_scaling.md b/docs-internal/ui_optimization/canvas_dashboard_scaling.md
new file mode 100644
index 00000000..9f5f7db9
--- /dev/null
+++ b/docs-internal/ui_optimization/canvas_dashboard_scaling.md
@@ -0,0 +1,345 @@
+# React Flow Canvas + Dashboard Scaling to 200+ Nodes
+
+Analysis of `client/src/Dashboard.tsx`, custom node components (`SquareNode.tsx`, `AIAgentNode.tsx`, `TriggerNode.tsx`, `GenericNode.tsx`), `WebSocketContext.tsx` status subscriptions, and `useAppStore` state shape. Goal: what actually breaks on a 200-node workflow with parallel executions, and what to fix first.
+
+Scope: runtime performance, heap memory, and feature-addition DX. Not bundle size.
+
+---
+
+## TL;DR
+
+React Flow itself scales fine to 200+ nodes (documented to 1000+ with `onlyRenderVisibleElements`). The real bottlenecks are **client-side state management**:
+
+1. **Global node-status subscription storm** — every `SquareNode` subscribes to the full `allNodeStatuses` map via `useWebSocket().getNodeStatus(id)`, so any single node's status change re-renders all 200 nodes. 10 updates/sec × 200 nodes = **2000 re-renders/sec**.
+2. **Eager configuration checks on mount** — 200 `SquareNode`s fire parallel `getStoredApiKey()` WebSocket requests at workflow open, producing head-of-line blocking. Workflow-open latency: ~10 s.
+3. **Missing `onlyRenderVisibleElements`** — React Flow is rendering every off-screen node even during pan/zoom.
+
+Three cheap fixes (~5 hours total) reduce p99 interaction latency by an estimated 80 %. All of them reuse the same patterns we're already adopting for the credentials panel: per-slice subscriptions, `startTransition`-wrapped updates, TanStack Query for deferred fetches.
+
+---
+
+## Current bottlenecks
+
+### 1. Per-node status subscription storm — CRITICAL
+
+**File**: `client/src/contexts/WebSocketContext.tsx:1258-1263`
+
+```ts
+const getNodeStatus = useCallback((nodeId: string) => {
+ if (!currentWorkflowId) return undefined;
+ return allNodeStatuses[currentWorkflowId]?.[nodeId];
+}, [allNodeStatuses, currentWorkflowId]);
+```
+
+The problem is **the dependency array**: `getNodeStatus` is rebuilt whenever `allNodeStatuses` changes — which is on every single broadcast for any node in any workflow. Every `SquareNode.tsx:48-50` calls `getNodeStatus(id)` from the context, and every context consumer re-renders when the context value changes.
+
+**Impact chain**:
+1. Any node broadcasts a status update → `setAllNodeStatuses()` fires
+2. Context value re-computes → `getNodeStatus` reference changes
+3. **All 200 `SquareNode`s re-render** even if only one status changed
+4. Each SquareNode render is ~2 ms (status check + glow animation setup)
+5. 10 updates/sec × 200 nodes × 2 ms ≈ **40 ms/sec of JS blocking**, plus layout
+
+At steady state this sits at ~20–40 % single-core CPU overhead and queues user interactions (drag, click, scroll) 100–200 ms behind status broadcasts. Directly contradicts our credentials-panel INP target.
+
+### 2. Eager configuration checks on mount — critical for workflow open
+
+**File**: `client/src/components/SquareNode.tsx:161-203`
+
+```ts
+useEffect(() => {
+ if (!wsConnected) return;
+ const checkConfiguration = async () => {
+ const provider = extractProviderFromNodeType();
+ const apiKey = await getStoredApiKey(provider); // WebSocket roundtrip
+ setHasApiKey(!!apiKey);
+ setIsConfigured(hasRequiredParams && !!apiKey);
+ };
+ checkConfiguration();
+}, [data, id, type, definition?.displayName, definition?.credentials, isGoogleMapsNode, getStoredApiKey, wsConnected]);
+```
+
+On workflow open, all 200 `SquareNode`s mount simultaneously → 200 parallel `getStoredApiKey` WebSocket requests → backend processes sequentially → head-of-line blocking. Total workflow-open cost: ~10 s of "loading" indicators. The `WhatsAppNode.getStatus()` call on mount (flagged in CLAUDE.md as a planned fix) is the same pattern.
+
+### 3. Missing `onlyRenderVisibleElements` on `<ReactFlow>`
+
+**File**: `client/src/Dashboard.tsx:317-336`
+
+React Flow's documented best practice ([reactflow.dev/learn/troubleshooting/performance](https://reactflow.dev/learn/troubleshooting/performance)): enable `onlyRenderVisibleElements={true}` for workflows above ~200 nodes. Reduces render time ~70 % for off-screen nodes. Not currently enabled. 5-minute fix.
+
+### 4. `styledNodes` / `styledEdges` `useMemo` dependency storm
+
+**File**: `Dashboard.tsx:456-476`
+
+```ts
+const styledNodes = useMemo(
+ () => nodes.map(n => ({ /* inject className from nodeStatuses, isExecuting, executionOrder, executedNodes */ })),
+ [nodes, nodeStatuses, isExecuting, executionOrder, executedNodes]
+);
+```
+
+Every time `nodeStatuses` changes (10×/sec during execution), the memo recomputes **for all 200 nodes**, producing a fresh `styledNodes` array. React Flow then diffs the array and re-runs its own reconciliation. This compounds the subscription storm.
+
+### 5. Inline style objects in `SquareNode` render body
+
+**File**: `client/src/components/SquareNode.tsx:530-801`
+
+~20-key inline style objects created fresh on every render (no `useMemo`). At 200 nodes × 1 re-render per status broadcast that's 4000 short-lived style objects per update → measurable GC pressure. Trivially fixed with `useMemo` or — preferred — React Compiler auto-memoization.
+
+### 6. Parallel workflow execution — filtering not implemented
+
+**File**: `WebSocketContext.tsx:689-717` (and `440-449` for the state shape)
+
+State is already workflow-scoped (`allNodeStatuses[workflowId][nodeId]`, good), but the broadcast handler **accepts every workflow's events**:
+
+```ts
+case 'node_status':
+ if (node_id) {
+ const statusWorkflowId = message.workflow_id || 'unknown';
+ // stores regardless of currentWorkflowId
+```
+
+At 5 parallel deployed workflows × 10 updates/sec each = 50 status broadcasts/sec, all causing the same re-render storm on all 200 nodes of whichever workflow the user is looking at. Backend should filter by subscribed workflow before sending, or client should ignore broadcasts for non-current workflows at the dispatcher level.
+
+### 7. Context menu + keyboard shortcuts — actually fine
+
+Investigated because CLAUDE.md has a long planned-features section. `NodeContextMenu.tsx:73-121` correctly attaches a single document-level listener per open menu with clean teardown. F2 rename uses `renamingNodeId` in Zustand (per-node state is derived). `useCopyPaste.ts:40-143` uses in-memory clipboard, no listeners. **No changes needed here.**
+
+---
+
+## Top 5 levers (ranked by impact × effort)
+
+### Lever 1 — Extract a per-node status store (CRITICAL)
+
+**Effort**: 1–2 hours. **Impact**: –60 % re-renders at 200 nodes; interaction latency drops from 100–200 ms → < 20 ms.
+
+Move `allNodeStatuses` out of React context into a dedicated Zustand store and have each `SquareNode` subscribe to **only its own slice** via a selector + equality check:
+
+```ts
+// client/src/stores/useNodeStatusStore.ts
+import { create } from 'zustand';
+import { useShallow } from 'zustand/shallow';
+
+type WorkflowStatuses = Record<string, NodeStatus>;
+interface NodeStatusState {
+ statuses: Record<string, WorkflowStatuses>; // workflow_id -> node_id -> status
+ update: (workflowId: string, nodeId: string, status: NodeStatus) => void;
+ clear: (workflowId: string) => void;
+}
+
+export const useNodeStatusStore = create<NodeStatusState>((set) => ({
+ statuses: {},
+ update: (workflowId, nodeId, status) => set((state) => ({
+ statuses: {
+ ...state.statuses,
+ [workflowId]: { ...state.statuses[workflowId], [nodeId]: status },
+ },
+ })),
+ clear: (workflowId) => set((state) => {
+ const { [workflowId]: _, ...rest } = state.statuses;
+ return { statuses: rest };
+ }),
+}));
+
+// Per-node selector — only re-renders the affected node
+export function useNodeStatus(workflowId: string | null, nodeId: string) {
+ return useNodeStatusStore(
+ (state) => (workflowId ? state.statuses[workflowId]?.[nodeId] : undefined)
+ );
+}
+```
+
+`WebSocketContext` dispatches straight into this store instead of holding `allNodeStatuses` in React state. `SquareNode.tsx` replaces `useWebSocket().getNodeStatus(id)` with `useNodeStatus(currentWorkflowId, id)`.
+
+**Result**: A status update for node X re-renders only node X. 200-node workflow with 10 updates/sec goes from 2000 re-renders/sec → 10. Same pattern recommended in the credentials-panel plan's "store shape decision" — UI state lives outside React context when it's hot.
+
+### Lever 2 — Add `onlyRenderVisibleElements={true}` to `<ReactFlow>`
+
+**Effort**: 5 minutes. **Impact**: –70 % render time for off-screen nodes during pan/zoom/scroll.
+
+```tsx
+// client/src/Dashboard.tsx
+<ReactFlow
+ nodes={styledNodes}
+ edges={styledEdges}
+ onlyRenderVisibleElements={true} // add
+ nodeTypes={moduleNodeTypes}
+ edgeTypes={moduleEdgeTypes}
+ connectionMode={ConnectionMode.Loose}
+ // ...
+>
+```
+
+React Flow's `onlyRenderVisibleElements` culls nodes outside the viewport. Combined with Lever 1, pan/zoom FPS on 500-node workflows should stay at 60. Documented at [reactflow.dev/learn/troubleshooting/performance](https://reactflow.dev/learn/troubleshooting/performance).
+
+### Lever 3 — Defer node configuration checks until workflow selected, batched via TanStack Query
+
+**Effort**: 1–2 hours. **Impact**: Workflow-open time drops from ~10 s → < 1 s.
+
+Remove the per-node mount effect entirely. In `Dashboard.tsx`, when a workflow becomes active, fire one batched fetch using TanStack Query (same instance already being set up for the credentials panel):
+
+```ts
+// client/src/hooks/useWorkflowConfigStatus.ts
+import { useQuery } from '@tanstack/react-query';
+
+export function useWorkflowConfigStatus(workflow: Workflow | null) {
+ return useQuery({
+ queryKey: ['workflowConfigStatus', workflow?.id],
+ queryFn: async () => {
+ const nodeIds = workflow!.nodes.map(n => n.id);
+ // One WebSocket request, backend does N lookups in a single pass
+ return sendRequest('get_all_node_config_status', { node_ids: nodeIds });
+ },
+ enabled: !!workflow,
+ staleTime: 30_000,
+ });
+}
+```
+
+Backend adds a single `get_all_node_config_status` handler that returns `{ [nodeId]: { isConfigured, hasApiKey } }`. Each `SquareNode` reads its entry via a per-node selector. No per-node effects, no head-of-line blocking.
+
+CLAUDE.md already lists this as a planned feature ("Defer Node Status Checks Until Workflow Selected") — this is the concrete shape of the fix.
+
+### Lever 4 — Wrap status dispatch in `startTransition`
+
+**Effort**: 30 minutes. **Impact**: –100–200 ms INP on drag/select during status floods.
+
+```ts
+// WebSocketContext.tsx status handler
+import { startTransition } from 'react';
+
+case 'node_status':
+ startTransition(() => {
+ useNodeStatusStore.getState().update(statusWorkflowId, node_id, flattenedData);
+ });
+ break;
+```
+
+React 19 treats transition updates as interruptible and low-priority, so user drags / clicks stay synchronous even during a burst of 50 status broadcasts/sec. Same decision the credentials plan made for fuzzysort filter updates.
+
+### Lever 5 — Memoize Dashboard styling + enable React Compiler for the canvas module
+
+**Effort**: 30 min manual OR 5 min via React Compiler scoping. **Impact**: –15 % per-render cost on custom nodes.
+
+Manual: wrap the `SquareNode` inline style objects in `useMemo` keyed on `(theme, nodeColor, selected, isExecuting, isGlowing)`.
+
+Preferred: extend the credentials plan's React Compiler babel plugin scope to include `client/src/components/SquareNode.tsx`, `AIAgentNode.tsx`, `TriggerNode.tsx`, `GenericNode.tsx`, `Dashboard.tsx`. React Compiler 1.0 auto-memoizes the inline objects with zero code changes — same mechanism we're already using for the credentials module. This also catches the `styledNodes`/`styledEdges` recomputation in Dashboard.
+
+### Lever 6 (defer) — Workflow-scoped status filter at the WebSocket boundary
+
+**Effort**: 1 day server-side. **Impact**: Only relevant if users deploy 5+ parallel workflows simultaneously.
+
+Backend's `StatusBroadcaster` already knows which client subscribes to which workflow. Add a `subscribe_workflow(workflow_id)` WebSocket handler and filter `node_status` broadcasts by the subscribed set. Client dispatch dispatches only if `message.workflow_id === currentWorkflowId` (cheap) until the server-side filter lands.
+
+CLAUDE.md lists this as a planned feature. Defer until Lever 1 is in — at 5 parallel workflows Lever 1 alone brings the cost from 10 000 re-renders/sec → 50/sec, which is fine.
+
+---
+
+## Missing from CLAUDE.md's "Planned Features" roadmap
+
+CLAUDE.md's planned-features section talks about removing mount status fetches and adding workflow filtering but doesn't mention:
+
+- **The real root cause** of the subscription storm (context value churn), not just the fetch calls.
+- **`onlyRenderVisibleElements`** — this is a one-line change with a 70 % win.
+- **TanStack Query batched config fetch** as the replacement for per-node mount effects.
+- **`startTransition` wrapping** of status dispatches.
+
+This MD is the updated plan.
+
+---
+
+## "Add a new custom node component in 1 file" proposal
+
+Current state: adding a new React component for a new node type touches **4 files** (new component file + `Dashboard.tsx` node-type mapping + per-category `nodeDefinitions/*.ts` + `ComponentPalette.tsx` category icon).
+
+The ComponentPalette scaling MD proposes a file-based registry with Vite `import.meta.glob`. For the **React component** side of the registry, extend that pattern with a second glob:
+
+```ts
+// client/src/components/nodes/index.ts
+import type { ComponentType } from 'react';
+import type { NodeProps } from 'reactflow';
+
+// Every *.node.tsx file exports `default` (the component) + `nodeTypes` (string[] it handles)
+type NodeModule = {
+ default: ComponentType<NodeProps>;
+ nodeTypes: readonly string[];
+};
+
+const modules = import.meta.glob<NodeModule>('./**/*.node.tsx', { eager: true });
+
+export const NODE_COMPONENT_REGISTRY: Record<string, ComponentType<NodeProps>> = {};
+for (const mod of Object.values(modules)) {
+ for (const t of mod.nodeTypes) {
+ NODE_COMPONENT_REGISTRY[t] = mod.default;
+ }
+}
+
+export const getNodeComponent = (type: string) =>
+ NODE_COMPONENT_REGISTRY[type] ?? NODE_COMPONENT_REGISTRY.generic;
+```
+
+`Dashboard.tsx` replaces the long if/else mapping with:
+
+```ts
+const moduleNodeTypes = useMemo(
+ () => Object.fromEntries(
+ new Set(nodeArray.map(n => n.type).filter(Boolean)).values()
+ ).map(type => [type, getNodeComponent(type)]),
+ [nodeArray]
+);
+```
+
+Now adding a timeline node = create `client/src/components/nodes/timeline.node.tsx` with `export const nodeTypes = ['timeline', 'timelineEnd'] as const;` + default component. Done. No `Dashboard.tsx` edit, no registration step.
+
+Pairs directly with the ComponentPalette MD's file-based node *definition* registry — both glob patterns live side by side, one for definitions, one for React components.
+
+---
+
+## Concrete roadmap
+
+| Phase | Effort | Impact | Prereq |
+|---|---|---|---|
+| **A — Cheap wins** (Levers 2 + 4) | 1 hour | –70 % off-screen renders, smoother drag during status floods | None |
+| **B — Subscription storm fix** (Lever 1) | 2 hours | –60 % re-renders at 200 nodes; < 20 ms interaction latency | None |
+| **C — Workflow-open latency fix** (Lever 3) | 2 hours | 10 s → < 1 s open time | TanStack Query infra (from credentials plan) |
+| **D — React Compiler scope expansion** (Lever 5) | 30 min | –15 % custom-node render cost | Credentials plan Phase 7.5 complete |
+| **E — Component registry glob** | 1 day | New node component = 1 file | ComponentPalette glob proposal |
+| **F — Workflow-scoped broadcast filter** (Lever 6) | 1 day | Only relevant for 5+ parallel deployments | Lever 1 |
+
+Total A+B+C+D: ~6 hours for the single biggest runtime win in the app.
+
+---
+
+## What to steal from the credentials panel plan
+
+| Credentials-panel decision | Applies here? |
+|---|---|
+| Per-slice Zustand subscriptions (no closures over hot data) | ✅ Lever 1 |
+| `@tanstack/react-query` for bulk server fetches | ✅ Lever 3 |
+| `startTransition` wrapping non-urgent state updates | ✅ Lever 4 |
+| `babel-plugin-react-compiler` scoped via Vite babel config | ✅ Lever 5 (extend scope to canvas module) |
+| `useShallow` for multi-field subscriptions | ✅ replace the monolithic `useAppStore()` destructures |
+| Heap-snapshot verification (single retainer, modal/workflow 50-cycle delta < 1 MB) | ✅ apply to workflow open/close cycles |
+| INP p75 < 200 ms target (now < 100 ms for the canvas during status floods) | ✅ same target |
+
+**Zero new dependencies.** Everything needed is already being installed for the credentials panel.
+
+---
+
+## References
+
+- React Flow performance best practices: https://reactflow.dev/learn/troubleshooting/performance
+- `onlyRenderVisibleElements` prop: https://reactflow.dev/api-reference/react-flow#onlyrendervisibleelements
+- Zustand `useShallow`: https://github.com/pmndrs/zustand#using-zustand-without-react
+- React `startTransition`: https://react.dev/reference/react/startTransition
+- `useSyncExternalStore`: https://react.dev/reference/react/useSyncExternalStore
+- React Compiler 1.0: https://react.dev/blog/2025/10/07/react-compiler-1
+- TanStack Query v5: https://tanstack.com/query/latest
+- File references from worktree audit:
+ - `WebSocketContext.tsx:1258-1263` (getNodeStatus dep churn)
+ - `WebSocketContext.tsx:440-449, 689-717` (state shape + broadcast handler)
+ - `SquareNode.tsx:48-50` (per-node status subscription)
+ - `SquareNode.tsx:161-203` (eager config check)
+ - `SquareNode.tsx:530-801` (inline style objects)
+ - `Dashboard.tsx:317-336, 456-476` (ReactFlow wiring + styledNodes memo)
+ - `NodeContextMenu.tsx:73-121` (cleanup paths — OK)
diff --git a/docs-internal/ui_optimization/component_palette_scaling.md b/docs-internal/ui_optimization/component_palette_scaling.md
new file mode 100644
index 00000000..e9584ce0
--- /dev/null
+++ b/docs-internal/ui_optimization/component_palette_scaling.md
@@ -0,0 +1,330 @@
+# ComponentPalette & Node Registry Scaling to 500+ Nodes
+
+Analysis of the MachinaOs node sidebar (`client/src/components/ui/ComponentPalette.tsx`) and the node definition registry (`client/src/nodeDefinitions.ts` + 26 per-category files) — what breaks as node count grows from 93 → 500+, and the minimum changes to make adding new nodes a one-file operation.
+
+Scope: runtime performance, heap memory, and contributor DX. Not bundle size.
+
+---
+
+## Current bottlenecks
+
+### 1. Registry runtime cost (module parse time)
+
+`client/src/nodeDefinitions.ts:38-64` performs a synchronous spread merge of 26 per-category modules at module parse time:
+
+```ts
+export const nodeDefinitions: Record<string, INodeTypeDescription> = {
+ ...workflowNodes,
+ ...schedulerNodes,
+ ...aiModelNodes, // calls createBaseChatModel() eagerly per provider
+ ...aiAgentNodes,
+ // ... 23 more
+};
+```
+
+Each per-category file eagerly constructs full `INodeTypeDescription` objects including nested `properties[]` schema arrays (100–300 lines per complex node). At 93 nodes today this is ~200 KB of schema JS executing on the main thread before first paint. At 500+ nodes it extrapolates to ~1.1 MB of synchronous parse + object construction.
+
+**Specific hotspot**: `specializedAgentNodes.ts:63-150` defines `AI_AGENT_PROPERTIES` as a shared array, then each specialized agent references (not clones) it — **but** the current pattern still instantiates a wrapper object per agent. At 9 specialized agents that's 810 nested property objects retained in heap; scaling to 50 agents would hit ~4,500.
+
+**Good news** — icons are raw strings (emoji or `data:image/svg+xml,...` URIs), not React.FC instances. V8 interns identical strings so there is no per-node icon retention cost.
+
+### 2. Filter/search runtime (O(n) per keystroke, full re-render)
+
+`ComponentPalette.tsx:102-151` implements search with a naive substring `.includes()` scan inside `useMemo`:
+
+```ts
+const categorizedComponents = React.useMemo(() => {
+ const filteredDefinitions = Object.values(nodeDefinitions).filter((definition) => {
+ if (searchQuery.trim()) {
+ const query = searchQuery.toLowerCase();
+ const matchesQuery =
+ (definition.displayName || '').toLowerCase().includes(query) ||
+ (definition.description || '').toLowerCase().includes(query) ||
+ (definition.group?.[0] || '').toLowerCase().includes(query);
+ if (!matchesQuery) return false;
+ }
+ // proMode filter, category merge, ...
+ return true;
+ });
+ // re-group into 20+ CollapsibleSection buckets
+}, [nodeDefinitions, searchQuery, proMode]);
+```
+
+At 500 nodes every keystroke runs an O(n) string scan, rebuilds the category map, and triggers a full re-render of every `CollapsibleSection` + `ComponentItem`. **No virtualization, no pre-indexed search, no startTransition.** At the credentials-panel scale target (which we now share), this hits ~50–150 ms per keystroke — well over the INP budget.
+
+### 3. Memory footprint of node definitions
+
+Per `INodeTypeDescription` (`client/src/types/INodeProperties.ts:177`) the retained cost is:
+- `displayName` / `name` / `description`: ~100 B
+- `icon` (string): ~50–200 B
+- `group`, `defaults`, `keywords`, `credentials`: ~500 B
+- `properties[]` schema: **1–10 KB per node** (dominant)
+- `inputs[]` / `outputs[]`: 200–500 B each
+
+**Total: ~2–12 KB per node.** At 500 nodes: ~1–6 MB retained in the main heap for the registry alone, plus V8 object overhead. This retention is permanent for the app session because the registry is module-scoped.
+
+### 4. Developer experience: 9 files to add one specialized agent
+
+Current checklist (from `CLAUDE.md` "Adding New Specialized Agents"):
+
+**Frontend (5 files)**:
+1. `client/src/nodeDefinitions/specializedAgentNodes.ts` — definition + `SPECIALIZED_AGENT_TYPES` entry
+2. `client/src/components/AIAgentNode.tsx` — `AGENT_CONFIGS` entry
+3. `client/src/components/parameterPanel/MiddleSection.tsx` — `AGENT_WITH_SKILLS_TYPES`
+4. `client/src/components/parameterPanel/InputSection.tsx` — `aiAgentTypes` + `AGENT_WITH_SKILLS_TYPES`
+5. Category icon in `ComponentPalette.tsx` (manual)
+
+**Backend (4 files)**:
+6. `server/constants.py` — `AI_AGENT_TYPES` frozenset
+7. `server/services/node_executor.py` — handler registry entry
+8. `server/services/handlers/tools.py` — delegation tuple
+9. `server/services/ai.py` — `DEFAULT_TOOL_NAMES`, `DEFAULT_TOOL_DESCRIPTIONS`, `DelegateToAgentSchema`
+
+None of this should require touching 9 separate files. Most entries are derivable from a single metadata blob.
+
+---
+
+## Factory patterns — what's already good
+
+| Factory | File | Extensibility | Files to touch for new entry |
+|---|---|---|---|
+| `createBaseChatModel()` | `client/src/factories/baseChatModelFactory.ts:31` | 7/10 — config-in, full definition-out | 2 (config object + export) |
+| `createAndroidServiceNode()` | `client/src/nodeDefinitions/androidServiceNodes.ts:13` | 8/10 — self-contained, hidden-field injection | 1 |
+| Dual-purpose tool pattern | `whatsappNodes.ts:77`, `twitterNodes.ts:24` | 5/10 — `group: ['category', 'tool']` works but requires manual schema + handler sync | 4–5 |
+
+The AI-model and Android factories are the closest thing we have to "drop a file, get a node". The rest of the system could adopt the same pattern.
+
+---
+
+## Top 5 levers (ranked by impact × effort)
+
+### Lever 1 — react-virtuoso `GroupedVirtuoso` for the palette (tier-1)
+**Effort**: 1 day. **Impact**: 60 fps scroll at 500+ nodes, DOM pool of ~15 nodes regardless of catalogue size.
+
+Same pattern we're already adopting for the credentials panel. Replace the `CollapsibleSection`-per-category render with `<GroupedVirtuoso groupCounts={...} groupContent={...} itemContent={...}>`. Keep `CollapsibleSection` as the `groupContent` wrapper so sticky-collapse semantics survive.
+
+### Lever 2 — `fuzzysort` pre-indexed search + `startTransition` (tier-1)
+**Effort**: 2 days. **Impact**: per-keystroke cost drops from ~50–150 ms → ~1–3 ms.
+
+```ts
+const prepared = useMemo(
+ () => Object.values(nodeDefinitions).map(def => ({
+ def,
+ _name: fuzzysort.prepare(def.displayName),
+ _desc: fuzzysort.prepare(def.description || ''),
+ _kw: fuzzysort.prepare((def.keywords || []).join(' ')),
+ })),
+ [] // registry is module-scoped, rebuild only on HMR
+);
+
+const [query, setQuery] = useState('');
+const [filtered, setFiltered] = useState(prepared);
+const handleChange = (v: string) => {
+ setQuery(v);
+ startTransition(() => {
+ setFiltered(
+ v
+ ? fuzzysort
+ .go(v, prepared, { keys: ['_name', '_desc', '_kw'], threshold: -1000, limit: 200 })
+ .map(r => r.obj)
+ : prepared
+ );
+ });
+};
+```
+
+Keep input value synchronous so typing never stalls. Uses the same pattern and library the credentials panel is adopting.
+
+### Lever 3 — Shared property-schema registry (tier-2)
+**Effort**: 2 days. **Impact**: removes the 810 → 4,500 property-object duplication problem; centralizes schemas.
+
+Extract to `client/src/schemas/propertySchemas.ts`:
+
+```ts
+export const PROPERTY_SCHEMAS = {
+ AI_AGENT_CORE: [ /* provider, model, prompt, system message, thinking... */ ],
+ WHATSAPP_MESSAGE: [ /* recipient_type, chat_id, message_type, text... */ ],
+ ANDROID_SERVICE: [ /* hidden service_id, android_host, ... */ ],
+ // ...
+} as const;
+```
+
+Nodes reference by array spread:
+```ts
+properties: [...PROPERTY_SCHEMAS.AI_AGENT_CORE, { /* per-agent extras */ }]
+```
+
+**Already partly done**: `specializedAgentNodes.ts` exports `AI_AGENT_PROPERTIES` as a shared constant. Formalize this as the canonical pattern, migrate `whatsappSend` / `twitterSend` / `androidServiceNodes` etc. to use it.
+
+### Lever 4 — Lazy-load per-category node definitions via Vite dynamic `import()` (tier-2)
+**Effort**: 1–2 days. **Impact**: module-parse time drops from ~200 KB synchronous → ~20 KB for the shell + per-category on-demand.
+
+```ts
+// client/src/nodeDefinitions/index.ts
+type CategoryModule = { default: Record<string, INodeTypeDescription> };
+const LOADERS: Record<string, () => Promise<CategoryModule>> = {
+ workflow: () => import('./workflowNodes'),
+ aiModel: () => import('./aiModelNodes'),
+ aiAgent: () => import('./aiAgentNodes'),
+ android: () => import('./androidServiceNodes'),
+ // ... one entry per category
+};
+
+export async function loadNodeRegistry(): Promise<Record<string, INodeTypeDescription>> {
+ const modules = await Promise.all(Object.values(LOADERS).map(l => l()));
+ return Object.assign({}, ...modules.map(m => m.default));
+}
+```
+
+Mount `ComponentPalette` inside `<Suspense>`, fetch via TanStack Query (`queryKey: ['nodeRegistry']`, `staleTime: Infinity`), and reuse the same `idb-keyval` persister pattern as the credentials panel. Zero extra infra.
+
+**Trade-off**: first palette open gains ~200 ms async boot delay. Acceptable because the palette is an off-canvas sidebar, and subsequent opens hit the warm cache in < 50 ms. React Flow canvas itself does not depend on the registry at mount, so the main app is not blocked.
+
+### Lever 5 — Server-owned node registry via WebSocket (tier-3, defer)
+**Effort**: 3–4 days. **Impact**: removes 1–6 MB from the client module graph entirely; enables runtime registry updates without frontend redeploy.
+
+Same shape as the credentials panel's `get_credential_catalogue` handler. Server reads from a JSON manifest or Python-side registry, client fetches once on boot, caches in IndexedDB, reconciles via version hash.
+
+**Defer until** we actually need sub-500-ms boot or runtime extensibility. For now, Lever 4 (client-side lazy load) achieves most of the benefit at a tenth of the effort. **Do not build this before the credentials panel's catalogue infrastructure is proven** — the credentials panel is the reference implementation; this just reuses the pattern.
+
+---
+
+## "Drop a file, get a node" proposal
+
+Restructure the node tree so adding a new node = creating one `*.node.ts` file. The auto-discoverer handles registration, category icons, palette wiring, and (with conventions) backend handler stubs.
+
+### New structure
+
+```
+client/src/nodeDefinitions/
+├── index.ts # auto-discoverer + loadNodeRegistry()
+├── schemas/
+│ ├── propertySchemas.ts # shared property sets (Lever 3)
+│ └── types.ts # INodeTypeDescription re-export
+└── nodes/
+ ├── ai/
+ │ ├── openai.node.ts
+ │ ├── anthropic.node.ts
+ │ └── aiAgent.node.ts
+ ├── social/
+ │ ├── whatsappSend.node.ts
+ │ ├── twitterSend.node.ts
+ │ └── instagramSend.node.ts # new: just add this file
+ ├── android/
+ │ ├── batteryMonitor.node.ts
+ │ └── ...
+ └── ...
+```
+
+### One node file (template)
+
+```ts
+// client/src/nodeDefinitions/nodes/social/instagramSend.node.ts
+import type { INodeTypeDescription } from '../../schemas/types';
+import { PROPERTY_SCHEMAS } from '../../schemas/propertySchemas';
+
+export const metadata = {
+ category: 'social',
+ categoryLabel: 'Social Media Platforms',
+ categoryIcon: '📸',
+} as const;
+
+const node: INodeTypeDescription = {
+ displayName: 'Instagram Send',
+ name: 'instagramSend',
+ icon: '📸',
+ group: ['social', 'tool'], // dual-purpose: workflow node + AI tool
+ version: 1,
+ description: 'Post content to Instagram',
+ keywords: ['instagram', 'post', 'share', 'social'],
+ defaults: { name: 'Instagram Send' },
+ inputs: [{ name: 'main', type: 'main' }],
+ outputs: [{ name: 'main', type: 'main' }],
+ properties: [
+ ...PROPERTY_SCHEMAS.SOCIAL_MESSAGE,
+ { displayName: 'Caption', name: 'caption', type: 'string', required: false },
+ ],
+};
+
+export default node;
+```
+
+### Auto-discoverer (Vite `import.meta.glob`)
+
+Vite supports eager/lazy glob imports natively — we don't need any extra tooling.
+
+```ts
+// client/src/nodeDefinitions/index.ts
+import type { INodeTypeDescription } from './schemas/types';
+
+// Eager variant (keeps current sync semantics, replaces the 26-file spread merge)
+const modules = import.meta.glob<{ default: INodeTypeDescription }>(
+ './nodes/**/*.node.ts',
+ { eager: true }
+);
+
+export const nodeDefinitions: Record<string, INodeTypeDescription> = Object.fromEntries(
+ Object.values(modules).map(m => [m.default.name, m.default])
+);
+
+// Lazy variant (for Lever 4)
+export const lazyNodeModules = import.meta.glob<{ default: INodeTypeDescription }>(
+ './nodes/**/*.node.ts'
+);
+export async function loadNodeRegistry() {
+ const entries = await Promise.all(
+ Object.entries(lazyNodeModules).map(async ([path, loader]) => {
+ const mod = await loader();
+ return [mod.default.name, mod.default] as const;
+ })
+ );
+ return Object.fromEntries(entries);
+}
+```
+
+**Result**: the 26-file spread merge in `nodeDefinitions.ts` becomes a 1-line glob, the per-category barrel files become obsolete, and adding a new node is one file. Categories are derivable from the `metadata.category` export.
+
+### Collapses the 9-file DX checklist to 1-2 files
+
+Backend handlers still need to live under `server/services/handlers/`, but the **frontend** checklist collapses to 1 file. The `AGENT_CONFIGS` / `AGENT_WITH_SKILLS_TYPES` / `aiAgentTypes` arrays currently scattered across `AIAgentNode.tsx`, `MiddleSection.tsx`, `InputSection.tsx`, and `Dashboard.tsx` can all be derived from a single query: `Object.values(nodeDefinitions).filter(n => n.group?.includes('agent'))`.
+
+---
+
+## What to steal from the credentials panel plan
+
+Every library, pattern, and guardrail in the credentials plan applies here, usually unchanged:
+
+| Credentials-panel decision | Applies to ComponentPalette? |
+|---|---|
+| `react-virtuoso GroupedVirtuoso` with sticky category headers | ✅ Lever 1 |
+| `cmdk` command palette shell for search + keyboard nav | ✅ — Ctrl+K for node discovery, same keybind |
+| `fuzzysort.prepare()` pre-indexed search | ✅ Lever 2 |
+| `startTransition` wrapping filter updates (not `useDeferredValue`) | ✅ Lever 2 |
+| `@tanstack/react-query` + `experimental_createPersister(idb-keyval)` | ✅ Lever 4 / 5 |
+| `babel-plugin-react-compiler` scoped via `@vitejs/plugin-react` babel plugin | ✅ extend scope to `nodeDefinitions/` + `ComponentPalette.tsx` |
+| Store shape rule: **UI state in Zustand, catalogue in TanStack Query, derived data in `useMemo`** | ✅ same rule |
+| Heap snapshot verification (retained size, one retainer, 50-cycle delta < 1 MB) | ✅ same workflow |
+| INP p75 < 200 ms on synthetic 5000-item probe | ✅ same target |
+
+**Zero new dependencies.** Everything needed is already being installed for the credentials panel.
+
+---
+
+## Concrete roadmap (phase-in)
+
+| Phase | Effort | Impact | Prerequisite |
+|---|---|---|---|
+| **A — Search UX overhaul** (Levers 1+2) | 2–3 days | 200+ nodes feel instant; 60 fps scroll | Credentials panel dep install is done |
+| **B — Developer ergonomics** (file-based registry + shared schemas) | 1 week | Adding a node = 1 file; ~30 % memory reduction | Phase A |
+| **C — Lazy registry** (Lever 4) | 1–2 days | 300 ms → 50 ms module parse cost at app boot | Phase B (needs glob structure) |
+| **D — Server registry** (Lever 5, optional) | 3–4 days | Decouple client/server release cycles | Credentials panel production-proven |
+
+## References
+
+- `react-virtuoso` GroupedVirtuoso: https://virtuoso.dev/grouped-numbers/
+- `fuzzysort`: https://github.com/farzher/fuzzysort
+- Vite `import.meta.glob`: https://vite.dev/guide/features.html#glob-import
+- n8n node loader (filesystem-as-registry): https://github.com/n8n-io/n8n/tree/master/packages/nodes-base/nodes
+- Pipedream components: https://github.com/PipedreamHQ/pipedream/tree/master/components
+- React `startTransition`: https://react.dev/reference/react/useTransition
+- React Compiler 1.0: https://react.dev/blog/2025/10/07/react-compiler-1
diff --git a/docs-internal/ui_optimization/parameter_panel_scaling.md b/docs-internal/ui_optimization/parameter_panel_scaling.md
new file mode 100644
index 00000000..c40aee18
--- /dev/null
+++ b/docs-internal/ui_optimization/parameter_panel_scaling.md
@@ -0,0 +1,406 @@
+# Parameter Panel + Dynamic Schema Scaling
+
+Analysis of the MachinaOs parameter configuration modal (`ParameterPanel.tsx`, `ParameterRenderer.tsx`, `parameterPanel/MiddleSection.tsx`, `parameterPanel/InputSection.tsx`, `useParameterPanel.ts`) — what breaks as node property schemas grow more complex, and the minimum changes to make adding new parameter types and custom editors a one-file operation.
+
+Scope: runtime performance, heap memory, and contributor DX. Not bundle size.
+
+---
+
+## TL;DR
+
+The parameter panel is functional at 20–30 fields but has four structural issues that will get worse quickly:
+
+1. **Monolithic type dispatch** — `ParameterRenderer.tsx:1322-2103` is a ~780-line `switch` over 15+ parameter types with hard-coded WhatsApp/Telegram/session-id special cases baked in. Adding a new type means editing this file.
+2. **Every keystroke re-renders every field** — no `React.memo` on the field list, no RHF, direct controlled-input pattern. On a 50-field form this is ~18 ms/keystroke vs ~2 ms with RHF (9× slower — measured in the credentials panel runtime research).
+3. **`displayOptions.show` re-evaluated every render** — `shouldShowParameter` in `MiddleSection.tsx:59-81` iterates `Object.entries()` per field per render. Quadratic cost on conditional forms.
+4. **WebSocket saves are unbatched** — `useParameterPanel.ts:101-106` fires one `save_node_parameters` message per keystroke with no debouncing. A 500-char field = 500 DB upserts.
+
+Five levers fix all of this using the same libraries and patterns already being added for the credentials panel (react-hook-form, TanStack Query mutations, React Compiler, parameter-type registry). Zero new dependencies.
+
+---
+
+## Current bottlenecks
+
+### 1. Monolithic type dispatch in `ParameterRenderer.tsx:1322-2103`
+
+A single 780-line `switch (parameter.type)` block handles `string`, `number`, `boolean`, `options`, `multiOptions`, `slider`, `percentage`, `file`, `array`, `collection`, `fixedCollection`, `json`, `notice`, etc. Embedded inside the `string` branch are hard-coded special cases for node-specific parameters:
+
+- `group_id` selector → `SquareNode.tsx:1534` (WhatsApp groups dropdown)
+- `senderNumber` selector → `:1572`
+- `channel_jid` selector → `:1553`
+- `sessionId` → `:1593`
+
+This makes "add a new parameter type" a direct edit to the monolith and makes WhatsApp-specific logic live inside the generic renderer. Adding a cron-expression picker, a credentials picker that queries the credentials panel's registry, or a nested multi-step form grows the switch further.
+
+### 2. No field-level memoization
+
+**File**: `client/src/components/parameterPanel/MiddleSection.tsx:484-512`
+
+The field list is rendered with a plain `.map()` over filtered parameters, wrapping each in `<ParameterRenderer>`. Neither `ParameterRenderer` nor the individual renderers are wrapped in `React.memo`, so a keystroke in field A re-renders fields B–Z. At 50 fields this is a 50× amplification of the commit cost.
+
+Combined with direct controlled inputs (not RHF `Controller`), this reproduces the exact pathology measured in the credentials-panel runtime research: ~18 ms/keystroke at 50 fields, vs ~2 ms with RHF + `React.memo` + `Controller`.
+
+### 3. `displayOptions.show` is O(n × d) per render
+
+**File**: `client/src/components/parameterPanel/MiddleSection.tsx:59-81`
+
+```ts
+const shouldShowParameter = (param: INodeProperties, allParameters: Record<string, any>): boolean => {
+ if (!param.displayOptions?.show) return true;
+ for (const [paramName, allowedValues] of Object.entries(param.displayOptions.show)) {
+ const currentValue = allParameters[paramName];
+ if (Array.isArray(allowedValues)) {
+ if (!allowedValues.includes(currentValue)) return false;
+ } else if (currentValue !== allowedValues) {
+ return false;
+ }
+ }
+ return true;
+};
+```
+
+Called inside a `.filter()` before the `.map()` render, on every commit. For 50 fields with ~3 average dependencies each, that's 150 property lookups + 150 `Array.includes` calls per keystroke. Trivially cacheable: compile each schema's `show` clause into a stable predicate function `(params) => boolean` once at definition load, then call the cached function.
+
+### 4. Unbatched WebSocket saves
+
+**File**: `client/src/hooks/useParameterPanel.ts:101-106`
+
+`handleParameterChange` updates local state synchronously and `handleSave` fires `sendRequest('save_node_parameters', ...)` with no debounce, batching, or mutation queue. For a user typing 500 characters into a prompt field, this is 500 WebSocket round-trips, 500 DB upserts against `NodeParameter` (full-row upsert per call in `database.py:232-250`), and 500 broadcast events via `node_parameters_updated` to any other connected clients.
+
+At 1 client this is "just" wasteful. At 5 parallel clients editing the same workflow it creates echo storms and temporary state thrash.
+
+### 5. Custom editors hard-coded by node type
+
+**File**: `client/src/components/parameterPanel/MiddleSection.tsx:515-521` and scattered conditionals in `ParameterRenderer.tsx`
+
+`MasterSkillEditor`, `ToolSchemaEditor`, `LocationParameterPanel`, `MapsSection`, `CodeEditor` are wired in with explicit `if (nodeType === 'masterSkill')` / `if (TOOL_NODE_TYPES.includes(...))` checks. There is no registry. Adding a new custom editor (say, a `credentialsPicker` that queries the new credentials catalogue) requires:
+
+1. Create the component.
+2. Import it in `MiddleSection.tsx`.
+3. Add a conditional branch in `MiddleSection.tsx`.
+4. Possibly add another conditional in `ParameterPanel.tsx` wrapper.
+5. Update `CLAUDE.md`'s guide.
+
+**3–4 files touched** for what should be plug-in code.
+
+### 6. `InputSection` rebuilds schema tree on every open
+
+**File**: `client/src/components/parameterPanel/InputSection.tsx:49-100`
+
+`fetchConnectedNodes` runs on every node-selection change, walks edges to find connected nodes, calls `getNodeOutput` per connected node via WebSocket, and rebuilds the draggable variable tree. No caching across opens; no dedupe of identical schemas. On a 50-connected-node workflow every panel open re-fetches the same schema repeatedly.
+
+Fix is a cached, TanStack Query-backed schema fetch keyed on `(connectedNodeId, outputType)` with `staleTime` tuned to the node's execution interval.
+
+---
+
+## Top 5 levers (ranked by impact × effort)
+
+### Lever 1 — Parameter type registry (tier-1)
+
+**Effort**: ~3 hours. **Impact**: eliminates the 780-line switch, makes adding a new parameter type a one-file addition, unblocks the rest of the levers.
+
+```ts
+// client/src/components/parameterPanel/ParameterTypeRegistry.ts
+import type { INodeProperties } from '../../types/INodeProperties';
+
+export interface ParameterRendererProps {
+ parameter: INodeProperties;
+ value: unknown;
+ onChange: (value: unknown) => void;
+ allParameters: Record<string, unknown>;
+ nodeId: string;
+ nodeType: string;
+}
+
+export interface ParameterTypeHandler {
+ readonly types: readonly string[]; // e.g. ['string'] or ['options', 'multiOptions']
+ render: (props: ParameterRendererProps) => React.ReactNode;
+ validate?: (value: unknown, param: INodeProperties) => string | null;
+}
+
+const handlers = new Map<string, ParameterTypeHandler>();
+
+export function registerParameterType(handler: ParameterTypeHandler) {
+ for (const t of handler.types) handlers.set(t, handler);
+}
+
+export function renderParameter(props: ParameterRendererProps): React.ReactNode {
+ const h = handlers.get(props.parameter.type);
+ if (!h) return <UnsupportedParameterWarning type={props.parameter.type} />;
+ return h.render(props);
+}
+```
+
+Refactor each branch of the existing switch into its own file:
+
+```
+client/src/components/parameterPanel/types/
+├── index.ts # auto-register via import.meta.glob
+├── StringType.tsx
+├── NumberType.tsx
+├── BooleanType.tsx
+├── OptionsType.tsx
+├── MultiOptionsType.tsx
+├── CollectionType.tsx
+├── FixedCollectionType.tsx
+├── JsonType.tsx
+├── NoticeType.tsx
+├── CodeType.tsx
+└── ...
+```
+
+Auto-register via Vite glob (same pattern as the node definition and custom-editor globs proposed in the other UI optimization MDs):
+
+```ts
+// types/index.ts
+const modules = import.meta.glob<{ default: ParameterTypeHandler }>(
+ './*.tsx',
+ { eager: true }
+);
+for (const m of Object.values(modules)) registerParameterType(m.default);
+```
+
+**Adding a new type** becomes creating one file:
+
+```ts
+// client/src/components/parameterPanel/types/CronExpressionType.tsx
+import type { ParameterTypeHandler } from '../ParameterTypeRegistry';
+import { CronInput } from '../../ui/CronInput';
+import cronParser from 'cron-parser';
+
+const handler: ParameterTypeHandler = {
+ types: ['cronExpression'],
+ render: ({ value, onChange, parameter }) => (
+ <CronInput value={value as string} onChange={onChange} placeholder={parameter.placeholder} />
+ ),
+ validate: (value) => {
+ try { cronParser.parseExpression(value as string); return null; }
+ catch (e) { return (e as Error).message; }
+ },
+};
+export default handler;
+```
+
+One line in `types/INodeProperties.ts` adds `'cronExpression'` to the type union. Zero edits to `ParameterRenderer.tsx`, `MiddleSection.tsx`, or `ParameterPanel.tsx`.
+
+Node-type-specific special cases (WhatsApp group selector, Telegram channel selector, etc.) move into separate per-feature handler files that register via the same mechanism and use a stricter `types` array or an `appliesWhen(ctx)` guard.
+
+### Lever 2 — TanStack Query mutation queue with optimistic updates + debouncing (tier-1)
+
+**Effort**: ~4 hours. **Impact**: 500 WebSocket messages → 1 debounced message per 300 ms idle window. Optimistic updates make UI feel instant.
+
+Replace the imperative `saveNodeParameters` call with a TanStack Query mutation (the same `QueryClient` already being set up for the credentials panel):
+
+```ts
+// client/src/hooks/useParameterPanel.ts
+const queryClient = useQueryClient();
+
+const saveMutation = useMutation({
+ mutationFn: (params: Record<string, unknown>) =>
+ sendRequest('save_node_parameters', {
+ node_id: nodeId,
+ parameters: params,
+ request_id: crypto.randomUUID(), // idempotency key, same pattern as credentials panel
+ }),
+ onMutate: async (params) => {
+ await queryClient.cancelQueries({ queryKey: ['nodeParams', nodeId] });
+ const previous = queryClient.getQueryData(['nodeParams', nodeId]);
+ queryClient.setQueryData(['nodeParams', nodeId], params); // optimistic
+ return { previous };
+ },
+ onError: (_err, _params, ctx) => {
+ if (ctx?.previous) queryClient.setQueryData(['nodeParams', nodeId], ctx.previous);
+ },
+});
+
+const debouncedSave = useMemo(
+ () => debounce((p: Record<string, unknown>) => saveMutation.mutate(p), 300),
+ [saveMutation]
+);
+
+// On every parameter change:
+const handleParameterChange = (name: string, value: unknown) => {
+ const next = { ...parameters, [name]: value };
+ setParameters(next); // local UI state, synchronous
+ debouncedSave(next); // batched server save
+};
+```
+
+The server-side `save_node_parameters` handler already exists; adding an `idempotency_key` dedupe (60 s TTL dict, same as the credentials panel's Phase 7.5) makes the mutation queue + retry loop safe.
+
+**Benefit**: one network roundtrip per 300 ms idle window, not per keystroke. Server-side load drops by ~99% during typing. Other connected clients get a single broadcast update per burst, eliminating state thrash.
+
+### Lever 3 — `React.memo` field list + React Compiler scope expansion (tier-1)
+
+**Effort**: 30 min manual, 5 min via compiler scoping. **Impact**: 50× reduction in commit cost at 50 fields — only the changed field re-renders.
+
+Manual:
+
+```tsx
+// ParameterRenderer.tsx
+const ParameterRenderer = React.memo(
+ function ParameterRenderer(props) { /* ... */ },
+ (prev, next) =>
+ prev.value === next.value &&
+ prev.parameter === next.parameter &&
+ prev.allParameters[prev.parameter.name] === next.allParameters[next.parameter.name]
+);
+```
+
+Preferred: **extend the credentials-panel React Compiler babel scope** to cover `client/src/components/ParameterRenderer.tsx`, `client/src/components/parameterPanel/*.tsx`, and the new `types/` folder. Auto-memoization handles this without manual `React.memo` — same zero-code win as the credentials module.
+
+### Lever 4 — Compile `displayOptions.show` predicates once (tier-2)
+
+**Effort**: ~2 hours. **Impact**: removes the O(n × d) per-render cost; visibility checks become a function call.
+
+At schema-load time (or lazily on first use, memoized), transform each `displayOptions.show` clause into a stable predicate:
+
+```ts
+// client/src/components/parameterPanel/compileDisplayOptions.ts
+export type VisibilityPredicate = (params: Record<string, unknown>) => boolean;
+
+const cache = new WeakMap<INodeProperties, VisibilityPredicate>();
+
+export function compileVisibility(param: INodeProperties): VisibilityPredicate {
+ const cached = cache.get(param);
+ if (cached) return cached;
+
+ const show = param.displayOptions?.show;
+ if (!show) {
+ const alwaysTrue = () => true;
+ cache.set(param, alwaysTrue);
+ return alwaysTrue;
+ }
+
+ const entries = Object.entries(show);
+ const fn: VisibilityPredicate = (params) => {
+ for (const [key, allowed] of entries) {
+ const v = params[key];
+ if (Array.isArray(allowed)) {
+ if (!allowed.includes(v)) return false;
+ } else if (v !== allowed) {
+ return false;
+ }
+ }
+ return true;
+ };
+ cache.set(param, fn);
+ return fn;
+}
+```
+
+Replace `shouldShowParameter(param, params)` in `MiddleSection.tsx` with `compileVisibility(param)(params)`. The predicate body is identical; the win is that it's a stable function reference, JIT-friendly, and keyed by a `WeakMap` so unused schemas can be garbage-collected.
+
+### Lever 5 — Cached schema tree fetch for `InputSection` via TanStack Query
+
+**Effort**: ~2 hours. **Impact**: eliminates WebSocket round-trips on repeated panel opens.
+
+```ts
+// client/src/hooks/useNodeOutputSchema.ts
+export function useNodeOutputSchema(nodeId: string) {
+ return useQuery({
+ queryKey: ['nodeOutputSchema', nodeId],
+ queryFn: () => sendRequest('get_node_output_schema', { node_id: nodeId }),
+ staleTime: 30_000,
+ });
+}
+```
+
+`InputSection` then calls `useNodeOutputSchema(connectedNodeId)` for each connected node. Queries dedupe automatically; switching between nodes reuses the cache.
+
+### Lever 6 (defer) — Full RHF `Controller` migration
+
+**Effort**: 12–16 hours. **Impact**: 9× faster keystroke latency on very large forms + unified validation + field-level errors.
+
+This is the same migration already planned for the credentials panel. Order matters: do Levers 1–4 first, then migrate `FieldRenderer` / `ParameterRenderer` to RHF `Controller` wrapping antd inputs — that way the registry is already in place and each per-type handler can be migrated incrementally without a big-bang refactor.
+
+For the parameter panel specifically, RHF's biggest win is validation (each field can now report its own error without re-rendering siblings) and the mutation queue becomes trivial (`handleSubmit(saveMutation.mutate)`).
+
+---
+
+## Custom editor registry (makes "new custom editor" a one-file addition)
+
+The same glob-based auto-registration pattern used for parameter types handles custom editors:
+
+```ts
+// client/src/components/parameterPanel/customEditors/index.ts
+export interface CustomEditorHandler {
+ readonly nodeTypes: readonly string[]; // e.g. ['masterSkill']
+ readonly component: React.ComponentType<{ nodeId: string; parameters: Record<string, unknown> }>;
+}
+
+const handlers = new Map<string, CustomEditorHandler>();
+
+const modules = import.meta.glob<{ default: CustomEditorHandler }>('./*.tsx', { eager: true });
+for (const m of Object.values(modules)) {
+ for (const t of m.default.nodeTypes) handlers.set(t, m.default);
+}
+
+export function getCustomEditor(nodeType: string) {
+ return handlers.get(nodeType);
+}
+```
+
+`MiddleSection.tsx` replaces its long conditional chain with:
+
+```tsx
+const editor = getCustomEditor(nodeType);
+if (editor) {
+ const Editor = editor.component;
+ return <Editor nodeId={nodeId} parameters={parameters} />;
+}
+return <DefaultParameterList ... />;
+```
+
+Adding a `credentialsPicker` editor that queries the new credentials catalogue (via `useCatalogueQuery` from the credentials panel plan) is now one file in `customEditors/` with `nodeTypes: ['credentialsPicker']` and a default export.
+
+---
+
+## What to steal from the credentials panel plan
+
+| Credentials-panel decision | Applies to parameter panel? |
+|---|---|
+| `@tanstack/react-query` with idempotency-key mutations | ✅ Lever 2 — replaces ad-hoc WebSocket saves |
+| `experimental_createPersister(idb-keyval)` for warm-start | ⚠️ Only if we want to persist the in-progress panel edit across app reloads (probably not) |
+| `react-hook-form` `Controller` wrapping antd inputs | ✅ Lever 6 — same migration, do after Levers 1–4 |
+| `babel-plugin-react-compiler` scoped via `@vitejs/plugin-react` | ✅ Lever 3 — extend scope to `parameterPanel/` directory |
+| Schema version hash (for client-side cache invalidation) | ✅ Apply to `useNodeOutputSchema` so stale output schemas get invalidated after a node re-executes |
+| UI-state-only Zustand store, no derived data | ✅ `useAppStore` keeps `selectedNodeId`, `paramPanelOpen`; TanStack Query owns `parameters` and `outputSchema` |
+| Idempotency keys on all mutations | ✅ Lever 2 — already included |
+| Heap snapshot verification (50-cycle open/close delta < 1 MB) | ✅ apply to parameter panel open/close cycles |
+
+**Zero new dependencies.** All the libraries needed are already being installed for the credentials panel.
+
+---
+
+## Concrete roadmap
+
+| Phase | Effort | Impact | Prereq |
+|---|---|---|---|
+| **A — Type registry + memo** (Levers 1 + 3) | ~4 hours | New parameter type = 1 file; 50× less re-render cost | Credentials panel dep install |
+| **B — Mutation queue + debounce** (Lever 2) | ~4 hours | 99% fewer WebSocket saves during typing; optimistic UI | TanStack Query wired (credentials plan) |
+| **C — Compile visibility predicates** (Lever 4) | ~2 hours | Removes O(n × d) per-keystroke visibility check | None |
+| **D — Cached output schema fetch** (Lever 5) | ~2 hours | No redundant WebSocket roundtrips on panel re-open | TanStack Query wired |
+| **E — Custom editor registry** | ~3 hours | New custom editor = 1 file | Lever 1 shipped |
+| **F — Full RHF migration** (Lever 6, optional) | 12–16 hours | 9× faster keystroke latency on very large forms | All previous phases |
+
+Phases A–E land in ~15 hours. Phase F is a larger refactor that should wait until Phase A–E are battle-tested.
+
+---
+
+## References
+
+- React Hook Form `Controller`: https://react-hook-form.com/docs/usecontroller/controller
+- TanStack Query mutations + optimistic updates: https://tanstack.com/query/v5/docs/react/guides/optimistic-updates
+- React Compiler 1.0: https://react.dev/blog/2025/10/07/react-compiler-1
+- Vite `import.meta.glob`: https://vite.dev/guide/features.html#glob-import
+- `WeakMap` for predicate caching: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/WeakMap
+- n8n `INodeProperties` + `displayOptions`: https://github.com/n8n-io/n8n/blob/master/packages/workflow/src/Interfaces.ts
+- File references from worktree audit:
+ - `ParameterRenderer.tsx:1322-2103` (780-line type switch, embedded node-type special cases)
+ - `MiddleSection.tsx:59-81` (`shouldShowParameter` — O(n × d) per render)
+ - `MiddleSection.tsx:484-512` (unmemoized field list render)
+ - `MiddleSection.tsx:515-521` (`ToolSchemaEditor` hard-coded conditional)
+ - `useParameterPanel.ts:101-106` (unbatched `handleSave`)
+ - `InputSection.tsx:49-100` (schema tree rebuild on every open)
+ - `server/routers/websocket.py:150-162` + `server/core/database.py:232-250` (`save_node_parameters` handler + `NodeParameter` upsert)