Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
docs: reflect Wave 12.B daemon framework + Stripe OAuth refinements
trohitg55c0b274fec8
diff --git a/CLAUDE.md b/CLAUDE.mdindex bd303090..a44bb550 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -25,7 +25,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire| **[Pricing Service](./docs-internal/pricing_service.md)** | Centralized cost tracking for LLM tokens and API services (Twitter, Google Maps) with HTTPX event hooks || **[Proxy Service](./docs-internal/proxy_service.md)** | Residential proxy provider management with template-based URL formatting, health scoring, and transparent HTTP node injection || **[Email Service](./docs-internal/email_service.md)** | IMAP/SMTP integration via Himalaya CLI with EmailService orchestrator, provider presets, custom credential fallback, and polling triggers |-| **[Stripe Service](./docs-internal/stripe_service.md)** | Stripe CLI integration — `stripeAction` (CLI pass-through) + `stripeReceive` (signed-webhook trigger). Reference plugin for the Wave 12 event framework AND the **CLI-managed-auth pattern**: browser OAuth driven by `stripe login --non-interactive` + auto-installed binary (`ensure_stripe_cli` from GitHub releases) + marker-token reuse of `auth_service.store_oauth_tokens` + generic `credential_catalogue_updated` broadcast — zero per-provider hardcoding in the frontend. |+| **[Stripe Service](./docs-internal/stripe_service.md)** | Stripe CLI integration — `stripeAction` (CLI pass-through) + `stripeReceive` (signed-webhook trigger). Reference plugin for the Wave 12 event framework AND the **CLI-managed-auth pattern**: browser OAuth via `stripe login --non-interactive` + `--complete <url>` (URL extracted from `next_step` shell command via `shlex.split`) + auto-installed binary (`ensure_stripe_cli` from GitHub releases) + marker-token reuse of `auth_service.store_oauth_tokens` + CloudEvents-shaped `broadcast_credential_event("credential.oauth.connected" \| ".disconnected", provider=...)` — zero per-provider hardcoding in the frontend. Daemon stdout/stderr ingested via `ProcessService`'s `line_handler` callback (no log-file tailing); credential gate routed through `await self.has_credential()` so non-api-key auth (`is_logged_in()`) plugs in via subclass override. || **[CI/CD Pipeline](./docs-internal/ci_cd.md)** | GitHub Actions workflows, predeploy validation, release publishing, and composite setup action || **[Workflow Schema](./docs-internal/workflow-schema.md)** | JSON schema for workflows, edge handle conventions, config node architecture || **[Execution Engine Design](./docs-internal/DESIGN.md)** | Architecture patterns, design standards, and implementation details for the workflow execution engine |diff --git a/docs-internal/node_creation.md b/docs-internal/node_creation.mdindex 26800464..5769e227 100644--- a/docs-internal/node_creation.md+++ b/docs-internal/node_creation.md@@ -152,17 +152,19 @@ reused:)```-2. **Generic broadcast on state change.** Plugin emits the existing- `credential_catalogue_updated` event — handled in- [`WebSocketContext.tsx`](../client/src/contexts/WebSocketContext.tsx)- line 671 since before Stripe shipped. The frontend invalidates- the catalogue, refetches, and the modal re-renders. No- per-provider broadcast type, no `case 'stripe_status'` —+2. **CloudEvents-shaped broadcast on state change.** Plugin emits a+ `WorkflowEvent` (CloudEvents v1.0) via the canonical helper —+ wrapped under the existing `credential_catalogue_updated`+ wire-format type. Same shape `save_api_key` / `twitter_logout` /+ `google_logout` use; locked by+ `tests/credentials/test_credential_broadcasts.py`. The frontend+ invalidates the catalogue, refetches, and the modal re-renders.+ No per-provider broadcast type, no `case 'stripe_status'` —**zero node-specific code in the frontend**.```python- await get_status_broadcaster().broadcast(- {"type": "credential_catalogue_updated"}+ await get_status_broadcaster().broadcast_credential_event(+ "credential.oauth.connected", provider="stripe", # or .disconnected on logout)```diff --git a/docs-internal/plugin_system.md b/docs-internal/plugin_system.mdindex 779a209f..3840a1e6 100644--- a/docs-internal/plugin_system.md+++ b/docs-internal/plugin_system.md@@ -1001,21 +1001,32 @@ The shared dispatch path lives in `routers/webhook.py` — it consultsFor plugins that wrap a long-lived CLI tool or SDK loop (Stripe CLI,future GitHub-CLI / Cloudflare-Wrangler / etc.). Delegates lifecycleto `ProcessService` (battle-tested PATHEXT-aware launching, kill_tree-cleanup, log capture):+cleanup, log capture, Terminal-tab broadcast). The base subscribes to+ProcessService's per-line callback hook (`line_handler`), so plugins+just provide the parser:```python+import shlexfrom services.events import DaemonEventSource, WorkflowEventclass StripeListenSource(DaemonEventSource):type = "stripe.listen"process_name = "stripe-listen"- binary_name = "stripe"+ binary_name = "" # see "binary_name" note belowworkflow_namespace = "_stripe"install_hint = "https://stripe.com/docs/stripe-cli#install"credential = StripeCredentialdef build_command(self, secrets: Dict) -> str:- return f"stripe listen --print-secret --api-key {secrets['api_key']}"+ # `shlex.quote` the binary so the path round-trips through+ # ProcessService's POSIX-mode `shlex.split` unchanged.+ binary = shlex.quote(str(stripe_cli_path() or "stripe"))+ return f"{binary} listen --forward-to ... --print-secret"++ async def has_credential(self) -> bool:+ # Override the default ``secrets["api_key"]`` gate when auth lives+ # outside MachinaOs (e.g. the CLI's own ~/.config/stripe/config.toml).+ return is_logged_in()def parse_line(self, stream: str, line: str) -> Optional[WorkflowEvent]:# called per-line for both stdout and stderr@@ -1025,8 +1036,19 @@ class StripeListenSource(DaemonEventSource):```The base provides `start()` / `stop()` / `restart()` / `status()` /-`has_credential()` / lifecycle locking / pre-flight `shutil.which`-check / log tailing.+lifecycle locking / pre-flight `shutil.which` check (skip via+`binary_name = ""` if the plugin resolves the binary itself).++**Output ingestion**: `DaemonEventSource` registers `self._on_line` as+the `line_handler` callback on `ProcessService.start()`. ProcessService+runs the single `stream.readline()` loop per stdout/stderr; on each+decoded line it writes the log file, broadcasts to the Terminal tab,+and invokes the callback — which calls `parse_line(stream, line)`. No+log-file tailing.++**Credential gate**: `start()` consults `await self.has_credential()`+before spawning. Default implementation tests `secrets["api_key"]`;+subclasses override for non-api-key auth (Stripe → `is_logged_in()`).### Webhook verifiers@@ -1172,14 +1194,15 @@ project that publishes pre-built binaries via GitHub releases.| HTTP webhook ingress | `routers/webhook.py` consults `WEBHOOK_SOURCES` registry | `register_webhook_source(MySource())` || Event dispatch into workflows | `event_waiter.dispatch(source.type, event)` from `WebhookSource.handle` | provider-specific `shape()` returning `WorkflowEvent` || Trigger waiting + filtering | `WebhookTriggerNode.build_filter` (CloudEvents glob) | optional `_extra_filter(params)` |-| Daemon lifecycle | `DaemonEventSource.start/stop/restart` via `ProcessService` | `build_command(secrets)` + `parse_line(stream, line)` |+| Daemon lifecycle | `DaemonEventSource.start/stop/restart` via `ProcessService` | `build_command(secrets)` + `parse_line(stream, line)` (subscribed via `ProcessService.start(line_handler=...)` — no log-file tailing) |+| Daemon credential gate | `DaemonEventSource.start` consults `await self.has_credential()` before spawning | optional override when auth is non-api-key (Stripe → `is_logged_in()`); default tests `secrets["api_key"]` || Lifecycle WebSocket commands | `make_lifecycle_handlers(prefix, source, extra=…)` | provider-specific extra handlers || Status refresh on WS connect | `make_status_refresh(source, status_key, broadcast_type)` | nothing — auto-derived from source || CLI subprocess invocation | `run_cli_command(binary=…, argv=…, credential=…)` | nothing — credential injection is automatic || Credentials Modal panel | `server/config/credential_providers.json` (read by `services.credential_registry`) | one provider entry: name, category, color, `kind: "oauth"`, `icon_ref`, `status_hook`, `ws.{login,logout,status}` handler names, fields list, instructions string. Frontend modal renders it automatically — no React file edits. || AI tool surface | `services/ai.py` `DEFAULT_TOOL_NAMES` + `DEFAULT_TOOL_DESCRIPTIONS` | one row per dual-purpose ActionNode mapping `<nodeType>` → `<snake_case_of_node_type>` || Skill (LLM teaching markdown) | `server/skills/<agent>/<skill-name>/SKILL.md` (auto-discovered by `SkillLoader`) | the markdown itself, plus the linkage in `visuals.json` (`"<nodeType>": { ..., "skill": "<skill-name>" }`) |-| Connection state surfaced to the modal (CLI-managed auth) | `auth_service.store_oauth_tokens(status_hook, "cli-managed", "cli-managed")` + the generic `credential_catalogue_updated` broadcast | one `_mark_logged_in` / `_mark_logged_out` helper pair calling `store_oauth_tokens` / `remove_oauth_tokens`; one `broadcast({"type": "credential_catalogue_updated"})` after every state change |+| Connection state surfaced to the modal (CLI-managed auth) | `auth_service.store_oauth_tokens(provider, "cli-managed", "cli-managed")` + `StatusBroadcaster.broadcast_credential_event` (CloudEvents v1.0 envelope wrapping `WorkflowEvent`; locked by `tests/credentials/test_credential_broadcasts.py`) | `_mark_logged_in` / `_mark_logged_out` helper pair; one `broadcaster.broadcast_credential_event("credential.oauth.connected", provider="<id>")` after login and `…disconnected` after logout. Same shape Twitter / Google logout use. || Auto-install of an external CLI binary | `_install.py` with `ensure_<cli>_cli()` + GitHub-releases asset map | pinned `_VERSION` constant, `(system, machine) -> (asset, kind, member)` table, and an override on `DaemonEventSource.start()` that `await`s the helper before `super().start()` (also set `binary_name = ""` to skip the framework's `shutil.which` pre-check) |### Tool / skill / visuals naming contractdiff --git a/docs-internal/stripe_service.md b/docs-internal/stripe_service.mdindex c2c9656c..c3fe8a52 100644--- a/docs-internal/stripe_service.md+++ b/docs-internal/stripe_service.md@@ -142,8 +142,14 @@ handle_stripe_login()│ ││ └── get_listen_source().start() ← daemon auto-starts│- └── _broadcast_catalogue_updated() ← generic broadcast- broadcaster.broadcast({"type": "credential_catalogue_updated"})+ └── _broadcast_credential_event("credential.oauth.connected")+ → broadcaster.broadcast_credential_event(+ "credential.oauth.connected", provider="stripe")+ (CloudEvents v1.0 envelope wrapped under the+ 'credential_catalogue_updated' wire-format type.+ Same shape twitter_logout / google_logout /+ save_api_key use; locked by+ tests/credentials/test_credential_broadcasts.py.)frontend's existing case-handler invalidates the cataloguequery → modal sees provider.stored = true → connectionindicator flips immediately@@ -166,25 +172,32 @@ handle_stripe_logout()StripeListenSource.start() (lock-protected, idempotent)│├── ensure_stripe_cli() ← override before super().start()- ├── has_credential() → is_logged_in() ← override- │ (file check on ~/.config/stripe/config.toml)+ ├── has_credential() → is_logged_in() ← override of the daemon gate+ │ (filesystem check on ~/.config/stripe/config.toml)├── binary_name = "" ← framework PATH check skipped├── ProcessService.start(│ name="stripe-listen",- │ command="<resolved-binary> listen+ │ command="<shlex-quoted-binary> listen│ --forward-to http://localhost:{port}/webhook/stripe│ --print-secret", # no --api-key│ workflow_id="_stripe_global",│ working_directory={workspace}/_stripe,+ │ line_handler=self._on_line, # ← per-line callback│ )- │ (CLI reads credentials from its config file)+ │ (CLI reads credentials from its config file; binary path is+ │ shlex.quote'd so Windows backslashes survive ProcessService's+ │ POSIX-mode shlex.split round-trip)│- └── tail stderr.log- on match r"whsec_[A-Za-z0-9_]+":- auth_service.store_api_key("stripe_webhook_secret", …)+ └── ProcessService loops `stream.readline()` per stdout/stderr,+ decodes UTF-8, writes to stdout.log / stderr.log, broadcasts to+ the Terminal tab, AND calls our line_handler:+ _on_line(stream, line) → parse_line(stream, line)+ on stderr match r"whsec_[A-Za-z0-9_]+":+ auth_service.store_api_key("stripe_webhook_secret", …)+ (No file-tailing; we hook into the same readline loop ProcessService+ already runs.)StripeListenSource.stop()- ├── cancel tail task└── ProcessService.stop("stripe-listen", "_stripe_global")```@@ -203,7 +216,8 @@ StripeListenSource.stop()| `server/nodes/stripe/stripe_action.py` | `StripeActionNode` — pass-through over the CLI via `run_cli_command`. || `server/nodes/stripe/stripe_receive.py` | `StripeReceiveNode(WebhookTriggerNode)` — filter overrides + output reshape. || `server/services/events/__init__.py` | Public framework surface — exports every base class + helper. |-| `server/services/events/daemon.py` | `DaemonEventSource` — supervises subprocess via `ProcessService`, tails logs. |+| `server/services/events/daemon.py` | `DaemonEventSource` — supervises subprocess via `ProcessService`. Subscribes to ProcessService's per-line callback (`line_handler`) instead of re-tailing the on-disk log files. Credential gate is `await self.has_credential()` so non-api-key auth (Stripe → `is_logged_in()`) plugs in via subclass override. |+| `server/services/process_service.py` | Spawns + supervises long-lived subprocesses. Loops `stream.readline()` per stdout/stderr, writes to `.log`, broadcasts to Terminal, and forwards each decoded line to the optional `line_handler` async callback (Wave 12.B addition for typed event-source subscribers — see `DaemonEventSource._on_line`). || `server/services/events/webhook.py` | `WebhookSource` + `WEBHOOK_SOURCES` registry + `register_webhook_source`. || `server/services/events/triggers.py` | `WebhookTriggerNode` + `BaseTriggerParams`. || `server/services/events/cli.py` | `run_cli_command` helper. |@@ -251,8 +265,9 @@ Storage:### `StripeListenSource(DaemonEventSource)`Supervises `stripe listen` as a long-lived process. Inherits the-full `DaemonEventSource` lifecycle (start / stop / restart / status /-log tailing). The Stripe-specific overrides are minimal:+full `DaemonEventSource` lifecycle (start / stop / restart / status)+plus the per-line callback subscription via `ProcessService`'s+`line_handler` hook. The Stripe-specific overrides are minimal:| Method / attr | Purpose ||---|---|@@ -262,9 +277,9 @@ log tailing). The Stripe-specific overrides are minimal:| `install_hint` | Surfaced in the "install failed" error path when the auto-installer can't reach GitHub releases. || `credential = StripeCredential` | Resolved by the framework before `build_command` is called. || `start()` (override) | `await ensure_stripe_cli()` then `super().start()`. Caches the resolved binary path so `build_command` (sync) can pick it up. |-| `build_command(secrets)` | Returns `<resolved-binary> listen --forward-to … --print-secret` (no `--api-key` — CLI reads its own config file; binary path comes from `stripe_cli_path()` cache populated by `start()`). |-| `has_credential()` (override) | Returns `is_logged_in()` — a filesystem check on `~/.config/stripe/config.toml`. The framework's auto-reconnect path uses this as the gate for restarting the daemon when a WS client connects. |-| `parse_line(stream, line)` | On each stderr line, regex-matches `whsec_…`; on first match, persists the secret via `auth_service.store_api_key("stripe_webhook_secret", …)`. The Stripe daemon doesn't emit events itself — events arrive via the webhook receiver. |+| `build_command(secrets)` | Returns `<shlex.quote'd-binary> listen --forward-to … --print-secret`. The binary path is `shlex.quote`d so it round-trips through `ProcessService`'s POSIX-mode `shlex.split` unchanged. No `--api-key`: CLI reads its own config file. |+| `has_credential()` (override) | Returns `is_logged_in()` — a filesystem check on `~/.config/stripe/config.toml`. Consulted by `DaemonEventSource.start` (the credential gate) and by `make_status_refresh` on every WS-client connect (auto-reconnect). |+| `parse_line(stream, line)` | Invoked once per decoded stdout/stderr line via `ProcessService`'s `line_handler` callback. On `whsec_…` match, persists the secret via `auth_service.store_api_key("stripe_webhook_secret", …)`. The Stripe daemon doesn't emit workflow events itself — they arrive via the webhook receiver. |### `StripeWebhookSource(WebhookSource)`@@ -413,11 +428,12 @@ the modal's Connect button are `stripe_login` and `stripe_logout`:Background `_complete_login(binary, next_step)` flow:-1. `run_cli_command([binary, "login", "--complete", next_step], timeout=600s)` blocks until OAuth completes.-2. `is_logged_in()` flips true (the CLI wrote `_api_key` lines into `config.toml`).-3. `_mark_logged_in()` writes `auth_service.store_oauth_tokens("stripe", "cli-managed", "cli-managed")`.-4. `get_listen_source().start()` spawns the supervised `stripe listen` daemon (which captures the `whsec_…` from its banner).-5. `_broadcast_catalogue_updated()` emits `{type: "credential_catalogue_updated"}` — the existing generic broadcast handler in `WebSocketContext.tsx` invalidates the catalogue, the modal refetches, and `provider.stored = true` for stripe flips the connection indicator.+1. `next_step` from step 1 is a literal shell command (`stripe login --complete '<URL>'`); extract the auth URL with `shlex.split(next_step)[-1]` before passing it to `--complete`.+2. `run_cli_command([binary, "login", "--complete", complete_url], timeout=600s)` blocks until OAuth completes.+3. `is_logged_in()` flips true (the CLI wrote `_api_key` lines into `config.toml`).+4. `_mark_logged_in()` writes `auth_service.store_oauth_tokens("stripe", "cli-managed", "cli-managed")`.+5. `get_listen_source().start()` spawns the supervised `stripe listen` daemon; the `whsec_…` banner is captured via the `line_handler` callback.+6. `_broadcast_credential_event("credential.oauth.connected")` emits a CloudEvents v1.0 envelope (`WorkflowEvent`) via `StatusBroadcaster.broadcast_credential_event`, wrapped under the `credential_catalogue_updated` wire-format type. The frontend invalidates the catalogue and `provider.stored = true` flips the connection indicator.## AI tool surface (`stripe_action`)@@ -569,15 +585,25 @@ without any provider-specific code in the catalogue handler. The`auth_service.store_oauth_tokens` API doesn't validate the strings;the marker exists purely to flip the existence check.-### 2. The generic `credential_catalogue_updated` broadcast+### 2. The CloudEvents-shaped credential broadcast-After `_mark_logged_in()` / `_mark_logged_out()`, the plugin-broadcasts:+After `_mark_logged_in()` / `_mark_logged_out()`, the plugin emits a+CloudEvents v1.0 envelope via the canonical helper:-```json-{ "type": "credential_catalogue_updated" }+```python+await get_status_broadcaster().broadcast_credential_event(+ "credential.oauth.connected", provider="stripe", # or .disconnected on logout+)```+`StatusBroadcaster.broadcast_credential_event` wraps a `WorkflowEvent`+(from `services.events.envelope`) and ships it under the+`credential_catalogue_updated` wire-format type — the same shape+`save_api_key`, `delete_api_key`, `twitter_logout`, `google_logout`+already use. The contract is locked by+`tests/credentials/test_credential_broadcasts.py` (`inspect.getsource`+introspection over each handler).+`WebSocketContext.tsx` already has a generic case for this event(line 671 — predates Stripe). Its handler calls `invalidateCatalogue`on the TanStack Query client; the catalogue refetches; the modal