Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
docs: Wave 15.4 sweep — reflect APScheduler + Redis-Streams retirement
trohitg1af2e009389c
diff --git a/CLAUDE.md b/CLAUDE.mdindex a38e3487..93e827a8 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -43,7 +43,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire| **[CLI Services Integration](./docs-internal/cli_services_integration.md)** | Guide for integrating CLI-based services (Temporal, etc.) with proper lifecycle management || **[Temporal Architecture](./docs-internal/TEMPORAL_ARCHITECTURE.md)** | Distributed workflow execution: activities, FIRST_COMPLETED scheduling, horizontal scaling || **[Native LLM SDK](./docs-internal/native_llm_sdk.md)** | Native SDK layer in services/llm/: Protocol-based providers, config-driven base URLs, 12 providers (xai native-chat-only; ollama/lmstudio local servers), native vs LangChain path routing |-| **[Event Waiter System](./docs-internal/event_waiter_system.md)** | Generic asyncio.Future/Redis-Streams waiter for push-based trigger nodes (WhatsApp, Telegram, Webhook, Chat, Task completion) |+| **[Event Waiter System](./docs-internal/event_waiter_system.md)** | In-memory asyncio.Future waiter for push-based trigger nodes on the canvas-Run path (WhatsApp, Telegram, Webhook, Chat, Task completion). Deployed triggers ride the Temporal canary path; the Redis-Streams backend was retired in Wave 15.3 || **[Credentials Encryption](./docs-internal/credentials_encryption.md)** | Fernet + PBKDF2 encryption pipeline, separate credentials.db, two credential systems (OAuth vs API keys), multi-backend abstraction || **[Status Broadcaster](./docs-internal/status_broadcaster.md)** | WebSocket-first communication: StatusBroadcaster singleton, live count via `len(MESSAGE_HANDLERS) + len(get_ws_handlers())`, broadcast message types, Android two-state model || **[RLM Service](./docs-internal/rlm_service.md)** | Recursive Language Model agent with REPL-based execution (llm_query, rlm_query, FINAL) |@@ -121,7 +121,6 @@ server/services/├── himalaya_service.py # HimalayaService CLI wrapper for IMAP/SMTP (any email provider)├── email_service.py # EmailService orchestrator (credential resolution, provider presets)├── todo_service.py # TodoService singleton for writeTodos tool (JSON per-session state)-├── scheduler.py # APScheduler singleton for cron job management├── memory.py # Markdown-based conversation memory helpers├── memory_store.py # In-memory conversation store (LangChain 0.3+ compatible)├── skill_prompt.py # Skill system prompt builder (injects SKILL.md for personality skills)diff --git a/docs-internal/DESIGN.md b/docs-internal/DESIGN.mdindex e51de5f8..cc91c17a 100644--- a/docs-internal/DESIGN.md+++ b/docs-internal/DESIGN.md@@ -588,23 +588,13 @@ async def scan_on_startup(self) -> List[str]:### Cross-Thread Event Dispatch-APScheduler runs callbacks in thread pools. The event waiter handles this:--```python-def dispatch(event_type: str, data: Dict) -> int:- """Thread-safe event dispatch."""- try:- # Check if we're in async context- loop = asyncio.get_running_loop()- asyncio.create_task(dispatch_async(event_type, data))- except RuntimeError:- # No event loop - we're in a thread- if _main_loop and _main_loop.is_running():- asyncio.run_coroutine_threadsafe(- dispatch_async(event_type, data),- _main_loop- )-```+Historical pattern (Wave 15 note): APScheduler thread-pool callbacks were+the original motivation; APScheduler was retired in Wave 15.2 and the+Redis-Streams `dispatch_async` sibling in Wave 15.3. `event_waiter`+still captures the main loop at startup (`capture_main_loop()` in+`main.py`) so any future thread-context caller can hop onto it with+`asyncio.run_coroutine_threadsafe(...)`; every production caller today+runs on the main event loop and uses the sync `dispatch()` directly.---diff --git a/docs-internal/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md b/docs-internal/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.mdindex 7fefdc44..184667c4 100644--- a/docs-internal/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md+++ b/docs-internal/TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md@@ -5,6 +5,16 @@- **Baseline**: `5b384b7` (v0.0.92)- **Supersedes**: the Wave 12/13 tracker (shipped; see `docs-internal/event_framework.md`)+## Implementation status++| Phase | Status |+|---|---|+| 15.1 — drop `WorkflowEvent.connection_status` | ✅ shipped `8bc96c1` |+| 15.2 — retire APScheduler cron stack | ✅ shipped `eda3a20` (-272 LOC; Temporal hard-required for cron) |+| 15.3 — retire Redis-Streams branch (D2b) | ✅ shipped `76576b1` (-347/+57 LOC; `dispatch_async` gone, `set_cache_service` → `capture_main_loop`, cache consumer-group methods gone; `_streams_available` flag KEPT — the surviving `stream_add` guards on it, a detail §4 originally missed) |+| 15.4 — docs sweep | ✅ shipped (this commit) |+| 16.x / 17.x / 18.x | ⏳ in progress |+---## §1 Motivationdiff --git a/docs-internal/event_framework.md b/docs-internal/event_framework.mdindex 6c72d01e..4f03eae2 100644--- a/docs-internal/event_framework.md+++ b/docs-internal/event_framework.md@@ -37,7 +37,7 @@ phase plan lives in `~/.claude/plans/properly-fix-the-tech-dreamy-tarjan.md`.| Phase | State ||---|---|| D2 — Custom `event_dlq` SQLModel table | ❌ **dropped** (commit `89b15bd`, docs only). Temporal Event History + Visibility queries cover the ops-inspection use case; reinventing them would contradict Wave 12's "Temporal-native, no custom infra" thesis. See § "Failure inspection — no separate DLQ table". |-| D2b — Retire `event_waiter.py` Redis-Streams branch | ⏳ pending — flag default now `True`; gated on production-stability window before draining the legacy collector. |+| D2b — Retire `event_waiter.py` Redis-Streams branch | ✅ shipped as Wave 15.3 (see [TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md](./TEMPORAL_CLEANUP_AND_RESILIENCE_PLAN.md)). `event_waiter` is memory-mode-only now; Temporal owns durable delivery. The in-memory collector still backs canvas-Run + non-canary (Twitter) triggers. || D4 — Drain remaining dual-emit on message/newsletter/history wire keys | ⏳ pending — paired with FE migration to envelope-aware readers on those channels (`whatsapp_message_received` et al). || WorkflowEnvironment integration smoke test | ⏳ pending — full 7-canary in-process Temporal cluster. Existing unit tests + per-canary producer tests + `TestCanaryRegistryCoverage` cover the static surface; the integration smoke would catch real-cluster regressions only. |@@ -53,7 +53,7 @@ If the canary fan-out causes regressions in production:2. Restart the server (`npm run start` / `uvicorn` reload). Pydantic Settings re-reads on startup.3. Confirm pass-through: `dispatch.emit()` logs `event-framework disabled — emit no-op` at DEBUG.-No DB migrations, no schema changes — the rollback is one env var + restart. The legacy `event_waiter` collector/processor keeps trigger nodes firing because plugin producers still call `event_waiter.dispatch(...)` alongside `dispatch.emit(...)` (the dual-dispatch pattern stays until D2b retires the Redis-Streams branch).+No DB migrations, no schema changes — the rollback is one env var + restart. The legacy `event_waiter` collector/processor keeps trigger nodes firing because plugin producers still call `event_waiter.dispatch(...)` alongside `dispatch.emit(...)` (the dual-dispatch pattern stays for the in-memory canvas-Run path; the Redis-Streams branch itself was retired in Wave 15.3).Locked by `tests/test_event_framework_phase_a.py::TestEventFrameworkEnabledDefault::test_event_framework_enabled_defaults_true` (source-introspection check that the `Field(default=True, ...)` declaration is present) and `TestCanaryRegistryCoverage::test_seven_canary_types_registered` (every canary plugin opted in via `register_canary_trigger_type`).diff --git a/docs-internal/event_waiter_system.md b/docs-internal/event_waiter_system.mdindex f3754418..8126fe6d 100644--- a/docs-internal/event_waiter_system.md+++ b/docs-internal/event_waiter_system.md@@ -3,7 +3,7 @@> **⚠️ Pre-Wave-11 — historical reference only.**> Node authoring now happens on the backend: each node is a Python plugin under `server/nodes/<category>/<node>.py` that emits a `NodeSpec`. The frontend reads specs via [client/src/lib/nodeSpec.ts](../client/src/lib/nodeSpec.ts) + [adapters/nodeSpecToDescription.ts](../client/src/adapters/nodeSpecToDescription.ts). See [plugin_system.md](./plugin_system.md) and [server/nodes/README.md](../server/nodes/README.md) for the current model. The snippets below that reference `client/src/nodeDefinitions/*` are kept for historical context.-Trigger nodes in MachinaOS suspend workflow execution until an external event arrives (WhatsApp message, webhook request, Telegram message, chat input, delegated task completion, etc.). The Event Waiter system is the generic primitive that backs all push-based triggers. It has two backends: in-memory (`asyncio.Future`) and Redis Streams (for multi-worker durability).+Trigger nodes in MachinaOS suspend workflow execution until an external event arrives (WhatsApp message, webhook request, Telegram message, chat input, delegated task completion, etc.). The Event Waiter system is the in-memory (`asyncio.Future`) primitive that backs push-based triggers on the canvas-Run path. Deployed triggers ride the Temporal-durable canary path (`services/events/dispatch.py` → `TriggerListenerWorkflow` / `PollingTriggerWorkflow`) instead; the Redis-Streams backend that previously offered cross-restart waiter persistence was retired in Wave 15.3 because Temporal owns durability now.Source file: `server/services/event_waiter.py`@@ -31,9 +31,9 @@ class Waiter:node_id: str # workflow node waitingnode_type: str # 'whatsappReceive', 'webhookTrigger', ...event_type: str # 'whatsapp_message_received', ...- params: Dict # node parameters (used to rebuild filter in Redis mode)+ params: Dict # node parameters (filter provenance)filter_fn: Callable[[Dict], bool]- future: Optional[asyncio.Future] # only used in memory mode+ future: Optional[asyncio.Future] # resolved by dispatch() on matchcancelled: boolcreated_at: float```@@ -55,7 +55,7 @@ TRIGGER_REGISTRY: Dict[str, TriggerConfig] = {}```-`cronScheduler` is **not** in this registry: it uses APScheduler directly and does not wait for events.+`cronScheduler` is **not** in this registry: it is a Temporal Schedule (created by the deployment manager via `services/temporal/schedules.py`) and does not wait for events.### Filter Builders@@ -92,42 +92,28 @@ event_waiter.wait_for_event(waiter) (suspends)External event arrives (WhatsApp RPC, Telegram long-polling, webhook HTTP request, ...)|v- event_waiter.dispatch(event_type, data) (sync, thread-safe)- or event_waiter.dispatch_async(event_type, data) (Redis mode)+ event_waiter.dispatch(event_type, data) (sync; also accepts a WorkflowEvent)|vFor each Waiter with matching event_type:if waiter.filter_fn(data):- waiter.future.set_result(data) (memory mode)- or stream_ack(...) (Redis mode)+ waiter.future.set_result(data)|vWorkflow resumes, trigger node completes, downstream nodes execute```-## Two Backends+## Backend-### Memory Mode--Default for local development. Uses `asyncio.Future` with a module-level `_waiters` dict.+Single in-memory backend using `asyncio.Future` with a module-level `_waiters` dict.- `register()` creates an `asyncio.Future` and stores it in `_waiters[waiter.id]`.- `wait_for_event()` awaits the future.- `dispatch()` iterates `_waiters`, runs `filter_fn(data)` on each, calls `future.set_result(data)` on matches.-Thread-safe: `dispatch()` detects whether it is running in an async loop or a thread (e.g. APScheduler callback) and uses `asyncio.run_coroutine_threadsafe(..., _main_loop)` when needed.--### Redis Streams Mode--Activated automatically when `CacheService.is_streams_available()` returns True. Used in Temporal multi-worker deployments where waiters may live on different processes than dispatchers.--- Events are appended to `events:{event_type}` streams via `XADD`.-- Each waiter creates its own consumer group `waiter_group_{waiter_id}` for broadcast semantics (every waiter sees every event, not load-balanced).-- `_wait_redis()` polls with blocking `XREADGROUP` in 5-second chunks so cancellation is responsive.-- On match, the stream entry is `XACK`ed and the waiter is cleaned up.-- Waiter metadata is stored in `waiters:{waiter_id}` with 24-hour TTL for visibility in other workers.+`capture_main_loop()` (called during app startup in `main.py`) stores the main event loop so future thread-context callers can hop onto it via `asyncio.run_coroutine_threadsafe`.-The Redis mode is initialized by `set_cache_service(cache)` during app startup in `main.py`.+Durability note: waiter state does NOT survive a process restart — that is by design. Deployed triggers get restart durability from the Temporal canary path (`TriggerListenerWorkflow` / `PollingTriggerWorkflow` / Temporal Schedules); the event waiter only backs interactive canvas-Run waits, where a dead process means the user's canvas session is gone anyway. (The Redis-Streams backend that previously covered this was retired in Wave 15.3.)## Polling Triggers vs Event Triggers@@ -141,8 +127,8 @@ Some triggers do not fit the push model because the upstream service has no webh| `taskTrigger` | Event (push via delegation) | `event_waiter.py` || `telegramReceive` | Event (push via long-polling) | `event_waiter.py` + `TelegramService` || `twitterReceive` | **Polling** | `deployment/triggers.py` + `asyncio.Queue` |-| `googleGmailReceive` | **Polling** | `deployment/triggers.py` + `asyncio.Queue` |-| `cronScheduler` | APScheduler | `deployment/triggers.py` |+| `googleGmailReceive` | **Polling** | `PollingTriggerWorkflow` (Temporal canary) |+| `cronScheduler` | Temporal Schedule | `services/temporal/schedules.py` |Polling triggers are routed in `DeploymentManager._create_poll_coroutine()` and use `TriggerManager.setup_polling_trigger()` with a custom poll function and an `asyncio.Queue`. See [workflow-schema.md](workflow-schema.md) for the full trigger list.@@ -152,7 +138,7 @@ Users can cancel a waiting trigger from the UI (Cancel button on the trigger nod1. Frontend sends `cancel_event_wait` WebSocket message with `waiter_id` or `node_id`.2. `handle_cancel_event_wait()` in `server/routers/websocket.py` calls either `event_waiter.cancel(waiter_id)` or `event_waiter.cancel_for_node(node_id)`.-3. `cancel()` sets `w.cancelled = True`, calls `future.cancel()` in memory mode, and deletes the Redis metadata key in Redis mode.+3. `cancel()` sets `w.cancelled = True` and calls `future.cancel()`.4. The suspended `wait_for_event()` raises `asyncio.CancelledError`, which bubbles up through the node executor.## Debuggingdiff --git a/docs-internal/node-logic-flows/workflow_triggers/cronScheduler.md b/docs-internal/node-logic-flows/workflow_triggers/cronScheduler.mdindex ff32095d..3322abae 100644--- a/docs-internal/node-logic-flows/workflow_triggers/cronScheduler.md+++ b/docs-internal/node-logic-flows/workflow_triggers/cronScheduler.md@@ -11,7 +11,8 @@## PurposeRecurring time-based trigger. In **deployed** workflows the scheduling-happens inside `DeploymentManager` using APScheduler; this handler only+happens inside `DeploymentManager` via a Temporal Schedule (the+APScheduler path was retired in Wave 15.2); this handler onlyruns when the node is executed directly (manual run) or as an AI tool. Inthat mode it simply sleeps for the computed interval once, then returnsmetadata describing the schedule. It is NOT an event-waiter - it does notdiff --git a/docs-internal/plugin_system.md b/docs-internal/plugin_system.mdindex 76822e89..5887a1ff 100644--- a/docs-internal/plugin_system.md+++ b/docs-internal/plugin_system.md@@ -957,7 +957,9 @@ EventSource (abstract)tail stdout/stderr; parse_line() → events```-Cron / scheduled events stay on APScheduler — they don't need this+Cron / scheduled events are Temporal Schedules (created by the+deployment manager via `services/temporal/schedules.py`; the+APScheduler path was retired in Wave 15.2) — they don't need thisbase. Internal in-process events go through `event_waiter.dispatch`directly.