Skip to content

Self-improving AI Employees for everyone

30.8 KB
Raw Blame

Execution Engine Design Document

Overview

The OpenCompany execution engine implements a robust workflow orchestration system combining industry-standard patterns from Netflix Conductor, Prefect 3.0, Temporal, and Redis Streams. This document details the architectural decisions, design patterns, and standards used.

Related docs:


Execution Modes

WorkflowService in server/services/workflow.py is a thin facade (~855 lines) that routes every workflow run based on available infrastructure. In every shipped configuration the effective routing is Temporal -> sequential:

workflow.execute(workflow_id, workflow_data)
            |
            v
    +---------------------+
    | TEMPORAL_ENABLED    |  yes --> _execute_temporal()
    | and Temporal up?    |             |
    +---------------------+             v
            | no                   TemporalExecutor
            v                      (per-node activities,
    _execute_sequential()           retries, horizontal scaling)
    Sequential fallback
    (simple topological walk)

1. Temporal Distributed (primary production mode)

When TEMPORAL_ENABLED=true and the Temporal server is reachable, every workflow node executes through Temporal. Three dispatch paths coexist (legacy execute_node_activity / per-type node.{type}.v{version} / Agent-as-child-workflow AgentWorkflow) gated by TEMPORAL_PER_TYPE_DISPATCH and TEMPORAL_AGENT_WORKFLOW_ENABLED flags. Per-node retries, per-node timeouts, horizontal scaling via worker pool, and the FIRST_COMPLETED orchestrator pattern.

Full dispatch matrix + activity inventory (legacy + per-type + the F4.B agent.* activities — read the live set from collect_agent_activities() in services/temporal/agent_activities.py) + worker configuration + heartbeat semantics live in TEMPORAL_ARCHITECTURE.md. Tool-call dispatch under F4.A: tool_building_pipeline.md §9.

Newly admitted deployment generations record execution control version 1. Stop closes scheduling gates before model requests, tools, polling, compaction and child starts. Admitted parallel work finishes under its existing retry policy, and its results/bookkeeping settle before the generation acknowledges Stopped. Resume releases the same pending continuation. The generation controller records independently living roots; native execution descriptions discover attached descendants after child-start acknowledgement.

Signals retain accepted trigger events, completed Updates acknowledge control and root membership, and Queries expose status. Event admission stays immediate when running; no inactivity batching is introduced. Controller and agent continuation carry explicit state across Continue-As-New. Existing generations and pre-marker histories keep their recorded command paths. The control contract defines timeout reconciliation, fail-closed recovery, replay validation and operating limits. These Temporal guarantees do not extend to the sequential fallback.

2. Sequential Fallback

When Temporal is not available, workflow execution falls back to a simple topological walk with no parallelism, no caching, and no retry. Intended only for minimal environments and CI smoke tests.

Redis-parallel branch (unreachable in shipped configs)

workflow.py still contains a _execute_parallel() branch that routes to WorkflowExecutor in services/execution/executor.py when Redis is enabled:

  • Conductor's decide pattern in _workflow_decide() under distributed Redis SETNX locks
  • Continuous scheduling via asyncio.wait(FIRST_COMPLETED) (_execute_with_continuous_scheduling — the only scheduling implementation; the legacy Fork/Join _execute_parallel_nodes helper was removed)
  • Prefect-style result caching keyed by hash_inputs() (defined in services/execution/models.py; the cache read/write lives in services/execution/cache.py)
  • Crash recovery via RecoverySweeper with heartbeats
  • Dead Letter Queue for failed nodes (services/execution/dlq.py)

No shipped configuration sets REDIS_ENABLED=true, so this branch is unreachable in practice; treat Temporal -> sequential as the routing contract.


RLM Agent Execution

rlm_agent is the agent type that does not use the shared native tool-calling loop. Its RLMAgentNode.execute_op plugin collects the usual connections and routes to RLMService in services/rlm/, which owns the recursive REPL runtime.

The Recursive Language Model pattern replaces the standard tool-calling loop with a Python REPL (exec()) where the agent writes code that can call:

  • llm_query(prompt) - invoke the small model connected to input-model
  • rlm_query(prompt) - recursively invoke the agent itself
  • FINAL(answer) - signal completion

This yields 81-98% token savings on tool-heavy workflows because the agent can orchestrate multiple LLM calls in a single code block instead of paying the round-trip cost of tool calling.

See rlm_service.md for the full design.


Architecture Principles

1. Isolated Execution Contexts

Each workflow execution has its own ExecutionContext - no shared global state between concurrent runs.

┌─────────────────────────────────────────────────────────────┐
│                    WorkflowService                           │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │ Execution    │  │ Execution    │  │ Execution    │      │
│  │ Context A    │  │ Context B    │  │ Context C    │      │
│  │ (workflow 1) │  │ (workflow 2) │  │ (workflow 1) │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
└─────────────────────────────────────────────────────────────┘

2. Separation of Concerns

  • Models (models.py): Pure data structures, no business logic
  • Cache (cache.py): Persistence layer abstraction
  • Executor (executor.py): Orchestration logic
  • Recovery (recovery.py): Fault tolerance

3. Dependency Injection

Services receive dependencies via constructor, enabling testing and flexibility:

class WorkflowExecutor:
    def __init__(self, cache: ExecutionCache,
                 node_executor: Callable,
                 status_callback: Callable = None):
        self.cache = cache
        self.node_executor = node_executor
        self.status_callback = status_callback

4. A node result is small, because it is copied many times

Node results are not stored once. One result is written to node_outputs 3x per store, broadcast 2x over the status WebSocket, retained in StatusBroadcaster._status for the process lifetime and replayed to every newly connecting client, aggregated into the workflow result, copied into every downstream activity's input, and — when the node is usable_as_tool — serialized verbatim into an LLM message.

Temporal caps a blob at 2,097,152 B for activity results and inputs, and warns at 524,288 B. Exceeding the cap fails the activity as PayloadsTooLarge (the SDK core's failure type since temporalio 1.31). It is in NON_RETRYABLE_ERROR_TYPES (1b.26), so whatever produced the payload runs once; before, an oversized result cost three attempts, re-invoking (and re-billing) the producer each time.

So: large data lives on disk in the workflow workspace, and the node returns a reference. services/media/ provides that for binary media (AudioRef, write_audio, coerce_file_param), with an HTTP route for the frontend to fetch through. A capped inline opt-in is explicitly rejected as a design: it makes the tool path sometimes catastrophic, which is worse than always-broken because it passes review.

See media_transport.md for the measurements and the API.


Design Patterns

1. Conductor's Decide Pattern

Source: Netflix Conductor OSS

The decide pattern is the core orchestration loop. It evaluates current state, finds ready tasks, executes them, and recurses until completion.

async def _workflow_decide(self, ctx: ExecutionContext):
    """Core orchestration loop - Conductor's decide pattern."""
    async with distributed_lock(f"execution:{ctx.execution_id}:decide"):
        # 1. Find nodes ready to execute (dependencies satisfied)
        ready_nodes = self._find_ready_nodes(ctx)

        if not ready_nodes:
            return  # All done or stuck

        # 2. Execute with continuous scheduling -- newly-ready
        #    dependents start as soon as their dependency completes
        await self._execute_with_continuous_scheduling(ctx, ready_nodes)

        # 3. Checkpoint and persist
        await self.cache.save_execution_state(ctx)

Benefits:

  • Single point of orchestration logic
  • Distributed lock prevents race conditions
  • Natural support for parallel execution
  • Easy to add features (retries, timeouts)

2. Continuous Scheduling (Temporal/Conductor Pattern)

Source: Temporal + Netflix Conductor

Independent nodes execute in parallel, and each completion immediately unlocks its dependents — no layer barrier. _execute_with_continuous_scheduling is the only scheduling implementation (the earlier layer-based Fork/Join helper _execute_parallel_nodes, which asyncio.gather()-ed a whole batch and waited for all of it, was removed as dead code).

async def _execute_with_continuous_scheduling(self, ctx, initial_nodes):
    """When any node completes, immediately start newly-ready dependents."""
    task_to_node = {
        asyncio.create_task(self._execute_node_with_caching(ctx, n)): n
        for n in initial_nodes
    }

    while task_to_node:
        done, _ = await asyncio.wait(task_to_node, return_when=asyncio.FIRST_COMPLETED)
        for task in done:
            node = task_to_node.pop(task)
            # ... record result, then start any newly-ready dependents
            for ready in self._find_ready_nodes(ctx):
                task_to_node[asyncio.create_task(
                    self._execute_node_with_caching(ctx, ready))] = ready

DAG Layer Computation (Kahn's Algorithm):

Layer 0: [Start]           ← No dependencies
Layer 1: [A, B, C]         ← Depend only on Start (parallel)
Layer 2: [D]               ← Depends on A and B
Layer 3: [E, F]            ← Depend on D (parallel)
Layer 4: [End]             ← Depends on E and F

3. Prefect-Style Task Caching

Source: Prefect 3.0 Task Caching

Results are cached by hashing inputs, enabling idempotent execution.

async def _execute_node_with_caching(self, ctx, node, enable_caching):
    # Generate cache key from inputs
    inputs = self._gather_node_inputs(ctx, node.node_id)

    if enable_caching:
        # Check cache first
        cached = await self.cache.get_cached_result(
            ctx.execution_id, node.node_id, inputs
        )
        if cached:
            node.status = TaskStatus.CACHED
            return cached

    # Execute and cache result
    result = await self.node_executor(...)

    if enable_caching and result.get("success"):
        await self.cache.set_cached_result(
            ctx.execution_id, node.node_id, inputs, result
        )

    return result

Cache Key Generation:

def hash_inputs(inputs: Dict[str, Any]) -> str:
    """Create deterministic hash of inputs for cache key."""
    # Sort keys for determinism
    sorted_json = json.dumps(inputs, sort_keys=True, default=str)
    return hashlib.sha256(sorted_json.encode()).hexdigest()[:16]

def generate_cache_key(execution_id: str, node_id: str, inputs: Dict) -> str:
    """Generate full cache key."""
    input_hash = hash_inputs(inputs)
    return f"result:{execution_id}:{node_id}:{input_hash}"

4. State Machine Pattern

Source: Standard FSM + Conductor TaskStatus

Tasks and workflows follow explicit state machines:

TaskStatus State Machine:
                    ┌─────────────────┐
                    │     PENDING     │
                    └────────┬────────┘
                             │ schedule
                    ┌────────▼────────┐
                    │    SCHEDULED    │
                    └────────┬────────┘
                             │ start
          ┌──────────────────▼──────────────────┐
          │               RUNNING               │
          └──┬──────────┬──────────┬──────────┬─┘
             │          │          │          │
        complete    cache hit    fail      cancel
             │          │          │          │
    ┌────────▼───┐ ┌────▼────┐ ┌──▼───┐ ┌───▼─────┐
    │ COMPLETED  │ │ CACHED  │ │FAILED│ │CANCELLED│
    └────────────┘ └─────────┘ └──────┘ └─────────┘

WorkflowStatus State Machine:
    PENDING → RUNNING → COMPLETED
                ↓           ↓
              PAUSED      FAILED
                ↓           ↓
            CANCELLED ←────┘

5. Distributed Locking

Source: Redis SETNX pattern

Prevents concurrent decide operations on the same execution:

@asynccontextmanager
async def distributed_lock(self, lock_name: str, timeout: int = 60):
    """Acquire distributed lock using Redis SETNX."""
    lock_key = f"lock:{lock_name}"
    lock_token = str(uuid.uuid4())

    # Try to acquire lock
    acquired = await self.cache.redis.set(
        lock_key, lock_token,
        nx=True,  # Only set if not exists
        ex=timeout  # Auto-expire after timeout
    )

    if not acquired:
        raise TimeoutError(f"Could not acquire lock: {lock_name}")

    try:
        yield
    finally:
        # Release lock (only if we own it)
        current = await self.cache.redis.get(lock_key)
        if current == lock_token:
            await self.cache.redis.delete(lock_key)

6. Event Sourcing (Partial)

Source: Event Sourcing pattern + Redis Streams

All state changes are recorded as immutable events:

# Event types
EVENT_TYPES = [
    "workflow_started",
    "node_started",
    "node_completed",
    "node_failed",
    "node_cached",
    "workflow_completed",
    "workflow_failed",
    "workflow_cancelled"
]

# Store event
await cache.add_event(execution_id, "node_completed", {
    "node_id": node.node_id,
    "execution_time": duration,
})

# Events stored in Redis Stream
# Key: execution:{id}:events
# Enables: replay, debugging, audit trail

7. Heartbeat Pattern

Source: Conductor + Distributed Systems

Running nodes send periodic heartbeats for crash detection:

# During execution
await cache.update_heartbeat(execution_id, node_id)

# Recovery sweeper checks
for node_id, node_exec in ctx.node_executions.items():
    if node_exec.status == TaskStatus.RUNNING:
        last_heartbeat = await cache.get_heartbeat(execution_id, node_id)

        if time.time() - last_heartbeat > HEARTBEAT_TIMEOUT:
            # Node is stuck - trigger recovery
            needs_recovery = True

8. Adapter Pattern

Source: Gang of Four

Bridges new executor to existing node handlers without modification:

async def _node_executor_adapter(self, node_id, node_type, parameters, context):
    """Adapter bridges WorkflowExecutor to existing execute_node()."""
    return await self.execute_node(
        node_id=node_id,
        node_type=node_type,
        parameters=parameters,
        nodes=context.get("nodes"),
        edges=context.get("edges"),
        session_id=context.get("session_id"),
        execution_id=context.get("execution_id"),
    )

Redis Key Schema

# Execution State (HASH)
execution:{id}:state
  - status: WorkflowStatus
  - workflow_id: string
  - session_id: string
  - started_at: timestamp
  - completed_at: timestamp
  - current_layer: int
  - checkpoints: JSON array

# Node States (HASH)
execution:{id}:nodes
  - {node_id}: JSON(NodeExecution)

# Node Outputs (HASH)
execution:{id}:outputs
  - {node_id}: JSON(output data)

# Result Cache (STRING with TTL)
result:{execution_id}:{node_id}:{input_hash}
  - JSON(cached result)
  - TTL: 3600 seconds

# Event History (STREAM)
execution:{id}:events
  - Immutable append-only log
  - Fields: event_type, timestamp, data

# Heartbeats (STRING with TTL)
heartbeat:{execution_id}:{node_id}
  - timestamp
  - TTL: 60 seconds

# Distributed Locks (STRING with TTL)
lock:execution:{id}:decide
  - lock_token (UUID)
  - TTL: 60 seconds

# Active Executions (SET)
executions:active
  - Set of execution_ids currently running

Data Models

ExecutionContext

@dataclass
class ExecutionContext:
    """Isolated state for a single workflow execution."""

    # Identity
    execution_id: str          # Unique ID (auto-generated)
    workflow_id: str           # Workflow definition ID
    session_id: str            # User session

    # State
    status: WorkflowStatus     # Current workflow status
    node_executions: Dict[str, NodeExecution]  # Per-node state
    outputs: Dict[str, Any]    # Node outputs for template resolution

    # Workflow Definition (immutable during execution)
    nodes: List[Dict]          # Node definitions
    edges: List[Dict]          # Edge definitions

    # Execution Progress
    execution_order: List[str] # Topological order (flat list)
    current_layer: int         # Current layer index
    checkpoints: List[str]     # Completed node IDs

    # Timing
    created_at: float
    updated_at: float
    started_at: Optional[float]
    completed_at: Optional[float]

    # Errors
    errors: List[Dict]         # Error records

The per-workflow workspace directory (<DATA_DIR>/workspaces/<workflow_slug>/, Wave 14 — keyed by the human-readable slug, not the UUID id; see CLAUDE.md "Workflow Naming") is not a field on ExecutionContext. WorkflowService resolves it and injects workspace_dir into the execution context dict that node handlers receive (services/workflow.py).

NodeExecution

@dataclass
class NodeExecution:
    """State for a single node within an execution."""

    node_id: str
    node_type: str
    status: TaskStatus

    # Execution details
    input_hash: str            # For cache key
    output: Dict[str, Any]
    error: str

    # Timing
    started_at: float
    completed_at: float

    # Retry tracking (future)
    retry_count: int = 0

Execution Flow

┌─────────────────────────────────────────────────────────────────────┐
│                         execute_workflow()                           │
└────────────────────────────────┬────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│  1. Create ExecutionContext                                          │
│     - Generate unique execution_id                                   │
│     - Initialize node_executions (all PENDING)                       │
│     - Compute execution layers (DAG analysis)                        │
└────────────────────────────────┬────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│  2. Persist Initial State                                            │
│     - Save to Redis: execution:{id}:state                           │
│     - Add to executions:active set                                   │
│     - Emit workflow_started event                                    │
└────────────────────────────────┬────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│  3. workflow_decide() Loop                                           │
│     ┌─────────────────────────────────────────────────────────────┐ │
│     │  a. Acquire distributed lock                                 │ │
│     │  b. Find ready nodes (deps satisfied)                        │ │
│     │  c. Execute batch (parallel if multiple)                     │ │
│     │  d. Update node states                                       │ │
│     │  e. Checkpoint to Redis                                      │ │
│     │  f. Release lock                                             │ │
│     │  g. Recurse until no ready nodes                             │ │
│     └─────────────────────────────────────────────────────────────┘ │
└────────────────────────────────┬────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│  4. Finalize                                                         │
│     - Set final workflow status                                      │
│     - Remove from executions:active                                  │
│     - Emit workflow_completed/failed event                           │
│     - Return results                                                 │
└─────────────────────────────────────────────────────────────────────┘

Recovery Mechanism

Crash Detection

┌─────────────────────────────────────────────────────────────────────┐
│                      RecoverySweeper (Background)                    │
│                                                                      │
│  Every 60 seconds:                                                   │
│  1. Get all execution IDs from executions:active                     │
│  2. For each execution:                                              │
│     a. Load state from Redis                                         │
│     b. Check if already complete → remove from active                │
│     c. Find RUNNING nodes                                            │
│     d. Check heartbeat age                                           │
│        - If > 5 minutes → mark as stuck                              │
│  3. Trigger recovery callback for stuck executions                   │
└─────────────────────────────────────────────────────────────────────┘

Startup Recovery

async def scan_on_startup(self) -> List[str]:
    """Find executions that need recovery on server restart."""
    needs_recovery = []

    for execution_id in await cache.get_active_executions():
        ctx = await cache.load_execution_state(execution_id)

        if ctx.status == WorkflowStatus.RUNNING:
            age = time.time() - ctx.updated_at
            if age > HEARTBEAT_TIMEOUT:
                needs_recovery.append(execution_id)

    return needs_recovery

Thread Safety

Cross-Thread Event Dispatch

Historical pattern (Wave 15 note): APScheduler thread-pool callbacks were the original motivation; APScheduler was retired in Wave 15.2 and the Redis-Streams dispatch_async sibling in Wave 15.3. event_waiter still captures the main loop at startup (capture_main_loop() in main.py) so any future thread-context caller can hop onto it with asyncio.run_coroutine_threadsafe(...); every production caller today runs on the main event loop and uses the sync dispatch() directly.


Configuration

Deployment Settings

_deployment_settings = {
    "delay_between_runs": 1.0,      # Seconds between iterations
    "stop_on_error": False,          # Stop on first error
    "max_iterations": 0,             # 0 = unlimited
    "use_parallel_executor": True    # Use new parallel engine
}

Execution Toggle

# services/workflow.py — Temporal first, then the (unreachable) Redis branch, then sequential
if use_temporal and self._temporal_executor is not None:
    return await self._execute_temporal(...)
# use_temporal with no executor logs an ERROR (silent fallthrough looked like success)
if use_parallel and self.settings.redis_enabled:
    return await self._execute_parallel(...)
return await self._execute_sequential(...)

All three return the same Run fields, which the editor's Run dialog reads: errors, error (the first failure's message), total_nodes and completed_nodes. See Temporal Architecture → The run result.


Comparison: Before vs After

Aspect Sequential (Before) Parallel (After)
Execution BFS traversal, one-by-one DAG layers, concurrent
State In-memory only Redis-persisted
Recovery None (lost on crash) Automatic via sweeper
Idempotency None Input-hashed caching
Concurrency Global flag blocks Isolated contexts
Locking None Distributed Redis locks
Events None Redis Streams

Implemented Features

Runtime Conditional Branching (conditions.py)

Edge conditions are evaluated at runtime using Prefect-style dynamic branching:

  • 19 operators (the OPERATORS table in conditions.py): eq, neq, gt, lt, gte, lte, contains, not_contains, exists, not_exists, is_empty, is_not_empty, matches, in, not_in, starts_with, ends_with, is_true, is_false
  • Supports nested field access via dot notation (e.g., result.status)
  • AND/OR condition groups with recursive evaluation

Dead Letter Queue (dlq.py)

Failed nodes (after all retries exhausted) are stored for inspection and replay:

  • DLQHandler - Active handler storing failed nodes with full context
  • NullDLQHandler - No-op handler when DLQ disabled (Null Object pattern)
  • WebSocket handlers: get_dlq_entries, get_dlq_entry, replay_dlq_entry, remove_dlq_entry, purge_dlq, get_dlq_stats

Temporal Distributed Execution (Optional)

When TEMPORAL_ENABLED=true, each node executes as an independent Temporal activity:

  • Per-node retry (3 attempts by default; plugin retry_policy wins) and a 24 h start_to_close ceiling (_NODE_ACTIVITY_START_TO_CLOSE); liveness is the 30 s heartbeat against a 2 min heartbeat_timeout
  • Horizontal scaling across worker pool
  • FIRST_COMPLETED pattern for dependency resolution
  • Connection pooling for WebSocket activity execution

Future Enhancements

Planned

  1. Sub-Workflows - Execute workflow as a node
  2. Metrics - Prometheus integration
  3. Workflow Versioning - Track definition versions

Potential

  1. Distributed Workers - Redis Streams consumer groups
  2. Priority Queues - Task prioritization

References