Skip to content

Self-improving AI Employees for everyone

docs(rfc): add RFC-0005 event OS and RFC-0004 Temporal SDK adoption

trohitg53ac0d2396e6
diff --git a/RFC-0004-TEMPORAL-SDK-FEATURE-ADOPTION.md b/RFC-0004-TEMPORAL-SDK-FEATURE-ADOPTION.md
new file mode 100644
index 00000000..0375fe97
--- /dev/null
+++ b/RFC-0004-TEMPORAL-SDK-FEATURE-ADOPTION.md
@@ -0,0 +1,732 @@
+# RFC-0004 — Temporal: Using the Python SDK and Platform Correctly and Fully
+
+Status: Draft. Owner decisions recorded in §17 (2026-10-16 and 2026-10-23); Q5, Q9, Q11, Q13 and Q15 are still open. The event path is designed in RFC-0005
+Created: 2026-10-16
+Scope: `server/services/temporal/`, `server/services/deployment/` (control, recovery, Reset), `server/services/events/dispatch.py`, `server/services/plugin/{base,scaling,polling}.py`, `server/nodes/scheduler/cron_scheduler/`, `server/nodes/workflow/approval_gate/`, `server/routers/webhook.py`, the Temporal CLI packaging (`server/services/temporal/_install.py`, `docker/Dockerfile`, the desktop shell), `server/tests/temporal/`, `.env.template` (`TEMPORAL_*`)
+Relationship: builds on [TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md) and [temporal-workflow-control.md](./docs-internal/temporal-workflow-control.md), which stay normative until a phase here lands. Corrects one CLAUDE.md durability rule (§8.5). Does not change RFC-0002.
+Baseline: temporalio 1.30.0 (`server/uv.lock`), Temporal CLI unpinned ("latest" at first boot), repository state of 2026-10-16. Line numbers are as of that date and drift.
+
+## How to read this
+
+- **Owner:** §1, §3 (decisions) and §17 (questions only you can answer).
+- **Reviewer:** §4 (what we use and what we don't) and §5 (defects). Each defect gives the evidence, the Temporal rule it breaks, the fix, and whether the fix changes workflow commands (and so needs a patch gate).
+- **Implementer:** §6 to §15 by area, §16 (phases) and §18 (acceptance gates).
+
+Every defect in §5 was found by one audit pass and re-checked by an independent verification pass that tried to refute it against the code, the installed SDK source and the docs. Corrections from verification are folded in. Statements that rest on the Temporal server or CLI source rather than the docs say so.
+
+## 1. Summary
+
+OpenCompany uses Temporal more deeply than most applications: a DAG orchestrator, an LLM agent loop, a per-deployment controller entity that continues-as-new, Updates with validators, a cooperative pause state machine, typed search attributes, Schedules, one worker per task queue, and patch-gated changes with replay tests. Much of this follows Temporal's documented patterns (§4.1).
+
+The audit found three groups of problems.
+
+1. **The platform under it is the development server.** Every install runs `temporal server start-dev` on a SQLite file, downloaded as "latest" on first boot with no version pin or checksum. Temporal documents both as development-only; a SQLite database created by one server version is never migrated by a later one; closed workflows are deleted after 24 hours; and no SDK metrics are exported, so none of Temporal's documented alerts (stuck workflow tasks, nondeterminism, oversized messages) can exist. (§5 H4, H5, §6, §7)
+2. **Temporal's semantics expose real defects.** Every backend restart fires each deployment's Start node again. A large webhook can wedge a deployment's controller. Inbound events are routed through eventually consistent Visibility and silently dropped. Node activities swallow cancellation and are recorded as completed failures. The LLM step and the legacy tool path retry without limit. Cron's overlap policy never applies to the graph run. Activity type names are computed from live plugin code at replay time, three patch markers were deleted without the deprecation step, and CI never replays histories recorded by a previous release. (§5)
+3. **Features that fix current pain are unused:** addressing Schedules and controllers by their known ids, `schedule_to_close` and `schedule_to_start` timeouts, the BENIGN error category, user metadata in the Web UI, `TemporalReportedProblems`, task queue priority, queue-wide rate limits, time-skipping tests, a replay corpus, and (in preview) External Storage. (§4.2)
+
+The RFC makes 18 decisions in five phases. Phase 0 fixes defects without changing workflow commands; Phase 1 builds the safety net (pinned server, SDK 1.34, metrics, replay gate); Phase 2 makes the command-changing fixes behind patch gates; Phase 3 adopts features; Phase 4 settles the production platform. Some choices are the owner's (§17), chiefly the upgrade contract (must runs that are open or paused survive an upgrade?) and which server durable installs run on.
+
+## 2. Method and sources
+
+- **Docs.** 282 pages of docs.temporal.io, downloaded as markdown on 2026-10-16 and read in full: the whole Python SDK guide (57 pages), the concept pages (83), cross-SDK worker performance and priority/fairness (12), best practices (9), design patterns (46), self-hosting and worker deployments (24), troubleshooting, references and guides (23), the CLI reference (9), and the AI cookbook and Python integrations (29).
+- **Releases.** temporalio release notes 1.25.0 to 1.34.0; Temporal CLI v1.3 to v1.9.2 and server 1.27 to 1.32; the installed 1.30.0 SDK source and the 1.34.0 sdist; the CLI and server repositories at exact tags for dev-server behavior the docs don't cover.
+- **Code.** Seven read-only audits of `server/` (core workflows, agent workflows, activities, workers and runtime, triggers and dispatch, client-side control, tests and docs and patches), then five verification passes.
+- **Not done.** Nothing was run against a live server. The effect of a newer dev server opening an older SQLite file is derived from source, not reproduced.
+
+## 3. Decisions
+
+**[Proposed]** is the author's recommendation, open to review. **[Owner]** needs the owner's call; §17 lists the options and a recommendation.
+
+| # | Decision | Status |
+|---|---|---|
+| D1 | Pin the Temporal CLI per OpenCompany release (exact version and per-platform SHA-256, versioned path); record the server version beside each `temporal.db`; refuse a database created by a different server minor | Proposed |
+| D2 | How an install moves an existing `temporal.db` to a newer server | Owner |
+| D3 | Which server durable installs run on: pinned dev server with SQLite, PostgreSQL, or Temporal Cloud | Owner |
+| D4 | Set namespace retention and the needed dynamic config explicitly instead of relying on dev-server defaults | Proposed |
+| D5 | Upgrade temporalio 1.30.0 to 1.34.0, behind the replay gate of D10 | Proposed |
+| D6 | Export SDK metrics, turn on `TemporalReportedProblems`, alert on Temporal's documented set | Proposed |
+| D7 | Bound inbound events at the edge; pass graphs, parameter snapshots and large events by reference | Proposed |
+| D8 | Route events and manage Schedules by their known ids from the app database; Visibility only for operations | Proposed (the delivery guarantee is Owner) |
+| D9 | Make cancellation real: re-raise it in activities, stop swallowing it in workflow loops, cancel before terminating | Proposed |
+| D10 | Upgrade contract and versioning: replay corpus gate, startup verification, patch lifecycle, dispatch frozen into run input | Owner (the contract); Proposed (the mechanics) |
+| D11 | An explicit retry policy and time budget on every activity | Proposed (the LLM budget values are Owner) |
+| D12 | What a failed run and a bug in workflow code look like in Temporal | Owner |
+| D13 | Human waits (approvals, reviewed delivery) move from polling activities to workflow messages and timers | Owner |
+| D14 | Label work in the Web UI; classify expected failures as BENIGN | Proposed |
+| D15 | Worker topology: protect the API event loop, make the standalone worker register what the embedded one does, queue-wide rate limits, priority, a registration check at boot | Proposed (which topologies are supported is Owner) |
+| D16 | Testing: replay corpus, time skipping, activity environment, determinism lint, golden identity test | Proposed |
+| D17 | Features evaluated and not adopted, each with the condition that would reopen it | Proposed |
+| D18 | Fix the security defects found on the way (webhook authentication, event scoping, secrets in history) ahead of and outside the phases | Proposed |
+
+## 4. What OpenCompany uses today
+
+### 4.1 Used, and used the documented way
+
+| Feature | Where | Note |
+|---|---|---|
+| Updates with validators for control | `services/temporal/workflow.py:249-263`, `workflow_control_workflow.py:106-200` | Revisioned state; stale requests rejected before they reach history; deterministic Update ids `control:{rev}:...` (`services/deployment/execution_control.py`) |
+| `@workflow.init` where handlers read input | MachinaWorkflow, controller, EmployeeJob, WorkspaceTaskController | Handlers can run before `run()`; the docs require this |
+| Continue-as-new only from the main loop, after `all_handlers_finished` | `workflow_control_workflow.py:605-716` | Matches "don't call Continue-as-New from the handlers" |
+| Deterministic "first completed" wait | `workflow.py:1101-1133` | Walks an insertion-ordered dict; never `asyncio.wait`, whose set results the SDK calls non-deterministic |
+| Typed search attributes | `executor.py:186-193`, `workflow_control_workflow.py:742-749` | The dict API is deprecated; not used |
+| Schedules, not Cron Jobs | `services/temporal/schedules.py` | "We recommend using Schedules instead of Cron Jobs" |
+| No workflow execution or run timeouts | `workflow.py:148-155` | "We generally do not recommend setting Workflow Timeouts" |
+| Deterministic ids, ABANDON and `ALLOW_DUPLICATE_FAILED_ONLY` for trigger runs | `trigger_listener_workflow.py:412-445` | The event id is the idempotency key |
+| Update-with-Start with `USE_EXISTING` | `services/node_invocations.py:47-58` | Workspace tasks |
+| At-most-once for approved sends | `approved_tool_call_workflow.py:30-80`, `services/approvals/tool_calls.py:367-379` | `maximum_attempts=1` plus a claim token |
+| Durable timer for the 5 s Undo window | `approved_tool_call_workflow.py:41-46` | `workflow.now()` arithmetic, tested with time skipping |
+| LLM errors classified at the activity boundary | `agent_activities.py:246-303` | Category decides `non_retryable`; Retry-After becomes `next_retry_delay`; provider SDK retries off |
+| Token streaming outside history | `services/chat/stream.py`, `services/chat/hub.py` | Per-attempt segments; the shape Temporal's AI demo uses |
+| Interceptors | `client.py:92-97` | `TracingInterceptor` registered once on the client; workers inherit it |
+| Worker rebuilt on every crash restart | `services/temporal/lifecycle.py`, `worker.py` | A Worker object is single-use |
+| Poller autoscaling; resource-based tuner for I/O-bound LLM work | `worker.py:538-571` | Needs server 1.28 or later (see H4) |
+| Replay tests | 18 files in `server/tests/temporal/` | Including synthetic pre-patch histories and `max_cached_workflows=0` |
+| Media by reference, byte budgets | `services/media/`, `agent_context_pressure.py`, `plugin/base.py:1015-1038` | Keeps results under the 2 MB payload limit |
+
+### 4.2 Not used, and relevant
+
+| Feature | What it gives | Availability | Where it fits | Phase |
+|---|---|---|---|---|
+| `schedule_to_close_timeout` | A total time budget across retries | GA | LLM step, legacy tools, polls | 0 |
+| `schedule_to_start_timeout` | Detects that a routed worker is gone | GA | Browser owner queues, Pi driver queue | 0 |
+| Handles by known id | Describe, signal, pause or delete without Visibility | GA | `dispatch.emit`, Schedule management | 0-1 |
+| SDK metrics (Prometheus, OpenTelemetry or `MetricBuffer`) | Alerts and tuning | GA | `Runtime` | 1 |
+| `TemporalReportedProblems` | Marks executions stuck in workflow-task failures | Server 1.30.1+, dynamic config | Boot reconcile, health view | 1 |
+| Time skipping, `ActivityEnvironment`, replay corpus | The documented test toolkit | GA | `server/tests/temporal/` | 1 |
+| `activity.cancellation_details()`, `activity.is_worker_shutdown()` | Tell pause, reset and shutdown apart | SDK 1.12 / 1.18 | The per-type activity wrapper | 0-2 |
+| `failure_exception_types` | Choose between "fail" and "stuck until fixed" for bugs | Experimental | Every workflow | 2 |
+| `workflow.info().get_current_history_size()` | Byte-based rollover | GA | Controller, AgentWorkflow | 2 |
+| `first_execution_run_id` on handles | Server-enforced run-chain identity | GA | Control plane | 2 |
+| `ApplicationErrorCategory.BENIGN` | Expected failures logged at DEBUG, kept out of failure metrics and error spans | SDK 1.12 | LLM retries, permit waits | 3 |
+| `static_summary`, `static_details`, `summary=`, `set_current_details` | Readable rows and timelines in the Web UI | SDK 1.9, UI 2.34.6 | Every start, LLM and tool activities, waits | 3 |
+| Heartbeat details | Resume long work from a checkpoint | GA | Browser, coding, polls | 3 |
+| Task queue Priority | Chat runs ahead of background work | Server GA 1.31 | Chat runs, agent steps | 3 |
+| Fairness keys | Per-employee or per-user fairness | Server GA 1.31 plus `matching.enableFairness` | Multi-employee installs | 3 |
+| `max_task_queue_activities_per_second`, `temporal task-queue config` | Queue-wide rate limits, changeable at runtime | Config API server 1.29 | Per-provider LLM queues | 3 |
+| `start_delay` | A delay with no workflow code running | GA | The Undo window | 3 |
+| Eager workflow start | Lower start latency when starter and worker share a process | Server 1.29; labelled preview | Chat-run starts | 3 (experiment) |
+| Activity operations (pause, reset, update options) | Operator recovery without restarting | Preview; server 1.28 | Runbook | 3 |
+| External Storage | Built-in claim check for large payloads | Preview; SDK 1.25 | Transcripts, events | 4 (evaluate) |
+| Payload codec and codec server | Encryption or compression of history | GA | Docker and VM installs | 4 (Owner) |
+| Standalone Activities | Durable single jobs without a workflow | GA in server 1.32, SDK 1.33 | Direct node runs | 4 (evaluate) |
+| `temporalio.envconfig`, TLS, API keys | Connect to a remote cluster or Temporal Cloud | GA | `DISTRIBUTED_MODE` | 4 (Owner) |
+
+## 5. Defects and risks
+
+Ranked. "Replay impact" says whether the fix changes the commands a workflow emits. If it does, the fix needs a patch gate or another versioning approach (§8.5).
+
+### Critical
+
+**C1. One large webhook can wedge a deployment's controller, or be lost silently.**
+- *What happens.* The generic webhook route reads the whole body with no limit (`server/routers/webhook.py:78`) and stores a JSON body twice, raw and parsed (`:93-94`), along with every request header (`:91`). Webhook events are not scoped to a workflow, so every controller with a webhook trigger receives the signal. When the controller spawns the run it builds `{**data, "_event_envelope": event}` (`trigger_listener_workflow.py:392-395`), four copies of the body, and adds the run graph and the deployment's full parameter snapshot to the child's start input (`:446-449`, `:489-490`). Nothing checks the size before `start_child_workflow`. The outcome depends on the body size:
+ - Roughly 0.5-1 MB (less with a large parameter snapshot): the signal fits under 2 MB but the child-start command does not. When the server reports its payload limits, SDK 1.30 fails the controller's workflow task with `PAYLOADS_TOO_LARGE` (`temporalio/worker/_worker.py:769-779`, `_workflow.py:487-500`) and the workflow stays open; every retry rebuilds the same command, so the deployment processes no further events until Reset. When the server does not report limits, the server closes the controller instead. Which happens depends on the unpinned dev-server version.
+ - Above about 1 MB: the signal itself is rejected; the webhook still answers 200 and the event is lost.
+ - Roughly 0.25-0.5 MB: the per-firing MachinaWorkflow wedges instead, because its activity inputs carry the trigger output twice.
+- *Rule.* 2 MB per payload, 4 MB per gRPC request. With Python SDK 1.23+ an oversized payload fails the workflow task and "The Workflow is not terminated and remains open, so you can deploy a fix and allow the Workflow to continue" (https://docs.temporal.io/troubleshooting/blob-size-limit-error).
+- *Fix (D7).* Cap request bodies at the edge (HTTP 413); keep one copy of the body; pass bodies over a threshold by reference (a workspace `FileRef`); check the child input size in the controller and dead-letter the event instead of starting. *Replay impact:* smaller payloads change no commands; the in-workflow size check does (patch gate).
+
+**C2. An upgrade can strand open runs, and nothing would notice.**
+- *What happens.* There are 19 patch markers at 24 call sites and no `workflow.deprecate_patch`. Three markers were deleted outright (`agent-delegation-acquire-cancellation-v1`, `agent-tool-call-identity-v2`, `agent-task-manager-delegation-v1`); the SDK's core fails replay of any history holding a marker the code no longer calls ("Non-deprecated patch marker encountered", `workflow_machines.rs:1771-1782`, read from the core source). Activity type names `node.{type}.v{version}`, their queues and retry policies are resolved from the live plugin registry during replay (`workflow.py:1053-1080`), and a `workflow.patched` call is itself conditional on a plugin attribute (`:856`); a plugin `version` bump changes the scheduled activity type for every open run. The agent child id format changed without a patch. CI replays almost only histories that the current code just produced: two fixtures exist and one is orphaned, the four replay tests that need a real CLI never run in CI, there is no verification at startup, and the app does not surface nondeterminism errors.
+- *Rule.* "Fail CI if any error is encountered during replay" (https://docs.temporal.io/develop/python/best-practices/testing-suite); replay recent histories before deploying (https://docs.temporal.io/develop/safe-deployments); patch, then deprecate, then remove after the old runs are gone (https://docs.temporal.io/develop/python/workflows/versioning).
+- *Fix (D10).* The owner sets the upgrade contract; the mechanics follow in §8.6. *Replay impact:* none for the tests and the startup check; freezing dispatch into run input is designed to produce today's commands.
+
+**C3 (security). Webhook trigger authentication is never enforced, and its secret is written to history.**
+- *What happens.* The webhook trigger offers `authentication: header` with `header_name` and `header_value` (`server/nodes/trigger/webhook_trigger/__init__.py:43-59`). No code reads them: both filters compare only the path (`:90-99`, `server/services/event_waiter.py:171-189`), `/webhook/` is a public path (`server/middleware/auth.py:34`), and the default empty path matches every webhook. The secret reaches history through the raw filter parameters and the parameter snapshot (`services/deployment/manager.py:584`, `:881`), and every request header is stored (`routers/webhook.py:91`); history is not encrypted.
+- *Fix (D18).* Enforce the header in the router before `emit` with a constant-time compare; require a non-empty path; keep secrets and raw headers out of workflow inputs. This does not wait for the phases.
+
+**C4. Every backend restart fires each deployment's Start node again.**
+- *What happens.* Boot reconcile re-arms running and paused generations by replaying the user-Start deploy path from the generation snapshot (`services/temporal/lifecycle.py:125` → `reconcile_active_controls_on_boot`, `services/deployment/handlers.py:1596-1617` → `_rearm_generation`, `:1767-1835` → `handle_deploy_workflow`, `:1822`). That path fires every `start` node (`services/deployment/manager.py:277-282`, `_fire_start_trigger` at `:546-560`) as a new MachinaWorkflow run with a random id, `<slug>-<uuid8>`, and no id reuse or conflict policy (`services/temporal/executor.py:127`, `:181-194`). Nothing marks the call as a re-arm. So after every boot, clean or not, a start-node graph runs again and repeats its side effects. A paused generation queues one more held run per restart, and all of them execute on Resume. After an unclean shutdown the re-arm runs before the recovery pause, so the duplicate starts while the generation is still running. temporal-workflow-control.md:653-656 calls re-arm "idempotent"; the mechanisms it lists do not cover start nodes.
+- *Rule.* The workflow id is the idempotency key of a start; a deterministic id with `WorkflowIDReusePolicy.REJECT_DUPLICATE` rejects a second start (https://docs.temporal.io/workflow-execution/workflowid-runid).
+- *Fix.* Give the start-node run a deterministic per-generation id (for example `<slug>-start-g<generation>`) with `REJECT_DUPLICATE`, and skip start nodes when re-arming. The owner confirms the intended behavior (§17 Q16). *Replay impact:* none (client side).
+
+### High
+
+**H1. Inbound events are routed through Visibility and dropped silently.**
+- *What happens.* `dispatch.emit` runs one Visibility List query per event and signals each match by workflow id (`services/events/dispatch.py:53-55`, `:132-158`, `:250-257`). The event is dropped, with only a log line, when Temporal is not connected (`:125-129`), the query fails (`:162-164`), it returns nothing (`:185-190`), or a signal fails (`:195-206`). Producers still answer 200 (`routers/webhook.py:69-76`, `:99-106`). A controller that has just started, has a newly registered event type, or is between two runs of a continue-as-new may not be in the index yet. A controller without push triggers never sets `ControlEventTypes`, and dispatch treats it as matching every event (`dispatch.py:177-183`, `:216-247`; `workflow_control_workflow.py:330-334`, `:739-740`).
+- *Rule.* Visibility is "eventually consistent", "an empty result is a valid response", and is "intended for operational discovery rather than a critical path for application logic. ... To coordinate between workflows, use signals, updates, or child workflows rather than reading Visibility" (https://docs.temporal.io/visibility).
+- *Fix (D8).* Route from the app database, which already holds every live generation and its deterministic controller id (`workflow-control-<id>-g<gen>`); signal by id with retries; tell the producer when delivery fails (5xx for webhooks, which providers retry). *Replay impact:* none (client side).
+
+**H2. Node activities swallow cancellation and are recorded as completed failures.**
+- *What happens.* `NodeExecutor` turns `asyncio.CancelledError` into `{success: False, error: "Cancelled"}` (`services/node_executor.py:231-239`) for every node type except `browser_agent`, `mobile_use_agent` and the Android tool. The per-type wrapper returns that dict (`plugin/base.py:1313-1388`), and the SDK records a normal return as Completed even after a cancel request (`temporalio/worker/_activity.py:358-363`). MachinaWorkflow then counts the node as failed, and a trigger-started run adds one to the deployment's pause-on-failure streak. It happens when a graceful backend shutdown outlasts `TEMPORAL_GRACEFUL_SHUTDOWN_SECONDS` (10 s on desktop, 30 s elsewhere) and when agent tool calls are cancelled with their workflow; with activity pause (§9.5) it would happen on every pause.
+- *Rule.* "For the Activity to appear canceled the exception needs to be re-raised" (https://docs.temporal.io/develop/python/workflows/cancellation).
+- *Fix (D9).* On the Temporal path re-raise; turn a worker-shutdown cancellation into a retryable `ApplicationError`, as `NodeWaitInterrupted` already does (`plugin/base.py:1347-1358`). *Replay impact:* none.
+
+**H3. Several activities retry without limit.**
+- *The LLM step:* 10 min start-to-close, 1 min heartbeat, `LLM_STEP_RETRY` with `maximum_attempts=0` and 5 s to 5 min backoff; only `NodeUserError` and `InvalidEvent` are non-retryable (`agent_workflow.py:90`, `:98`, `:3168-3173`; `_retry_policies.py:94-100`). Any other exception (a missing key, a parsing bug after the call) and every timeout retry forever, so a turn that needs more than 10 minutes, such as a long output from a local model on a CPU, is cut off and re-billed indefinitely.
+- *The legacy tool path* (`agent_workflow.py:108-141`): no retry policy (the server default is unlimited) and a fixed 10 min timeout, ignoring the plugin's declared policy. It serves manual canvas runs and generations admitted before execution control; `claude_code_agent`, `codex_agent` and `rlm_agent` declare `maximum_attempts=1` (`nodes/agent/claude_code_agent/__init__.py:251-254`) but on this path are re-run indefinitely, so a Claude Code session longer than 10 minutes never finishes.
+- *Activities with no policy at all* (so unlimited): `store_node_output_activity` (`workflow.py:618-626`) and `load_persisted_workflow_graph_activity` (`trigger_listener_workflow.py:358-362`).
+- *An oversized result* fails with `PayloadSizeError`, which Temporal makes retryable on purpose and which is not in `NON_RETRYABLE_ERROR_TYPES` (`_retry_policies.py:52-55`); under `LLM_STEP_RETRY` it repeats forever.
+- *Rule.* The default retry policy is unlimited; bound total time with `schedule_to_close` rather than attempts (https://docs.temporal.io/encyclopedia/retry-policies); "Infinite retries without circuit breaking" is listed as an anti-pattern (https://docs.temporal.io/best-practices/error-handling).
+- *Fix (D11, §9.2).* *Replay impact:* none; retry policies and timeouts are not compared on replay (§8.5).
+
+**H4. Durable state sits on the development server.**
+- *What happens.* `_install.py` downloads `https://temporal.download/cli/archive/latest` (`:46`), caches it under a fixed name (`:121`) with `known_hash=None` (`:124-136`), so the first boot fixes the version forever and nothing verifies the bytes; `company clean` keeps it and the Docker image copies it into the volume once (`docker/Dockerfile:39-43`, `:117`). Installs run whatever server their first boot downloaded. `start-dev` creates the SQLite schema only when the file is new and keeps no schema version; between server 1.27 and 1.31 the schema grew from 33 to 37 tables, and a database created by 1.27 or earlier cannot load any workflow when opened by 1.28 or later (missing `chasm_node_maps`; read from the CLI and server source). The namespace gets 24-hour retention (CLI source), while `.env.template:147-149` promises that "Temporal UI keeps showing every workflow that ever ran". `--metrics-port 0` opens `/metrics` on a random port; the comment at `_runtime.py:118` says it disables it.
+- *Rule.* "The development server is not intended for production use. It skips certain HTTP security checks" (https://docs.temporal.io/cli/command-reference/server); SQLite persistence "is meant only for development and testing, not production usage" (https://docs.temporal.io/temporal-service/persistence); a file-based SQLite Visibility schema cannot be upgraded, "you must delete the database and create it again" (https://docs.temporal.io/self-hosted-guide/visibility/sqlite); upgrade one minor version at a time (https://docs.temporal.io/self-hosted-guide/upgrade-server).
+- *Fix.* D1 to D4 (§6).
+
+**H5. There is no way to see that Temporal work is stuck.**
+- *What happens.* The runtime exports no metrics: `Runtime(telemetry=TelemetryConfig(logging=LoggingConfig(filter="ERROR")), worker_heartbeat_interval=None)` (`client.py:76-83`, `worker.py:125-136`). The log filter also hides Core's WARN lines, and nothing forwards Core logs into the app's logging. No workflow sets `failure_exception_types`, so a bug or a nondeterminism error leaves the execution Running with workflow-task failures retried forever, and nothing reads `TemporalReportedProblems`.
+- *Rule.* Start alerting with zero pollers, `NonDeterminismError`, `GrpcMessageTooLarge`, workflow-task schedule-to-start latency and `RESOURCE_EXHAUSTED` (https://docs.temporal.io/best-practices/worker-alerting); a non-determinism failure leaves executions "in Running status the whole time" (https://docs.temporal.io/encyclopedia/detecting-workflow-failures).
+- *Fix.* D6 (§7).
+
+**H6. Cron's overlap policy never applies to the graph run.**
+- *What happens.* The Schedule uses `ScheduleOverlapPolicy.SKIP` (`schedules.py:150`, `:226-229`), but the workflow it starts, `CronTriggerWorkflow`, starts the graph run as an ABANDON child and returns as soon as the start is acknowledged (`nodes/scheduler/cron_scheduler/_workflow.py:176-198`). SKIP only covers that short-lived workflow, so slow runs pile up without limit. The comment at `schedules.py:187-189` says the opposite. Schedule action ids also carry a literal `{{.ScheduledStartTime}}` (`schedules.py:65-71`), and the graph run's id uses `workflow.now()` rather than the scheduled time.
+- *Rule.* SKIP discards an action only while "a previously started Workflow Execution is still running" (https://docs.temporal.io/schedule); "The Action's timestamp is appended to the Workflow Id" (same page); no templating is documented.
+- *Fix.* Either await the child in `CronTriggerWorkflow` or let the Schedule start the run directly and read `TemporalScheduledStartTime`; the owner chooses the overlap behavior (§17). *Replay impact:* yes (patch gate or a new workflow type).
+
+**H7. The documented worker-pool rollback strands runs, and the standalone worker registers different code.**
+- *What happens.* `.env.template:100-107` presents `TEMPORAL_WORKER_POOL_ENABLED=false` as a rollback. Routing is frozen into each run's input (`executor.py:29-48`) and into the controller's trigger specs (`manager.py:886-887`), so MachinaWorkflow keeps routing node activities to plugin queues (`workflow.py:556`, `:863`, `:1078`) that nothing polls once the flag is off (`lifecycle.py:226-229`; the plugins' default queue is `machina-default`). With no `schedule_to_start` timeout those activities wait forever; only cron Schedules are refreshed in place on reboot. The standalone worker (`worker.py:792-947`) registers no `plugins=` (so no `CronTriggerWorkflow`), no `poll.*` activities and no pool queues, hard-codes namespace `default` and queue `machina-tasks`, and without `DISTRIBUTED_MODE` imports no plugins at all, so its workflow code schedules `execute_node_activity` where the embedded worker schedules `node.<type>.v<n>`; a workflow task that moves between the two fails replay.
+- *Rule.* "All Workers polling the same Task Queue must register the same Workflow Types and Activity Types" (https://docs.temporal.io/task-queue).
+- *Fix (D15).* Correct the rollback text and make the flag affect only new runs (refuse to turn it off while frozen runs exist); build every worker's registration from one function; refuse to start a standalone worker that differs.
+
+**H8. Workflow loops swallow cancellation.**
+- *What happens.* When awaited activity work is cancelled, the awaiting workflow code receives `ActivityError` whose cause is `temporalio.exceptions.CancelledError`; with the default `TRY_CANCEL`, core resolves the activity at once (`temporalio/worker/_workflow_instance.py:1897-1937`). `ActivityError` is an ordinary `Exception` (`temporalio/exceptions.py:13`, `:288`), so `except Exception` catches it. The SDK's helper `is_cancelled_exception` (`temporalio/exceptions.py:459`) is used in five places in the repo, none of them these:
+ - Legacy polling listener: a cancel during the poll activity is swallowed and polling continues (`polling_trigger_workflow.py:193`); a cancel during the graph lookup is swallowed and the child run is still started (`:297`). Legacy deployments stop listeners with `cancel()` (`services/deployment/manager.py:1159-1162`).
+ - Legacy trigger listener: swallowed, and the listener keeps listening; after a cancel during the graph lookup or the idle broadcast, the child is still started (`trigger_listener_workflow.py:276` and the handlers inside it).
+ - Controller poll tasks (`workflow_control_workflow.py:849-866`), cancelled by `task.cancel()` from reset, `replace_graph` or continue-as-new: the loop catches `asyncio.CancelledError`, but the cancelled activity arrives as `ActivityError` and falls into `except Exception`. A trigger retired by Apply can keep polling, and during continue-as-new the `asyncio.gather` at `:615` can wait on the still-running task and stall the rollover; execution control narrows this window without closing it.
+- *Rule.* Cancellation must be re-raised to take effect; `temporalio.exceptions.is_cancelled_exception(e)` tells it apart from other failures.
+- *Fix (D9).* Re-raise when `is_cancelled_exception(e)`; check the retired flag after every await in poll loops; the determinism lint (§8.1) keeps it fixed. *Replay impact:* yes, for runs whose history recorded a swallowed cancel (patch gate).
+
+**H9. Boot recovery can stall, and one transient error can fail a generation for good.**
+- *What happens.* After an unclean shutdown, with the default `WORKFLOW_CONTROL_CRASH_RECOVERY=pause`, the serial boot loop awaits each recovery pause in turn (`handlers.py:1614-1617` → `handle_pause_workflow` → the shielded drain at `:2307-2308`). The drain sends `wait_for_checkpoint` Updates with no client deadline (`services/deployment/execution_control.py:145-169`), and the SDK polls until the Update has an outcome (`temporalio/client/_workflow.py:1953-1976`). Admitted actions include 24 h node activities and LLM steps that retry without limit, so recovery can wait for hours; nothing bounds the loop, and the dev-server watchdog starts only after it returns (`lifecycle.py:125-127`). Separately, a single `NOT_FOUND`, or any error whose text contains "not found" or "already completed", counts as "controller missing" (`handlers.py:736-748`, `:903-912`; `services/deployment/execution_control.py:63-76`) and moves a v1 generation to the terminal `failed` state (`handlers.py:834-841`), which only Reset and Start clear. The code never tells a missing workflow from a missing namespace, although the SDK exposes `RPCError.grpc_status` and both failure detail types.
+- *Rule.* Give Update calls a client timeout; with no worker polling, the client keeps retrying (https://docs.temporal.io/design-patterns/early-return). `DescribeWorkflowExecution`, not Visibility, answers whether an execution exists (https://docs.temporal.io/visibility).
+- *Fix.* Run recovery pauses as background transitions, as the boot loop already does for pausing and resuming rows (`handlers.py:1600-1606`); start the watchdog first; give every Update a deadline; decode `NotFoundFailure` against `NamespaceNotFoundFailure` and require two Describe confirmations before failing a generation. *Replay impact:* none.
+
+**H10. The controller's continue-as-new input carries the whole graph once per trigger, and an oversized carry fails the deployment.**
+- *What happens.* Each trigger spec embeds the nodes, edges and parameter snapshot (`manager.py:874-889`; rewritten by `replace_graph`, `workflow_control_workflow.py:303-306`) and is carried across continue-as-new (`:684-688`). Above 1.9 MB the controller raises a non-retryable `ControllerContinuationTooLarge` (`:705-711`), which fails it; boot recovery or Resume rebuilds it with the same specs and it fails again at the next rollover. Triggers are added after the start by unacknowledged `register_trigger` signals, one full graph copy each, resent on every boot (`manager.py:945-956`).
+- *Rule.* "Pass minimal state"; "Exceeding the 2 MB payload limit on Continue-As-New input ... Use external storage" (https://docs.temporal.io/design-patterns/continue-as-new).
+- *Fix (D7).* Carry trigger metadata and a generation reference; load the graph in activities; pass triggers in the start input. *Replay impact:* gate on a carried version so that only post-rollover runs change.
+
+**H11 (security). Task-completion events are not scoped to their workflow and can cross users.**
+- *What happens.* `_broadcast_task_event` sets `data.workflow_id` but not the envelope's `workflow_id` (`server/nodes/agent/_events.py:109-134`), so dispatch adds no workflow scope, and the filter that runs (`event_waiter.py:214-259`) matches everything with default parameters. A Task Completed trigger in another workflow, or another user's workflow in `AUTH_MODE=multi`, fires with this task's result as its input.
+- *Fix (D18).* Set `envelope.workflow_id`. *Replay impact:* none.
+
+**H12. Cleanup is skipped on failure, cancellation and Reset, and Reset misses Pi inline runners.**
+- *What happens.* Agent children start with the default parent close policy, TERMINATE (`workflow.py:820-828`; `employee_job_workflow.py:117`, `:120`; `agent_workflow.py:593-597` for the teammate child of `DelegatedTaskWorkflow`, `:1728-1732` for direct delegates). When an uncontrolled run fails, MachinaWorkflow stops without cancelling its running handles (`workflow.py:905-908`); the children are then terminated by policy. Reset terminates every execution it finds instead of cancelling it (`services/deployment/handlers.py:992-994`, `:2591`), and termination never reaches workflow code. That skips `DelegatedTaskWorkflow`'s delegation cancellation and permit release (`agent_workflow.py:627-725`), AgentWorkflow's `employee.job.failed` and browser-claim release (`:834-856`) and delegation cleanup (`:1506-1554`), EmployeeJobWorkflow's failure marker and MachinaWorkflow's unregister. Reset compensates browser claims and Workspace tasks, but nothing sweeps team tasks, subagent permits or employee jobs; the leftovers stay "active" without blocking the next generation. This fits the open "Reset leaves team tasks and permits" bug. Separately, `PiInlineDelegationWorkflow` is started from a tool handler with no search attributes (`services/handlers/tools.py:378-386`, `pi_inline_delegation.py:219-226`), so it is its own root and neither Reset query finds it (`handlers.py:844-900`). It registers in the controller's `live_roots`, which Reset never reads, so inline Claude Code or RLM delegations to a Coding Agent survive Reset and workflow deletion.
+- *Rule.* "In most cases, canceling is preferable because it allows the Workflow to finish gracefully" (https://docs.temporal.io/develop/python/workflows/cancellation); "Prefer Cancellation" (https://docs.temporal.io/encyclopedia/workflow/cancellation-and-termination).
+- *Fix (D9).* `REQUEST_CANCEL` with `WAIT_CANCELLATION_COMPLETED` for agent children; cancel and drain on failure; Reset cancels, waits a bounded grace period, then terminates stragglers, and also acts on `live_roots`; start the Pi runner with `EventWorkflowId` like its siblings. *Replay impact:* the parent close policy is not compared on replay (§8.5); the new cancel commands in MachinaWorkflow are (patch gate); Reset and the Pi start options are client side.
+
+### Medium
+
+**M1. All activities run on the API server's event loop.** The 48 `@activity.defn` functions are `async def`, no worker passes `activity_executor`, and the embedded workers run on uvicorn's loop (`main.py:393`). A blocking call in any of about 150 plugins stalls HTTP, WebSocket, polling and heartbeats; one known case is `json.dumps` of transcripts up to about 1 MB on every LLM step (`agent_activities.py:668`). Docs: "By default, Activities should be synchronous rather than asynchronous"; blocking an async activity "blocks your event loop and the rest of Temporal" (https://docs.temporal.io/develop/python/best-practices/python-sdk-sync-vs-async); don't deploy workers "side-by-side with other application code" (https://docs.temporal.io/develop/safe-deployments). Fix in §11.1.
+
+**M2. Schedules are found through Visibility, and ids built from labels collide.** Pause, delete, Apply and Reset find Schedules with `list_schedules` (`schedules.py:337-355`, `:398-409`; `services/employees/safe_apply.py:85`, `:108`, `:177-200`; `handlers.py:2561`; `manager.py:997`), although the ids are deterministic; a Schedule created moments earlier can be missed while strict mode reports success. Listener and Schedule ids come from slugged labels, so "Orders 1" and "Orders-1" collide: the second push trigger is silently ignored (`workflow_control_workflow.py:325-329`) and the second cron node overwrites the first Schedule (`schedules.py:254-295`). Fix: address Schedules by id; derive ids from immutable node ids (a one-time migration, §17).
+
+**M3. Polling triggers lose events.** An id whose detail fetch fails is still recorded as seen and never emitted; `post_emit` (for example Gmail mark-as-read) runs inside the poll activity before the workflow has recorded the event (`services/plugin/polling.py:207-270`), so a lost attempt hides the message from the `is:unread` query; a failed child start is marked seen and never retried (`polling_trigger_workflow.py:203-227`). The poll activity has no heartbeat timeout and reuses one activity id every cycle (`workflow_control_workflow.py:809`, `:834-838`). Fix in §10.5.
+
+**M4. Human waits hold worker slots for days.** The approval gate polls the database every 10 s inside an activity with a 24 h start-to-close and unlimited retries (`nodes/workflow/approval_gate/__init__.py:48`, `:181-185`, `:235-277`); reviewed employee delivery waits up to 31 days in one activity (`employee_job_workflow.py:134-139`); permit waits poll every second. Pause checkpoints wait for them. Temporal's pattern is a Signal or Update plus `workflow.wait_condition(..., timeout=...)` (https://docs.temporal.io/design-patterns/approval). D13 (Owner).
+
+**M5. Failures don't look like failures.** MachinaWorkflow and AgentWorkflow complete with `success: False` (`workflow.py:913-930`), so Temporal status, Visibility and metrics never show a failed run. Without `failure_exception_types`, an unexpected exception in workflow code retries the workflow task forever; a malformed `register_trigger` signal does that to a controller (`workflow_control_workflow.py:325-334`), and a signal handler that raises any other exception is "stuck" by design (https://docs.temporal.io/handling-messages). D12 (Owner).
+
+**M6. Inputs grow with the graph.** Every node activity and agent child input carries the full node list, edges, parameter snapshot and upstream inputs (`workflow.py:708-771`). All ready nodes are scheduled in one workflow task, and a task whose commands exceed the 4 MB gRPC limit terminates the execution ("Terminated Executions do not retry", https://docs.temporal.io/troubleshooting/blob-size-limit-error). The run result returns every output (`:924-930`). Fix (D7): pass references; the activity already loads parameters from the database.
+
+**M7. A failed search-attribute registration is reported as harmless.** Registration catches the error and returns a status that the caller ignores (`search_attributes.py:175-183`, `client.py:126-137`), but every start sets custom search attributes, which the server rejects with "search attribute ... is not defined". Fix: fail loudly; pass `--search-attribute` when the dev server creates the namespace.
+
+**M8. Routed queues have no `schedule_to_start`.** Browser owner queues (`services/browser_owners.py:46-47`; `agent_workflow.py:144-155`; `workflow.py:788-792`) and the Pi driver queue check owner liveness only when work is bound (`browser_owners.py:60-68`). An owner that dies afterwards strands its tasks silently. Docs: set `schedule_to_start_timeout` on worker-specific queues and retry the sequence elsewhere (https://docs.temporal.io/design-patterns/worker-specific-taskqueue).
+
+**M9. LLM calls bypass the queue built for them.** The LLM and compaction steps run on `machina-tasks` (`agent_workflow.py:3168-3173`), not `ai-heavy` as `agent_activities.py:9-13` says, so the ai-heavy tuner and rate limit never apply, and they share 100 slots with 31-day deliveries. Rate limits are per worker only (`worker.py:464-474`, `:618`). Fix in §11.3.
+
+**M10. The determinism sandbox is off everywhere.** All 14 workflow definitions pass `sandboxed=False`. Edge conditions run a user-supplied regular expression on the workflow thread and log through structlog rather than `workflow.logger` (`services/execution/conditions.py:30-32`, `:217-223`); a catastrophic pattern would block the thread past the SDK's deadlock detector. Docs: "Skipping Workflow Sandboxing results in a lack of determinism checks" (https://docs.temporal.io/develop/python/best-practices/python-sdk-sandbox). Fix in §8.1.
+
+**M11. AgentWorkflow history grows quickly and its continue-as-new input carries dead weight.** Each LLM step input records the whole transcript, the tool definitions and the system prompt (about 12 + 18k events per turn with k tools); with a transcript near its byte budget, history passes 10 MB in about 25 turns. Continue-as-new happens only at a clean turn end, and its input carries the stale seed conversation (up to 1 MB), duplicated tools and parameter snapshot, and unbounded thinking text, which can exceed 1.9 MB and raise `AgentContinuationTooLarge` (`agent_workflow.py:2989-3014`). Fix in §13.4.
+
+**M12. Boot recovery is not safe with several backends.** The crash marker is one key in the shared cache table, read and overwritten at every boot and stamped clean by every graceful shutdown (`handlers.py:1452`, `:1481-1508`, `:2835`), and every backend with Temporal enabled reconciles every active row (`main.py:379-396`, `lifecycle.py:125`, `core/database.py:3105-3120`). `services/deployment` never consults `DISTRIBUTED_MODE` or a replica id. A replica that boots while another runs reads "running", treats it as a crash and pauses every running generation; one replica's clean stop can hide another's crash; every replica re-arms every generation (C4). Revision checks keep the pauses themselves from conflicting. Fix with a per-replica marker and one elected reconciler, if multi-backend installs are supported (§17 Q14).
+
+### Low
+
+- **L1.** Dead code: `emit_event_activity` (`activities.py:389`), `WSConnectionPool` (`ws_client.py`), the "heartbeat on every non-matching broadcast" branch (`activities.py:275-276`; `/ws/internal` is never a broadcast target), unused helpers in `agent_activities.py:2599-2612`, and the `EventSource` and `EventReceivedAt` search attributes, registered and never set.
+- **L2.** Update calls with no client timeout (`node_invocations.py:52`, `handlers.py:708`); with no worker polling, the client waits indefinitely.
+- **L3.** Pi Updates without validators (`pi_workflow.py:79-107`); Pi signal handlers that run activities (`pi_control.py:36-55`); the Workspace controller rejects inside Update handlers instead of validators, so rejected calls still cost history and the per-run Update quota (`workspace_tasks_workflow.py:82-148`).
+- **L4.** Three LLM call sites keep their SDK's own retries under Temporal's (`nodes/vision/vision_analyze/__init__.py:310`, `:348`, `:399`; `services/memory/vector_store.py:239`; `nodes/translate/_providers/llm.py:168`).
+- **L5.** `WebhookSource` puts every event on an unbounded queue that nothing drains (`services/events/webhook.py:121`, `push.py:27-31`).
+- **L6.** The `temporal_start` and `temporal_stop` WebSocket commands have no owner check (`services/temporal/_handlers.py:35-55`); with login off (the default) any page on the machine that passes the Origin check can stop the engine.
+- **L7.** Worker identities are not unique across hosts; `worker.py:290` imports the private `load_default_build_id`.
+- **L8.** `_get_node_inputs` and the dependency map iterate sets (`workflow.py:940`, `:1192-1194`). Only in-memory key order varies: the default JSON converter writes payloads with `sort_keys=True` (`temporalio/converter/_payload_converter.py:639-643`), so neither commands nor payload bytes change. Iterate lists anyway, to keep the rule simple.
+- **L9.** The execution-control Updates carry no explicit id (`execution_control_activities.py:17-27`); their handlers are idempotent, so this is hygiene.
+
+## 6. Platform: server, persistence and versions
+
+### 6.1 What runs today
+
+| Item | Today |
+|---|---|
+| Python SDK | temporalio 1.30.0 (2026-07-02) in `server/uv.lock`; floor `>=1.25.0` in `server/pyproject.toml:170` |
+| Latest SDK | 1.34.0 (2026-09-30) |
+| Dev server | `temporal server start-dev` spawned by `services/temporal/_runtime.py:137-159` with `--port --ui-port --db-filename --metrics-port 0 --log-level warn --namespace` and three SQLite pragmas |
+| CLI version | "latest" at first boot, cached forever; today that resolves to v1.9.1 (server 1.32.0, UI 2.54.1). v1.9.2 is a pre-release build with a server release candidate |
+| Retention | 24 hours (CLI source); never changed by the app |
+| Dynamic config | None passed |
+| Metrics | SDK: none. Server: `/metrics` on a random port, never scraped |
+
+### 6.2 The SQLite schema hazard
+
+The CLI calls `SetupSchema` only when the database file does not exist, in every release from v1.3.0 to v1.9.1; the script uses plain `CREATE TABLE` and keeps no schema version table. The schema changed between server minors:
+
+| Server | Tables | New |
+|---|---|---|
+| 1.27 | 33 | |
+| 1.28 | 34 | `chasm_node_maps` |
+| 1.29 | 36 | `tasks_v2`, `task_queues_v2` |
+| 1.31 | 37 | `current_chasm_executions` (schema v1.11) |
+
+A database created by 1.27 or earlier, opened by 1.28 or later, cannot load any workflow. One created by 1.30 or earlier, opened by 1.31 or 1.32, fails only for newer features such as standalone activities. Opening a newer database with an older server is unsupported. (CLI and server source at release tags; not reproduced.)
+
+Today no install changes server version by itself, because the cached binary is never replaced. The hazard appears the day the version is pinned or bumped (D1), and on any machine where the cache is deleted but the database is kept.
+
+### 6.3 Pinning (D1)
+
+- Pin `version = 1.9.1` (server 1.32.0) and the per-platform SHA-256 from that release's `checksums.txt` in a committed file. The versioned URL is `https://temporal.download/assets/temporalio/cli/releases/download/v1.9.1/temporal_cli_1.9.1_<os>_<arch>.<zip|tar.gz>`; it returns the same bytes and ETag as the GitHub asset and `checksums.txt` sits beside it. This URL shape is observed behavior, not documented; the documented fallback is the GitHub release asset.
+- Download with `pooch.retrieve(known_hash="sha256:...")` into a versioned directory and file name, so an upgrade keeps the old binary for rollback.
+- Write `temporal.db.server.json` (server version, CLI version, creation time) when the database is created, and refuse to start the dev server on a different server **minor**; patch releases of the same minor are allowed, so security fixes can ship.
+- Apply the same pin to the Docker image (replace the copy-once volume step with a versioned path) and the desktop bundle.
+- Record the pin in `cli/` predeploy checks so CI downloads the same binary.
+
+### 6.4 Moving an existing database (D2, Owner)
+
+| Option | How | Cost |
+|---|---|---|
+| A. Keep the binary that made the database | Store each database's server version; run that binary; offer an explicit, staged upgrade one minor at a time on a copy of the file | Most code; old servers keep running on old installs and miss security fixes |
+| B. Recreate the Temporal database and re-arm from app state | Pause deployments, stop the server, move `temporal.db` aside, start the pinned server, run the existing boot reconcile (`reconcile_active_controls_on_boot`), which re-arms generations from their stored graph snapshots | Loses in-flight runs and history (closed history is already deleted after 24 h); needs a "your running workflows will restart" notice |
+| C. Move durable installs to PostgreSQL | The documented production store with a schema tool (`temporal-sql-tool`) | Larger footprint; see D3 |
+
+Recommendation: B for desktop installs, because their state of record is the app database and runs are already paused on every unclean restart; C, or B with a pinned server, for Docker and VM installs, chosen under D3. Option B depends on the upgrade contract (D10): if open runs must survive upgrades, B is allowed only between releases that declare it.
+
+### 6.5 Which server durable installs run on (D3, Owner)
+
+| Option | For | Notes |
+|---|---|---|
+| Pinned dev server with SQLite (D1, D4) | Desktop; small single-user installs | Documented as development-only: single writer, one history shard, no authorization, no schema upgrades |
+| PostgreSQL-backed Temporal server | Docker and cloud VM installs meant to run for months | Production-grade store with schema upgrades; Temporal publishes server images and Helm charts; adds a database and a process to supervise |
+| Temporal Cloud | Hosted, multi-tenant or paid installs | Needs TLS or API keys and `envconfig` in `client.py` (not supported today); payload codec strongly advised |
+
+### 6.6 Retention and dev-server flags (D4)
+
+- Set retention explicitly (recommendation: 7 days) after the namespace exists, with the operator service (`UpdateNamespace`) or `temporal operator namespace update --retention`, and correct `.env.template:147-149`. Retention applies only to executions that close afterwards.
+- Pass `--search-attribute Name=Type` for the seven custom attributes; it applies only when the namespace is created, so keep the operator-API registration for existing databases (M7).
+- Pass explicit `--dynamic-config-value` entries for what the app relies on, instead of each CLI version's defaults:
+ - `system.numConsecutiveWorkflowTaskProblemsToTriggerSearchAttribute=5` (D6)
+ - `frontend.WorkerHeartbeatsEnabled=true`, `frontend.ListWorkersEnabled=true` if worker heartbeating is turned on
+ - `matching.enableFairness=true` only if fairness keys are adopted
+ - `history.enableStandaloneActivityOperatorCommands` only if standalone activities are adopted
+- Bind the server and UI to loopback explicitly (`--ip 127.0.0.1`, `--ui-ip 127.0.0.1`); consider `--headless` on Docker and VM installs, where the Web UI shows every payload in plain text; `--ui-disable-news-fetch` on all installs.
+- Fix the `--metrics-port` comment; give the server a fixed metrics port only if something scrapes it.
+
+### 6.7 SDK upgrade 1.30.0 to 1.34.0 (D5)
+
+No API the repo uses was removed (checked by grep and an AST diff); core dependencies are unchanged (`nexus-rpc==1.4.0`). What changes:
+
+- `payload_limits` moved from `DataConverter` to `Client.connect(payload_limits=PayloadLimitsConfig(...))` and `PayloadSizeWarning` was removed (1.31); `services/media/limits.py:14` names the removed class.
+- From 1.33, calling continue-as-new from an Update handler fails the workflow task (the repo never does).
+- Fixes that matter here: cancellations no longer swallowed (1.31); try-cancel child and duplicate activity-cancel nondeterminism fixed (1.32, 1.33); sticky pollers no longer starve new workflows when `max_cached_workflows` is small, for up to about 60 s (1.33; local mode uses 50); shutdown drains completions (1.33).
+- New and useful: `workflow.uuid7`, replay-safe OpenTelemetry providers (1.32); `ActivityHandle` operator commands and an OpenTelemetry metrics exporter (1.33); `WorkflowAlreadyStartedError.first_run_id` (1.34).
+- Gate: record histories with 1.30.0 before the bump (D16) and replay them with 1.34.0.
+
+### 6.8 Security posture of the dev server
+
+The dev server has no authorizer ("allows every API request"), and its Web UI shows every payload. It must stay on loopback, which needs the explicit flags above. The `temporal_start` and `temporal_stop` WebSocket commands need an owner check (L6). Encrypting payloads with a codec (§12.4) is an owner decision for Docker and VM installs.
+
+## 7. Observability and operations (D6, D14)
+
+### 7.1 Metrics and alerts
+
+Create the runtime with metrics before any other Temporal object: `Runtime(telemetry=TelemetryConfig(metrics=PrometheusConfig(bind_address="127.0.0.1:<port>"), global_tags={...}))` for Docker and VM installs, and `MetricBuffer` drained into the app's health view on desktop, where nothing scrapes. Keep tag values low-cardinality (no workflow ids).
+
+Start with Temporal's documented set (https://docs.temporal.io/best-practices/worker-alerting), plus two of ours:
+
+| Alert | Signal |
+|---|---|
+| No pollers on a queue with demand | `num_pollers` = 0 while tasks are scheduled |
+| Nondeterminism | `temporal_workflow_task_execution_failed{failure_reason="NonDeterminismError"}` |
+| Oversized task | `failure_reason` `GrpcMessageTooLarge` or `PayloadsTooLarge` |
+| Workflow tasks waiting | workflow-task schedule-to-start p99 above 5 s |
+| Server pushback | `RESOURCE_EXHAUSTED` on start, signal or update |
+| Retry storm (ours) | rate of attempts above N on `LLM_STEP_RETRY`, `PERMIT_WAIT_RETRY` and browser cleanup, counted with `activity.metric_meter()` in the existing interceptor |
+| Stuck executions (ours) | count of `TemporalReportedProblems IS NOT NULL` |
+
+Use the rate of attempts, not cumulative counts, and do not use `worker_task_slots_available` on the resource-tuned queues (the docs say it is not meaningful there).
+
+### 7.2 Stuck executions
+
+Set `system.numConsecutiveWorkflowTaskProblemsToTriggerSearchAttribute` (server 1.30.1+). The server then sets `TemporalReportedProblems` after five consecutive workflow-task failures. Boot reconcile and the deployment status view query `TemporalReportedProblems IN ('category=WorkflowTaskFailed','category=WorkflowTaskTimedOut')` and show "needs attention" instead of "running".
+
+### 7.3 Labels in the Web UI (D14)
+
+- `static_summary` (one line, 200 bytes) and `static_details` (Markdown, 20 KB) on every start: employee, deployment, generation, trigger; never prompts or secrets.
+- `summary=` on LLM steps (model, turn), tool activities (tool name), timers and child starts.
+- `workflow.set_current_details()` while a run waits (approval, permit, pause); it is served by a query and adds no history events.
+- Adding `summary=` to an activity already in history is not documented as replay-safe; confirm with the replay corpus before rollout.
+
+### 7.4 Logs and expected failures
+
+Replace `ExpectedLLMRetryFilter` (`core/logging.py:256-283`) with `ApplicationError(..., category=ApplicationErrorCategory.BENIGN)` for expected outcomes (provider rate limits, permit waits). BENIGN only changes logging level, failure metrics and span status; retries are unchanged. Forward Core logs at WARN through `LoggingConfig(filter=TelemetryFilter(core_level="WARN", other_level="ERROR"), forwarding=LogForwardingConfig(logger=...))`.
+
+### 7.5 Tracing
+
+`TracingInterceptor` is correct but "the earlier integration": workflow spans have no duration, and runs started by a Schedule get no workflow span unless `always_create_workflow_spans=True`. `OpenTelemetryPlugin` with `create_tracer_provider()` gives real durations (experimental; register on the client only). Spans are recorded but dropped today: `core/tracing.py` installs no exporter unless console spans are on. Choose an exporter before investing here.
+
+### 7.6 Operator runbook (CLI)
+
+Add a runbook page to docs-internal covering, with the pinned CLI:
+
+- `temporal workflow describe | show --reverse | trace | stack` to see why a run is stuck.
+- `temporal workflow reset --type LastWorkflowTask` after shipping a fix for a workflow-task failure (resetting a closed run "is not idempotent").
+- `temporal activity pause | unpause | reset | update-options` for a misbehaving activity, once H2 is fixed (preview; server 1.28+).
+- `temporal task-queue describe` for backlog, and `temporal task-queue config set --queue-rps-limit` for a provider outage.
+- `temporal schedule describe` for missed actions (`schedule_missed_catchup_window`).
+
+## 8. Workflow design (D9, D10, D12)
+
+### 8.1 Determinism and the sandbox
+
+Today's workflow code avoids every non-deterministic call the docs list: no system time, randomness, `uuid4`, environment reads or I/O, and no `asyncio.wait` or set iteration that drives commands. The exceptions are the edge-condition helper (M10) and code constants and the live plugin registry deciding commands (C2).
+
+With the sandbox off, nothing enforces this. Recommendation:
+
+1. A lint test that walks every module defining a `@workflow.defn` class and the helpers it imports, and fails on `datetime.now`, `time.time`, `random`, `uuid.uuid4`, `os.environ`, `Settings()`, `asyncio.wait`, `asyncio.as_completed`, `print`, stdlib or structlog loggers, and `except Exception` around awaited activities or children unless the block re-raises cancellation.
+2. Sandbox the small, pure workflows first (ApprovedToolCallWorkflow, NodeInvocationWorkflow, EmployeeJobDeliveryWorkflow, CronTriggerWorkflow), with `workflow.unsafe.imports_passed_through()` around imports and `SandboxRestrictions.default.with_passthrough_modules(...)` for app packages. The large workflows stay unsandboxed until their module imports are free of side effects (`services/temporal/__init__.py` registers handlers on import).
+3. Optionally replace the hand-rolled `_wait_any_complete` with `workflow.wait(..., return_when=asyncio.FIRST_COMPLETED)`, which returns ordered lists; this changes no commands.
+
+### 8.2 Messaging
+
+The control plane is in good shape (§4.1). Changes:
+
+- Turn signals that need an answer into Updates with validators: `register_trigger` (today a poison signal can wedge the controller, M5), and the Pi `pause` and `resume` handlers, which run activities.
+- Add validators to the Pi Updates and move the Workspace controller's rejections into validators (L3).
+- Give every Update call a client deadline (`rpc_timeout` or `asyncio.wait_for`) and a stable `id=` where it is retried (L2, L9).
+- Count Signals and Updates per run, not only events: limits are 10,000 signals, 2,000 total and 10 in-flight Updates per run. Each fired run costs the controller two Updates and a signal.
+
+### 8.3 Continue-as-new
+
+- Controller: carry metadata, not graphs (H10); add `get_current_history_size()` next to the event count; keep continuing-as-new only from the main loop.
+- A quiet controller never rolls over, so it never picks up new code and its markers can never be retired. Roll over periodically (for example after 7 days or N patch changes) as well as under pressure; the docs suggest periodic continue-as-new so "you're running only a couple of versions".
+- AgentWorkflow: trim the carried input (M11).
+
+### 8.4 Children and cancellation (D9)
+
+- Agent children: `parent_close_policy=ParentClosePolicy.REQUEST_CANCEL`, `cancellation_type=ChildWorkflowCancellationType.WAIT_CANCELLATION_COMPLETED`.
+- MachinaWorkflow: on failure or cancellation, cancel every handle in `running`, await them, then finish.
+- Workflow loops: never let `except Exception` swallow cancellation; check `is_cancelled_exception(e)` and re-raise (H8).
+- Reset: signal, cancel, wait a bounded grace period (for example 30 s), then terminate what is left; keep the existing database sweeps as the backstop.
+
+### 8.5 What needs a patch (corrects CLAUDE.md)
+
+CLAUDE.md says "Removing/adding start options or retry policies on recorded commands requires a `workflow.patched(...)` guard". That is stricter than Temporal needs. On replay, the SDK's core always checks the kind and order of commands: each recorded event must match the next command the code emits (`workflow_machines.rs:928-1000`). Among command attributes it compares only an activity's id and type (`activity_state_machine.rs:371-396`) and a child workflow's id and type (`child_workflow_state_machine.rs:233-262`), and only for histories that recorded the `IdAndTypeDeterminismChecks` flag, which current core records on every workflow task (`internal_flags.rs:20-23`, `:98-109`; the Rust sources ship in the temporalio 1.30.0 wheel). Task queues, timeouts, retry policies, inputs, headers, cancellation types, parent close policies and id reuse policies are never compared. The docs agree: it is "not safe to change the types or IDs of Child Workflows or Activities", while changing their options does not cause nondeterminism errors (https://docs.temporal.io/workflow-definition). One Python exception is documented: changing a timer's duration from or to zero is non-deterministic (https://docs.temporal.io/develop/python/workflows/timers).
+
+The working rule becomes:
+
+- **Needs a patch gate:** adding, removing or reordering commands (activities, children, timers, signals to external workflows, search-attribute upserts, markers); changing an activity type name or activity id; changing a child workflow type or id; changing whether a `workflow.patched` call happens.
+- **Does not need a patch, but needs the replay corpus to pass:** timeouts, retry policies, task queues, cancellation types, parent close policies, `summary=`, input and result payload shapes (the code must still accept old shapes).
+
+### 8.6 Patch lifecycle and the replay corpus (D10)
+
+- **Freeze dispatch.** Record each node's activity name, queue, timeouts and retry policy in the run input when the run is admitted, as agent tools already do (`_tool_activity_policy`), and keep registering old `node.X.vN` names for one release after a version bump. Old inputs fall back to the registry, so today's commands are unchanged.
+- **Corpus.** Record one history per workflow type and per important path (manual run, trigger run, agent turn with tools, delegation, controller rollover, cron firing, approval, workspace task, Pi run) with each release, redacted, under `server/tests/temporal/fixtures/<release>/`. CI replays every fixture of the previous release (and the one before, if the owner wants N-2) with `Replayer(workflows=<the production list>)`, built from the same function the worker uses, and fails on any error.
+- **Golden identity test.** A test lists the activity type names, child id formats and patch markers each workflow can emit and fails when one changes without a patch entry.
+- **Startup verification** (Owner, §17): on upgrade, fetch the histories of open executions (`list_workflows(...).map_histories()`) and replay them before starting workers. On failure, either refuse to start workers, or start them and mark the affected generations "needs Reset".
+- **Retirement.** Keep a table of markers in TEMPORAL_ARCHITECTURE.md with the release that added each one. A marker becomes `deprecate_patch` once no supported upgrade source can hold a run that predates it, and is removed one release after that. Never delete a marker in one step again.
+
+### 8.7 Failure semantics (D12, Owner)
+
+Two questions:
+
+1. When a run fails (a node failed, an agent gave up), should the Temporal workflow end Failed, so that Temporal status, Visibility (`ExecutionStatus='Failed'`) and metrics show it, or stay Completed with `success: False` and a `RunOutcome` search attribute?
+2. When workflow code hits a bug, should the run stay stuck until a fixed release replays it (Temporal's default, now visible through §7.2), or fail fast (`failure_exception_types=[Exception]`), losing the run but not the deployment?
+
+Recommendation: fail graph and agent runs as Failed after their bookkeeping (a patch-gated final command), and keep bugs stuck but visible, because a fixed release resumes them without losing state.
+
+## 9. Activities (D11)
+
+### 9.1 Async activities on the API loop
+
+See M1 and §11.1. Short term: a lint that flags known blocking calls (`requests`, `time.sleep`, `subprocess.run`, `sqlite3`, file I/O without `to_thread`) inside `async def` activities and plugin operations, and move CPU work such as transcript encoding into `asyncio.to_thread`.
+
+### 9.2 Timeouts and retries
+
+| Activity | Today | Proposed |
+|---|---|---|
+| LLM step | 10 min attempt, unlimited, 5 s to 5 min | `start_to_close` from the model's declared limit (local models longer); `schedule_to_close` budget (owner value, §17); non-provider exceptions non-retryable; `PayloadSizeError` non-retryable; BENIGN for rate limits |
+| Compaction | 3 attempts | Same policy as the LLM step, so a 429 burst does not end the run |
+| Legacy agent tools | 10 min, no policy (unlimited) | The plugin's declared policy and timeout on every path |
+| Node activities | 24 h attempt, 2 min heartbeat, plugin policy | The plugin's declared timeout; heartbeats that report progress, not only liveness; `schedule_to_close` per plugin |
+| Store output, load graph | No policy (unlimited) | Explicit bounded policy |
+| Poll activities | `max(30, 4 x interval)` s, no heartbeat timeout | Heartbeat timeout; per-cycle activity id (patch) |
+| Owner-routed activities | No `schedule_to_start` | `schedule_to_start_timeout` plus failover |
+| Own-permit re-acquire | 3 attempts of 1 h | `PERMIT_WAIT_RETRY` |
+| Employee delivery | 31 days, 3 attempts | Workflow wait (D13) |
+
+Add `PayloadSizeError` to `NON_RETRYABLE_ERROR_TYPES`. Plugin exceptions are converted to returned results inside the activity (`plugin/base.py:683-761`), so plugin retry policies never see an HTTP 5xx or 429. The owner decides whether graph nodes retry transient errors (§17); if so, plugins raise a typed retryable error and keep returning envelopes for user errors.
+
+### 9.3 Cancellation and heartbeats
+
+Fix H2. Use `activity.cancellation_details()` to tell pause, reset, timeout and worker shutdown apart; re-raise in each case. For long activities, heartbeat a small checkpoint (`activity.heartbeat(details)`) and read `activity.info().heartbeat_details` on retry, as the browser and coding agents could.
+
+### 9.4 Idempotency
+
+Activities run at least once. Graph-node sends (Telegram, WhatsApp, email, HTTP POST) have no idempotency key or resend guard; agent tool sends already refuse a second attempt. The owner chooses at-most-once or at-least-once with provider idempotency keys (§17). Keys must not use the run id, which changes across continue-as-new and workflow retries; derive them from the workflow id and the activity id, which stay the same across activity retries.
+
+### 9.5 Activity operations
+
+Pause, unpause, reset and update-options let an operator stop a runaway activity without restarting anything (preview; server 1.28+; raw gRPC from Python for workflow activities). They are safe only after H2: today a paused node activity would report a completed failure. Document them in the runbook (§7.6); no product UI.
+
+### 9.6 Standalone Activities
+
+GA in server 1.32 and SDK 1.33: a client-started durable activity with its own id space and no workflow. It fits single-node direct runs; it does not fit agents, controllers or chat runs, and Schedules cannot start it. Evaluate in Phase 4.
+
+## 10. Events, triggers and Schedules (D7, D8)
+
+### 10.1 Routing without Visibility
+
+1. The producer looks up live generations that subscribe to the event type in the app database (the control rows already hold the deterministic controller id and the graph snapshot).
+2. It signals `workflow-control-<id>-g<gen>` by id, without a run id, with a few retries and backoff. A plain signal, not Signal-with-Start: a controller's existence is lifecycle state.
+3. If no live generation subscribes, the event is recorded as "no consumer". If delivery fails, webhooks answer 5xx so the provider retries; chat marks the run `not_delivered`, as it does today.
+4. Visibility stays for operations: reconcile jobs, status views, cleanup sweeps.
+
+### 10.2 Limits at the edge and references
+
+- A configurable body limit on `/webhook/*` (HTTP 413), one stored copy of the body, and a byte budget in `emit`.
+- Bodies over a threshold become a workspace `FileRef`, like media.
+- Trigger runs receive the graph by generation reference; node activities load parameters from the database, as they already do.
+
+### 10.3 Cron
+
+The owner chooses the overlap behavior (§17). For "skip while a run is going", let the Schedule start a workflow that awaits the run, or start the run directly; for "allow overlap", say so in the comment and set `ScheduleOverlapPolicy.ALLOW_ALL`. Drop the literal `{{.ScheduledStartTime}}` from action ids (Temporal appends the timestamp) and use the scheduled time, not `workflow.now()`, in the run id. Consider `jitter` for many crons on one minute, `pause_on_failure`, and alerts on `schedule_missed_catchup_window`.
+
+### 10.4 Schedules by id
+
+Compute Schedule ids from the generation snapshot and use `client.get_schedule_handle(id)` for describe, pause, update and delete; keep the `list_schedules` sweep only as a fallback that reports what it finds.
+
+### 10.5 Polling triggers
+
+- Return only the ids that were emitted or deliberately skipped.
+- Move `post_emit` (mark-as-read) into its own activity after the workflow has recorded the event (patch gate).
+- Retry a failed child start instead of marking it seen.
+- Heartbeat timeout and per-cycle activity ids (patch gate).
+- For inboxes polled every 30-60 s, the docs' "infrequent polling" shape (an activity that fails with a BENIGN "not ready" error and retries on a fixed interval) records no history per quiet cycle; today each quiet cycle costs about 11 events. Worth measuring before changing.
+
+## 11. Workers and performance (D15)
+
+### 11.1 Topology and the event loop
+
+| Profile | Recommendation |
+|---|---|
+| Desktop | Keep the embedded worker. Protect the loop (§9.1), size the graceful shutdown inside the shell's deadline, and keep cancellation correct |
+| Docker, VM | Run the worker as a separate process from uvicorn (the standalone path, once fixed). Temporal recommends workers apart from application code; one more process also gives a second core |
+| Distributed | Several worker processes with unique identities, the same registration everywhere, `envconfig` and TLS for a remote cluster |
+
+### 11.2 One registration function
+
+Build the list of workflows, activities and plugins in one function used by the embedded manager, the pool, the standalone worker and the test `Replayer`. At boot, assert that every activity name a workflow can schedule is registered on the queue it targets: an unregistered activity fails as "Not Found", which Temporal retries, so it loops silently today.
+
+### 11.3 LLM providers, rate limits and priority
+
+- Put LLM steps on their own queue (or one per provider) with queue-wide `max_task_queue_activities_per_second` and a small `max_concurrent_activities`; the per-worker limit multiplies with worker count. Raise or lower it at runtime with `temporal task-queue config set` (server 1.29+).
+- Set `priority=Priority(priority_key=1)` on chat-run starts and their agent steps, and `priority_key=3` or higher for background triggers, so a busy install answers the owner first (server 1.31+; Priority does not preempt running tasks). Fairness keys per employee only if the owner wants it and `matching.enableFairness` is set.
+
+### 11.4 Tuning
+
+- Sticky cache: local mode keeps 50 workflows. SDK 1.33 fixes the poller starvation that small caches caused; after the upgrade, measure cache misses and forced evictions with long-lived controllers before keeping 50.
+- The docs recommend fixed-size slot suppliers "for most workloads" and resource-based ones for I/O-bound work. ai-heavy fits the latter; the browser queue, whose tasks use a lot of memory each, is the docs' counter-example. Use a fixed-size supplier there.
+
+### 11.5 Shutdown
+
+Workers are stopped by cancelling their run task; `Worker.shutdown()` is never called, and the SDK discourages cancel-based shutdown. Call `shutdown()` with the graceful timeout and await it. The worst case on desktop (about 35 s before the clean-shutdown marker, against a 65 s deadline) fits; keep the comments in `core/desktop.py:64-71` and `cli/_common.py:90-93` in step, including the Pi driver pool stop.
+
+## 12. Data and payloads (D7)
+
+### 12.1 Limits
+
+| Limit | Warn | Hard | On breach |
+|---|---|---|---|
+| One payload | 512 KB (256 KB on some pages) | 2 MB | Workflow task fails, workflow stays open (SDK 1.23+) |
+| One gRPC request (all commands of one workflow task) | | 4 MB | Execution terminated, no retry |
+| History events | 10,240 | 51,200 | Terminated |
+| History size | 10 MB | 50 MB | Terminated |
+| Pending activities, children, signals sent, cancels | | 2,000 each (keep under 500) | Command fails, task retried |
+| Signals received per run | | 10,000 | Further signals rejected |
+| Updates per run | | 10 in flight, 2,000 total | |
+| Search attribute value | | 2 KB | Workflow task fails |
+
+### 12.2 By reference
+
+Graphs, parameter snapshots, large events and transcripts already exist in the app database or the workspace. Pass ids (generation, node, file) and load inside activities. This removes most of H10, M6 and M11 without a new storage system.
+
+### 12.3 External Storage
+
+External Storage is the SDK's built-in claim check (`ExternalStorage(drivers=[S3StorageDriver(...)], payload_size_threshold=256 * 1024)` on the data converter). It names AI agent conversations as a use case, but it is Public Preview with two breaking changes since 1.25, Temporal never deletes stored payloads ("If your Workflows run indefinitely ... there is no finite TTL that guarantees safety", which collides with our no-run-timeout policy), and the only desktop-fit driver is a local-disk driver documented as development-only. Do not adopt now; reopen when it is GA and a driver fits the desktop.
+
+### 12.4 Codecs and encryption (Owner)
+
+History in `temporal.db` holds prompts, transcripts, emails and error text in plain text, and the Web UI shows them. A `PayloadCodec` (AES-GCM, key id in the payload metadata) with `DefaultFailureConverterWithEncodedAttributes`, plus a codec server inside the backend for the Web UI, would encrypt them. Keys must live as long as the longest-running workflow, months, not only the retention window. Search attributes are never encoded: never put secrets or personal data in them. Recommendation: encrypt on Docker and VM installs; leave desktop installs, where the file sits beside the app database, unencrypted.
+
+## 13. AI agents on Temporal
+
+### 13.1 Our loop against Temporal's integrations
+
+Temporal's Python integrations (OpenAI Agents, Strands, Deep Agents, Google ADK and GenAI, LangGraph) all run the framework's loop inside the workflow and make each model call one activity carrying the whole transcript; tools and MCP calls are activities; SDK client retries are turned off. OpenCompany's AgentWorkflow does the same with its own provider-neutral loop, plus a conversation store, cooperative stop, approvals and a token-streaming path. Keep the native loop; adopting a framework plugin would put a foreign loop, mostly in preview, into the sandbox-less workflow code.
+
+### 13.2 Streaming
+
+Every integration streams through Workflow Streams (Public Preview, SDK 1.27): batched signals in and Update long-polls out, all recorded in history and limited by the 10,000-signal and 2,000-Update caps. Our activity-to-WebSocket path records nothing in history and handles retried attempts with per-attempt segments; Temporal's own AI demo uses the same shape. Keep it. Use queries and `set_current_details` for coarse progress.
+
+### 13.3 LLM retries
+
+Already right: errors are classified, Retry-After drives `next_retry_delay`, provider retries are off. To add:
+
+- The budget from §9.2.
+- Read `x-should-retry` and decide 409 (the OpenAI plugin retries it).
+- Bound honoured Retry-After values (the cookbook ignores delays over 60 s).
+- An OpenRouter HTTP 200 whose body is an error is classified as a non-retryable protocol error with a "check the base URL" message (`services/llm/providers/openai.py:142-172`); classify it by the inner code instead. (LLM-layer finding, not Temporal.)
+- Turn off SDK retries at the three call sites in L4.
+
+### 13.4 History growth
+
+- Pass the transcript by reference: the conversation store already persists it after each step, so the LLM step input can be a cursor plus the new messages.
+- Trim the continue-as-new input (seed conversation, duplicated tools and parameter snapshot, thinking text).
+- Add a byte-based rollover check with `get_current_history_size()`.
+
+### 13.5 Human in the loop
+
+The cookbook's shape is a signal or Update that sets state, plus `workflow.wait_condition(..., timeout=...)`. Use an Update with a validator when the approver needs an immediate "accepted / too late / already decided". This is D13; it also applies to the Undo window, where an Undo Update can wake the 5 s wait early.
+
+## 14. Testing (D16)
+
+- **Replay corpus and CI gate** (§8.6).
+- **Run the CLI-gated tests in CI** with the pinned CLI and full git history, on Linux at least; Windows and macOS if CI time allows, since the desktop targets them.
+- **Time skipping** (`WorkflowEnvironment.start_time_skipping()`) for timers, catch-up windows, backoff and approvals; never-run paths such as TriggerListenerWorkflow, PollingTriggerWorkflow, EmployeeJobDeliveryWorkflow and real Schedules.
+- **`ActivityEnvironment`** with `on_heartbeat`, `cancel()` and `worker_shutdown()` for the per-type wrapper (would have caught H2).
+- **A visibility query contract test**: every query string the repo issues runs against the pinned server (server 1.32 changed the query converter; 1.33 removes the old one).
+- **The determinism lint** (§8.1) and the **golden identity test** (§8.6).
+- Workflow logic is mostly tested by monkeypatching `temporalio.workflow`; keep those tests, but every workflow type needs at least one real-SDK test.
+
+## 15. Evaluated and not adopted (D17)
+
+| Feature | Why not now | Reopen when |
+|---|---|---|
+| Nexus | One team, one namespace; Temporal lists that case under "when not to use" | Workloads split across namespaces or teams |
+| Serverless workers | Require Worker Versioning and Temporal-invoked compute | A Cloud-hosted tier |
+| Worker Versioning | Needs old builds to keep polling until drained; a single-process desktop cannot | Distributed installs with side-by-side deploys; then PINNED for short runs and upgrade-on-continue-as-new for entities |
+| Workflow Pause | Pre-release; server 1.30+ plus a flag; no Python API; rejects Updates and Queries while paused, which the controller needs | GA with Update support |
+| Workflow Streams for tokens | Every chunk lands in history | Never for tokens; maybe for coarse progress |
+| Framework plugins (OpenAI Agents, Strands, ADK, LangGraph) | Foreign loops, mostly preview, tight upstream pins | Never as the core loop |
+| External Storage | Preview; no deletion; no desktop driver | GA with a fitting driver |
+| Dual Visibility | A migration tool for real servers | A move to a production server |
+| Local activities | Small gains; whole-task re-runs; signals wait | Measured latency on bookkeeping steps |
+| Standalone Nexus Operations, NexGen | Pre-release | |
+
+## 16. Phases
+
+**Phase 0: defects that change no workflow command.**
+1. Security (D18): enforce webhook authentication and a non-empty path; scope task-completion events; keep secrets and raw headers out of workflow inputs; owner check on `temporal_start` and `temporal_stop`.
+2. Restart safety (C4, H9): start-node runs get a deterministic per-generation id with `REJECT_DUPLICATE` and are skipped on re-arm; recovery pauses run in the background with Update deadlines; `NOT_FOUND` is decoded before a generation is failed.
+3. Edge limits: webhook body cap and one stored copy.
+4. Retries and timeouts (D11): bounded policies on every activity, `PayloadSizeError` non-retryable, the legacy tool path uses declared policies, the LLM step budget, `schedule_to_start` on routed queues.
+5. Cancellation (H2, H12): re-raise in node activities; Reset also acts on `live_roots`; the Pi inline runner is started with `EventWorkflowId`.
+6. Schedules by id; search-attribute registration fails loudly.
+7. Worker parity: one registration function; refuse a divergent standalone worker; correct the pool-flag rollback text.
+8. Docs and comments listed in §20.
+
+**Phase 1: the safety net.**
+1. D1 and D4: pin the CLI with checksums, database marker, retention, explicit flags and dynamic config.
+2. D5: SDK 1.34.0, after recording 1.30.0 fixtures.
+3. D6: metrics, `TemporalReportedProblems`, the alert table, "needs attention" in the status view.
+4. D16: replay corpus and CI gate, golden identity test, CLI-gated tests in CI, time-skipping and activity-environment tests, query contract test, determinism lint.
+5. D8 client side: route events from the app database.
+
+**Phase 2: fixes that change commands, behind patch gates.**
+1. Freeze dispatch into run input.
+2. Controller: size check and dead-letter, slim carry, triggers in the start input, periodic rollover.
+3. Agent children `REQUEST_CANCEL`, cancel-and-drain on failure, workflow loops re-raise cancellation; Reset cancels before terminating.
+4. Cron overlap (owner choice).
+5. Polling: `post_emit` after record, per-cycle ids.
+6. D12 and D13 per the owner's answers.
+7. AgentWorkflow: transcript by reference, trimmed continue-as-new input.
+
+**Phase 3: features.** Web UI labels; BENIGN; LLM queues with queue-wide limits; Priority for chat; heartbeat checkpoints; the sandbox for small workflows; an eager-start experiment for chat runs; the operator runbook with activity operations.
+
+**Phase 4: production platform (Owner).** D2 and D3; Temporal Cloud and TLS support; payload codec; Worker Versioning for distributed installs; Standalone Activities and External Storage evaluations.
+
+## 17. Owner decisions
+
+The owner answered on 2026-10-16. Further answers came on 2026-10-23 while designing [RFC-0005](./RFC-0005-EVENT-OS-ON-TEMPORAL.md), which implements this RFC's event-path decisions. Answers recorded here are decisions. For questions still open, the recommendation is the working assumption.
+
+| # | Question | Options | Recommendation | Answer |
+|---|---|---|---|---|
+| Q1 | Upgrade contract (D10): must runs that are open or paused survive an upgrade to the next release? | Always (N-1 replay gate) / Within a release line; major releases may require Reset / Any release may require Reset | Always, for N-1; a release that cannot honour it must say so and use D2 option B | Always, for N-1 (2026-10-16) |
+| Q2 | Server for durable installs (D3) | Pinned dev server with SQLite everywhere / PostgreSQL for Docker and VM / also support Temporal Cloud | Pinned dev server on desktop; PostgreSQL for Docker and VM; Cloud later | 2026-10-16: pinned dev server with SQLite. Superseded on 2026-10-23 (RFC-0005 E1-E3): one instance per owner with free Temporal only, no Temporal Cloud. The dev server runs up to about 100 employees; above that, a hosted instance runs the open-source server on its own PostgreSQL. Hosted instances use PostgreSQL for the app database |
+| Q3 | Moving an existing database (D2) | Keep its binary / recreate and re-arm / PostgreSQL | Recreate and re-arm on desktop, with notice | Keep the install's current server; moving is opt-in (2026-10-16). RFC-0005 §20 defines the dev-server-to-full-server move as a re-arm in the maintenance window |
+| Q4 | Failed runs (D12) | Temporal Failed / Completed plus `RunOutcome` | Temporal Failed | Temporal Failed (2026-10-16) |
+| Q5 | Bugs in workflow code (D12) | Stuck and visible until a fix / fail fast | Stuck and visible | Open |
+| Q6 | LLM step budget (D11) | A total budget per step (for example 30 min) / unlimited with alerts / unlimited for outages, capped for timeouts | A total budget, with longer attempts for local models | A total budget per step (2026-10-16) |
+| Q7 | Event delivery (D8) | Guaranteed (5xx and retries) / best effort without Visibility / keep today's | Guaranteed | Guaranteed (2026-10-16); designed in RFC-0005 §8-§9 |
+| Q8 | Graph-node sends (§9.4) | At-least-once with idempotency keys / at-most-once / per plugin | At-least-once with keys where the provider supports them, at-most-once otherwise | As recommended. Where a provider has no idempotency key, an uncertain send is shown as `unknown` with Retry and is never resent automatically (2026-10-23, RFC-0005 E12) |
+| Q9 | Transient errors in graph nodes (§9.2) | Retried by Temporal / returned as failures (today) | Retried, for plugins marked safe | Open |
+| Q10 | Human waits (D13) | Workflow messages and timers / keep activities | Workflow messages and timers | Update plus durable timer (2026-10-16); RFC-0005 §17.5 |
+| Q11 | Cron overlap (§10.3) | Skip while a run is going / allow overlap / buffer one | Skip | Open. Cron stays on Temporal Schedules (2026-10-23, RFC-0005 E5); RFC-0005 defaults overlap to skip |
+| Q12 | Preview features in shipped builds | Behind a flag, off by default / where they solve a problem / GA only | Behind a flag, off by default | Behind a flag, off by default (2026-10-16) |
+| Q13 | History encryption and retention (§12.4, D4) | Encrypt on Docker and VM, 7 days / no encryption, 7 days / other | Encrypt on Docker and VM, 7 days | Open. Telemetry is metrics and errors only (2026-10-23, RFC-0005 E14) |
+| Q14 | Supported topologies (D15) | Embedded only / plus a separate worker process / plus multi-backend distributed mode | Embedded plus a separate worker process; distributed later | One instance per owner. Embedded on desktop and in small hosted instances; separate worker processes in large hosted instances (2026-10-23, RFC-0005 §20) |
+| Q15 | Startup verification of open runs (§8.6) | Refuse to start workers on failure / start and mark "needs Reset" / none | Start and mark "needs Reset" | Open |
+| Q16 | Start nodes (C4): when does a deployment's Start node run? | Once per generation / again after every restart (today) | Once per generation | Once per generation (approved with RFC-0005 Phase 0.4, 2026-10-23) |
+
+## 18. Acceptance gates
+
+| Gate | Check |
+|---|---|
+| AG1 | The CLI download URL names an exact version and `known_hash` is set; a test fails on "latest" |
+| AG2 | Starting with a database created by another server minor refuses with a clear message |
+| AG3 | Retention is set to the configured value after namespace creation (integration test) |
+| AG4 | A metrics endpoint or buffer exists; the alert table's signals are emitted in a test run |
+| AG5 | `TemporalReportedProblems` turns a failing workflow task into "needs attention" in the status view |
+| AG6 | CI replays every fixture of the previous release with the production workflow list and fails on any error |
+| AG7 | The golden identity test fails when an activity name, child id format or marker changes without a patch entry |
+| AG8 | The determinism lint passes for every workflow module |
+| AG9 | Every activity call site has an explicit retry policy; none is unlimited without a `schedule_to_close_timeout` |
+| AG10 | A cancelled node activity is recorded as Cancelled (ActivityEnvironment test) |
+| AG11 | A webhook without the configured header gets 401; one over the size cap gets 413 |
+| AG12 | An event is delivered to a controller that started 10 ms earlier (no Visibility on the path) |
+| AG13 | A slow cron run causes the next tick to be skipped (or, if the owner chooses overlap, the policy says so) |
+| AG14 | A standalone worker with a different registration refuses to start |
+| AG15 | Every query string in the repo runs against the pinned server (contract test) |
+| AG16 | The upgrade from the previous release keeps a paused generation resumable (end-to-end, with the pinned CLI) |
+| AG17 | Restarting the backend twice starts no second start-node run for a running or paused generation |
+| AG18 | Reset cancels, then terminates, every execution of the generation, including Pi inline runners reached through `live_roots` |
+
+## 19. What this RFC does not change
+
+- The plugin model, the WebSocket protocol, the chat protocol and RFC-0002's conversation store.
+- The cooperative pause design; native Workflow Pause stays unused (§15).
+- The decision to keep workflow execution and run timeouts unset.
+- The provider-neutral agent loop (§13.1).
+
+## 20. Docs and comments to correct
+
+| Where | Today says | Should say |
+|---|---|---|
+| CLAUDE.md, durability item 2 | Option changes need a patch | §8.5 rule |
+| CLAUDE.md, Temporal heartbeats note | Heartbeats on every non-matching broadcast | That branch is dead; the 30 s receive timeout is the heartbeat |
+| CLAUDE.md, cron naming row | `<slug>-<label>-<ScheduledStartTime>` | The real ids (H6) |
+| CLAUDE.md and `services/deployment/handlers.py:1520` | Native pause needs server 1.28+ | Server 1.30+ plus `frontend.WorkflowPauseEnabled` |
+| `.env.template:100-107` | The pool flag rolls back cleanly | It affects new runs only |
+| `.env.template:147-149` | Every workflow stays visible | Closed workflows are deleted after the retention period |
+| `.env.template:212-216`, `core/config.py:179-183` | `EVENT_FRAMEWORK_ENABLED=false` rolls back | It no longer does (event_framework.md:54-56) |
+| `services/temporal/_runtime.py:118` | `--metrics-port 0` disables metrics | It picks a random port |
+| `services/temporal/schedules.py:187-189` | SKIP drops a firing while the run is going | See H6 |
+| `services/temporal/agent_activities.py:9-13` | LLM step lives on ai-heavy | It runs on `machina-tasks` |
+| `services/temporal/agent_workflow.py:2445-2447` | One-shot retry | Unlimited |
+| `services/temporal/client.py:126-129` | Search attributes fall back to broadcast-only | Starts fail without them |
+| `services/media/limits.py:14` | `PayloadSizeWarning` | Removed in SDK 1.31 |
+| TEMPORAL_ARCHITECTURE.md:962 | 9 live markers | 19, with a table (§8.6) |
+| TEMPORAL_ARCHITECTURE.md:771, temporal-workflow-control.md:622-623 | Fixtures cover graph, agent and invocation histories | Two agent fixtures, one orphaned |
+| temporal-workflow-control.md:653-656 | Boot re-arm is idempotent | Not for start nodes (C4) |
+| event_framework.md:439 | The filter activity exists because of the sandbox | Every workflow is unsandboxed |
+| `server/routers/webhook.py:8-9` | Plugins pull from `source.emit()` | Nothing does (L5) |
+
+## 21. References
+
+Python SDK guide: https://docs.temporal.io/develop/python (workflows, activities, workers, client, testing-suite, python-sdk-sandbox, python-sdk-sync-vs-async, error-handling, message-passing, continue-as-new, child-workflows, cancellation, schedules, versioning, workflow-streams, data-conversion, data-encryption, external-storage, observability, enriching-ui, interceptors, standalone-activities, integrations).
+Concepts: https://docs.temporal.io/visibility, /search-attribute, /schedule, /patching, /workflow-execution/limits, /encyclopedia/retry-policies, /encyclopedia/detecting-workflow-failures, /encyclopedia/workflow/cancellation-and-termination, /encyclopedia/workflow/workflow-pause, /activity-operations, /standalone-activity, /external-storage, /worker-versioning, /task-queue, /handling-messages.
+Operations: https://docs.temporal.io/troubleshooting/blob-size-limit-error, /troubleshooting/execution-failures, /troubleshooting/schedule-missed-actions, /references/sdk-metrics, /references/dynamic-configuration, /best-practices/worker-alerting, /best-practices/worker, /develop/worker-performance, /develop/task-queue-priority-fairness, /develop/safe-deployments.
+Self-hosting: https://docs.temporal.io/temporal-service/persistence, /self-hosted-guide/visibility/sqlite, /self-hosted-guide/upgrade-server, /self-hosted-guide/embedded-server, /cli/command-reference/server.
+Design patterns: https://docs.temporal.io/design-patterns (approval, entity-workflow, continue-as-new, signal-with-start, early-return, worker-specific-taskqueue, polling, downstream-rate-limiting, priority-task-queues, fairness).
+AI: https://docs.temporal.io/ai, /ai/cookbook, /develop/python/integrations.
+Releases: https://github.com/temporalio/sdk-python/releases (1.25.0 to 1.34.0), https://github.com/temporalio/cli/releases, https://github.com/temporalio/temporal/releases.
diff --git a/RFC-0005-EVENT-OS-ON-TEMPORAL.md b/RFC-0005-EVENT-OS-ON-TEMPORAL.md
new file mode 100644
index 00000000..8006ae8b
--- /dev/null
+++ b/RFC-0005-EVENT-OS-ON-TEMPORAL.md
@@ -0,0 +1,1159 @@
+# RFC-0005 — Event OS: Event-Driven AI Employees on Temporal
+
+- **Status:** Accepted, 2026-10-23 (owner decisions in §3). Implementation is phased (§24); nothing in it has landed yet.
+- **Created:** 2026-10-23
+- **Scope:** the path from every event source to every effect:
+ - `server/services/events/` and `server/routers/webhook.py`;
+ - the receive and send paths of the channel plugins (`server/nodes/{telegram,whatsapp,whatsapp_business,discord,twitter,email,google,stripe,dodo_payments,...}`);
+ - `server/services/chat/` and `server/services/deployment/`;
+ - `server/services/temporal/`: new workflow types, and changes to MachinaWorkflow and AgentWorkflow behind input flags;
+ - `server/core/database.py` and `server/services/status_broadcaster.py`;
+ - the deploy logic (`cli/terraform/`, `cli/commands/deploy/`, `install.sh`, `docker/`).
+- **Relationship:**
+ - Implements, for the event path, RFC-0004 decisions D7 (bounded events), D8 (routing by known ids), D9 (cancellation), D10 (upgrade contract), D13 (human waits) and D18 (security fixes).
+ - Keeps RFC-0002's conversation store and adds a thread to its key.
+ - Until a phase lands, [event_framework.md](./docs-internal/event_framework.md), [chat_protocol.md](./docs-internal/chat_protocol.md), [temporal-workflow-control.md](./docs-internal/temporal-workflow-control.md) and [TEMPORAL_ARCHITECTURE.md](./docs-internal/TEMPORAL_ARCHITECTURE.md) stay normative for what it covers.
+- **Baseline:** HEAD a8e390ca (2026-10-23), temporalio 1.30.0, Temporal CLI unpinned. Line numbers are as of that date and drift. Figures marked *est.* are unmeasured and are confirmed by the gates in §26.
+
+## How to read this
+
+- **Owner:** §1, §3 (decisions), §25 (defaults chosen and what is still open).
+- **Reviewer:**
+ - §4: what is broken today and where;
+ - §6: the architecture and its invariants;
+ - §9.5: the delivery protocol;
+ - §23: failure modes.
+- **Implementer:** §7 to §21 by layer, §22 (scenarios), §24 (phases), §26 (acceptance gates).
+
+## 1. Summary
+
+OpenCompany's AI employees should run like the processes of an operating system that sleeps until something happens. A message, a webhook, a timer or another employee wakes one. It reacts, schedules its own follow-ups, shares the machine fairly with every other employee, and delivers what it produces through queues that apply backpressure instead of dropping. This RFC designs that system on Temporal, borrowing the Linux kernel's answers to the same problems:
+- interrupt top and bottom halves;
+- NAPI's switch from push to pull under load;
+- scheduling classes with reserved shares;
+- epoll-style match rules;
+- timer wheels;
+- qdisc and BQL on the way out;
+- supervisors that cap restart intensity;
+- cgroup memory limits applied to model spend.
+
+**Today's event path is at-most-once.** Every event is routed by a Visibility query that is eventually consistent by design. When Temporal is down, a query lags or a Signal fails, the event is dropped with a log line while the producer is told it succeeded. Telegram discards every message sent while the backend was down. One large webhook can wedge a deployment. Start nodes fire again on every restart. Inbound message bodies are broadcast to every connected socket. (§4)
+
+**The design** (§6):
+1. **Ingress commits before acknowledging.** Every event is appended to an inbox in the instance's own database (SQLite on the desktop, PostgreSQL on hosted instances) before anything acknowledges it. Nothing else happens in the request path.
+2. **A router (DB only) assigns streams.** Match rules compiled from each generation's graph give every input a place in one conversation stream, under per-stream sequence numbers.
+3. **A relay delivers by known id.** It sends each stream's pending inputs in bounded batches to a deterministic Temporal entity, one Signal-With-Start in flight per stream, and records progress only after Temporal accepts. Duplicates are dropped by sequence number. An input counts as consumed in the same transaction that saves it into the agent's history.
+4. **Each conversation is one entity.** An entity workflow (`ConversationWorkflow`) debounces bursts, starts one turn at a time, and offers new messages to the running turn at its next step.
+5. **An employee supervisor workflow holds control and time.** It owns control state, alarms and the daily budget.
+6. **Classes and budgets decide who runs.** Four scheduling classes with reserved shares (owner, customer, internal, background) and a daily budget per employee decide who runs when capacity or money is short.
+7. **Timers live in Temporal.** Cron jobs and polls are Temporal Schedules addressed by immutable ids. Reminders are timers in the supervisor.
+8. **Employees talk through the inbox.** Messages between employees are inbox rows carrying causal ids, a hop limit and a breaker.
+9. **Sends go through an outbox.** Each send has an idempotency key, and its token buckets slow the producing turn instead of dropping. A send whose outcome is uncertain is shown as `unknown`, never resent on its own.
+10. **Temporal is rebuildable.** Everything needed to resume lives in the app database, so Temporal can be rebuilt from it (I10). An instance can therefore move from the free dev server to the free open-source server, or recover a lost Temporal database, without losing an acknowledged event.
+
+**Deployment model (owner decisions, §3).**
+- Every owner runs their own instance, each with free Temporal: a desktop app, a container or a VM. There is no shared cluster and no Temporal Cloud.
+- Instances with up to about 100 employees use the CLI dev server. Larger hosted instances, up to about 1,000 employees, use the open-source server on the instance's PostgreSQL.
+- Releases reach the hosted fleet together in a weekly window.
+- Instances report metrics and errors, never content.
+
+**Rollout.** Phase 0 ships nine fixes for verified defects without changing any workflow command. Phase 1 builds the replay safety net. Phases 2 to 8 introduce the inbox, drivers, entities, scheduler, timers, messaging, egress and supervision, each versioned per generation. Phase 9 covers instance tiers and fleet operations. Phase 10 retires the old path. (§24)
+
+## 2. Method and sources
+
+- **Research.** Fifteen parallel research passes (RFC-0004's Temporal mirror and audits) and four new parallel design passes (current-state audit, target architecture, bus and scale, migration).
+- **Verification.** Claims that changed the design were checked against the code at HEAD; the citations in §4 were re-read.
+- **Temporal sources:**
+ - a local mirror of 282 docs.temporal.io pages (2026-10-16), especially message passing, entity workflows, the event-accumulator pattern, Schedules, task-queue priority and fairness, multi-tenant patterns, the CLI server reference, the upgrade guide and the production checklist;
+ - the temporalio 1.30.0 source (`temporalio/worker/_tuning.py`, `temporalio/common.py`);
+ - forum threads on signal throughput and Kafka-to-Temporal consumers.
+- **Linux sources:**
+ - the kernel documentation for NAPI, generic IRQ handling, EEVDF and CFS scheduling, cgroup v2 (`cpu.max`, `memory.high`, `memory.max`), PSI, timers, qdiscs and BQL, RCU and the soft-lockup watchdog;
+ - the `epoll`, `inotify` and `netlink` man pages;
+ - systemd's `Restart=`, `StartLimit*=` and `Persistent=`.
+- **Agent-OS prior art:**
+ - AIOS (arXiv 2403.16971), Letta, Erlang/OTP supervisors, Orleans grains and reminders, Akka mailboxes;
+ - peer platforms: Inngest (debounce, batch, throttle), Restate (virtual objects), Hatchet (concurrency strategies), LangGraph (double-texting), DBOS (partitioned queues).
+- **Reference systems built on Temporal:** Temporal's chatbot sample (one workflow per conversation), Cursor's agent runs (Signal-With-Start follow-ups, replayed histories in CI), Quo's voice agent (interruptions as Signals).
+- **Not done:** no load test and no run against a live server. Throughput figures are estimates.
+
+## 3. Decisions
+
+### 3.1 Owner decisions (2026-10-23)
+
+| # | Decision |
+|---|---|
+| E1 | **One instance per owner.** Every owner's desktop app, container or VM runs OpenCompany, its database and its own free Temporal. No shared hosted backend, no Temporal Cloud. Target: 100,000+ employees across about 10,000 instances. |
+| E2 | **Instance tiers.** The desktop and hosted instances up to about 100 employees run the Temporal CLI dev server. A hosted instance past about 100 employees (up to about 1,000) moves to the free open-source Temporal server on its own PostgreSQL. (§20) |
+| E3 | **Databases.** Hosted instances run PostgreSQL for the app database and the inbox. The desktop app stays on SQLite. |
+| E4 | **Bus.** The inbox in the instance's database is the only durable store for events. gRPC carries events where a transport is needed: the protocol for out-of-process drivers (the WhatsApp bridge first), with acks and resume, and the relay's calls into Temporal. No message broker. (§7.3, §9) |
+| E5 | **Timers.** Cron jobs, polling triggers and reminders run on Temporal Schedules and timers. (§15) |
+| E6 | **Merging.** A new message for an employee that is still working is merged into the running work at its next step. Quick bursts are batched into one input. (§11) |
+| E7 | **Internal wake sources:** messages from other employees, self-scheduled wake-ups, task results and delegations, and changes to watched shared data. (§16) |
+| E8 | **Employee messaging.** Any employee may message, delegate to and wake any other employee of the same owner without linking. Loop limits apply. Owners live in separate instances, so traffic between owners cannot happen. |
+| E9 | **Customer histories.** Each customer conversation keeps its own history. Owner chat stays one conversation. Shared facts go through the Memory tool. (§12) |
+| E10 | **Priority.** When capacity is short the order is: owner chat and approvals, then customer conversations, then internal events, then background work. Every class keeps a reserved share. (§13) |
+| E11 | **Daily budget per employee.** Past 80%, only owner and customer conversations run. At 100% the employee pauses and the owner gets a notice with Resume. It resumes on its own at midnight in the owner's time zone. (§14) |
+| E12 | **Uncertain sends.** A send whose outcome is unknown, to a provider without idempotency keys (WhatsApp bridge, Telegram), is never resent automatically. It shows `unknown` with a Retry button. (§17) |
+| E13 | **Fleet upgrades.** Releases reach all hosted instances together in a fixed weekly maintenance window. (§21) |
+| E14 | **Telemetry.** Hosted instances report metrics and errors centrally, never content. (§18.5) |
+| E15 | **The Linux model** draws on all three diagrams: the kernel map, the network stack and the storage stack. |
+
+### 3.2 RFC-0004 decisions this RFC implements
+
+| RFC-0004 | Decision | Here |
+|---|---|---|
+| Q7 / D8 | Guaranteed delivery: route by the app database to known ids, retry, answer 5xx when not stored | §8, §9 |
+| Q1 / D10 | Runs open on release N-1 survive an upgrade | §24.1, §24.3 |
+| Q4 / D12 | Failed runs end as Failed workflows | §10 |
+| Q10 / D13 | Human waits become an Update plus a durable timer | §17.5 |
+| Q12 | Preview features only behind a flag, off by default | §10.4 |
+| Q16 | The Start node runs once per generation | Phase 0.4 |
+| Q8 | Sends: at-least-once with idempotency keys where the provider supports them, otherwise at-most-once (E12) | §17 |
+
+RFC-0004 Q2 (which server durable installs run on) is answered by E1 to E3.
+
+## 4. Current state (audit at HEAD, 2026-10-23)
+
+### 4.1 How an event flows today
+
+- **Entry point.** Every source calls `emit` in `server/services/events/dispatch.py:63-106`.
+- **Routing.** `emit` runs one Visibility List query per event to find running consumers (`:132-164`), then sends a plain Signal `on_event` to each one (`:250-257`).
+- **The consumer.** The consumer is the deployment's `WorkflowControlWorkflow`. It:
+ - dedupes the last 4,096 keys;
+ - queues FIFO per generation;
+ - spawns one `MachinaWorkflow` per event, one at a time, with no cap on concurrent runs (`server/services/temporal/workflow_control_workflow.py:336-484`; `trigger_listener_workflow.py:305-512`).
+- **Outside the controller:**
+ - Cron uses Schedules that start runs directly.
+ - Polling runs inside the controller.
+ - Owner chat admits a run in the chat ledger, then emits.
+
+### 4.2 Guarantees today
+
+- **At-most-once ingress.** An event is dropped with a log line when Temporal is down, the Visibility query fails or lags, no consumer is found, or a Signal fails (`dispatch.py:125-206`). Producers always see success: webhooks answer 200 regardless (`server/routers/webhook.py:69-106`).
+- **Retries are never deduped for generic webhooks.** They get a random event id. Signed providers (Stripe `evt_`, Dodo `webhook-id`, WhatsApp Business `wamid`) have stable ids.
+- **One run per chat session.** Owner chat allows one live run per session, and a second message is refused with `run_in_progress` (`server/services/chat/ledger.py:286-293`). WhatsApp and Telegram messages each start their own run, so replies in one conversation can race.
+- **No priority and no per-employee cap.**
+ - LLM steps run on `machina-tasks`, not the rate-limited `ai-heavy` queue (RFC-0004 M9).
+ - Provider rate limits are per worker (`server/services/temporal/worker.py:618`).
+ - One deterministic spawn failure retries the head of a generation's queue every 5 s and blocks everything behind it (`workflow_control_workflow.py:477-480`).
+- **Internal events.** Only `agent.task.completed` can wake work, and it is unscoped (H11). An employee cannot schedule its own wake-up or message another employee.
+
+### 4.3 Defects this RFC fixes
+
+| Id | Defect | Evidence | Fixed in |
+|---|---|---|---|
+| C1 (RFC-0004) | One large webhook wedges a deployment: the body is copied about four times into a child's input, past the 2 MB payload limit | `trigger_listener_workflow.py:394-397`; `workflow_control_workflow.py:705-711` | 0.3, controller lane, P4 |
+| C3 (RFC-0004) | Webhook header authentication is never enforced, and an empty path matches every webhook | `server/routers/webhook.py`; `nodes/trigger/webhook_trigger` filter | 0.1 |
+| C4 (RFC-0004) | A deployment's Start node fires again on every backend restart | `server/services/deployment/handlers.py:1612-1617`; `manager.py:282` | 0.4 |
+| H1 (RFC-0004) | Routing through Visibility drops events silently | `dispatch.py:109-206` | P2 |
+| H6 (RFC-0004) | Cron overlap is not prevented: the Schedule's policy covers only the short workflow it starts | `server/services/temporal/schedules.py` | P6 |
+| H8 (RFC-0004) | Workflow loops swallow cancellation | RFC-0004 §5 | controller lane |
+| H11 (RFC-0004) | Task events reach every workflow and every socket | `server/nodes/agent/_events.py:109-134` | 0.2, P6 |
+| M3 (RFC-0004) | Polling loses events: a failed detail fetch is marked seen, and mark-as-read runs before the event is recorded | `server/services/plugin/polling.py:233-259` | P3 |
+| N1 | Telegram discards every message sent while the backend was down (`drop_pending_updates=True`) | `server/nodes/telegram/_service.py:663` | 0.5, P3 |
+| N2 | The WhatsApp bridge loop awaits `emit` inline and acknowledges nothing | `server/nodes/whatsapp/_service.py:111-127` | P3 |
+| N3 | Discord interactions are acknowledged, then dispatched fire-and-forget | `server/nodes/discord/_router.py:110-117` | P2 |
+| N4 | With login on, Discord's interaction POSTs get 401 (not a public path) | `server/middleware/auth.py:14-34` | 0.7 |
+| N5 | A deployed `twitterReceive` never fires: its dispatcher is never called | `server/nodes/twitter/_events.py:68` | 0.8 |
+| N6 | On PostgreSQL the chat lane index is a full `UNIQUE(session_id)`, so a conversation's second run fails | `server/models/chat.py:69` (only `sqlite_where`) | 0.6 |
+| N7 | Inbound bodies are broadcast to every socket, twice for WhatsApp (raw frame plus envelope) | `server/services/status_broadcaster.py:228-281`; `server/nodes/whatsapp/_events.py:214-223` | 0.9 |
+| N8 | Every reserved transaction on PostgreSQL takes one global advisory lock: about 60 call sites and 8-12 per turn, so an instance tops out at est. 20-50 turns/s | `server/core/database.py:530`, `:763` | P9b |
+| N9 | `workflow.db` has no WAL and no `busy_timeout`; a long reader blocks every commit | only `server/services/temporal/_runtime.py:153-158` sets WAL, and only for Temporal | P2 |
+| N10 | PostgreSQL has no schema migrations: a missing column fails startup | `server/core/database.py:193` | P9b |
+| N11 | A webhook source puts every event on a queue nothing drains | `server/services/events/webhook.py:121` | 0.3 |
+
+## 5. Principles from the Linux stack
+
+### 5.1 Eight principles
+
+1. **Sleep by default; wake on an edge, reconcile the level.** An event is a hint and state is the truth (NO_HZ idle, Orleans virtual actors). A waiting Temporal workflow holds no worker.
+2. **Split every event into a tiny top half and a deferred bottom half.**
+ - The top half authenticates, dedupes, enqueues and acks.
+ - The bottom half is schedulable and may sleep (hard IRQ, then softirq, workqueue or threaded IRQ).
+ - Never do slow work at the edge.
+3. **Bound every queue and name its overflow policy:** block, coalesce, drop-oldest, or reject plus resync (inotify `IN_Q_OVERFLOW`, netlink `ENOBUFS`). Queue wait time is the primary health signal (PSI, the USE method).
+4. **Under load, switch from push to pull and drain a batch against a budget.** NAPI: the first interrupt arms a poll that drains up to a budget before interrupts are re-enabled.
+5. **Schedule with strict classes, weighted shares, quotas and reserved floors** (stop, deadline, rt, fair and idle classes; EEVDF weights; `cpu.max`; RT throttling keeps 5% for others). Running work is not preempted, so isolate capacity per class. Bound the depth of downstream queues to protect latency (Kyber, BQL).
+6. **Failure is normal.** Supervise hierarchically, back off, cap restart intensity and escalate (systemd `Restart=` and `StartLimitBurst=`, OTP supervisors' MaxR/MaxT). Enforce budgets before thrash: `memory.high` throttles, `memory.max` stops.
+7. **Storms need circuit breakers that degrade gracefully** (spurious-IRQ disable, journald rate limits). Agent form: hop and TTL limits, causal ids, a token bucket per edge, and a breaker that turns a hot edge into a periodic digest.
+8. **Instrument every edge and measure waiting, not just utilisation** (tracepoints, PSI).
+
+### 5.2 Mapping
+
+| Linux | Agent OS | Here |
+|---|---|---|
+| process, `task_struct` | an employee and each of its conversations | supervisor and conversation entity workflows (§10) |
+| device model: bus, driver, device, probe, hotplug | channel adapters bound to employees | `ChannelDriver` registry (§7) |
+| hard IRQ / bottom half | ingress / conversation turn | durable append and ack (§8), then router and relay (§9) |
+| NAPI | doorbell and batch drain | at most one delivery in flight per stream, a bounded window, backlog left in the database (§9.4) |
+| signal delivery on return to user space | merging new messages | the agent loop's step boundary (§11) |
+| scheduling classes, cgroups, HTB | owner, customer, internal, background | Priority keys, class queues, a slot pool with floors, admission (§13) |
+| `memory.high` / `memory.max` | spend throttle and pause | daily budget ledger (§14) |
+| epoll interest lists, D-Bus match rules, netlink groups | subscriptions | `bus_subscriptions` compiled at Start (§9.3) |
+| timer wheel, hrtimers, systemd timers | cron, polls, reminders | Schedules and durable timers (§15) |
+| pipes, signals, D-Bus | employees messaging each other | inbox rows with causal ids (§16) |
+| qdisc (TBF, HTB, fq_codel), BQL, writeback throttling | the outbound path | outbox, token buckets, the producer slows down (§17) |
+| RCU | a running generation keeps its version | `resource_manifest` flags frozen per generation (§10.4) |
+| watchdogs, OOM killer, oomd | stuck and runaway employees | deadlines, restart intensity, quarantine, budgets (§18) |
+| journaling (jbd2) | write-ahead before acknowledging | inbox append before any ack (I2) |
+
+### 5.3 Prior art adopted
+
+- **AIOS:** per-resource queues and syscalls with priority and time limits; confirmation before irreversible actions (approvals).
+- **Letta:** loop control belongs to the runtime, not the model.
+- **Erlang/OTP:** asynchronous messages, let it crash, supervision with restart intensity.
+- **Orleans:** virtual actors that activate on demand, durable reminders, turn-based single-threaded execution.
+- **Akka:** bounded mailboxes that overflow to dead letters, control-aware priority mailboxes, passivation of idle entities.
+- **Peer platforms:**
+ - Inngest debounce, batch and throttle are the entity's debounce and the class buckets.
+ - LangGraph's double-texting strategies (reject, enqueue, interrupt, rollback) and Hatchet's `CANCEL_IN_PROGRESS` are the entity's merge, next-turn and Interrupt.
+ - Restate virtual objects are entity workflows.
+ - DBOS partitioned queues are class queues plus fairness keys.
+
+## 6. Architecture
+
+### 6.1 Shape
+
+```
+DRIVERS (RX) TOP HALF (request path) BOTTOM HALF (database) RELAY PROCESSES (Temporal) EGRESS (TX)
+WhatsApp bridge -> verify, cap, normalize, -> router: subscriptions, -> per conversation stream: -> ConversationWorkflow per conversation: -> outbox row ->
+(gRPC), Telegram, dedupe, INSERT storm guards, stream seq, gates (generation, budget), debounce, admit, turn child, send activity
+Discord, webhooks, bus_ingress, commit, ack bus_deliveries window, Signal-With-Start merge at step boundaries (token bucket,
+polls, schedules, (no Temporal, no LLM) "deliver" with the batch EmployeeSupervisorWorkflow per employee: priority,
+IPC, owner chat control, alarms, budget, upkeep idempotency key)
+```
+
+### 6.2 Invariants
+
+| # | Invariant | Principle |
+|---|---|---|
+| I1 | A Signal is a hint. The inbox and its cursors are the truth; every consumer reconciles from them. | 1 |
+| I2 | No request path calls Temporal or a model. Every request path ends at a durable commit. | 2 |
+| I3 | Every queue is bounded and names its overflow policy. Every drop is a row with a reason. | 3 |
+| I4 | One relay per stream at a time and one consumer per stream. An entity runs at most one turn, and an input belongs to exactly one turn. | — |
+| I5 | Serialize only where state is shared (an agent's conversation key). Everything else runs in parallel under class floors. | 5 |
+| I6 | Routing goes from the app database to known workflow ids. Visibility is for operations only. | — |
+| I7 | Nothing over 64 KB enters Temporal; larger bodies travel by reference. An entity holds at most 32 inputs or 512 KB, and any backlog stays in the database. | 4 |
+| I8 | Every event carries `chainid`, `hop` and `causationid`. Internal wakes are checked for hops, cycles and rate. | 7 |
+| I9 | Every effect is idempotent, or at-most-once with an explicit `unknown`. Never a silent duplicate. | — |
+| I10 | Temporal is rebuildable from the app database: generation snapshots, conversation histories, inbox cursors, alarms, budgets and Schedule definitions all live there. | 6 |
+
+### 6.3 Layers
+
+| Layer | Linux analog | Responsibility | Runs in | Section |
+|---|---|---|---|---|
+| Drivers | device model, NIC driver | probe, start/stop, normalize, ack after commit, send | app process; edgymeow (Go) | §7 |
+| Ingress | hard IRQ | verify, cap, dedupe, append, ack | request path | §8 |
+| Router | softirq, match rules | subscriptions, storm guards, stream sequence numbers | in-process task | §9.3 |
+| Relay | receive queue, TCP window | gates, window, Signal-With-Start, `delivered_seq` | in-process task | §9.4 |
+| Processes | task_struct, run queue | supervisor, conversation entity, turn, agent | Temporal | §10 |
+| Scheduler | sched classes, cgroups, HTB | classes, floors, admission, Priority, fairness | DB + Temporal | §13 |
+| Budgets | `memory.high` / `memory.max` | daily ledger, holds, pause, auto-resume | DB + supervisor | §14 |
+| Timers | timer wheel, systemd timers | Schedules, alarms, durable timers | Temporal | §15 |
+| IPC | pipes, D-Bus | employee messages, delegations, watched data | inbox | §16 |
+| Egress | qdisc, BQL | outbox, buckets, idempotency, approvals | Temporal activities + DB | §17 |
+| Supervision | systemd, OTP, watchdogs, PSI | deadlines, restart intensity, attention, metrics | supervisor + DB | §18 |
+
+## 7. Drivers (the device model)
+
+### 7.1 Contract
+
+A `ChannelDriver` implements:
+- `probe(account)` and `start(account)` / `stop(account)`;
+- `normalize(raw) -> IngressRecord` (envelope, dedupe key, conversation id, counterpart, size);
+- `ack(raw)`, called only after the inbox append has committed;
+- `send(outbound) -> SendResult`;
+- a `capabilities` declaration: who holds unacknowledged events, ordering, send idempotency, reply window.
+
+The `DriverRegistry` runs a connection driver only while a running generation subscribes to its account (principle 1). It re-probes on credential events (`com.opencompany.credential.*`) and on generation Start or Reset. A lease in `bus_leases` guarantees one reader per account; Telegram, for example, allows only one `getUpdates` consumer per token. An account that stays down for more than 5 minutes becomes an attention item.
+
+Plugins keep their own drivers in `server/nodes/<plugin>/_driver.py` and register them from their package `__init__.py`, like the other plugin registries. The old `register_webhook_source`, `register_filter_builder` and `register_trigger_precheck` remain as shims for one release.
+
+### 7.2 Per-driver behaviour
+
+| Driver | Who holds unacked events | Dedupe key | Ack after commit | Send idempotency |
+|---|---|---|---|---|
+| WhatsApp (edgymeow bridge) | WhatsApp's server, through whatsmeow `SynchronousAck` plus the bridge's outbox | `whatsapp:<acct>:<chat>:<msgid>` | gRPC `Ack(seq)`; the bridge then acks WhatsApp | none: at-most-once (E12) |
+| WhatsApp Business | Meta, retried up to 7 days | `wamid` | 200 | none; 24-hour reply window |
+| Telegram | Telegram, 24 hours | `update_id` | the `offset` advances only after commit | none |
+| Discord gateway | gateway resume by sequence; REST catch-up after a full reconnect | message id | n/a | `nonce` with `enforce_nonce` |
+| Discord interactions | not redelivered; 3-second window | interaction id | deferred reply (type 5) after commit, else 500 | n/a |
+| Owner chat | the client re-sends with `client_message_id` | message uid | WebSocket answer after commit | database |
+| Signed webhooks (Stripe, Dodo, GitHub, Standard Webhooks, WhatsApp Business) | the provider (Stripe retries for 3 days) | `evt_…`, `webhook-id`, `X-GitHub-Delivery` | 2xx | the provider's `Idempotency-Key` |
+| Generic webhook | the caller | a configured header, else a body hash within 10 minutes | 2xx | caller-defined |
+| Pollers (Gmail, IMAP, Outlook, Twitter) | provider state | message id, UID, delta token, tweet id | cursor saved in the append transaction; `post_emit` (mark-as-read) after commit | email `Message-ID` check |
+| Cron, Start, alarms | Temporal | `<schedule>:<scheduled_time>`, `start:<gen>`, `alarm:<id>` | n/a | n/a |
+| Internal (messages, tasks, data changes) | the database | `<producer workflow id>:<activity id or tool_call_id>` (never the run id) | n/a | n/a |
+
+### 7.3 gRPC driver protocol (E4)
+
+Out-of-process drivers stream events into the bus over gRPC: on loopback or a named pipe on the desktop, and on localhost with mTLS where the driver runs in another container.
+
+```proto
+service EventIngress {
+ // The driver resends every unacknowledged event after a reconnect, in order.
+ rpc Publish(stream DriverEvent) returns (stream Ack);
+}
+message DriverEvent {
+ string driver_id = 1; // "whatsapp:<account>"
+ uint64 seq = 2; // per-driver monotonic sequence from the driver's outbox
+ string source_key = 3; // account scope
+ string dedupe_key = 4; // e.g. whatsapp:<acct>:<chat>:<msgid>
+ string type = 5; // CloudEvents type
+ bytes envelope = 6; // CloudEvents JSON, at most 64 KB
+ string body_ref = 7; // for media and large bodies
+}
+message Ack { uint64 seq = 1; enum Status { ACCEPTED = 0; DUPLICATE = 1; REJECTED = 2; } Status status = 2; string reason = 3; }
+```
+
+- The bus sends `Ack` only after the append commits.
+- At most 1,000 events are in flight per stream, with deadlines on every call.
+- The driver advances its upstream cursor only after the `Ack`: the WhatsApp acknowledgement, the Telegram offset, the Discord sequence, the IMAP UID.
+
+**edgymeow changes.** edgymeow is its own Go project, published to npm and pinned at `[email protected]` in `server/nodes/whatsapp/_install.py:39`. It changes in its own repository:
+- Turn on whatsmeow `SynchronousAck`.
+- Write each decrypted event to a local SQLite outbox before acknowledging WhatsApp.
+- Stream the outbox over `Publish`, and delete each row on `Ack`.
+- Resend from the last acknowledged sequence after a reconnect.
+
+Today the bridge speaks JSON-RPC over WebSocket without acks (`server/nodes/whatsapp/_service.py:111-127`).
+
+gRPC is not used as the bus itself, and no router-side log is built (§9.7).
+
+### 7.4 Driver-level fixes
+
+- **Telegram:** a manual `get_updates(offset)` loop replaces `start_polling(drop_pending_updates=True)`.
+- **WhatsApp:** the `_recv` loop only appends (the bridge protocol above).
+- **Discord interactions:** append before the deferred reply.
+- **Pollers:** a failed detail fetch is not marked seen, and becomes a dead row after 5 failures.
+- **Twitter receive** joins the polling path.
+
+## 8. Ingress (the top half)
+
+### 8.1 Pipeline
+
+1. **Resolve the endpoint** from the active generations' subscriptions.
+2. **Check size.** Check `Content-Length`, then stream the body with a cap: 256 KiB by default, raisable per endpoint up to 10 MiB.
+3. **Verify** with a constant-time comparison of the configured header, or an HMAC over the raw bytes for signed providers.
+4. **Normalize** to a CloudEvents envelope. Keep allowlisted headers only; never `authorization`, `cookie`, `proxy-authorization` or the configured secret header.
+5. **Store large bodies by reference.** A body over 64 KB is streamed to `event_blobs` (a file plus its sha256). The row keeps `body_ref` and a 4 KB preview.
+6. **Append.** `INSERT ... ON CONFLICT (source_key, dedupe_key) DO NOTHING`, commit, and answer.
+
+**Never `reserved_session`.** Ingress uses its own short transaction. On PostgreSQL, `reserved_session` takes the global advisory lock (N8).
+
+**Owner chat.** `admit_message` (`server/services/chat/ledger.py:252`) writes the chat message and the ingress row in one transaction, so it becomes the transactional outbox.
+
+**Internal producers** (the tools in §16, `alarms.fire`, task completion) append with deterministic keys from inside their activity. A failed append raises, so Temporal retries the activity.
+
+`dispatch.emit()` remains as a facade that appends an internal ingress row, so existing producers keep working.
+
+### 8.2 Answers
+
+| Condition | Answer |
+|---|---|
+| Appended, or a duplicate | 200 (Discord: deferred reply, type 5) |
+| Body over the endpoint cap | 413, before reading when `Content-Length` is known |
+| Bad signature or header secret; an empty generic path | 401 |
+| No running or paused generation declares the path | 404 |
+| Wrong method | 405 |
+| Per-source rate or backlog over high water | 429 with Retry-After |
+| Secret unavailable, or the append failed | 503 with Retry-After |
+| Verified but malformed | 200, stored as `dead(malformed)` so the provider stops retrying |
+
+## 9. The bus
+
+### 9.1 Contract
+
+```python
+class Bus(Protocol):
+ async def append(self, records: list[IngressRecord]) -> list[AppendResult] # Accepted | Duplicate | Rejected(reason); durable on return
+ async def next_ingress(self, router_id: str, limit: int) -> list[IngressRow] # per-source order
+ async def route(self, ingress_ids: list[int], deliveries: list[Delivery]) -> None # atomic
+ async def claim_stream(self, relay_id: str, classes: list[int]) -> StreamLease | None # (top class, oldest pending) first
+ async def pending(self, lease: StreamLease, max_items: int, max_bytes: int) -> list[Delivery] # seq order, not held
+ async def delivered(self, lease: StreamLease, through_seq: int) -> None # after Temporal accepted
+ async def hold(self, scope: Scope, reason: HoldReason) -> int # budget | paused | window
+ async def release(self, scope: Scope, reason: HoldReason) -> int
+ async def settle(self, stream: StreamKey, through_seq: int, outcome: Outcome) -> None
+ async def dead_letter(self, delivery_id: int, reason: str) -> None
+ async def replay(self, selector: ReplaySelector) -> int # re-route retained ingress; operator-confirmed
+ async def lag(self) -> BusLag # per class: oldest pending age, held count, relay lag
+```
+
+There are two drivers, `SqliteBus` and `PostgresBus` (`server/services/bus/sql.py`). Producers, the router, the relay and the entities never see which one is in use. A test-only `DirectBus` performs Signal-With-Start without an inbox.
+
+### 9.2 Tables (`server/models/events.py`)
+
+| Table | Columns (main) |
+|---|---|
+| `bus_ingress` | id, source_key, dedupe_key, type, class, received_at, envelope (≤ 64 KB) or body_ref, size, chainid, hop, causationid, status (`accepted` / `routed` / `no_consumer` / `dead` / `expired`), dup_count, error |
+| `bus_dedupe` | (source_key, dedupe_key) UNIQUE, expires_at (dedupe window 7 days). On PostgreSQL it is kept separate from the day-partitioned `bus_ingress`, so the check spans days. |
+| `bus_subscriptions` | employee, generation, trigger node or implicit kind, event types (canonical plus legacy), source scope, filter parameters (secret refs only), class, `thread_by`, lane, intake agent, merge, debounce, `on_busy`, reply-capable |
+| `bus_streams` | stream_key = (workflow, generation, lane, thread), next_seq, delivered_seq, consumed_seq, top_pending_class, oldest_pending_at, hold_reason, entity_id, default_route, state |
+| `bus_deliveries` | (stream_key, seq) UNIQUE, ingress_id, subscription_id, route, class, payload or ref, causal fields, status (`pending` / `held` / `delivered` / `done` / `failed` / `dead`), reason, attempts |
+| `bus_leases` | name, owner, until, fencing token |
+| `channel_accounts` | id, driver, external id, credential ref, status, cursor JSON, lease, last_seen_at |
+| `event_blobs` | id, sha256, path, size, created_at |
+
+**SQLite (desktop):** WAL, `synchronous=FULL` (an acknowledged event must survive power loss) and `busy_timeout=5000`. Appends are single autocommit statements. One relay task. Consumed rows are deleted in chunks after retention.
+
+**PostgreSQL (hosted):**
+- `bus_ingress` is range-partitioned by day, so retention is `DROP PARTITION`.
+- Relays claim streams with `FOR UPDATE SKIP LOCKED`.
+- `idle_in_transaction_session_timeout` and `statement_timeout` are set.
+- Wake-ups use a post-commit kick plus a 1 s poll of a partial index. `LISTEN/NOTIFY` is used only for idle-to-pending edges, if at all, because every notifying commit serializes on one lock.
+
+### 9.3 Router
+
+Subscriptions are compiled from the generation snapshot in the same transaction that moves the generation to `running`. They replace the `register_trigger` Signals and the `ControlEventTypes` search attribute.
+
+For each `accepted` ingress row, in per-source order, one transaction:
+1. **Match.** Match subscriptions by type and scope, then by the plugin's `build_filter`. A filter that raises makes a `dead(filter_error)` row; it neither admits nor drops silently.
+2. **Guard.** Apply the storm guards (§16.3).
+3. **Number.** Lock the stream row, allocate `seq = next_seq` and write the `bus_deliveries` row. This serializes commits per stream.
+4. **Mark.** Mark the ingress row `routed`, `no_consumer` or `dead`.
+
+Fan-out is capped at 50 subscriptions per event. Re-running the router after a crash is a no-op.
+
+### 9.4 Relay
+
+The relay claims streams ordered by (highest pending class, oldest pending). When Temporal pushes back, it shares the Signal rate among classes by deficit round-robin with the floors of §13.1. For each claimed stream it checks, in order:
+1. **The generation is active.** Otherwise the deliveries become `dead(reset)`.
+2. **Budget and pause holds** (§14). `high` holds C3 and C4; `max` or a paused employee holds every class.
+3. **The window.** If `delivered_seq - consumed_seq` has reached 32 inputs or 512 KB, the stream is held with reason `window`, and the backlog stays in the database (principle 4).
+4. **Send.** It reads at most 16 pending deliveries or 256 KB and calls `client.start_workflow("ConversationWorkflow", id=<entity id>, start_signal="deliver", start_signal_args=[batch], id_reuse_policy=ALLOW_DUPLICATE, priority=Priority(priority_key=<class>, fairness_key=<employee id>))`.
+5. **Commit.** It commits `delivered_seq` only after Temporal accepts.
+
+**Limits:** at most one Signal-With-Start in flight per stream, and at most 4 per entity per second.
+
+**When Temporal fails:**
+- A breaker opens on unavailability, with backoff and a 5 s probe.
+- RESOURCE_EXHAUSTED slows the lowest class first.
+- A deterministic rejection marks the delivery dead after 5 attempts. The stream keeps moving, so there is no head-of-line blocking.
+
+### 9.5 Delivery protocol: exactly one effect per event
+
+1. **Ingress or driver:** append, commit, acknowledge.
+2. **Router:** writes the delivery with its stream `seq`.
+3. **Relay:** sends `deliver(rows, upto_seq)` and commits `delivered_seq` after Temporal accepts.
+4. **Entity:** drops any row with `seq <= applied_seq`. That counter is carried across continue-as-new, so a relay crash between the accept and the commit makes only a harmless duplicate.
+5. **Consumption.** For agent turns, `save_conversation` writes the transcript and advances the stream's `consumed_seq` in one transaction, so an input counts as consumed exactly when its text is in the agent's history. Graph turns without an agent settle at turn end with `settle`.
+6. **Effects.** Effects carry idempotency keys (§17). The `run_runtime_mutation` ledger (`server/core/database.py:731`) covers the database effects.
+
+### 9.6 Why the relay pushes rather than having entities pull
+
+The alternative was a payload-free doorbell Signal with the entity fetching from the database. The relay pushes bounded batches instead, for three reasons:
+- Every driver presents the same contract to the entity.
+- A wake costs one action instead of three (signal, fetch, claim).
+- Workflow workers never read the inbox.
+
+This is the documented consumer pattern: Signal-With-Start, then commit after the server accepts. NAPI's property is kept: once the window fills, the relay stops pushing and the entity drains its window in one turn.
+
+### 9.7 Alternatives rejected
+
+| Option | Why not |
+|---|---|
+| Signal-With-Start straight from ingress, no inbox (RFC-0004's in-workflow mailbox) | Ingress must refuse or lose events whenever Temporal is down or throttling. No replay, no dead letters, nowhere to hold work for a budget, and every event costs 2-3 state transitions. |
+| gRPC streams between services as the bus | gRPC moves events and never stores them: a crash loses whatever was buffered. Adding a log means building a broker. |
+| A broker (Kafka, NATS JetStream, KurrentDB, Pub/Sub) | One more stateful service in every instance (desktops have none; Redis Streams were retired in Wave 15.3). An instance peaks at about 50 events/s, while PostgreSQL handles est. 2,000-5,000/s. |
+| Dapr pub/sub | Adds a sidecar and no durability of its own. |
+
+### 9.8 Conformance suite
+
+`server/services/bus/conformance/` runs against both drivers:
+- kill -9 at every step loses no acknowledged event;
+- duplicate appends;
+- FIFO per stream;
+- hold and release order;
+- the window;
+- replay;
+- 10,000 messages to one conversation cost fewer than 1,000 Signals per entity run.
+
+## 10. Processes (the workflow types)
+
+### 10.1 Granularity and ids
+
+| Workflow | Id | Started by |
+|---|---|---|
+| EmployeeSupervisorWorkflow | `emp-<workflow_id>-g<gen>` | Start, with `REJECT_DUPLICATE` |
+| ConversationWorkflow (the entity) | `conv-<workflow_id>-g<gen>-<lane>-<thread>` | the relay, by Signal-With-Start with `ALLOW_DUPLICATE` |
+| Turn (MachinaWorkflow, input `event_os_version=1`) | `turn-<conv id>-<n>` | the entity: a child with `REQUEST_CANCEL` and `WAIT_CANCELLATION_COMPLETED` |
+| AgentWorkflow (input `merge_version=1`) | existing child ids (`_agent_child_workflow_id`) | the turn |
+| ScheduleFireWorkflow and the poll cycle | Schedule actions | Schedules `cron-…`, `poll-…`, `alarm-…` (§15) |
+| ApprovalWorkflow | `approval-<id>` | approval creation |
+
+Ids use the immutable workflow id, never the slug. Every new type:
+- uses `@workflow.init`;
+- has signal handlers that only buffer;
+- continues as new only from the main loop, after `workflow.all_handlers_finished()`.
+
+### 10.2 Per-type specification
+
+| Type | State | Messages | Timers | Continue-as-new |
+|---|---|---|---|---|
+| Supervisor | control state and revision; alarm `next_due` and version; budget state and version; health window | **Updates** with validators:<br>• `set_control_state` (fans Stop/Resume out to the live turns in `sched_leases`; a durable Stop barrier that fixes H9);<br>• `wait_for_checkpoint`;<br>• `set_limits`;<br>• `resume_after_budget`.<br>**Signals:** `alarms_changed`, `budget_changed`, `turn_failed`.<br>**Query:** `status`. | updatable alarm timer; budget period end (midnight in the owner's time zone); 5-minute upkeep | when suggested, or daily; carries about 2 KB |
+| Entity | pending window per class; `applied_seq`; offered set for the running turn; running turn handle; `turn_no`; debounce anchors; failure window | **Signals:** `deliver`, `inputs_consumed`, `turn_suspended`, `wake`.<br>**Update:** `interrupt_turn` (validator: that turn is running).<br>**Query:** `status`. | debounce (sliding plus cap); admission retry every 60 s; turn deadline per class; idle passivation; restart backoff | only between turns, when suggested or past 4,000 events or 4,000 Signals. Carries `applied_seq`, `turn_no` and at most 256 KB of pending inputs; any excess goes back to the database as `hold(window)`. |
+| Turn | today's MachinaWorkflow state, plus the offered inputs | **Signal:** `offer_inputs`, forwarded to the intake agent. If that agent has already finished, the turn answers `inputs_returned`. | none new | none. Loads the graph by reference (generation plus trigger node), which fixes H10 and M6. Its result adds the consumed inputs and usage. |
+| AgentWorkflow | plus offered inputs, consumed ids, merges, current class | **Signals:** `offer_inputs`, `budget_released` | none new; a turn suspended for budget waits without a timer, and an upkeep sweep fails one suspended for more than 7 days | its existing continue-as-new; the resume marker adds consumed ids, class and merges, and already carries `usage_total` (`agent_workflow.py:2997`) |
+| ScheduleFire / poll cycle | none | none | none | one activity: append a tick, or poll (fetch, append and save the cursor in one transaction, then `post_emit`) |
+| Approval | decision, revision, expiry | **Update:** `decide{decision, revision, key}`.<br>**Query:** `status`. | expiry, then the Undo window | one send activity, then `approval.decided` (C1) appended to the origin thread |
+
+### 10.3 Failure semantics
+
+- A turn that fails ends as a Failed workflow (RFC-0004 D12).
+- The entity retries its inputs once, then dead-letters them with an attention item. Restart intensity is described in §18.2.
+- A bug in workflow code leaves that entity or supervisor stuck and visible (`TemporalReportedProblems`). The inbox keeps accepting, and nothing acknowledged is lost (I10).
+
+### 10.4 Versioning
+
+- **Generation flags.** New generations get `resource_manifest.event_os_version=1` from `begin_generation` (`server/services/deployment/control.py:112`, `:160`). Older generations keep `WorkflowControlWorkflow` until they are Reset.
+- **Patches.** New types need no patches. Shared types (MachinaWorkflow, AgentWorkflow) branch only on input flags frozen at admission, which no history from an older release contains.
+- **Preview features** stay unused (RFC-0004 Q12): Workflow Streams, External Storage, Worker Versioning, Workflow Pause and eager start.
+
+## 11. Merging and batching (E6)
+
+1. **Debounce.** An idle entity that receives `deliver` waits with a sliding window and an absolute cap:
+
+ | Class | Sliding window | Cap |
+ |---|---|---|
+ | Owner chat | 0.4 s | 1.5 s |
+ | Customer chats | 2 s | 8 s |
+ | Customer groups | 3 s | 10 s |
+ | Other classes | none | none |
+
+ `workflow.wait_condition` with a timeout drives this.
+2. **Admit.** `admission.acquire(class)` (§13.3) answers granted, queued (the entity waits for `wake`), or denied by budget or pause (it also waits for `wake`).
+3. **Start.** The entity takes the longest pending prefix that shares one reply route (at most 50 inputs or 256 KB) and starts turn n with references to those inputs.
+4. **Offer.** When `deliver` arrives during turn n on the same route, the entity sends `offer_inputs` to the turn, which forwards it to the intake agent. The agent merges at two points:
+ - **B1.** At the top of each iteration, after `_wait_until_resumed` (`server/services/temporal/agent_workflow.py:1152-1172`), it appends offered inputs as user messages before the model call.
+ - **B2.** After a final answer that has not been sent (`:3078-3096`), at most twice per turn and on customer channels only (§25): it keeps the unsent draft as an assistant message, adds a one-line note that more messages arrived, and takes one more step. A customer who double-texts gets one coherent reply.
+
+ The save at that step advances `consumed_seq`, and the agent tells the entity with `inputs_consumed`. Tool batches are never interrupted.
+5. **Priority inheritance.** If a merged input has a higher class than the turn, the agent's later steps use that class's queue and Priority key, and the entity upgrades its admission lease (the PI-futex analog).
+6. **Turn end.** Offered but unmerged inputs, and inputs for another route, start the next turn immediately, without debounce. The consumed inputs of a failed turn are retried once, then dead-lettered with an attention item.
+
+**Graph runs** whose first node after the trigger is not an agent take one input per turn; `on_busy` controls overlap (`queue` by default, `skip` for cron). In a batch, the trigger output is the last input's data plus `batch: [...]`, so existing templates keep working. Subgraphs without an agent that has Context are stateless and get a stream per event, so they run in parallel.
+
+**Passivation.** After an idle period (owner 24 h, customer 30 min, others 10 min), the entity completes. A `deliver` that lands while it is closing is not lost: the server rejects completion while new Signals are unhandled, and the relay's re-ring of an undelivered window covers the rest.
+
+**Chat changes** (`server/services/chat/`, `docs-internal/chat_protocol.md`):
+- `send_chat_message` no longer answers `run_in_progress` or `engine_unavailable`. A message sent while Temporal is down is stored and shown as queued.
+- The entity is the only writer of `ChatRun`. `chat_run.open` runs at turn start, and a new `chat_run_inputs` table records merged messages.
+- New run events: `run.input_merged`.
+- An owner message during a running owner turn is merged at B1, or starts the next turn right away.
+
+## 12. Conversation streams, lanes and memory (E9)
+
+**Threads**
+
+| `thread` | Used for |
+|---|---|
+| `main` | owner chat, cron, webhooks, polls and task reviews (today's continuity) |
+| `t_<sha256(account, chat)[:24]>` | each customer conversation; ids carry no personal data |
+| `peer_<employee>` | messages from another employee |
+| origin thread | task results and approvals |
+| `e_<delivery id>` | ephemeral, per event, for stateless subgraphs |
+
+**Lanes.** Triggers whose subgraphs share an agent with Context form one lane.
+
+**Implicit subscriptions** need no node. They are routed to the front agent: the talk agent, else the lead, else the first agent.
+
+| Subscription | Class | Thread |
+|---|---|---|
+| `employee.inbox` | C3 | `peer_<sender>` |
+| `employee.task_result` | C3 | origin |
+| `alarm.fired` | C4 | the alarm's thread |
+| `approval.decided` | C1 | origin |
+
+**Conversation key.** It becomes (workflow, generation, agent node, thread). `agent_conversations` gains `thread_id` with default `main`, and its unique key widens to match. This is the first Alembic migration (§21.4).
+
+**Unchanged.** RFC-0002's rules stay in force:
+- requests are built from messages;
+- the store records what was sent;
+- loads are loud and saves are best effort;
+- a carried rollover transcript beats the stored one.
+
+## 13. Scheduler (E10)
+
+### 13.1 Classes
+
+| Class | Sources | Priority key | LLM queue | Floor | Turn deadline |
+|---|---|---|---|---|---|
+| C1 owner | owner chat, approval decisions | 1 | `llm-c1` | 30%, plus one reserved slot per employee | 15 min |
+| C2 customer | WhatsApp, WhatsApp Business, Telegram, Discord, email replies | 2 | `llm-c2` | 40% | 15 min |
+| C3 internal | employee messages, delegations, task results, signed provider webhooks | 3 | `llm-c3` | 15% | 30 min |
+| C4 background | alarms, cron, polls, watched data, generic webhooks, Start ticks | 4 | `llm-c4` | 15% | 60 min |
+| C5 upkeep | digests, sweeps | 5 | `machina-tasks` | none | n/a |
+
+A trigger may override its class. A class's unused share flows down the order.
+
+### 13.2 Temporal mapping
+
+- **Priority keys** are set on entity starts, turn children and every activity (inherited), and on sends.
+- **Fairness key = employee id**, with weight 1. Each instance has one owner, so there are no tenant keys.
+- **The dev server** gets `--dynamic-config-value matching.enableFairness=true`. Priority is on by default (`matching.useNewMatcher`).
+- **Both need the pinned server** at version 1.31 or later (RFC-0004 D1). Without them behaviour stays correct, only unfair.
+- **Task queues.**
+ - Workflow tasks share `machina-tasks`.
+ - LLM steps run on the four class queues and leave `machina-tasks` (RFC-0004 M9). Each run freezes its queue in its input.
+ - Tools stay on their plugin queues with Priority keys; sends go to `messaging`.
+- **Slot pool.** Each worker process runs one Worker per class queue. They share one `ClassSlotPool` built on `temporalio.worker.CustomSlotSupplier` (stable in SDK 1.30, `temporalio/worker/_tuning.py:160`). The pool:
+ - guarantees each class its floor;
+ - grants freed slots in strict class order;
+ - limits idle poller reservations to 1-2 per class, because `reserve_slot` runs before polling.
+
+ Fallback: fixed pool sizes. The existing `WorkerTuner.create_composite` usage in `worker.py` shows where it plugs in.
+- **Starvation shows.** Background queues get `schedule_to_start` timeouts.
+
+### 13.3 Admission
+
+Tables:
+- `sched_leases`: turn, employee, class, state `running` or `suspended`, control revision. It is also the live-turn registry used by Stop and Reset.
+- `sched_waiters`.
+
+`acquire` is one transaction:
+1. Budget and pause gate.
+2. Per-employee cap: 4 by default, plus the reserved owner slot.
+3. Grant if the class is below its floor. Grant from shared capacity only if no higher class is waiting. Otherwise queue the request.
+
+`release` serves waiters in this order:
+- classes below their floor;
+- strict class order;
+- round-robin by employee within a class.
+
+It then wakes the waiting entity with `wake`. The supervisor's upkeep reaps leases whose turn is confirmed closed.
+
+### 13.4 Provider depth and priority inversion
+
+- **Concurrency.** Worker slots cap concurrent LLM activities. Per-credential token buckets cap requests and tokens per minute for each class. Retry-After responses become BENIGN retries with `next_retry_delay`.
+- **Priority inversion is avoided by:**
+ - floors;
+ - priority inheritance on merge (§11);
+ - no lock held across turns;
+ - the owner's **Interrupt** (`interrupt_turn`): when a lower-class turn holds the same conversation, it is cancelled at its next merge point and its inputs return to pending.
+
+## 14. Budgets (E11)
+
+**Ledger:**
+- `employee_budgets`: daily limit, soft ratio 0.8, the owner's time zone, revision.
+- `employee_spend`: day, reserved, settled, state (`normal` / `high` / `max`), version.
+- `spend_events`: key `{workflow_id}:{activity_id}`, UNIQUE.
+
+The OpenCompany Gateway already records spend per call and replays retried calls at no cost. The ledger also covers calls made with the owner's own keys.
+
+| Where | Rule |
+|---|---|
+| Relay, before delivery | Reads the cached state (refreshed by transition events, 30 s TTL). `high` holds C3 and C4; `max` or paused holds every class. |
+| Entity, before each turn | `admission.acquire` reads the ledger, which is authoritative. |
+| LLM step (`agent.execute_llm_step`) | `budget.reserve(key, upper_estimate)` (input tokens times price, plus `max_tokens` times output price), then `settle(key, actual)` after the call. If the reservation would pass 100%, it raises `BudgetExceeded`: non-retryable and BENIGN. The agent suspends, it does not fail. A C3 or C4 turn that sees `high` suspends at its next step and holds no slot. |
+
+**Transitions:**
+- The transaction that crosses a threshold appends `employee.budget.changed` to the supervisor.
+- At `max`, the supervisor runs Stop with `pause_reason="budget"`, through the existing automatic-pause path (`_automatic_pause_values`, `server/services/deployment/handlers.py:1030`). The owner gets an attention item and a notice with **Resume**.
+- At midnight in the owner's time zone, the supervisor's period timer resets the day and resumes, with a notice.
+- `resume_after_budget` (raise or top up the limit) does the same at once.
+- Held inputs are released in (class, seq) order, and suspended turns get `budget_released`.
+
+**Continue-as-new and retries.** Enforcement uses only the ledger. Reservation keys are activity ids, and iteration numbers continue across continue-as-new, so retries never double-count.
+
+**Overshoot** is bounded by price-table error, because reservations use the upper estimate.
+
+## 15. Timers (E5)
+
+- **Cron.**
+ - Schedule `cron-<workflow_id>-g<gen>-<node_id>`, built from immutable node ids (fixes RFC-0004 M2).
+ - Its action is ScheduleFireWorkflow, which appends a C4 tick, so ticks obey class floors and the 80% gate.
+ - Overlap `BUFFER_ONE` covers the tiny fire workflow. Jitter is 60 s, and catch-up is one interval, at most 1 hour.
+ - The business overlap rule is the entity's `on_busy`, default `skip` (fixes H6).
+ - Stop pauses Schedules by id and Reset deletes them by id. Re-arm recreates them from the snapshot (I10).
+- **Polls.**
+ - Schedule `poll-<workflow_id>-g<gen>-<node_id>` with overlap `SKIP`.
+ - The poll cycle fetches, appends new items and saves the cursor in one transaction, then runs `post_emit` (fixes M3).
+- **Alarms (self-scheduled wake-ups).**
+ - Tools `set_alarm`, `cancel_alarm` and `list_alarms` write `alarms`:
+ - `id = hash(turn, tool_call_id)`, thread, `due_at` (UTC);
+ - `if_late`: fire with a note, or `skip_after:<duration>`;
+ - `cancel_on: inbound`, repeat, status.
+ - They then signal `alarms_changed`.
+ - The supervisor keeps one updatable timer for the earliest due alarm. `alarms.fire_due` moves the alarm to `fired` by compare-and-swap and appends an input stamped with `scheduled_for`, `fired_at` and its lateness.
+ - Recurring alarms become Schedules `alarm-<id>`. Limit: 500 per employee.
+- **Follow-ups** are alarms with `cancel_on=inbound`; the router cancels them in the same transaction as the inbound message.
+- **The Start node** fires once per generation (dedupe key `start:<gen>`; Phase 0.4).
+- **Approval expiry** is the ApprovalWorkflow's timer. `expire_due` stays as a backstop.
+
+## 16. Employee messaging and watched data (E7, E8)
+
+### 16.1 Messages and delegation
+
+- **Tools** (`server/nodes/tool/employee_mail/`): `send_to_employee`, `delegate_to_employee` and `complete_task`. They write `ipc_tasks` (task id, requester employee, generation and thread, assignee, state, deadline, `reply_to`) and append to the inbox in the sender's own transaction, as an outbox.
+- **Scope.** Any employee in the instance may address any other.
+- **Routing.**
+ - The target's implicit `employee.inbox` subscription puts the message in thread `peer_<sender>`.
+ - Results route through the explicit `reply_to`, never a broadcast. The router accepts a completion only from the assignee (`assigned` to `completed`, by compare-and-swap). A deadline alarm delivers `timed_out`.
+ - Team tasks inside one employee become scoped inputs to their origin thread (fixes H11).
+
+### 16.2 Watched shared data
+
+- The Context, Canvas and Memory stores emit `com.opencompany.data.<store>.<entity>.changed` (subject = the record id) from their write paths. Each event carries the writing turn's causal fields.
+- Subscriptions add match rules on store and subject.
+- Inputs coalesce per (subscription, record) and are delivered level-triggered: "changed N times; re-read the record". Up to 200 subjects, then a single `resync` input.
+
+### 16.3 Storm guards (`server/services/bus/causal.py`, `token_buckets`)
+
+- At most 8 hops.
+- **Cycle check.** An input whose target is already on its `path` (at most 16 entries) is suppressed as `loop_suppressed`.
+- No self-wake from an employee's own writes.
+- A token bucket per edge: 10 per minute, burst 20.
+- **Breaker.** After 3 exhausted windows the breaker opens: the edge becomes a 15-minute digest and an attention item names both employees. It half-opens after 1 hour.
+- **Merged turns.** A merged turn's outputs carry the maximum hop and the union of the input paths.
+
+## 17. Egress
+
+### 17.1 Outbox
+
+`outbound_messages` holds:
+- the idempotency key (UNIQUE), employee, thread, turn, account, recipient (encrypted), kind, class;
+- body or ref;
+- status: `held`, `queued`, `sending`, `sent`, `failed`, `unknown` or `expired`;
+- approval id, `not_before`, claim token, provider message id, receipt.
+
+The reply key is a hash of (thread, route, consumed input set, node). A retried turn that answers the same inputs is therefore deduplicated even when its text differs.
+
+### 17.2 Sending
+
+`egress.send` runs on `messaging` with Priority = class and fairness key = channel account:
+1. Claim the row by compare-and-swap.
+2. Take a token from the per-account and per-recipient bucket. If none is free, raise a retryable error with `next_retry_delay`, so no worker slot is held while waiting.
+3. Call the provider with its idempotency key where it has one.
+
+### 17.3 Uncertain outcomes (E12)
+
+For a provider without idempotency keys, a retry that finds an expired `sending` claim marks the row `unknown`. That raises an attention item and offers a Retry button. It never resends on its own.
+
+### 17.4 Order and backpressure
+
+- Sends are awaited inside turns, so each thread stays FIFO.
+- Above high water the producing turn waits (BQL, writeback throttling).
+- C4 sends may use only the top half of a bucket.
+
+### 17.5 Approvals
+
+- A send that needs approval is `held`.
+- ApprovalWorkflow (an Update `decide` plus a durable timer, RFC-0004 D13) sets `not_before` to the end of the Undo window and sends once. It replaces `ApprovedToolCallWorkflow`.
+- Approved sends go out in the order the owner decides them.
+
+### 17.6 Receipts and UI
+
+- Provider status events update the row, and a failure becomes an input to the thread.
+- `StatusBroadcaster.broadcast` requires an audience: owner, workflow or system. Frames that carry content need an owner or workflow audience, so content never reaches every socket.
+- Chat streaming through the chat hub is unchanged.
+
+## 18. Supervision, observability and telemetry
+
+### 18.1 Watchdogs
+
+- Turn deadlines per class, owned by the entity.
+- A stuck-turn alarm when there has been no progress for 10 minutes.
+- Router stalled: the oldest `accepted` row is older than 30 s.
+- A window full while its entity is idle (a credit leak).
+- `TemporalReportedProblems`.
+- The chat watchdog reads turns by known id, never Visibility.
+
+### 18.2 Restart intensity (OTP MaxR/MaxT)
+
+- **Per thread.** 3 failed turns in 10 minutes back off 1, 5, then 30 minutes. 5 in an hour quarantine the thread, with its inputs held.
+- **Per employee.** 30% of turns failing (at least 10 in 15 minutes), or 3 quarantined threads, makes the employee `degraded`: C3 and C4 pause and the owner is told.
+
+### 18.3 Attention items
+
+`attention_items` (kind, severity, ref, first and last seen, count, state) appear in the employee summary. The kinds are:
+- quarantined thread, dead input, unknown or failed send;
+- budget high, budget max;
+- driver down, Temporal problem, stuck turn;
+- loop suppressed or breaker open, missed alarm, expired approval.
+
+### 18.4 Metrics (measure waiting)
+
+| Area | Metrics |
+|---|---|
+| Ingress | append latency |
+| Bus | relay lag; oldest pending age and held count per class |
+| Scheduling | admission wait and slot wait per class; `schedule_to_start` per queue |
+| Temporal | Signals and history size per run |
+| Turns | duration and outcome |
+| Egress | depth and latency |
+| Health | dead letters by reason, open breakers, budget transitions, event-loop lag |
+
+A per-employee `employee_timeline` view joins ingress, deliveries, turns, outbox, alarms, messages and attention on `chainid`.
+
+### 18.5 Fleet telemetry (E14)
+
+- **Hosted instances export** metrics and error reports over OpenTelemetry (OTLP) to a central collector. Error reports carry the exception type, version, count and traceback frames without local values.
+- **Never exported:** message text, prompts, customer data and credentials.
+- **Enforcement.** A content-filter test checks every exported attribute name and value type.
+- **Desktop.** The desktop app exports nothing unless the owner opts in.
+
+## 19. Delivery semantics, ordering and overflow
+
+**Delivery at each stage**
+
+| Stage | Semantics |
+|---|---|
+| Provider to ingress | at-least-once |
+| Acceptance | exactly once within the 7-day dedupe window |
+| Routing | exactly once |
+| Relay delivery | at-least-once, deduplicated by `seq` |
+| Consumption | exactly once (transcript and cursor in one transaction) |
+| Processing | at-least-once, with one retry |
+| Effects | effectively once through idempotency keys; at-most-once with `unknown` where a provider has none |
+
+**Ordering**
+- FIFO per stream and reply route.
+- Across routes in one stream: sequence order at turn boundaries, but a running turn may absorb later inputs on its own route first.
+- No order across streams.
+- Sends are FIFO per thread.
+
+**Overflow (defaults)**
+
+| Source | Bound | When full | Staleness |
+|---|---|---|---|
+| Owner chat | 100 pending per session | `queue_full` at send; nothing saved | never expires |
+| Customer chats | entity window 32; 5,000 per account | never dropped; above high water the driver stops reading (Telegram, bridge) or answers 429 (WhatsApp Business) | delivered late with a lateness note; WhatsApp Business needs a template after 24 h |
+| Signed webhooks | 10,000 unrouted per endpoint; 50/s | 429 | none |
+| Generic webhooks | 1,000 unrouted; 10/s | 429 | none |
+| Polls | 1,000 unrouted | cycle skipped; cursor stays | n/a |
+| Cron | 1 pending tick per Schedule | the newest wins; `on_busy` | older than one interval: `expired` |
+| Alarms, open tasks | 500 alarms, 200 open tasks per employee | creation refused | `if_late`; task deadline |
+| Employee messages | 10 per minute per edge; 500 pending per target | breaker, then digest | 24 h TTL, then `expired` |
+| Watched data | 1 per (subscription, record); 200 subjects | coalesce, then `resync` | level-triggered |
+| Receipts, typing, presence | 1,000 in memory per account | drop oldest | 30 s |
+
+## 20. Instance tiers (E1-E3)
+
+| | Desktop | Hosted tier S | Hosted tier L |
+|---|---|---|---|
+| Employees | any, warning above about 100 | up to about 100 | up to about 1,000 |
+| App database and inbox | SQLite | PostgreSQL 16 | PostgreSQL 16 |
+| Temporal | CLI dev server on SQLite | CLI dev server on SQLite | open-source `temporal-server` on the instance's PostgreSQL (databases `temporal` and `temporal_visibility`), 64 history shards fixed at creation |
+| Processes | one | API with relay and workers | API with relay; a workflow worker; activity workers per class |
+| Peak load (est.) | below tier S | about 5 events/s, 50 Temporal actions/s, 65 LLM calls in flight | about 50 events/s, 500 actions/s, 650 LLM calls in flight |
+| Limit | dev server est. 100-500 actions/s on one shard | same | 64 shards; app database after the advisory-lock fix (N8) |
+
+The peaks are scaled from a 20-employee model: 2,600 turns/day, about 35k Temporal actions/day and 8-10 actions/s at peak. At tier L the owner's LLM provider limits bind before Temporal does, and the class floors decide who waits.
+
+**Tier S hardening.** The CLI documentation warns that the dev server "is not intended for production use" and skips some HTTP security checks. Tier S therefore:
+- pins the CLI version and checksum (RFC-0004 D1);
+- persists to `--db-filename` with the WAL pragmas already set (`_runtime.py:153-158`);
+- binds to 127.0.0.1 only;
+- sets the namespace retention to 7 days;
+- passes the dynamic config for priority and fairness;
+- keeps its database in the instance backups.
+
+If its database is lost, the inbox and I10 bring the instance back without losing an acknowledged event.
+
+**Tier L setup**, all in the deploy logic:
+- The single-process `temporal-server` runs all services.
+- Its schema is created and upgraded with `temporal-sql-tool`.
+- It binds to localhost.
+- It is upgraded one minor version at a time, as the production checklist requires.
+- The shard count is fixed when the cluster is created, so 64 is confirmed by the tier-L load test before the first tier-L instance exists.
+
+**Moving from S to L** runs in the maintenance window, and only once every generation runs on the new path (Phase 4 or later):
+1. Stop every generation at its checkpoint. Inputs keep arriving in the inbox.
+2. Wait for running turns. Turns past their deadline are cancelled and their inputs return to pending.
+3. Start the full server, and create the namespace and search attributes.
+4. Re-arm from the app database: supervisors, Schedules and alarms come back. Then Resume, and the inbox drains.
+
+Temporal histories are not migrated (I10). The old `temporal.db` is kept for 7 days.
+
+## 21. Fleet operations (E13, E14)
+
+### 21.1 Deploy logic
+
+Every instance change lives in the deploy logic, never in a hand-configured VM: `cli/terraform/gcp/`, `cli/terraform/aws/`, the startup scripts, `install.sh` and `docker/`. Hosted instances:
+- install PostgreSQL 16 and create the app and Temporal databases;
+- pin the OpenCompany and Temporal versions;
+- choose their tier;
+- configure the telemetry exporter.
+
+**Today's behaviour that must change.** The GCP startup script reinstalls the package on every boot, and the installer service follows npm latest within minutes of a publish. Hosted instances must instead stay on their pinned version until the next window.
+
+### 21.2 Weekly window
+
+All hosted instances upgrade together in one fixed weekly UTC window (default: Sunday 03:00).
+
+**Before the window:**
+- The release must pass the replay gate, the golden identity test and the bus conformance suite.
+- Our own staging instances run it from 3 days earlier.
+
+**During the window, each instance:**
+1. snapshots its databases;
+2. applies forward-only migrations;
+3. restarts;
+4. checks `/health/ready`;
+5. if the check fails, rolls back to the previous version and snapshot on its own, and reports the outcome through telemetry.
+
+**Temporal** moves at most one minor version per window. Inputs arriving during the restart wait at their providers (webhook retries, the Telegram backlog, the bridge's outbox) and land in the inbox afterwards.
+
+### 21.3 Backups and restore
+
+- **Daily backups.** A PostgreSQL dump with WAL archive, plus a snapshot of the data directory, go to the instance's bucket.
+- **Restore** always runs the reconcile:
+ - deliveries `delivered` but not consumed return to pending;
+ - Schedules and alarms are re-armed;
+ - orphaned chat runs fail.
+- **No duplicate sends.** Send keys come from workflow and activity ids, so in-flight activities that run again send nothing twice.
+
+### 21.4 Schema migrations
+
+- Alembic manages both databases with expand-then-contract changes: a release only adds, and removals ship two releases later.
+- This replaces the ad-hoc `_migrate_*` startup steps and `_validate_shared_schema` (N10).
+- The first migration adds `agent_conversations.thread_id`. Until Alembic lands (Phase 9b), Phase 0.6's index change uses an explicit startup step under the existing advisory lock.
+
+## 22. Scenarios
+
+**(a) Three WhatsApp messages in 2 s while turn 7 is replying to message 41**
+1. The bridge streams each event over `Publish`.
+2. The bus appends each one (key `whatsapp:A:C:<id>`) and acks.
+3. The bridge then acks WhatsApp and deletes its outbox row.
+4. The router writes deliveries 42, 43 and 44 to stream T as C2.
+5. The relay sends `deliver[42..44]` to `conv-W-g7-L-T` and commits `delivered_seq`.
+6. Turn 7 is running on the same route, so the entity offers them; the turn forwards them to the agent.
+7. The agent's step 2 returns a final answer. At B2 it keeps the unsent draft, appends the three messages and takes step 3, which writes one reply covering all four.
+8. The save advances `consumed_seq` to 44.
+9. The send step writes an `outbound_messages` row keyed over inputs 41-44, takes a token for account A and recipient C, and sends.
+
+If the employee had been idle, one debounced turn takes 42-44 at 3.6 s. If the reply had already gone out, 42-44 start turn 8 immediately.
+
+**(b) A Stripe webhook delivered twice, with a restart in between**
+1. Delivery 1: size check, signature verified over the raw bytes, appended with key `evt_X`, answered 200.
+2. The backend crashes. On boot, the router resumes from `accepted` rows and the relay from pending deliveries.
+3. If the crash came after Temporal accepted the Signal but before `delivered_seq` was committed, the entity drops the repeat by `seq`.
+4. Delivery 2 has the same `evt_X`: it is a duplicate, gets 200 and routes nothing.
+5. One turn handles the event. Its Stripe calls and emails carry keys derived from (input, node).
+
+If the crash came before the append, Stripe never saw a 2xx, and its retry becomes the first append.
+
+**(c) The owner chats while the same employee's cron job runs**
+1. The 08:00 Schedule appends a tick (C4, thread `main` of the worker lane), which runs on `llm-c4` with priority 4.
+2. At 08:03 the owner's message (message plus ingress row in one transaction) goes to the talk lane as C1.
+3. After the 0.4 s debounce, admission grants the employee's reserved owner slot, and the turn runs on `llm-c1`. The slot pool serves C1 first.
+4. The two turns use different conversation keys and run side by side.
+
+If an editor graph wires both triggers into one agent and one thread, the owner's message waits as "next", and the owner can press Interrupt.
+
+**(d) A reminder for tomorrow 09:00, with the laptop asleep at 09:00**
+1. `set_alarm` writes the row and signals the supervisor, which sets its timer.
+2. On wake at 09:47 the dev server fires the overdue timer. `alarms.fire_due` appends an input that is 47 minutes late.
+3. Signal-With-Start restarts the passivated entity.
+4. The agent reads "Reminder due 09:00, firing 47 min late".
+
+With `skip_after:2h`, a later wake marks the alarm `missed` and raises an attention item instead.
+
+**(e) Employee A delegates to B, and B's result wakes A**
+1. `delegate_to_employee` checks hops, the edge bucket and the open-task cap, then writes `ipc_tasks` and appends. A's turn ends with "assigned".
+2. B's inbox puts the task in thread `peer_A` as C3, and B runs.
+3. `complete_task` (assignee only, by compare-and-swap) appends a result with `reply_to = (A, T_A)`.
+4. The router puts it in T_A. A's next turn uses the same conversation key, so A remembers its plan.
+
+If the deadline passes first, A gets `timed_out`.
+
+**(f) A CRM record changes, waking two employees, and one would wake the other**
+1. Chain c0 creates inputs for E1 and E2 at hop 1 (C4).
+2. E1 writes the record: hop 2, path [E1]. E1 itself is not woken. E2's input is coalesced, or merged into its running turn.
+3. E2 writes the record: hop 3, path [E1, E2]. The input for E1 is a cycle, so it is suppressed.
+4. If the ping-pong continues through new chains, the edge bucket runs out, the breaker turns the edge into a digest, and an attention item names E1 and E2.
+
+**(g) The dev server is down for 10 minutes while Telegram messages and webhooks keep arriving**
+1. Drivers, ingress and the router depend only on the database.
+2. Telegram's offset advances after each append, webhooks get 200, and owner messages show as queued.
+3. The relay's breaker probes every 5 s.
+4. Backlogs past high water pause Telegram reads (Telegram keeps them for 24 h) or answer 429.
+5. On recovery the relay drains by class floors, owner first. Alarms and catch-up ticks fire late and coalesce.
+
+Nothing acknowledged is lost.
+
+**(h) A 5 MB webhook body**
+1. `Content-Length` is above the 256 KiB default cap, so the answer is 413 before the body is read.
+2. If the endpoint declares 10 MiB, the body is streamed to `event_blobs` with an incremental HMAC. The row keeps `body_ref`, a 4 KB preview and the allowlisted headers.
+3. Turns carry only the reference, and nodes read it through a FileRef.
+
+No Temporal payload can exceed a limit (C1), and the body is never broadcast.
+
+**(i) The budget crosses 80% and then 100% (limit $20 a day)**
+1. A C2 step reserves $0.20 at $15.90 spent, reaching 80.5%. The ledger moves to `high`.
+2. The relay holds the employee's C3 and C4 inputs, and the admission for a cron turn is denied. A running C4 turn suspends at its next step. C1 and C2 continue.
+3. A reservation that would pass $20 raises `BudgetExceeded`, and that turn suspends.
+4. The ledger moves to `max`. The supervisor stops the employee with `pause_reason=budget`, and the owner gets a notice with Resume.
+5. The owner adds $10, or midnight arrives. The state returns to `normal`, held inputs are released in (class, seq) order, and suspended turns get `budget_released`.
+
+## 23. Failure modes
+
+| Failure | Detected by | Effect | Recovery |
+|---|---|---|---|
+| Crash before the append | the provider saw no 2xx | none | the provider retries |
+| Crash after the append, before routing | age of the oldest `accepted` row | delay | the router resumes |
+| Crash after Temporal accepted, before `delivered_seq` | none needed | duplicate delivery | the entity drops it by `seq` |
+| Relay dies holding a stream lease | lease expiry | delay | another relay task claims it; the fencing token stops stale updates |
+| Temporal (dev server) down or restarting | relay breaker | backlog, 2xx continues | probe, then drain by class floors |
+| `temporal.db` lost or corrupt | server fails to start | in-flight turns lost | rebuild: start empty, re-arm from the app database (I10); unconsumed inputs re-run |
+| App database unavailable | ingress errors | 503; drivers stop reading | providers retry; Telegram holds 24 h; the bridge's outbox grows |
+| `RESOURCE_EXHAUSTED` from Temporal | relay error rate | slower delivery | lowest class slowed first; backoff |
+| Window leak | window full while the entity is idle | stream held | the entity reports consumption when idle; upkeep resets |
+| Entity or supervisor bug | `TemporalReportedProblems` | that thread or employee stuck | fix, then replay; attention item |
+| Turn fails | child failure | inputs `failed` | retried once, then dead-lettered; restart intensity |
+| Turn hangs | deadline timer; progress watchdog | slot held | the turn is cancelled, then the retry policy |
+| Poison input | repeated failures | thread quarantined | input dead-lettered; the thread continues |
+| Continue-as-new carry too large | size check before continue-as-new | none | excess returned to the database |
+| Signal or Update limits near | counters | none | earlier continue-as-new; relay batching |
+| Two readers on one account | lease fencing | duplicate ingest | dedupe |
+| Loop between employees | path, hop, bucket | suppressed or digest | breaker half-opens after 1 hour |
+| Budget overshoot | ledger | bounded by price error | pause |
+| Missed Schedule while asleep | lateness metrics | late or coalesced ticks | catch-up window, `on_busy`, `if_late` |
+| Provider outage or 429 | step retries (BENIGN) | slower turns | Retry-After; spend is not double-counted |
+| Unknown send outcome | expired claim | `unknown` | the owner's Retry (E12) |
+| Reset while work is in flight | generation fence | old entities cancelled; inputs `dead(reset)` | none needed |
+| A weekly upgrade fails on an instance | readiness check | instance down briefly | automatic rollback to the previous version and snapshot |
+
+## 24. Rollout
+
+### 24.1 Rules
+
+1. **Durable before fast.** Every phase leaves an append-only record, so a rollback never drops an acknowledged event.
+2. **Versioned per generation.** New behaviour is a `resource_manifest` flag frozen into workflow input. `workflow.patched` is used only inside shared types.
+3. **One path per generation.** Each generation uses exactly one event path, never a choice per event, so no conversation gets two replies during cutover.
+4. **Replay gate.** No change to workflow code merges until histories recorded on the previous release replay green.
+5. **Expand-only schema.** Additive migrations only.
+
+### 24.2 Phases
+
+| Phase | Content |
+|---|---|
+| 0 | Nine independent fixes without workflow-command changes:<br>• 0.1 webhook route table, 401/404/405 and empty path (C3);<br>• 0.2 scoped task events (H11);<br>• 0.3 body cap and the undrained queue (C1, N11);<br>• 0.4 Start once per generation (C4);<br>• 0.5 Telegram backlog (N1);<br>• 0.6 PostgreSQL chat lane (N6);<br>• 0.7 Discord public ingress (N4);<br>• 0.8 Twitter receive (N5);<br>• 0.9 broadcast audience (N7). |
+| 1 | Safety net:<br>• fixture recorder and replay gate;<br>• golden identity test;<br>• pinned Temporal CLI (RFC-0004 D1);<br>• `TEMPORAL_TEST_CLI` in CI. |
+| 2 | Durable ingress and router delivering to today's controllers by known id (H1, N3, N9):<br>• WAL on `workflow.db`;<br>• bus tables and both drivers;<br>• rollout per source behind `EVENT_INGRESS_MODE`, after a shadow soak that compares Visibility targets with registry targets. |
+| 3 | Drivers with durable acks (N1, N2, M3):<br>• registry and leases;<br>• Telegram offsets;<br>• the edgymeow gRPC protocol with outbox and `SynchronousAck` (cross-repo);<br>• Discord catch-up;<br>• polls as Schedules;<br>• send idempotency keys. |
+| 4 | Entities for new generations (`event_os_version=1`):<br>• supervisor, conversation entity, turn mode, merge points;<br>• chat as the entity's single writer;<br>• 4b: customer channels and per-thread histories. |
+| 5 | Scheduler and budgets:<br>• class queues, slot pool, Priority and fairness;<br>• admission;<br>• daily ledger, holds, pause, midnight resume. |
+| 6 | Timers and messaging:<br>• cron Schedules by immutable ids with `on_busy` (H6, M2);<br>• alarms and follow-ups;<br>• employee messaging;<br>• watched data;<br>• storm guards. |
+| 7 | Egress and approvals:<br>• outbox, buckets, `unknown`;<br>• ApprovalWorkflow;<br>• audience-scoped broadcasts. |
+| 8 | Supervision:<br>• restart intensity, quarantine, degraded mode;<br>• attention items;<br>• metrics, timeline. |
+| 9 | Instance tiers and fleet:<br>• 9a: pinned versions, the weekly-window upgrader with rollback, telemetry, staging, backups;<br>• 9b: PostgreSQL in hosted instances, Alembic, the advisory-lock replacement (N8, N10);<br>• 9c: tier L server, the S-to-L move, load tests. |
+| 10 | Retirement:<br>• R_k: new generations default to the new path;<br>• R_k+1: old generations are upgraded at boot (drain at the Stop checkpoint, then Reset and Start with the same snapshot and conversation store, with a notice);<br>• R_k+2: remove Visibility emit, the poll-coroutine registry and the legacy listener registrations once none is open. |
+
+**Critical path:** 0, then 1, 2, 4, 5, 6 and 10.
+
+**Parallel lanes:**
+- Phase 3 after Phase 2.
+- Controller hardening for old generations after Phase 1, serialized with any other controller edit:
+ - a size guard on child input (C1);
+ - poll loops re-raise cancellation (H8);
+ - a slim continue-as-new carry (H10).
+- Phases 7 and 8 after Phase 4.
+- 9a after Phase 1; 9b before the first hosted PostgreSQL instance; 9c after Phase 5.
+
+### 24.3 Patch markers
+
+A marker becomes `deprecate_patch` one release after no open history can lack it. That only becomes true through Phase 10's upgrade or a periodic rollover. It is removed one release later. No marker is ever deleted in one step.
+
+## 25. Defaults chosen and open items
+
+### 25.1 Defaults (change at review)
+
+- **B2 late merge** applies to customer channels only. In owner chat, a new message starts the next turn right away, so no streamed draft is ever replaced.
+- **Class floors** are 30/40/15/15, plus one reserved owner slot per employee.
+- **Webhook classes.** Signed provider webhooks are C3; generic webhooks and polls are C4; any trigger can override.
+- **Debounce windows, passivation times, overflow bounds and storm limits** are as given above.
+- **Retention:**
+ - ingress bodies: 7 days;
+ - deliveries and the timeline: 30 days;
+ - dead letters: 90 days;
+ - Temporal history: 7 days.
+- **Late alarms** fire with a lateness note; `skip_after` is optional per alarm.
+- **Messages that providers held during downtime** are delivered with a lateness note.
+- **Generic webhook dedupe** uses a configured header, otherwise a body hash within 10 minutes.
+- **The maintenance window** is Sunday 03:00 UTC; staging instances upgrade 3 days earlier.
+- **Tier L** uses 64 history shards, confirmed by its load test.
+
+### 25.2 Still open
+
+- **History encryption** with a payload codec on hosted instances (RFC-0004 Q13).
+- **Temporal retries for graph nodes** marked safe (RFC-0004 Q9).
+- **Startup verification of open runs**: refuse, or start and mark "needs Reset" (RFC-0004 Q15).
+- **When old generations are upgraded automatically** (§24.2, Phase 10), and what the owner is told.
+
+## 26. Acceptance gates
+
+| Gate | Check |
+|---|---|
+| EG1 | Under kill -9 at any step, no acknowledged event is lost (conformance suite on both drivers) |
+| EG2 | Turning a flag back off drains the inbox; nothing already appended is dropped |
+| EG3 | One user's socket never receives another user's content; webhook and provider payloads are never broadcast |
+| EG4 | A second chat run in one session succeeds on PostgreSQL |
+| EG5 | Five messages within the debounce window produce one turn; a message sent mid-turn appears in the next step's model input; tool batches are never split |
+| EG6 | A generation paused on the previous release resumes after every phase (pinned CLI, end to end) |
+| EG7 | 10,000 messages to one conversation cost fewer than 1,000 Signals per entity run, and its history stays under 10 MB |
+| EG8 | SQLite append p99 is under 25 ms at 50 events/s |
+| EG9 | Webhooks answer 401, 404, 405 and 413 as specified; 2xx only after commit; 503 when the append fails |
+| EG10 | Restarting the backend twice starts no second Start-node run (RFC-0004 AG17) |
+| EG11 | Budget: past 80% background inputs are held while chats run; at 100% the employee pauses with a notice; at midnight it resumes |
+| EG12 | An event for a generation that started 10 ms earlier is delivered (RFC-0004 AG12) |
+| EG13 | A Temporal database replaced by an empty one loses no acknowledged event; re-arm restores generations, Schedules and alarms |
+| EG14 | Tier S sustains 100 employees on the dev server; tier L sustains 1,000 employees at about 500 actions/s |
+| EG15 | The telemetry export contains no message text, prompts, customer data or credentials |
+| EG16 | A weekly-window upgrade with a forced readiness failure rolls the instance back by itself |
+| EG17 | Two customers' conversations never share an agent history |
+| EG18 | An employee-to-employee loop is cut at 8 hops, and a hot edge becomes a digest with an attention item |
+
+## 27. What this RFC does not change
+
+- The plugin model (`BaseNode`, the registries, NodeSpec as the source of truth).
+- The WebSocket protocol envelope and the CloudEvents `WorkflowEvent`.
+- The chat wire contract, except the additions in §11.
+- RFC-0002's store semantics, except the thread in its key.
+- Canvas Runs: `event_waiter` stays for manual Runs, and the relay notifies local waiters.
+- The cooperative pause design (Temporal's native Workflow Pause stays unused).
+- The decision to leave workflow execution and run timeouts unset.
+- The provider-neutral agent loop.
+
+## 28. Docs to update with each phase
+
+| Document | Change |
+|---|---|
+| `docs-internal/event_framework.md` | Inbox, router and relay replace Visibility routing; drivers; subscriptions; storm guards |
+| `docs-internal/chat_protocol.md` | Merged messages, no `run_in_progress` or `engine_unavailable`, `run.input_merged`, queued while Temporal is down |
+| `docs-internal/temporal-workflow-control.md` | Supervisor, entities, Start once per generation, re-arm and I10 |
+| `docs-internal/TEMPORAL_ARCHITECTURE.md` | New workflow types, class queues, slot pool, Priority and fairness, tiers |
+| `docs-internal/discord_service.md` | Public ingress prefix; append before the deferred reply |
+| `docs-internal/docker.md`, deploy runbooks | PostgreSQL per hosted instance, tiers, weekly window, telemetry, backups |
+| `CLAUDE.md` | The rows on the event framework, chat protocol, Temporal architecture and Docker |
+| `.env.template` | `WEBHOOK_MAX_BODY_BYTES`, `TELEGRAM_REPLAY_MAX_AGE_S`, `EVENT_INGRESS_MODE`, `EVENT_OS`, the telemetry endpoint, the maintenance window |
+
+## 29. References
+
+- **Temporal:**
+ - Python SDK guide: https://docs.temporal.io/develop/python (message passing, continue-as-new, child workflows, schedules, workers, testing).
+ - Task queue priority and fairness: https://docs.temporal.io/develop/task-queue-priority-fairness
+ - Design patterns (entity workflow, event accumulator, signal-with-start, approval, fairness): https://docs.temporal.io/design-patterns
+ - Limits: https://docs.temporal.io/workflow-execution/limits
+ - Multi-tenant patterns, the CLI server reference (https://docs.temporal.io/cli/command-reference/server), the upgrade guide (https://docs.temporal.io/self-hosted-guide/upgrade-server) and the production checklist (https://docs.temporal.io/self-hosted-guide/production-checklist).
+- **SDK source:**
+ - temporalio 1.30.0: `temporalio/worker/_tuning.py` (`CustomSlotSupplier`) and `temporalio/common.py` (`Priority`).
+- **Linux:**
+ - NAPI: https://docs.kernel.org/networking/napi.html
+ - EEVDF: https://docs.kernel.org/scheduler/sched-eevdf.html
+ - cgroup v2: https://docs.kernel.org/admin-guide/cgroup-v2.html
+ - RCU: https://docs.kernel.org/RCU/whatisRCU.html
+ - epoll(7): https://man7.org/linux/man-pages/man7/epoll.7.html
+ - systemd.service and systemd.timer (`Restart=`, `StartLimitBurst=`, `Persistent=`).
+- **Agent OS and actors:**
+ - AIOS (arXiv 2403.16971);
+ - Letta;
+ - Erlang/OTP design principles, supervisor behaviour;
+ - Microsoft Orleans (grains, reminders);
+ - Akka (mailboxes, passivation).
+- **Peer platforms:** Inngest (debounce, batch, throttle), Restate (virtual objects), Hatchet (concurrency strategies), LangGraph (double-texting), DBOS (queues).
+- **Drivers:** whatsmeow, https://pkg.go.dev/go.mau.fi/whatsmeow (`Client.SynchronousAck`).
+- **PostgreSQL:** "Postgres LISTEN/NOTIFY does not scale" (Recall.ai), https://recall.ai/blog/postgres-listen-notify-does-not-scale
+- **Related RFCs:** RFC-0002 (agent context and memory), RFC-0004 (Temporal SDK adoption).