Skip to content

AI-powered virtual executive team — a single coherent executive persona backed by 8 specialist agents (FastAPI + Next.js).

feat(workflows): let Python jobs run on a separate runner (#402)

Rufusd266972a9b4c
diff --git a/.env.example b/.env.example
index 6cbbb272..526767d5 100644
--- a/.env.example
+++ b/.env.example
@@ -435,6 +435,12 @@ GOOGLE_OAUTH_CLIENT_SECRET=
# PYTHON_SANDBOX_DIR=/opt/pysandbox
# PYTHON_JOB_TIMEOUT_S=120
# PYTHON_JOB_MEMORY_MB=1536
+# Run jobs on a separate runner instead of the local sandbox: each job is
+# POSTed there with the key as a bearer token, and the runner sends back the
+# result the local sandbox would. https only (plain http to localhost,
+# .internal or .flycast hosts). The runner must isolate jobs itself.
+# PYTHON_JOB_RUNNER_URL=
+# PYTHON_JOB_RUNNER_KEY=
# A script that works may be saved as a named tool and run again (run_script
# save_as / tool=, workflows/saved_tools.py; Settings -> Advanced -> Saved
# tools). It can only call what the chat or step running it may. false: no
diff --git a/packages/core/openexecutive/architecture/architecture-facts.yaml b/packages/core/openexecutive/architecture/architecture-facts.yaml
index 760673f8..cb73c70b 100644
--- a/packages/core/openexecutive/architecture/architecture-facts.yaml
+++ b/packages/core/openexecutive/architecture/architecture-facts.yaml
@@ -6123,6 +6123,17 @@ workflows:
and kills the process then; the exit code no longer counts once a result
parsed. _prune runs under a lock and treats a vanished folder as gone
(concurrent prunes raised FileNotFoundError and lost the reply's links).
+ Remote runner (PYTHON_JOB_RUNNER_URL + PYTHON_JOB_RUNNER_KEY): run_job
+ POSTs {code, inputs, files b64, max_*, timeout_s, memory_mb} with the key
+ as a bearer token and no redirects; the reply is read up to
+ _MAX_STDOUT_BYTES and only result/error/printed (length-capped), files,
+ skipped and usage {peak_mb, cpu_ms} are kept; HTTP errors, timeouts and
+ unreadable replies become job errors. available() is true with a runner
+ and no local sandbox. URL must be https, or http to localhost/.internal/
+ .flycast, and needs the key (settings refuse a URL without one). Result
+ files must be str -> valid base64 (b64decode validate=True) or the job
+ errors, so a bad runner reply can't raise in the handler. The runner is
+ trusted to isolate jobs.
Kept jobs, attachments, metering: run_python_job takes save_as +
description (kept as saved_tools kind 'python', tools [], code as the
script; only when the job worked) and tool + inputs to run one again
diff --git a/packages/core/openexecutive/architecture/prebuilt/workflows.json b/packages/core/openexecutive/architecture/prebuilt/workflows.json
index 82c3b6bc..ca7c3342 100644
--- a/packages/core/openexecutive/architecture/prebuilt/workflows.json
+++ b/packages/core/openexecutive/architecture/prebuilt/workflows.json
@@ -1,7 +1,7 @@
{
"section_id": "workflows",
"title": "Workflows",
- "markdown": "Workflows are deterministic, multi-step orchestrations. The built-ins are registered in `openexecutive.workflows.WORKFLOW_REGISTRY`; user-created (dynamic) workflows are stored as data (see below). Each follows the same contract: structured input feeds an ordered list of named steps that render a Markdown artifact, persisted via `complete_run` into `workflow_runs.artifact`. Every step emits a `WorkflowEvent` (start/progress/complete) streamed over SSE at `/workflows/{name}/runs`.\n\nThe registry holds 25 built-in workflows, plus any number of user-created ones. Representative built-ins:\n\n- `board_prep`, `quarterly_plan`, `annual_plan` — planning and governance deliverables.\n- `performance_review`, `comp_refresh`, `org_design`, `exec_search_brief` — people and org artifacts (the last is an advisory hiring brief / scorecard from the CHRO, with input from the hired function's specialist — `sales` for a sales role — kept when the recruiting pipeline was removed).\n- `crisis_comms`, `churn_deep_dive`, `competitive_teardown` — situational analyses.\n- `department_check_in` — runs on department cadence; its `goal_status` step returns structured verdicts persisted back to `department_goals.status` (`department_check_in.persist_goal_verdicts`, shared with `weekly_review`).\n- `weekly_review` — the principal's weekly look back and ahead (see *Weekly review* below).\n\n**Standing facts in unattended output.** `end_of_day_digest`, the weekly review's top-three call and the morning reflection append the standing facts (`memory.facts.with_standing_facts`; a teammate's marked `(per <name>)`) to their user turn, and the morning brief gets them through `render_briefing_context`. A workflow step that consults a specialist gets them through `route_to_specialist`. A correction made in chat therefore holds in every recurring brief and run. The morning brief also reads `facts.render_teammate_changes(since)` off the event loop and passes it to both `render_briefing_context` (a TEAMMATE CORRECTIONS SINCE LAST BRIEF block) and `build_brief_fingerprint`, so the principal hears of each correction a teammate made in the window, and that alone un-suppresses an otherwise unchanged brief. Proposals waiting for approval are listed only in a brief the principal reads privately (`include_proposed=_private_ok()`), and then mark the run private to them. The principal's own Always in the loop notes (`memory.history_brief`, only with their switch on) come in by a stricter rule, only on the scheduler's delivery (`PRINCIPAL_DELIVERY`), never a run from chat, whose tool result reaches the turn's shared audit row and peer memory: the morning brief's FROM YOUR NOTES — DUE SOON block and the end-of-day digest's FROM YOUR NOTES — TODAY block (what they committed to since the last digest, then what is due tomorrow). Either marks the run `private_to_principal` and moves its fingerprint; the digest, which had no private reads before, now emits that flag too.\n\n### Change-aware principal briefs\n\n`morning_brief` and `end_of_day_digest` are the two recurring briefs the scheduler sends to the principal. Both are now a *delta*, not a re-list: the window is \"since the last **delivered** brief\" (`briefing.brief_state.since_for`; 24 h on a cold store, clamped to 7 days), activity comes from `_build_activity(limit, since=…)`, proposals (action category only) split into *new since last brief* — enumerated, with each item's why-now and next move — versus a one-line carried-over count, a *rewritten* block lists carried items the review rewrote inside the window (`brief_state.rewritten_since`: verdict `changed`, still open — the brief reports them under \"what changed\", never as done), and a *handled overnight* block lists what the Executive's alert review **completed** on its own (`brief_state.handled_since`; the `changed` verdict is excluded because the alert is still open). The morning brief also reads the principal's **live world** since that window opened (`briefing.live_signals.gather_live_signals`): mail and chat the Executive handled (INBOUND, grouped by sender and subject), what got stuck (STUCK: a reply the outbound gate held, a non-roster sender who got no reply, a Google Chat message left unanswered, drafts waiting in the principal's Gmail), the principal's own chats, and — in both modes now, unless the solo top three already listed it — today's calendar, read fresh (`live_signals.refresh_calendar(max_age=0)`). The 07:30 reflection's **Flagged for the brief** bullets are carried across (`brief_state.reflection_flags_since`, the latest reflection run finished inside the window) instead of surviving only as an activity line with the run's title, and the rhythm runs themselves (`today.RHYTHM_WORKFLOWS`) are dropped from the brief's activity: their dated titles were noise and moved the fingerprint every day. At-risk departments name their slipping goals and each proposal carries its age. The period is the principal's local date. Section order: top call → **today** (the calendar's shape and anything stuck) → what changed (inbound first, naming who wrote) → handled overnight → needs you (new only) → waiting on → at risk (the slipping goal, or the department left out — never \"no detail yet\"). Quoted mail, chat, invite and reflection text is marked as data, never instructions. **Privacy:** the brief reads what is private to the principal — their contacts' alerts and mail, their drafts, their chats and their calendar — only on a run for them: the scheduler's (`morning_brief.PRINCIPAL_DELIVERY`) or the principal's own verified chat turn in a conversation only they can read (`delegation.settings.private_conversation`: the web chat, a Slack or Discord DM, a private Telegram chat — never a shared channel, where the reply is posted for everyone). A run anyone else starts (a teammate from chat in team mode, `POST /workflows/…/runs`) reads what everyone may see, as before. A run that did read a private row says so on its `result` event (`private_to_principal`: its live read differs from the shared one in any key or count), and the scheduler, `run_workflow` and `POST /workflows/{name}/runs` all then keep `persistence.PRIVATE_RUN_ARTIFACT` in the run history instead of the text. Text another pass wrote (the reflection's flags) renders as quoted ` > ` lines and every goal and headline field is one-lined, so none can start a line that reads as a block header. The fingerprint carries the live signals' time-free keys (who wrote about what, what is stuck, the calendar's shape) and the reflection flags when there are any, so \"nothing new\" means the principal's world was quiet too; the context summary is logged at INFO (`morning_brief: context since=… inbound=… stuck=… suppressed=…`). In **solo mode** (one person using Open Executive for themselves) both briefs speak to the principal, whatever their role: the morning brief runs top call → top three today → what changed (inbound and stuck first) → handled overnight → needs you → due this week (the principal's open loops overdue or due within 7 days, from `open_loops.principal_due_soon`) → goals at risk (named by area), the end-of-day digest what I did → open decisions → goals at risk tomorrow → sleep on this, over a context with goals at risk by area and no people-waiting or department block; the fingerprint carries the mode and ignores who is awaiting, and carries the due items as `(loop_id, state)` pairs when there are any. **Top three today** is picked in code, not by the model (`briefing.top_three`: overdue then due-today commitments, off-track then at-risk goals, commitments due later this week, then active projects, stalest first). The solo Briefing's *Top three today* card shows the same pick from the same code (`GET /today/top-three`, see Today). When the MCP gateway runs the Google Workspace server, one `google_workspace__get_events` call (4 s timeout) lists today's events on the principal's calendar — their address as the calendar id, read with the Executive's account (`primary` when the Executive is signed in as them) — and on a business day (`calendar_tools.is_business_day`, the rule the booking tools apply; at the weekend the day's events are still listed but no slot is suggested) each item gets its own free block inside `CALENDAR_BUSINESS_HOURS_START`–`_END`, rendered with a TODAY'S CALENDAR block — only a date-only event is all day, a zero-length one blocks nothing, and one whose times can't be read is left out; no gateway, no principal address, an error, a timeout or a reply it cannot read all mean no calendar and no slots, never a failed brief; the log records only the reply's length or the error's type, never calendar text. The fingerprint carries the three items' keys (no dates, no slot times) and only a coarse hash of the day (busy blocks rounded to 15 minutes, no titles, no dates), so an unchanged day still suppresses. A day whose calendar read fails after one that worked (or the reverse) changes the fingerprint once, so that brief is written in full. A date-free fingerprint of the inputs is compared to the last delivered one; if unchanged the artifact is the one-liner \"Nothing new since yesterday's brief — N items still waiting on you.\" and no model call is made (`PRINCIPAL_BRIEF_SUPPRESS_UNCHANGED`; `force_full=true` on a manual run bypasses it). The workflow yields a `result` event with the fingerprint; only the scheduler records it, after a successful delivery, in `briefing_narrative` under scope `brief:<kind>` — so manual runs never advance the window.\n\n**Brief delivery.** `scheduler.runner._deliver_to_principal` maps the principal's `preferred_channel` (`slack` / `discord` / `telegram` / `email`) to its delivery channel and tries it first, then the other chat channels the principal has an id for, in the order Slack → Discord → Telegram. Email goes through the active MCP gateway (`google_workspace__send_gmail_message` as the Executive's own address, to the principal, subject `<title> — <date>`) and is used only for an explicit `email` preference, or for `any` when no chat channel resolves — an owner linked by email at setup has `any`, so one who also has Slack does not start getting two emails a day. When nothing can deliver (a web-only principal, or email with no gateway), the brief is still generated and stored as a run with its artifact, as before — a web-only principal reads it on the Artifacts page — it is just not delivered (audited), so the window does not advance, and the next occurrence is chained. The same delivery path carries the client-rotation digest and the weekly review.\n\n**Grounding.** Nothing an unattended pass writes is read by anyone before it goes, so `openexecutive.briefing.grounding` checks it deterministically — no model judges the model. A name (a run of two or more capitalised words once section headers, weekdays, imperative openers and org nouns are trimmed, or one capitalised word in a person slot like \"ask Marcus\" or \"Marcus wrote\") is grounded only when a single line of the pass's inputs holds all of it, or it belongs to a roster person some input mentions — being on the roster alone is not enough. A figure (money, a percentage, a k/M/B number, any bare number of 11 or more; never a date, time, year, duration, id, list marker, `Q3` or small count) is grounded when an input number matches it within the precision it is written with (`$1.2M` matches `$1,234,567`; `47 units` does not match 48). The inputs are the rendered context split into lines labelled by section (a header also carries its bullet count), the company profile and the department titles. `morning_brief` renders its context itself and hands it to the synthesizer as `rendered_context`, so the check reads exactly what the model read; after the quiet fallback, `ground_brief` runs on both briefs. When something is ungrounded it makes one repair call to the fast model (same system prompt, the list of items, \"remove or correct using only the context\"), re-checks, then drops each line still ungrounded and any header left empty, and appends \"_Held back N lines that named people or figures I couldn't match to your data — /today has the full list._\" (everything ungrounded → one held-back line). Then each grounded figure gets a server-chosen ` [n]` — the matching input line with the most words in common with the figure's clause — and a `**Sources**` list closes the brief (`- [n] <section> — <snippet>`, at most 9, snippets quote inbound mail, so links are masked (\"(link removed)\"), Markdown, angle brackets and @-mentions stripped, and they are cut to 90 chars; never `[n]:` at a line start, which Markdown reads as a link definition). Text with nothing to change comes back byte-identical, the fingerprint hashes the inputs so suppression is unaffected, and the grounded text is what is delivered, stored and recorded as delivered. Both brief prompts and the digest prompts say the rule up front. `executive_reflection` wraps its handlers in a `GroundingScope`: `create_alert`, `message_person`, `send_department_message`, `send_company_broadcast` and `schedule_followup` refuse text with an ungrounded name or figure, returning a JSON error that names the items so the model can look the person up or leave the figure out (every tool result joins the inputs, so a `lookup_person` answer grounds the DM after it; the recipient's own name always counts). Its **Flagged for the brief** bullets are filtered the same way before the tool log is appended — the next brief quotes them, where they would ground themselves — and `brief_state.reflection_flags_since` now stops at the tool log's `---`. The reflection's `grounding` rows are private when a private-to-principal alert reached its context, and the digest's always are (its HANDLED block can quote one). Everything fails open. `GROUNDING_CHECKS=enforce` (default) acts, `report` only writes the `grounding` audit rows, `off` skips it; `GROUNDING_CITATIONS=false` drops the markers. Not yet grounded: the /today header (its cards sit beside it), `weekly_review`, the research synthesis, dates and meetings, and a real person named in the wrong role.\n\n### Weekly review (solo)\n\n`weekly_review` (background, inputs all defaulted) closes out the principal's week in six steps. **Goals**: every area with goals is graded by its own specialist, concurrently, with the department check-in's verdict contract (`_parse_verdicts`) and write-back (`persist_goal_verdicts` → `record_goal_review`, one `goal_status_review` audit row per area with actor `weekly_review`); an area with no goals is skipped, one without a specialist is listed ungraded, and one whose specialist fails is shown as last recorded. **Commitments**: `open_loops.principal_due_soon` — overdue or due within 7 days. **Projects**: active initiatives with no update for 7+ days. **Decisions**: the week's (`episodic.decisions_since`). **Revisit**: up to three decisions older than 30 days with an empty outcome (`episodic.decisions_awaiting_outcome`, newest first), listed as `[decision N]` under *How did these turn out?* — the principal's answer is recorded with `record_decision_outcome` (see Agents). **Assemble**: every section always renders (a one-line note when empty), ending with *Next week's top 3* from one fast-model call that follows the `weekly-review` playbook (a constant system prompt; the playbook rides in the user turn), trimmed to at most three list items — or, when that call fails or returns nothing, the same ranked pick the morning brief uses. In solo mode the scheduler fires it weekly (`principal_weekly_review`, see Scheduler) and delivers it through `_deliver_to_principal` like the briefs, recording the delivery outcome. The solo Briefing also shows the latest completed run to the principal (`GET /today/weekly-review`, see Today): `weekly_review.summarize_review` reads the artifact's title line (`TITLE_PREFIX`) and its last `TOP_THREE_HEADING` section back as plain text, so a principal no channel reaches still sees next week's top three. Wording is role-neutral; run by hand in team mode it says *department* instead of *area*.\n\n**Principal-only workflows.** A workflow can name the workspace modes in which only the principal may start it (`Workflow.principal_only_modes`): `weekly_review` in both modes (built from the principal's decision log, commitments and goals, and it re-grades the goals) and `morning_brief` in solo (built from the principal's commitments and calendar). Team `morning_brief` and every other workflow can be started by anyone as before. This decides who may start or trigger a run, not who may read one: stored runs, the scheduled briefs and weekly reviews included, are read under the existing `/workflows/runs` rules — which is why a morning brief that drew on the principal's private rows is stored as `PRIVATE_RUN_ARTIFACT`, not as its text.\n\n- **From chat**, `run_workflow` checks at dispatch, before any run row is created, that the turn is the principal on a verified surface — `create_goal`'s rule — and otherwise refuses with a plain error that does not spell out the rule.\n- **Over HTTP**, the Jobs page's `POST /workflows/{name}/runs` and `POST /evals/runs` (a workflow scenario runs the workflow on live data) call `refuse_principal_only_run`: only a caller signed in as the principal, or a request with no caller email (the CLI or local login), may start the run. Unlike the workspace settings, an install with no principal on the roster lets no signed-in caller in, as in chat. Anyone else gets a 403 (\"Only the principal can run this workflow.\") before any run row exists. The mode is the stored one, or an eval scenario's own `workspace_mode`. The eval route runs exactly the scenario list it checked.\n\nEvery refusal is audited `refused=true` with the workflow, the mode and the caller. The checks fail closed: a mode that cannot be read counts as principal-only, and a check that cannot read the roster refuses.\n\n**Knowledge in the reflection.** A chat turn looks the knowledge base up before the Executive answers; the reflection reads the org's state instead, so nothing looked it up for it. It now carries its own `search_knowledge` tool (`orchestrator.knowledge_tools`, not in the shared `_ALL_SKILL_TOOLS`), which runs the same `knowledge.retriever.retrieve` as chat — the review gate, the synced Drive / OneDrive / Confluence / Notion collections and the published-artifact rule (the pass has no viewer, so no one's private document comes back). The prompt (`_KNOWLEDGE_RULE`, both modes) asks it to search before acting on or flagging something that turns on a customer's or supplier's terms, a price, a policy, a plan or a past decision. Its passages come back up to 4,000 characters (`execute_tool_calls(wide_results=...)`) while other tools stay at their short preview, its result counts as grounding input like any tool result, and the run's tool log keeps each line to 160 characters. Read-only, so Take the lead never gates it. The loop has 5 iterations (was 4) to leave room for a search before acting.\n\nNeither the nightly reflection nor the research synthesis can call `ack_alert`: both run unattended with open alerts rendered into their context, and those headlines are minted from inbound mail and chat, so an injected \"the principal already dismissed 13\" would be acted on with nobody watching. Closing an alert is the alert review's job, with evidence.\n\n`executive_reflection` now carries memory: yesterday's standup (the previous completed run's artifact) is rendered into the context and the prompt forbids re-acting on anything listed there unless it changed; open alerts render with their id, review verdict and recommended move. Reflection never closes alerts — the alert review does, with evidence. It also renders an **OPEN LOOPS** block (up to 10, soonest due first, overdue flagged) from Attunement so the standup can connect a slipped promise to the rest of the org state; the nudge engine already chases overdue loops, so the prompt tells it not to DM about a loop merely for being overdue. A **WHAT LANDS** block (from the Attunement outcome ledger) shows, per person, how often each kind of proactive DM got answered over 30 days; the prompt steers DMs toward kinds that land and away from kinds a person reliably ignores. Reflection and research tag their tool execution as proactive (`tag_proactive`) so their DMs are counted. Resolving a wait-for-human approval (`resumer.apply_resolution`) marks the nudges that chased it `acted`. `close_open_loop` is withheld from reflection for the same reason as `ack_alert`: loop text is quoted from what people wrote. `unattended_toolkit` also drops `schedule_tools.UNATTENDED_WITHHELD_TOOLS` (today `create_goal`) from both passes in both modes: a goal is the principal's own decision, and these passes see only inbound text. A turn private to the principal (mail from one of their contacts, or mail they forwarded) cannot start a workflow either: `run_workflow` and `run_executive_research` are in `schedule_tools.PRIVATE_TURN_WITHHELD_TOOLS`, which the chat loop neither offers nor runs on such a turn, since a workflow's steps can reach anyone.\n\nIn **solo mode** the reflection is framed for the principal, whatever their role: they are the only person who uses Open Executive and the only one it takes direction from, it prefers `create_alert` and `schedule_followup` to the principal, and it never messages anyone else from an unattended pass — the people in the principal's world are contacts, so anything they should hear is flagged for the principal to raise. The team roster is replaced by one YOUR PRINCIPAL line (person_id and follow-up channels), its follow-up guard is seeded from the principal only, and department risk renders as areas with goals at risk. The research synthesis follows the same rule — its routing menu and roster name only the principal. In both, `send_department_message`, `send_company_broadcast` and `set_department_head` are withheld, and so are `create_calendar_event`, `create_instant_meeting` and `run_workflow` — a booking reaches its attendees and a workflow can do anything its steps do, so the principal books and starts workflows from chat — and `message_person` refuses anyone but the principal. These passes run unattended with inbound text in their context, so the rules are enforced in code, not only in the prompt.\n\nIn either mode, both passes now dispatch only the tools they offer: the handler map is built from the offered list, so a name the model emits without being offered it is skipped as an unknown tool. They used to look names up in the full handler registry, so tools they deliberately withhold — the raw DM tools (`send_slack_dm` has no roster check), `ack_alert`, `close_open_loop`, and in research the watchlist writes and `run_workflow` — still ran when the model emitted them. Both read the mode before binding their own session (so a caller's override, such as an eval's, counts) and pin it on that session, so their tool handlers use the same mode. They carry the principal's role across the same way (`effective_principal_role` of the caller's session, pinned as `turn_principal_role`), so a specialist or workflow started from either pass sees the role of the turn that started it. Team runs are unchanged.\n\n### Wait-for-human pause\n\n`wait_for_human` (`openexecutive/workflows/wait_for_human.py`) is a first-class step primitive that pauses a run: the runner calls `save_checkpoint`, the `workflow_runs` row moves to `status='awaiting_human'`, and the serialized `WaitForHumanEvent` is stored in `state_json`. (Department authority is a *separate* mechanism — `propose_only` departments file a briefing alert via `departments.authority.propose_via_alert` and mark the scheduled action done; they do not route through this gate.)\n\n**Delivery.** Both checkpoint sites — the SSE route and the `run_workflow` chat tool — call `workflows.gate_delivery.deliver_gate_question` immediately *before* checkpointing, so `state_json` carries the routing fields a reply is matched against. Two modes. When the approver is the person already in the conversation (the chat session's `origin_channel` is set — to anything but `email`, whose answer is no chat reply and would never be sent — and its `caller_person_id` is the gate's `person_id`), nothing is sent — the question rides in the Executive's own reply — and the gate records `origin_session_id` plus the session's channel. Otherwise the question is delivered through `schedule_tools.handle_message_person`, which picks the person's preferred channel, falls through on a failed send, and falls back to a briefing alert; the gate then records the channel, address and provider message id that send returned, plus the id of the DM session it landed in — that send is always 1:1, so the answer belongs in that conversation and nowhere else. Without it the gate matched on person and channel alone, so an unrelated @mention in a public channel resolved it and had the acknowledgement, run title included, posted where everyone could read it. Delivery is not guaranteed: the outbound guard can suppress it (duplicate, rate cap, quiet hours) and an unreachable person gets the alert fallback. Both cases leave the routing fields empty and report `delivery` as `suppressed` / `alerted` rather than claiming someone was asked, and the chat tool's presentation hint changes accordingly.\n\n**Matching a reply.** Each inbound adapter passes the message to `inbound_resolver.resolve_inbound_message` with its `session_id`. Matching is three-tier: an explicit `in_reply_to` against a stored `outbound_message_id`; otherwise a single open run on that channel; otherwise one utility-fast disambiguation call gated at 0.85 confidence. An unmatched explicit reference rules out only the candidates that carry a stored outbound id — the ones it could have matched — and the rest continue; rejecting outright meant an adapter that passes a thread id on every message (Slack's `thread_ts`) short-circuited the later tiers forever, and a global test let one delivered gate's id block an unrelated self-approval gate. Channel names are normalized (`slack_dm` ≡ `slack`), the delivered address is compared when both sides know it, a row whose `state_json` will not parse is refused (a corrupt row otherwise presents as a legacy one), and an empty stored channel is treated as a wildcard only for rows checkpointed before gate delivery existed. Two guards keep an open gate from swallowing unrelated traffic: a gate carrying an `origin_session_id` matches replies only from the conversations the adapter names (a Slack or Discord thread offers its parent channel session too, mirroring the history double-write, since a gate raised on a mention recorded that session while the answer arrives under the thread's — on Slack only when that person rooted the thread, or the alias would reach into a colleague's), and the parser can return `unrelated`, which declines to resolve and lets the message reach the normal chat path. The parser is handed the gate's own question, so a bare \"yes\" is judged against something. Its fallback is flagged by a key the model cannot emit rather than by a magic note string it could, and a fallback declines to resolve on *every* reply shape — recording the empty string a `free_text` fallback fabricates would close a sign-off on a parser outage. Every shape can answer `unrelated`, not just `approve_reject`: without it an open `free_text` gate turned the person's next message into its answer verbatim, and a `numeric` one swallowed any message containing a number. The parser has one caller outside workflows: its `roster_identity` shape — `{decision: approve|link|decline|unrelated, name, kind: team|contact|null}`, with `name` only as the person wrote it (never inferred) and a fallback of `unrelated` — reads the principal's emailed answer to a roster request (who a new sender is; see Org). It is not a gate reply shape, and a fallback or `unrelated` leaves the request waiting. `resumer.apply_resolution` then records the decision, marks the run `resolved`, and writes an audit row. A timeout applies the step's `on_timeout` policy (escalate / auto_proceed / fail). The outcome surfaces on the Today dashboard.\n\n**Resuming past the gate.** A paused run continues on its own. Re-entering it was long assumed to mean serializing a Python generator frame; it does not. The only state alive across a dynamic workflow's steps is its inputs (already persisted), a company-profile block that is recomputed, the completed steps' output strings, and the loop cursor — so the engine hands the checkpoint a small JSON `WorkflowResumeState` (workflow name, the gate's id and index, the next index, the outputs so far), stored in `workflow_runs.resume_state_json`, and `DynamicWorkflow.resume` replays its step loop from the recorded index. `run()` and `resume()` share that one loop, which is why a second gate needs no special handling — it is simply another pause raised from the same place. The payload rides on a `WaitForHumanEvent` field declared `exclude=True`, so it can never leak into `state_json`, whose exact key set the resolver branches on. A gate with no payload is *pause-only*: the decision is recorded and the run stops, which is what every caller got before resume existed.\n\n`resumer._process_resumable` claims a resolved run with one guarded UPDATE — `resolved` → `running`, stamping `resumed_at` and minting a claim token — and drives it to a terminal row. It runs two ways: immediately, via a task `apply_resolution` kicks off so a Slack approval starts moving in seconds, and every 60s from the poll loop as the durable backstop. Both share the claim, so a run executes exactly once even with two resumers against one database. The kicked task binds no chat session (`set_session(None)`), exactly like the poll loop: it is created inside the inbound handler that recorded the answer and would otherwise inherit that person's live turn — so a run resumed because the principal replied on Slack does not act with the principal's identity (it cannot, for example, reach the principal's contacts). It drops the handler's audit scope too (`audit.context.unscoped_audit_rows`): inherited, the principal's scope would make private every row of the run that names a contact, and whoever started the run would see those rows vanish after the approval. The poll runs the executor *after* the timeout loop, which is what finally makes `on_timeout='auto_proceed'` do what its name says: the timeout resolves the run and the same tick executes it. While the Executive is paused (`scheduler.pause`) the resumer does nothing — no startup sweep, no timeouts, no resumes — and `apply_resolution` still records the decision but skips the immediate kick; the first unpaused iteration catches up on all of it, so a timeout that lapsed during the pause applies its `on_timeout` policy then.\n\n**The claim is fenced, because it is not a lease.** The stale sweep decides a worker is dead from a wall-clock guess, so a slow but *alive* worker will eventually be declared dead and replaced. Every write the executor makes carries its claim token and is refused once superseded — otherwise the replaced worker's late completion lands on top of its replacement's result, or its checkpoint at a later gate resurrects an already-finished run and asks the approver a second time. The retry budget resets at each new gate (a gate is new work, not a retry), and a run that exhausts it is *ended* rather than dropped: an exhausted run sits in `running` with a payload, in neither queue, and would otherwise show as in-progress forever.\n\n**The gate fails closed.** Continuing is an allowlist — `approve` and the synthetic `auto_proceed` — not \"anything that is not a stop word\". `decision` is whatever a fast model extracted from free-form chat and it drifts; under a denylist, \"rejected\", \"Reject\", \"decline\" and \"no\" would all continue, so the approver declines and the deliverable is produced anyway. Reply shapes that ask a question rather than seek permission always continue, since the answer *is* the value. A stop ends the run at `error` with the human's note. Either way the decision becomes the gate step's own output, so it reads as a section *in* the artifact — capped and blockquoted, because that text reaches the synthesis specialist as prompt context.\n\n**A definition edited during the pause is refused.** Resume re-validates the payload's version, the recorded gate, *and* a fingerprint of the whole step list, and derives the index to continue at rather than reading it from the payload. Pinning only the gate would not be enough: definitions are overwritten by name and any rostered user can save one, so someone could keep the gate identical and replace every step after it — and the approver's sign-off would be recorded against work they never saw, which defeats the one thing an approval gate is for. No built-in workflow yields the gate; only user-created dynamic workflows do.\n\n### Launching from chat\n\nWorkflows are not only reachable from the `/jobs` UI — the Executive can list and run any of them mid-conversation. `openexecutive.orchestrator.workflow_run_tools` wires two tools into the main chat loop: `list_workflows` (the launchable catalog, with each workflow's input fields) and `run_workflow` (runs one to completion and returns its Markdown artifact, using the same engine as `POST /workflows/{name}/runs`). `run_workflow` resolves both built-ins and custom workflows via `get_workflow`, validates `inputs` against the workflow's input model, persists the run (`create_run` → `complete_run` / `fail_run`), and on an approval-gate step checkpoints the run as `awaiting_human` exactly as the HTTP route does — literally the same code, since both call `workflows.gate.checkpoint_gate`, which delivers the question and writes the checkpoint in one place. It never hangs waiting on a human, and it reports both whether the question reached them and whether the run will continue by itself once answered. The runners that *cannot* pause — the schedulers, evals, client rotation, onboarding, the research tools — call `gate.ensure_workflow_event`, which raises on a gate rather than letting one be swallowed; the scheduler's dynamic-workflow branch used to ignore it and record a successful run containing nothing. `executive_research` is excluded (it has the dedicated `run_executive_research` tool). Running a workflow from chat delivers nothing on its own — it renders the artifact into the conversation, and the recurring principal briefs (`morning_brief`, `end_of_day_digest`) reach anyone only via the scheduler's own `_deliver_to_principal`. The tool description used to claim those two DM the principal when run and to confirm before firing them, which had the model soliciting a confirmation on a false premise; it now names `executive_reflection` as the one workflow that really does execute tool calls of its own (DMs to heads, broadcasts, alerts). A successful run surfaces an `action_taken` chip linking to `/jobs/runs/{run_id}`.\n\n### Dynamic (user-created) workflows\n\nUsers create reusable workflows on the fly — from the executive chat, the conversational wizard at `/jobs/new`, or the step-by-step advanced editor — without shipping code. The Workflows page (`/jobs`) opens on a *Start here* panel that leads to all three: the user describes the job in a sentence and *Start* opens the wizard with that text sent as its first message (handed over in `sessionStorage`, so a link cannot make the wizard send text the user never typed; where storage is blocked it falls back to `/jobs/new?describe=`, which only fills the composer and sends nothing), or picks *Step by step* (advanced editor) or *Talk it through in chat* (`/?new=1&draft=`). Below it come the user's own workflows, then four ready-made starter picks with the full built-in list folded behind *Browse all* (`?browse=1`). A dynamic workflow is a stored JSON definition (the `dynamic_workflows` table) interpreted at runtime by one generic `DynamicWorkflow` engine (`openexecutive/workflows/dynamic.py`) that implements the same `Workflow` ABC as the built-ins, so the run path, SSE streaming, run persistence, and `/jobs/[name]` form rendering are unchanged. `get_workflow` / `list_workflows` fall through to the dynamic store after the static registry (built-ins win on a name collision).\n\nA definition declares free-text input fields and an ordered step list of four kinds: **specialist** (consult a specialist with a goal prompt, optionally grounded by RAG), **action** (get something done with tools — see below), **approval_gate** (a `wait_for_human` pause), and a terminal **synthesis** step that assembles the artifact. At least one specialist or action step is required. `validate_definition` enforces the structural rules: exactly one terminal synthesis step, valid specialists, `{placeholder}` tokens that reference declared input fields, rostered approval/cadence people (team members — never one of the principal's contacts, whom an unattended run may not message), and a parseable cadence.\n\nAuthoring from chat uses a **draft → confirm** handshake: the Executive calls `draft_workflow` (validate; return a `confirm_token` plus a plan summary), shows the user the plan, then calls `save_workflow` with the same definition and token — a token mismatch is rejected, so the saved workflow is exactly what the user approved. The step-by-step advanced editor drives the `/workflows/custom` CRUD endpoints directly.\n\n**The New workflow wizard** (`/jobs/new`) is the default way to create one from the UI. It works like conversational onboarding: the user describes the job, the `workflow_designer` Council agent (`openexecutive/workflows/designer.py`) asks one clarifying question per turn — deliverable, per-run inputs, sign-off, schedule, recipient — with up to four clickable suggested answers, then emits a complete definition plus a plain-language summary and the assumptions it made. Every turn offers the model exactly three fixed, name-sorted tools (`ask_clarifying_question`, `emit_workflow_draft` — reusing the chat tool's `DEFINITION_SCHEMA` — and `search_available_tools`, which is answered inside the same turn from the tool catalog, up to four searches per turn, with the tool_use/tool_result pairs never stored in the plain-text transcript; every tool a search returns is remembered on the session and replayed into later turns, so a forced draft or a repair turn — which cannot search — still has the exact names); `tool_choice` flips to the emit tool once, when the six-question budget is spent or the user clicks *Draft it now*. The emitted definition goes through `validate_definition`, the tool-availability check, and a check against saved custom names, with one repair turn that feeds the errors back. The system prompt is a cached constant; the live context — specialists, the roster with person ids, the user's timezone (the one set through `PUT /workspace`, else `USER_TIMEZONE`; cadences stay UTC), and taken names — is rendered once per session into the first user turn, and while refining, the exact current draft is prepended to the latest user turn so the model edits it rather than re-deriving it. Roster names and roles are flattened to one line and capped at 80 characters before they enter that context, and the system prompt tells the model to treat the whole block as data, never instructions. Sessions live in memory (`/workflows/designer/*`, 2 h TTL, 50-session cap), run one model turn at a time (a second concurrent `/message` or `/draft` gets 409, and a failed turn is rolled back by identity), and never write: the review card saves through the same `POST /workflows/custom`, which re-validates, and *Edit details* opens the draft in the advanced editor (`/jobs/new?designer=<id>`) before saving.\n\n**Editing is a conversation too.** *Edit* on a saved workflow (`/jobs/new?edit=<name>`) opens the same designer through `POST /workflows/designer/edit`: the stored definition is the starting draft, so the user's first message (\"have Mark sign off and run it Fridays\") is applied to it like any refinement, and a note in the context block (a user turn, never the cached system block) tells the model it is changing a saved workflow. The draft always keeps the workflow's name — `_check_draft` puts it back and skips the taken-name check — and the route keeps its on/off state, so an edit can neither fork a copy nor switch on a tool workflow waiting for approval. The stored draft is handed to the model marked as data, not instructions. The review card lists *What changes* in plain words against the saved version (`lib/workflowChanges.ts`): every field is compared, with named sentences for the sign-off person and what it asks for — a sign-off moved off approve/reject is called out because the run then continues whatever the answer — the schedule, steps added/removed/reordered, inputs, tools and the tool-call budget, and a catch-all line for any other field. It names only the newly added tools that may change things. *Save changes* calls `POST /workflows/custom/{name}/save-edit`, which saves the session's own draft after re-validating it, only when it is the draft the card showed (the card sends what it displayed; a message that landed since is a 409), and only if the stored workflow has not changed since the conversation opened (409 otherwise), so an edit started before someone switched the workflow off cannot switch it back on. The step-by-step form stays as the advanced editor (`/jobs/new?edit=<name>&mode=advanced`).\n\n**Action steps (workflows that act).** An `action` step (`ActionStepSpec`: `goal`, `tools`, `max_tool_calls` 1–50, default 20) is run by the `workflow_actor` Council agent (`openexecutive/workflows/action_step.py`) as a bounded tool-use loop. Its `tools` list is an allowlist of exact tool names — MCP gateway tools (`server__tool`, e.g. `google_workspace__append_table_rows`) and a small set of built-ins (`oe__read_file`, `oe__message_person`, `oe__create_alert`) — and **clicking Create is the user's approval** of that list: the review card shows every tool, flags the ones that may change things (anything whose name doesn't say it reads, and any tool with a URL/URI/endpoint/webhook parameter anywhere in its schema, or whose schema couldn't be read, since a URL can carry data out), and names them in a \"this workflow can use these without asking again\" notice. Because that card is the approval, a workflow with action steps saved from chat (`draft_workflow`/`save_workflow`, whose confirmation is a token the model holds itself) is stored **switched off** (`is_active=false`) and `save_workflow` returns `status: saved_pending_review` with a `/jobs/{name}` link. An inactive workflow can't run or schedule (`get_workflow` and the scheduler skip it); `/jobs/{name}` shows it on the same review card, where **Turn on workflow** (`POST /workflows/custom/{name}/activate`, which re-validates the tools) is the approval — the click sends the definition the card showed, and the server refuses (409) if the stored one has changed since, so chat overwriting it while the card is open can't get new tools switched on. Chat may revise a switched-off workflow but refuses to replace a switched-on one that has action steps (an inbound email steering the Executive could otherwise switch off an approved job and stage new tools under its name); those edits happen in the advanced editor. The Workflows catalog (`/jobs`) lists switched-off workflows in a compact *Waiting for your approval* strip, and saving edits in the advanced editor keeps one off. `tool_catalog.py` resolves names to schemas through the gateway's `search_tools` (extensible-mcp's Markdown output is parsed; resolving is also the per-session discovery its `call_tool` requires) and validates availability at save time (`POST`/`PUT /workflows/custom`, chat `draft_workflow`/`save_workflow`, designer drafts) — an unknown, misspelled or deny-listed tool is a 422. The gateway meta-tools (`search_tools`, `call_tool`, `load_mcp_server`) can never be step tools. `run()` resolves every action step's tools before any step runs, so a vanished tool stops the run before an earlier step acts; `resume()` checks only the steps still to run, after the gate and decision are validated. The last turn of an action step always switches tools off so the agent can report. At run time only the allowlisted tools are offered (sorted; constant cached system block, per-run goal/inputs/earlier results in the user turn), every `tool_use` is re-checked against the allowlist, MCP calls go through `MCPGateway.call_tool` so the config deny-list and the Gmail/Calendar/Drive recipient gates still apply, the call budget is enforced (then a final tools-off turn asks for the report; running out of turns fails the step and lists what was already done), tool errors (the gateway's `Error:` / `Error calling tool` / `Tool error:` prefixes, or a truthy `error` field) return to the model as `is_error` results, each call emits a `progress` event and an audit row (`workflow_tool_call`: tool and outcome; for tools that may change things also a short digest of target arguments — ids, recipients, URLs, paths — so a redirected write is visible; content arguments are never logged), and the step's output is the agent's report plus an \"Actions taken\" list. Tool output is framed as data, never instructions.\n\n**Scripts in action steps.** Besides the step's own tools, the agent is offered one more, `run_script` (`openexecutive/workflows/step_script.py`), for work that repeats over many items — every file in a folder, every row, every email — or chains calls with simple logic. It writes a short Python script once instead of spending a model turn per call; each of the step's tools is a function in the script (`drive__move_file(file_id=…)`, or `call_tool(name, arguments)` by exact name), and only the script's final value and printed output come back. The script runs in **Monty** (`pydantic-monty`), a sandboxed Python interpreter for model-written code, in a worker process spawned per script: no files, network, environment, subprocesses or third-party imports — the step's tools are the only things it can reach. **A script is a faster way to make the same calls, never a way around them:** every call it makes goes through the same per-call path as a direct `tool_use` (`action_step._StepCalls`) — the allowlist, a call budget (a script's calls have their own: 10× the step's `max_tool_calls`, capped at `WORKFLOW_SCRIPT_MAX_CALLS`, default 500, because they cost no model turn each — so a step can work through a folder of hundreds of files while its direct calls stay capped at 1–50), the first-write target check (a held write comes back to the script as `{\"status\": \"held\", …}`) and one `workflow_tool_call` audit row per call — and the script itself is audited as `run_script`, with how many calls it made and how long it took. The step's \"Actions taken\" list collapses repeats into one line with a count. A step whose direct calls are spent keeps its tools on while the script budget has room, so it can still script the rest (a direct call is refused as over budget). When a tool returns a list of five or more items and the round didn't use `run_script`, the step adds the same `FANOUT_HINT` as chat (once a step), pointing the per-item work at one script. The worker is stepped from the step's own task (each call pauses it and is answered there), so cancelling the run stops the script and its in-flight call together, and each call's progress shows as it happens. At most `SCRIPT_MAX_WORKERS` (default 2) scripts run at once across the server, chat and workflows together; another waits for a slot within its own wall clock, else returns \"the server was busy\". A worker exists only while its script runs (about 8 MB idle). Execution time (excluding time waiting on tools), memory and recursion depth are capped inside the worker, the script has a wall-clock limit, and its text is size-capped; printed output past its cap is dropped. A failing script returns its error to the agent as an `is_error` result, with the calls it already made listed first (ahead of any long result, so a cut never drops them) so a fixed re-run doesn't repeat writes; a call cut off by the time limit is listed as `may_have_run`, and gets a `cancelled (may have run)` audit row. A workflow definition still holds no code; a script lives on only if the agent saves it as a tool (below). Because a step can loop at run time, the workflow designer keeps work that repeats over items (each email, file, row or listed follow-up) in ONE action step whose goal spells out the items, rather than a step per item; `run_script` is never a step tool, and a definition that lists it is rejected (`validate_definition`) so the designer's repair turn removes it. `WORKFLOW_STEP_SCRIPTS=false` turns the tool off, and a step then only calls its tools one by one.\n\n**Python jobs (`run_python_job`, `openexecutive/workflows/python_job.py`).** Monty can't run libraries, so library work on files (split or merge PDFs with pypdf, spreadsheets with openpyxl and pandas, charts with matplotlib, images with Pillow, and documents made from scratch or from files: Word with python-docx, PowerPoint with python-pptx, laid-out PDFs with fpdf2) runs as a Python job: Pyodide (Python in WebAssembly) executed by **Deno** in a process of its own per job. Pyodide alone is not a sandbox — its Python can call into the JavaScript runtime, which under plain Node reaches the server's environment, files and network — so Deno is what isolates it: it is started with read access to the sandbox folder and the worker script only, with no network, environment, writes or processes, and an empty process environment. Files go in on stdin and results come back on stdout. A job gets only what the call names: Knowledge-library files, the files attached to the chat message being answered (`attachment_files`, the uploads as sent, bound for that turn only by `POST /chat/upload` in `workflows/turn_files.py`, so the model never retypes an attachment's text), text the caller passes, and `inputs` (a dict the code reads); result files are kept 7 days under the company folder and served to the principal at `GET /python-jobs/{job}/{file}`, which the Executive links in its reply. The library set is fixed at build time: the API image installs Pyodide and Deno from npm and the compiled libraries from Pyodide's CDN, each checked against the sha256 in Pyodide's lock file, and a build smoke runs a job. Pyodide can't infer the compiled packages the pure-Python libraries import (lxml, Pillow, fonttools), so the worker loads them when a job imports docx, pptx or fpdf. Pyodide, Deno and the pure-Python libraries are pinned by hash too (an npm lock installed with `npm ci`, pip `--require-hashes`). One job runs at a time per process (200–450 MB each), with a `PYTHON_JOB_TIMEOUT_S` limit and a memory cap (`PYTHON_JOB_MEMORY_MB`, default 1536, as the process's data limit, plus V8 heap and WebAssembly memory caps); the worker returns at most 20 result files and 25 MB, and the API stops the job as soon as its reply passes that. The job runs at a lower CPU priority and is first in line if the machine runs out of memory, so a heavy job slows or loses itself rather than the API; Deno's own caches go in a temporary folder removed with the job; and kept results are capped at 200 MB in all, oldest first. The tool is chat-only, offered only on the principal's own turns (`PRINCIPAL_ONLY_TOOLS`), withheld on private turns, unattended passes and under the Act as me lockdown, and not offered at all without the sandbox (`PYTHON_JOBS_ENABLED`, `PYTHON_SANDBOX_DIR`).\n\nA job that worked can be kept the same way as a script (below): `save_as` and `description` keep its code as a Custom tool of kind `python`, and `run_python_job(tool=<name>, inputs={...})` with its files runs it again, recording the run, so a job done before costs no code-writing tokens. Python tools run only on the principal's own chat turns: they are never listed for workflow steps and never turn on for workflows (`PUT /saved-tools/{name}` answers 409), and `run_script` refuses one (and `run_python_job` a script tool). Every job writes one `python_job` audit row (`ok`, `duration_ms`, `cpu_ms` and `peak_mb` read from the process's /proc entry just before it is stopped, files and bytes in and out, attachments, the Custom tool it ran or kept), which `GET /audit/usage` sums into `python_jobs` with the model usage of the turns that ran one.\n\n**Saved tools.** A script that works can be kept to run again by name (`openexecutive/workflows/saved_tools.py`); each tool has a `kind`, `script` (`run_script`) or `python` (`run_python_job`, above). `run_script` takes `save_as` and `description` with a new script: if the script succeeds it is saved, with the set of tools it actually called, and a failed script is never saved. Later, `run_script(tool=<name>, inputs={...})` runs the saved script instead of new code, with `inputs` available to it as a dict. Only the principal's own turns save: a chat turn where the principal is speaking on a verified, interactive surface (the same test as `load_mcp_server`), never an inbound email, a teammate, an unattended run or a workflow step — a step is offered `tool=` but not `save_as`. A saved tool runs with the authority of whoever runs it, so its author must be trusted at least as much as anyone who may run it. For the same reason only a workflow step (its tools owner-approved) or the principal's own turn may run one: anyone else's chat turn is refused `tool=` and not offered `list_saved_tools`, so a stranger can neither list the principal's automations nor run one with inputs they choose. Descriptions are kept to one line of printable text (they are shown to models), and every save writes a `saved_tool_changed` audit row (`actor: executive`) beside the owner's own changes. Saving needs no approval beyond that, because a saved tool can do no more than the context running it: its calls go through the same per-call path as any script (in a step: the allowlist, budget, target check and audit), and a step runs one only when the step already allows every tool it used (otherwise the result names the `missing_tools`) and only at the version the owner turned on for workflows (`workflow_version`, Settings → Advanced → Custom tools → Use in workflows). A tool kept on a turn that read outside content could otherwise carry injected code into an unattended run, so keeping stays automatic and works in chat at once, but workflows wait for the owner; a newer save doesn't change the version workflows run, and pruning never drops it. Each step's `run_script` description lists the saved tools that step may run; chat asks `list_saved_tools`. Every save of changed content is a new version (the newest 20 are kept, plus the one that ran until then, so a rolled-back version the owner trusts survives later saves), every run is recorded (`saved_tool_runs`, the newest 200 per tool), and the owner can turn a tool off or switch it back to an earlier version on Settings → Advanced → Saved tools (`/saved-tools` routes, the owner's alone). A turned-off tool never runs, and a new version saved by the model does not turn it back on. `SAVED_TOOLS_ENABLED=false` stops saving and running saved tools everywhere.\n\n**Targets are chosen at run time — and a new one is confirmed once.** Approving a workflow approves its *tools*; which sheet, doc, recipient or URL each call hits is chosen by the model at run time, by design (a tab per month, a reply to whoever sent the invoice, a doc per client). What bounds that is the allowlist, the gateway's recipient gates, the audit digest, and **first-write confirmation**: a `TargetPolicy` (one per run) checks every call to a tool that may change things. `resource_targets` walks its arguments recursively for resource keys (snake- or camel-case ids, `to`/`cc`/`bcc`, recipients, addresses, attendees, members, users, owners, parents, URLs/links/hrefs, paths, channels, spaces, rooms, phones, hosts, buckets, tables, spreadsheet/document/folder/calendar/file); a matching key marks everything beneath it, even a `name`. Outside a resource, only a new item's own `name`/`title`/`subject`, message/thread/draft/label ids, `range`/`sheet` and workspace-mcp's acting `user_google_email` are skipped — and whatever the key, a bare value shaped like an email address, a URL or an international phone number is a target, except inside what the call writes (`rows`, `values`, `body`, `text` …). Key names are a heuristic, so this is a best-effort guard layered on the allowlist, the gateway gates and the audit. Float ids are compared as integers. Arguments too deep, naming too many targets, or holding an over-long value **fail closed** (the call is refused). A target is trusted only when it was approved for this workflow before (`workflow_approved_targets`, by value) or is an id a write in this run created — a top-level id/link field (`id`, `spreadsheetId`, `webViewLink` …) of the **structured JSON object** result of a tool whose name says it creates (create / insert / upload …) and that takes no URL — a tool whose schema couldn't be read is assumed to (free text, nested records and readers' or updaters' results are never read, so a fetched page, an echoed title, a shared page's links or a stranger in `ccRecipients` can't plant one); those survive a pause in the resume payload (`run_created`). Values written in the workflow's own text are *not* trusted, since anyone who can edit the workflow could change them without review: they are asked once, then remembered. Approvals are by value, for any of the workflow's tools, and are **cleared when the workflow's tool set changes** (a newly added tool must earn them again). Any other target makes the call **held**: it is not run and costs no budget, the model is told it is waiting for approval and carries on, and it is audited `held for approval` (at most 10 held per step, 16 KB each; an identical repeat is held once). Whether a call is checked at all follows the read-only label, which now never marks a tool whose name carries a write verb anywhere (`find_and_replace_doc`). After the step finishes, the run pauses with a `WaitForHumanEvent` to the workflow's **owner** (`owner_person_id`, the person who created it on the Workflows page or in chat; falling back to the principal if unknown or archived), `on_timeout=auto_proceed`, 48 h, and a `WorkflowResumeState` of `kind=\"held_writes\"` carrying the exact held calls. On resume (`_resume_held_writes`, fingerprint-checked like a gate, with the remaining steps' tools checked first), only `approve` runs them — each still has to be in the live step's allowlist and resolve; its targets are remembered first — and anything else (decline, timeout, unreadable) skips them; either way a *Held for approval* section is added to the step's output and the run continues at the next step. With no one to ask, held writes are skipped and noted; a step that fails or crashes after holding, or a resume that stops early (definition changed, a later tool missing), audits its held writes as dropped and says they did not run. The question quotes each target JSON-escaped (a value can't add its own lines) and always shows a URL's full scheme and host, saying how much of the rest is hidden. The owner answers by chat reply or with **Approve / Decline** on the run page (`POST /workflows/runs/{id}/decision`, which the principal may also use); remembered targets are listed and removable on the workflow page.\n\n`oe__read_file` extracts text (PDF, Word, Excel, CSV, Markdown, text) only from files under `WORKFLOW_FILE_DIRS` (default: workspace-mcp's attachment folder), resolving symlinks and `..` before the check (`tool_catalog.resolve_readable_file`, shared with the Executive's `read_document` tool). A scanned PDF is converted by `knowledge.pdf_reader` instead of failing as \"may be a scanned image\"; one that still cannot be read returns the reason. Action steps are allowed in scheduled workflows. Because they have external side effects, a resumed run heartbeats its claim (`touch_resume_claim`, at most once a minute as events flow — each tool call emits one; a busy database is retried on the next heartbeat, never treated as a lost claim; the sweep's requeue re-checks the claim's age in the same UPDATE, so a heartbeat landing between the sweep's read and its write wins), so the stale-claim sweep never requeues and replays a slow but live run, and a worker that finds its claim superseded stops at its next event. A worker that actually crashes mid-resume is still requeued; its action steps then run again, so goals should say how to avoid duplicates (the actor reads current state before writing). `GET /workflows/tools/search?q=` and `GET /workflows/tools/describe?names=` back the advanced editor's tool picker and the review card's labels.\n\nAn optional cadence (`daily@` / `weekly@` / `quarterly@`) re-uses the scheduler via the `dynamic_workflow` action kind, which runs the workflow on schedule and DMs the artifact to a rostered recipient. A gate followed by further steps pauses at `awaiting_human` and then resumes once answered, so gates no longer have to sit immediately before synthesis. They remain forbidden in cadence-enabled definitions: a scheduled fire has no interactive human to ask.\n\nBuilt-ins the system runs on its own (`morning_brief`, `end_of_day_digest`, `executive_reflection`, `executive_research`) set `background = True`, surfaced as `WorkflowMeta.background`; the `/jobs` ready-made list files them under a **System** chip instead of the default view. They stay runnable by hand.\n\n**Workflows vs. playbooks.** `/jobs` has three tabs: **Build** (`?tab=catalog`, the default), **Runs**, and **Playbooks** (`?tab=playbooks`; `/skills` redirects there). A playbook is a skill (`knowledge/skills_repo.py`): *how* the Executive does a piece of work, with no inputs, schedule or side effects. A workflow is the runnable job. Workflows **follow** playbooks: a step that follows one gets its body appended to the step prompt via `workflows/playbooks.py` (`load_playbook` + `playbook_clause`). Six built-ins do — `board_prep` (`board-prep-deck`), `quarterly_plan` (`quarterly-okr-set`, `quarterly-forecast`), `mbr` (`monthly-business-review`, in the financial, KPI and function steps), `competitive_teardown` (`competitive-teardown`, in the four analysis steps), `fundraising_prep` (`fundraise-narrative`, narrative step) and `exec_search_brief` (`role-scorecard`, profile step). Each declares them in `Workflow.playbooks`; a unit test holds that declaration equal to the `load_playbook(\"…\")` calls in the module, and `WorkflowMeta.playbooks` exposes it. A custom workflow's `specialist` step may name a `playbook` (validated to exist at save; appended after `{field}` rendering, so braces in a playbook are never treated as placeholders). The lookup resolves the playbook in effect, so a company's customized copy changes the workflow's output, and a hidden, missing or malformed one falls back to the step's own prompt plus RAG — a workflow never fails over a playbook (`load_playbook` catches everything; a custom step whose playbook is unavailable emits a `progress` event saying so). Because an approved scheduled workflow reads its playbook at each run, the chat tools refuse to edit or delete any playbook a workflow follows, or to create one under a name a workflow still follows after its playbook was deleted; if the workflow list can't be read they refuse (`unverifiable`) rather than assume nothing follows it. Only a person on the Playbooks tab can change what a workflow follows, and the chat summary of a drafted workflow names each step's playbook. `playbook_users()` maps each playbook to the workflows that follow it — built-ins and every custom workflow, switched-off ones included (turning one on approves its definition, so what it follows must not have changed from chat meanwhile); the skills API reports it as `used_by`, and the Executive's `search_skills` hits carry it as `workflows`.\n\nCompleted artifacts are discoverable in the unified artifacts section (`/artifacts`, labelled **Documents** in the UI) alongside their run-detail page at `/jobs/runs/{run_id}`. A run someone starts by hand (`POST /workflows/{name}/runs`, `run_workflow`, research in chat) stores them as `workflow_runs.owner_person_id` and its history and output are theirs alone (plus the person a gate waits on); scheduled and system runs keep no owner and stay the team's. Workflow output stays Markdown — the scheduler DMs it and reflection rereads it as text — and gains other formats only on the way out: any run can be downloaded as Word (`/artifacts/run:<id>/download?as=docx`), and the Executive can repackage one with `get_artifact` → `draft_artifact(format=…, supersedes=\"run:<id>\")`.\n\n## Data-driven retrospectives\n\n`engagement_value_report` introduces a new workflow pattern: instead of the user typing the substance into a form (the `investor_update` / `mbr` shape), the workflow **reads its substance from the database**. A deterministic gatherer (`clients/engagement_activity.py`) pulls the period's decisions, initiatives, advisory consultations, delivered workflow artifacts (excluding prior value reports), follow-through counts, and alerts handled — from the live DB for the active client, or a parked client slot's `state.db` opened read-only. A single board-comms call narrates those facts (the prompt forbids inventing anything not in them; an empty period skips the LLM entirely), and the artifact's lists are assembled deterministically. Built for renewal conversations — the engagement record is the evidence.\n\nWith Take the lead as the Executive on (`orchestrator.take_the_lead`), reflection and research build their toolkit through `unattended_toolkit(..., source=...)`. That keeps solo's booking and `run_workflow` tools and an unrestricted `message_person` (teammates as well as the principal). Every acting tool goes through the gate, and a held action waits on Today as a `take_the_lead_action` decision. Reflection also gets the MCP gateway's `search_tools` and `call_tool` (`_with_lead_tools`; `call_tool` behind the gate, `load_mcp_server` left out), so it can find and use whatever the deployment has connected, such as building a sheet to track something. A note in the user turn (never the cached system prompt) tells it to act rather than only report, and it gets more room: 10 iterations, 4000 tokens, and tool results up to 4000 characters. The grounding check still refuses outward text naming people or figures the pass didn't see. New mail and chat also wake the reflection early (scheduler section). In training (`take_the_lead.training`) every acting call that isn't allowed waits too, as a training card for the principal, and both passes append `take_the_lead.learned_note()` to their user turn: that it's in training, and the principal's own edits on earlier cards as examples of how they want those done (never in the cached system blocks).",
+ "markdown": "Workflows are deterministic, multi-step orchestrations. The built-ins are registered in `openexecutive.workflows.WORKFLOW_REGISTRY`; user-created (dynamic) workflows are stored as data (see below). Each follows the same contract: structured input feeds an ordered list of named steps that render a Markdown artifact, persisted via `complete_run` into `workflow_runs.artifact`. Every step emits a `WorkflowEvent` (start/progress/complete) streamed over SSE at `/workflows/{name}/runs`.\n\nThe registry holds 25 built-in workflows, plus any number of user-created ones. Representative built-ins:\n\n- `board_prep`, `quarterly_plan`, `annual_plan` — planning and governance deliverables.\n- `performance_review`, `comp_refresh`, `org_design`, `exec_search_brief` — people and org artifacts (the last is an advisory hiring brief / scorecard from the CHRO, with input from the hired function's specialist — `sales` for a sales role — kept when the recruiting pipeline was removed).\n- `crisis_comms`, `churn_deep_dive`, `competitive_teardown` — situational analyses.\n- `department_check_in` — runs on department cadence; its `goal_status` step returns structured verdicts persisted back to `department_goals.status` (`department_check_in.persist_goal_verdicts`, shared with `weekly_review`).\n- `weekly_review` — the principal's weekly look back and ahead (see *Weekly review* below).\n\n**Standing facts in unattended output.** `end_of_day_digest`, the weekly review's top-three call and the morning reflection append the standing facts (`memory.facts.with_standing_facts`; a teammate's marked `(per <name>)`) to their user turn, and the morning brief gets them through `render_briefing_context`. A workflow step that consults a specialist gets them through `route_to_specialist`. A correction made in chat therefore holds in every recurring brief and run. The morning brief also reads `facts.render_teammate_changes(since)` off the event loop and passes it to both `render_briefing_context` (a TEAMMATE CORRECTIONS SINCE LAST BRIEF block) and `build_brief_fingerprint`, so the principal hears of each correction a teammate made in the window, and that alone un-suppresses an otherwise unchanged brief. Proposals waiting for approval are listed only in a brief the principal reads privately (`include_proposed=_private_ok()`), and then mark the run private to them. The principal's own Always in the loop notes (`memory.history_brief`, only with their switch on) come in by a stricter rule, only on the scheduler's delivery (`PRINCIPAL_DELIVERY`), never a run from chat, whose tool result reaches the turn's shared audit row and peer memory: the morning brief's FROM YOUR NOTES — DUE SOON block and the end-of-day digest's FROM YOUR NOTES — TODAY block (what they committed to since the last digest, then what is due tomorrow). Either marks the run `private_to_principal` and moves its fingerprint; the digest, which had no private reads before, now emits that flag too.\n\n### Change-aware principal briefs\n\n`morning_brief` and `end_of_day_digest` are the two recurring briefs the scheduler sends to the principal. Both are now a *delta*, not a re-list: the window is \"since the last **delivered** brief\" (`briefing.brief_state.since_for`; 24 h on a cold store, clamped to 7 days), activity comes from `_build_activity(limit, since=…)`, proposals (action category only) split into *new since last brief* — enumerated, with each item's why-now and next move — versus a one-line carried-over count, a *rewritten* block lists carried items the review rewrote inside the window (`brief_state.rewritten_since`: verdict `changed`, still open — the brief reports them under \"what changed\", never as done), and a *handled overnight* block lists what the Executive's alert review **completed** on its own (`brief_state.handled_since`; the `changed` verdict is excluded because the alert is still open). The morning brief also reads the principal's **live world** since that window opened (`briefing.live_signals.gather_live_signals`): mail and chat the Executive handled (INBOUND, grouped by sender and subject), what got stuck (STUCK: a reply the outbound gate held, a non-roster sender who got no reply, a Google Chat message left unanswered, drafts waiting in the principal's Gmail), the principal's own chats, and — in both modes now, unless the solo top three already listed it — today's calendar, read fresh (`live_signals.refresh_calendar(max_age=0)`). The 07:30 reflection's **Flagged for the brief** bullets are carried across (`brief_state.reflection_flags_since`, the latest reflection run finished inside the window) instead of surviving only as an activity line with the run's title, and the rhythm runs themselves (`today.RHYTHM_WORKFLOWS`) are dropped from the brief's activity: their dated titles were noise and moved the fingerprint every day. At-risk departments name their slipping goals and each proposal carries its age. The period is the principal's local date. Section order: top call → **today** (the calendar's shape and anything stuck) → what changed (inbound first, naming who wrote) → handled overnight → needs you (new only) → waiting on → at risk (the slipping goal, or the department left out — never \"no detail yet\"). Quoted mail, chat, invite and reflection text is marked as data, never instructions. **Privacy:** the brief reads what is private to the principal — their contacts' alerts and mail, their drafts, their chats and their calendar — only on a run for them: the scheduler's (`morning_brief.PRINCIPAL_DELIVERY`) or the principal's own verified chat turn in a conversation only they can read (`delegation.settings.private_conversation`: the web chat, a Slack or Discord DM, a private Telegram chat — never a shared channel, where the reply is posted for everyone). A run anyone else starts (a teammate from chat in team mode, `POST /workflows/…/runs`) reads what everyone may see, as before. A run that did read a private row says so on its `result` event (`private_to_principal`: its live read differs from the shared one in any key or count), and the scheduler, `run_workflow` and `POST /workflows/{name}/runs` all then keep `persistence.PRIVATE_RUN_ARTIFACT` in the run history instead of the text. Text another pass wrote (the reflection's flags) renders as quoted ` > ` lines and every goal and headline field is one-lined, so none can start a line that reads as a block header. The fingerprint carries the live signals' time-free keys (who wrote about what, what is stuck, the calendar's shape) and the reflection flags when there are any, so \"nothing new\" means the principal's world was quiet too; the context summary is logged at INFO (`morning_brief: context since=… inbound=… stuck=… suppressed=…`). In **solo mode** (one person using Open Executive for themselves) both briefs speak to the principal, whatever their role: the morning brief runs top call → top three today → what changed (inbound and stuck first) → handled overnight → needs you → due this week (the principal's open loops overdue or due within 7 days, from `open_loops.principal_due_soon`) → goals at risk (named by area), the end-of-day digest what I did → open decisions → goals at risk tomorrow → sleep on this, over a context with goals at risk by area and no people-waiting or department block; the fingerprint carries the mode and ignores who is awaiting, and carries the due items as `(loop_id, state)` pairs when there are any. **Top three today** is picked in code, not by the model (`briefing.top_three`: overdue then due-today commitments, off-track then at-risk goals, commitments due later this week, then active projects, stalest first). The solo Briefing's *Top three today* card shows the same pick from the same code (`GET /today/top-three`, see Today). When the MCP gateway runs the Google Workspace server, one `google_workspace__get_events` call (4 s timeout) lists today's events on the principal's calendar — their address as the calendar id, read with the Executive's account (`primary` when the Executive is signed in as them) — and on a business day (`calendar_tools.is_business_day`, the rule the booking tools apply; at the weekend the day's events are still listed but no slot is suggested) each item gets its own free block inside `CALENDAR_BUSINESS_HOURS_START`–`_END`, rendered with a TODAY'S CALENDAR block — only a date-only event is all day, a zero-length one blocks nothing, and one whose times can't be read is left out; no gateway, no principal address, an error, a timeout or a reply it cannot read all mean no calendar and no slots, never a failed brief; the log records only the reply's length or the error's type, never calendar text. The fingerprint carries the three items' keys (no dates, no slot times) and only a coarse hash of the day (busy blocks rounded to 15 minutes, no titles, no dates), so an unchanged day still suppresses. A day whose calendar read fails after one that worked (or the reverse) changes the fingerprint once, so that brief is written in full. A date-free fingerprint of the inputs is compared to the last delivered one; if unchanged the artifact is the one-liner \"Nothing new since yesterday's brief — N items still waiting on you.\" and no model call is made (`PRINCIPAL_BRIEF_SUPPRESS_UNCHANGED`; `force_full=true` on a manual run bypasses it). The workflow yields a `result` event with the fingerprint; only the scheduler records it, after a successful delivery, in `briefing_narrative` under scope `brief:<kind>` — so manual runs never advance the window.\n\n**Brief delivery.** `scheduler.runner._deliver_to_principal` maps the principal's `preferred_channel` (`slack` / `discord` / `telegram` / `email`) to its delivery channel and tries it first, then the other chat channels the principal has an id for, in the order Slack → Discord → Telegram. Email goes through the active MCP gateway (`google_workspace__send_gmail_message` as the Executive's own address, to the principal, subject `<title> — <date>`) and is used only for an explicit `email` preference, or for `any` when no chat channel resolves — an owner linked by email at setup has `any`, so one who also has Slack does not start getting two emails a day. When nothing can deliver (a web-only principal, or email with no gateway), the brief is still generated and stored as a run with its artifact, as before — a web-only principal reads it on the Artifacts page — it is just not delivered (audited), so the window does not advance, and the next occurrence is chained. The same delivery path carries the client-rotation digest and the weekly review.\n\n**Grounding.** Nothing an unattended pass writes is read by anyone before it goes, so `openexecutive.briefing.grounding` checks it deterministically — no model judges the model. A name (a run of two or more capitalised words once section headers, weekdays, imperative openers and org nouns are trimmed, or one capitalised word in a person slot like \"ask Marcus\" or \"Marcus wrote\") is grounded only when a single line of the pass's inputs holds all of it, or it belongs to a roster person some input mentions — being on the roster alone is not enough. A figure (money, a percentage, a k/M/B number, any bare number of 11 or more; never a date, time, year, duration, id, list marker, `Q3` or small count) is grounded when an input number matches it within the precision it is written with (`$1.2M` matches `$1,234,567`; `47 units` does not match 48). The inputs are the rendered context split into lines labelled by section (a header also carries its bullet count), the company profile and the department titles. `morning_brief` renders its context itself and hands it to the synthesizer as `rendered_context`, so the check reads exactly what the model read; after the quiet fallback, `ground_brief` runs on both briefs. When something is ungrounded it makes one repair call to the fast model (same system prompt, the list of items, \"remove or correct using only the context\"), re-checks, then drops each line still ungrounded and any header left empty, and appends \"_Held back N lines that named people or figures I couldn't match to your data — /today has the full list._\" (everything ungrounded → one held-back line). Then each grounded figure gets a server-chosen ` [n]` — the matching input line with the most words in common with the figure's clause — and a `**Sources**` list closes the brief (`- [n] <section> — <snippet>`, at most 9, snippets quote inbound mail, so links are masked (\"(link removed)\"), Markdown, angle brackets and @-mentions stripped, and they are cut to 90 chars; never `[n]:` at a line start, which Markdown reads as a link definition). Text with nothing to change comes back byte-identical, the fingerprint hashes the inputs so suppression is unaffected, and the grounded text is what is delivered, stored and recorded as delivered. Both brief prompts and the digest prompts say the rule up front. `executive_reflection` wraps its handlers in a `GroundingScope`: `create_alert`, `message_person`, `send_department_message`, `send_company_broadcast` and `schedule_followup` refuse text with an ungrounded name or figure, returning a JSON error that names the items so the model can look the person up or leave the figure out (every tool result joins the inputs, so a `lookup_person` answer grounds the DM after it; the recipient's own name always counts). Its **Flagged for the brief** bullets are filtered the same way before the tool log is appended — the next brief quotes them, where they would ground themselves — and `brief_state.reflection_flags_since` now stops at the tool log's `---`. The reflection's `grounding` rows are private when a private-to-principal alert reached its context, and the digest's always are (its HANDLED block can quote one). Everything fails open. `GROUNDING_CHECKS=enforce` (default) acts, `report` only writes the `grounding` audit rows, `off` skips it; `GROUNDING_CITATIONS=false` drops the markers. Not yet grounded: the /today header (its cards sit beside it), `weekly_review`, the research synthesis, dates and meetings, and a real person named in the wrong role.\n\n### Weekly review (solo)\n\n`weekly_review` (background, inputs all defaulted) closes out the principal's week in six steps. **Goals**: every area with goals is graded by its own specialist, concurrently, with the department check-in's verdict contract (`_parse_verdicts`) and write-back (`persist_goal_verdicts` → `record_goal_review`, one `goal_status_review` audit row per area with actor `weekly_review`); an area with no goals is skipped, one without a specialist is listed ungraded, and one whose specialist fails is shown as last recorded. **Commitments**: `open_loops.principal_due_soon` — overdue or due within 7 days. **Projects**: active initiatives with no update for 7+ days. **Decisions**: the week's (`episodic.decisions_since`). **Revisit**: up to three decisions older than 30 days with an empty outcome (`episodic.decisions_awaiting_outcome`, newest first), listed as `[decision N]` under *How did these turn out?* — the principal's answer is recorded with `record_decision_outcome` (see Agents). **Assemble**: every section always renders (a one-line note when empty), ending with *Next week's top 3* from one fast-model call that follows the `weekly-review` playbook (a constant system prompt; the playbook rides in the user turn), trimmed to at most three list items — or, when that call fails or returns nothing, the same ranked pick the morning brief uses. In solo mode the scheduler fires it weekly (`principal_weekly_review`, see Scheduler) and delivers it through `_deliver_to_principal` like the briefs, recording the delivery outcome. The solo Briefing also shows the latest completed run to the principal (`GET /today/weekly-review`, see Today): `weekly_review.summarize_review` reads the artifact's title line (`TITLE_PREFIX`) and its last `TOP_THREE_HEADING` section back as plain text, so a principal no channel reaches still sees next week's top three. Wording is role-neutral; run by hand in team mode it says *department* instead of *area*.\n\n**Principal-only workflows.** A workflow can name the workspace modes in which only the principal may start it (`Workflow.principal_only_modes`): `weekly_review` in both modes (built from the principal's decision log, commitments and goals, and it re-grades the goals) and `morning_brief` in solo (built from the principal's commitments and calendar). Team `morning_brief` and every other workflow can be started by anyone as before. This decides who may start or trigger a run, not who may read one: stored runs, the scheduled briefs and weekly reviews included, are read under the existing `/workflows/runs` rules — which is why a morning brief that drew on the principal's private rows is stored as `PRIVATE_RUN_ARTIFACT`, not as its text.\n\n- **From chat**, `run_workflow` checks at dispatch, before any run row is created, that the turn is the principal on a verified surface — `create_goal`'s rule — and otherwise refuses with a plain error that does not spell out the rule.\n- **Over HTTP**, the Jobs page's `POST /workflows/{name}/runs` and `POST /evals/runs` (a workflow scenario runs the workflow on live data) call `refuse_principal_only_run`: only a caller signed in as the principal, or a request with no caller email (the CLI or local login), may start the run. Unlike the workspace settings, an install with no principal on the roster lets no signed-in caller in, as in chat. Anyone else gets a 403 (\"Only the principal can run this workflow.\") before any run row exists. The mode is the stored one, or an eval scenario's own `workspace_mode`. The eval route runs exactly the scenario list it checked.\n\nEvery refusal is audited `refused=true` with the workflow, the mode and the caller. The checks fail closed: a mode that cannot be read counts as principal-only, and a check that cannot read the roster refuses.\n\n**Knowledge in the reflection.** A chat turn looks the knowledge base up before the Executive answers; the reflection reads the org's state instead, so nothing looked it up for it. It now carries its own `search_knowledge` tool (`orchestrator.knowledge_tools`, not in the shared `_ALL_SKILL_TOOLS`), which runs the same `knowledge.retriever.retrieve` as chat — the review gate, the synced Drive / OneDrive / Confluence / Notion collections and the published-artifact rule (the pass has no viewer, so no one's private document comes back). The prompt (`_KNOWLEDGE_RULE`, both modes) asks it to search before acting on or flagging something that turns on a customer's or supplier's terms, a price, a policy, a plan or a past decision. Its passages come back up to 4,000 characters (`execute_tool_calls(wide_results=...)`) while other tools stay at their short preview, its result counts as grounding input like any tool result, and the run's tool log keeps each line to 160 characters. Read-only, so Take the lead never gates it. The loop has 5 iterations (was 4) to leave room for a search before acting.\n\nNeither the nightly reflection nor the research synthesis can call `ack_alert`: both run unattended with open alerts rendered into their context, and those headlines are minted from inbound mail and chat, so an injected \"the principal already dismissed 13\" would be acted on with nobody watching. Closing an alert is the alert review's job, with evidence.\n\n`executive_reflection` now carries memory: yesterday's standup (the previous completed run's artifact) is rendered into the context and the prompt forbids re-acting on anything listed there unless it changed; open alerts render with their id, review verdict and recommended move. Reflection never closes alerts — the alert review does, with evidence. It also renders an **OPEN LOOPS** block (up to 10, soonest due first, overdue flagged) from Attunement so the standup can connect a slipped promise to the rest of the org state; the nudge engine already chases overdue loops, so the prompt tells it not to DM about a loop merely for being overdue. A **WHAT LANDS** block (from the Attunement outcome ledger) shows, per person, how often each kind of proactive DM got answered over 30 days; the prompt steers DMs toward kinds that land and away from kinds a person reliably ignores. Reflection and research tag their tool execution as proactive (`tag_proactive`) so their DMs are counted. Resolving a wait-for-human approval (`resumer.apply_resolution`) marks the nudges that chased it `acted`. `close_open_loop` is withheld from reflection for the same reason as `ack_alert`: loop text is quoted from what people wrote. `unattended_toolkit` also drops `schedule_tools.UNATTENDED_WITHHELD_TOOLS` (today `create_goal`) from both passes in both modes: a goal is the principal's own decision, and these passes see only inbound text. A turn private to the principal (mail from one of their contacts, or mail they forwarded) cannot start a workflow either: `run_workflow` and `run_executive_research` are in `schedule_tools.PRIVATE_TURN_WITHHELD_TOOLS`, which the chat loop neither offers nor runs on such a turn, since a workflow's steps can reach anyone.\n\nIn **solo mode** the reflection is framed for the principal, whatever their role: they are the only person who uses Open Executive and the only one it takes direction from, it prefers `create_alert` and `schedule_followup` to the principal, and it never messages anyone else from an unattended pass — the people in the principal's world are contacts, so anything they should hear is flagged for the principal to raise. The team roster is replaced by one YOUR PRINCIPAL line (person_id and follow-up channels), its follow-up guard is seeded from the principal only, and department risk renders as areas with goals at risk. The research synthesis follows the same rule — its routing menu and roster name only the principal. In both, `send_department_message`, `send_company_broadcast` and `set_department_head` are withheld, and so are `create_calendar_event`, `create_instant_meeting` and `run_workflow` — a booking reaches its attendees and a workflow can do anything its steps do, so the principal books and starts workflows from chat — and `message_person` refuses anyone but the principal. These passes run unattended with inbound text in their context, so the rules are enforced in code, not only in the prompt.\n\nIn either mode, both passes now dispatch only the tools they offer: the handler map is built from the offered list, so a name the model emits without being offered it is skipped as an unknown tool. They used to look names up in the full handler registry, so tools they deliberately withhold — the raw DM tools (`send_slack_dm` has no roster check), `ack_alert`, `close_open_loop`, and in research the watchlist writes and `run_workflow` — still ran when the model emitted them. Both read the mode before binding their own session (so a caller's override, such as an eval's, counts) and pin it on that session, so their tool handlers use the same mode. They carry the principal's role across the same way (`effective_principal_role` of the caller's session, pinned as `turn_principal_role`), so a specialist or workflow started from either pass sees the role of the turn that started it. Team runs are unchanged.\n\n### Wait-for-human pause\n\n`wait_for_human` (`openexecutive/workflows/wait_for_human.py`) is a first-class step primitive that pauses a run: the runner calls `save_checkpoint`, the `workflow_runs` row moves to `status='awaiting_human'`, and the serialized `WaitForHumanEvent` is stored in `state_json`. (Department authority is a *separate* mechanism — `propose_only` departments file a briefing alert via `departments.authority.propose_via_alert` and mark the scheduled action done; they do not route through this gate.)\n\n**Delivery.** Both checkpoint sites — the SSE route and the `run_workflow` chat tool — call `workflows.gate_delivery.deliver_gate_question` immediately *before* checkpointing, so `state_json` carries the routing fields a reply is matched against. Two modes. When the approver is the person already in the conversation (the chat session's `origin_channel` is set — to anything but `email`, whose answer is no chat reply and would never be sent — and its `caller_person_id` is the gate's `person_id`), nothing is sent — the question rides in the Executive's own reply — and the gate records `origin_session_id` plus the session's channel. Otherwise the question is delivered through `schedule_tools.handle_message_person`, which picks the person's preferred channel, falls through on a failed send, and falls back to a briefing alert; the gate then records the channel, address and provider message id that send returned, plus the id of the DM session it landed in — that send is always 1:1, so the answer belongs in that conversation and nowhere else. Without it the gate matched on person and channel alone, so an unrelated @mention in a public channel resolved it and had the acknowledgement, run title included, posted where everyone could read it. Delivery is not guaranteed: the outbound guard can suppress it (duplicate, rate cap, quiet hours) and an unreachable person gets the alert fallback. Both cases leave the routing fields empty and report `delivery` as `suppressed` / `alerted` rather than claiming someone was asked, and the chat tool's presentation hint changes accordingly.\n\n**Matching a reply.** Each inbound adapter passes the message to `inbound_resolver.resolve_inbound_message` with its `session_id`. Matching is three-tier: an explicit `in_reply_to` against a stored `outbound_message_id`; otherwise a single open run on that channel; otherwise one utility-fast disambiguation call gated at 0.85 confidence. An unmatched explicit reference rules out only the candidates that carry a stored outbound id — the ones it could have matched — and the rest continue; rejecting outright meant an adapter that passes a thread id on every message (Slack's `thread_ts`) short-circuited the later tiers forever, and a global test let one delivered gate's id block an unrelated self-approval gate. Channel names are normalized (`slack_dm` ≡ `slack`), the delivered address is compared when both sides know it, a row whose `state_json` will not parse is refused (a corrupt row otherwise presents as a legacy one), and an empty stored channel is treated as a wildcard only for rows checkpointed before gate delivery existed. Two guards keep an open gate from swallowing unrelated traffic: a gate carrying an `origin_session_id` matches replies only from the conversations the adapter names (a Slack or Discord thread offers its parent channel session too, mirroring the history double-write, since a gate raised on a mention recorded that session while the answer arrives under the thread's — on Slack only when that person rooted the thread, or the alias would reach into a colleague's), and the parser can return `unrelated`, which declines to resolve and lets the message reach the normal chat path. The parser is handed the gate's own question, so a bare \"yes\" is judged against something. Its fallback is flagged by a key the model cannot emit rather than by a magic note string it could, and a fallback declines to resolve on *every* reply shape — recording the empty string a `free_text` fallback fabricates would close a sign-off on a parser outage. Every shape can answer `unrelated`, not just `approve_reject`: without it an open `free_text` gate turned the person's next message into its answer verbatim, and a `numeric` one swallowed any message containing a number. The parser has one caller outside workflows: its `roster_identity` shape — `{decision: approve|link|decline|unrelated, name, kind: team|contact|null}`, with `name` only as the person wrote it (never inferred) and a fallback of `unrelated` — reads the principal's emailed answer to a roster request (who a new sender is; see Org). It is not a gate reply shape, and a fallback or `unrelated` leaves the request waiting. `resumer.apply_resolution` then records the decision, marks the run `resolved`, and writes an audit row. A timeout applies the step's `on_timeout` policy (escalate / auto_proceed / fail). The outcome surfaces on the Today dashboard.\n\n**Resuming past the gate.** A paused run continues on its own. Re-entering it was long assumed to mean serializing a Python generator frame; it does not. The only state alive across a dynamic workflow's steps is its inputs (already persisted), a company-profile block that is recomputed, the completed steps' output strings, and the loop cursor — so the engine hands the checkpoint a small JSON `WorkflowResumeState` (workflow name, the gate's id and index, the next index, the outputs so far), stored in `workflow_runs.resume_state_json`, and `DynamicWorkflow.resume` replays its step loop from the recorded index. `run()` and `resume()` share that one loop, which is why a second gate needs no special handling — it is simply another pause raised from the same place. The payload rides on a `WaitForHumanEvent` field declared `exclude=True`, so it can never leak into `state_json`, whose exact key set the resolver branches on. A gate with no payload is *pause-only*: the decision is recorded and the run stops, which is what every caller got before resume existed.\n\n`resumer._process_resumable` claims a resolved run with one guarded UPDATE — `resolved` → `running`, stamping `resumed_at` and minting a claim token — and drives it to a terminal row. It runs two ways: immediately, via a task `apply_resolution` kicks off so a Slack approval starts moving in seconds, and every 60s from the poll loop as the durable backstop. Both share the claim, so a run executes exactly once even with two resumers against one database. The kicked task binds no chat session (`set_session(None)`), exactly like the poll loop: it is created inside the inbound handler that recorded the answer and would otherwise inherit that person's live turn — so a run resumed because the principal replied on Slack does not act with the principal's identity (it cannot, for example, reach the principal's contacts). It drops the handler's audit scope too (`audit.context.unscoped_audit_rows`): inherited, the principal's scope would make private every row of the run that names a contact, and whoever started the run would see those rows vanish after the approval. The poll runs the executor *after* the timeout loop, which is what finally makes `on_timeout='auto_proceed'` do what its name says: the timeout resolves the run and the same tick executes it. While the Executive is paused (`scheduler.pause`) the resumer does nothing — no startup sweep, no timeouts, no resumes — and `apply_resolution` still records the decision but skips the immediate kick; the first unpaused iteration catches up on all of it, so a timeout that lapsed during the pause applies its `on_timeout` policy then.\n\n**The claim is fenced, because it is not a lease.** The stale sweep decides a worker is dead from a wall-clock guess, so a slow but *alive* worker will eventually be declared dead and replaced. Every write the executor makes carries its claim token and is refused once superseded — otherwise the replaced worker's late completion lands on top of its replacement's result, or its checkpoint at a later gate resurrects an already-finished run and asks the approver a second time. The retry budget resets at each new gate (a gate is new work, not a retry), and a run that exhausts it is *ended* rather than dropped: an exhausted run sits in `running` with a payload, in neither queue, and would otherwise show as in-progress forever.\n\n**The gate fails closed.** Continuing is an allowlist — `approve` and the synthetic `auto_proceed` — not \"anything that is not a stop word\". `decision` is whatever a fast model extracted from free-form chat and it drifts; under a denylist, \"rejected\", \"Reject\", \"decline\" and \"no\" would all continue, so the approver declines and the deliverable is produced anyway. Reply shapes that ask a question rather than seek permission always continue, since the answer *is* the value. A stop ends the run at `error` with the human's note. Either way the decision becomes the gate step's own output, so it reads as a section *in* the artifact — capped and blockquoted, because that text reaches the synthesis specialist as prompt context.\n\n**A definition edited during the pause is refused.** Resume re-validates the payload's version, the recorded gate, *and* a fingerprint of the whole step list, and derives the index to continue at rather than reading it from the payload. Pinning only the gate would not be enough: definitions are overwritten by name and any rostered user can save one, so someone could keep the gate identical and replace every step after it — and the approver's sign-off would be recorded against work they never saw, which defeats the one thing an approval gate is for. No built-in workflow yields the gate; only user-created dynamic workflows do.\n\n### Launching from chat\n\nWorkflows are not only reachable from the `/jobs` UI — the Executive can list and run any of them mid-conversation. `openexecutive.orchestrator.workflow_run_tools` wires two tools into the main chat loop: `list_workflows` (the launchable catalog, with each workflow's input fields) and `run_workflow` (runs one to completion and returns its Markdown artifact, using the same engine as `POST /workflows/{name}/runs`). `run_workflow` resolves both built-ins and custom workflows via `get_workflow`, validates `inputs` against the workflow's input model, persists the run (`create_run` → `complete_run` / `fail_run`), and on an approval-gate step checkpoints the run as `awaiting_human` exactly as the HTTP route does — literally the same code, since both call `workflows.gate.checkpoint_gate`, which delivers the question and writes the checkpoint in one place. It never hangs waiting on a human, and it reports both whether the question reached them and whether the run will continue by itself once answered. The runners that *cannot* pause — the schedulers, evals, client rotation, onboarding, the research tools — call `gate.ensure_workflow_event`, which raises on a gate rather than letting one be swallowed; the scheduler's dynamic-workflow branch used to ignore it and record a successful run containing nothing. `executive_research` is excluded (it has the dedicated `run_executive_research` tool). Running a workflow from chat delivers nothing on its own — it renders the artifact into the conversation, and the recurring principal briefs (`morning_brief`, `end_of_day_digest`) reach anyone only via the scheduler's own `_deliver_to_principal`. The tool description used to claim those two DM the principal when run and to confirm before firing them, which had the model soliciting a confirmation on a false premise; it now names `executive_reflection` as the one workflow that really does execute tool calls of its own (DMs to heads, broadcasts, alerts). A successful run surfaces an `action_taken` chip linking to `/jobs/runs/{run_id}`.\n\n### Dynamic (user-created) workflows\n\nUsers create reusable workflows on the fly — from the executive chat, the conversational wizard at `/jobs/new`, or the step-by-step advanced editor — without shipping code. The Workflows page (`/jobs`) opens on a *Start here* panel that leads to all three: the user describes the job in a sentence and *Start* opens the wizard with that text sent as its first message (handed over in `sessionStorage`, so a link cannot make the wizard send text the user never typed; where storage is blocked it falls back to `/jobs/new?describe=`, which only fills the composer and sends nothing), or picks *Step by step* (advanced editor) or *Talk it through in chat* (`/?new=1&draft=`). Below it come the user's own workflows, then four ready-made starter picks with the full built-in list folded behind *Browse all* (`?browse=1`). A dynamic workflow is a stored JSON definition (the `dynamic_workflows` table) interpreted at runtime by one generic `DynamicWorkflow` engine (`openexecutive/workflows/dynamic.py`) that implements the same `Workflow` ABC as the built-ins, so the run path, SSE streaming, run persistence, and `/jobs/[name]` form rendering are unchanged. `get_workflow` / `list_workflows` fall through to the dynamic store after the static registry (built-ins win on a name collision).\n\nA definition declares free-text input fields and an ordered step list of four kinds: **specialist** (consult a specialist with a goal prompt, optionally grounded by RAG), **action** (get something done with tools — see below), **approval_gate** (a `wait_for_human` pause), and a terminal **synthesis** step that assembles the artifact. At least one specialist or action step is required. `validate_definition` enforces the structural rules: exactly one terminal synthesis step, valid specialists, `{placeholder}` tokens that reference declared input fields, rostered approval/cadence people (team members — never one of the principal's contacts, whom an unattended run may not message), and a parseable cadence.\n\nAuthoring from chat uses a **draft → confirm** handshake: the Executive calls `draft_workflow` (validate; return a `confirm_token` plus a plan summary), shows the user the plan, then calls `save_workflow` with the same definition and token — a token mismatch is rejected, so the saved workflow is exactly what the user approved. The step-by-step advanced editor drives the `/workflows/custom` CRUD endpoints directly.\n\n**The New workflow wizard** (`/jobs/new`) is the default way to create one from the UI. It works like conversational onboarding: the user describes the job, the `workflow_designer` Council agent (`openexecutive/workflows/designer.py`) asks one clarifying question per turn — deliverable, per-run inputs, sign-off, schedule, recipient — with up to four clickable suggested answers, then emits a complete definition plus a plain-language summary and the assumptions it made. Every turn offers the model exactly three fixed, name-sorted tools (`ask_clarifying_question`, `emit_workflow_draft` — reusing the chat tool's `DEFINITION_SCHEMA` — and `search_available_tools`, which is answered inside the same turn from the tool catalog, up to four searches per turn, with the tool_use/tool_result pairs never stored in the plain-text transcript; every tool a search returns is remembered on the session and replayed into later turns, so a forced draft or a repair turn — which cannot search — still has the exact names); `tool_choice` flips to the emit tool once, when the six-question budget is spent or the user clicks *Draft it now*. The emitted definition goes through `validate_definition`, the tool-availability check, and a check against saved custom names, with one repair turn that feeds the errors back. The system prompt is a cached constant; the live context — specialists, the roster with person ids, the user's timezone (the one set through `PUT /workspace`, else `USER_TIMEZONE`; cadences stay UTC), and taken names — is rendered once per session into the first user turn, and while refining, the exact current draft is prepended to the latest user turn so the model edits it rather than re-deriving it. Roster names and roles are flattened to one line and capped at 80 characters before they enter that context, and the system prompt tells the model to treat the whole block as data, never instructions. Sessions live in memory (`/workflows/designer/*`, 2 h TTL, 50-session cap), run one model turn at a time (a second concurrent `/message` or `/draft` gets 409, and a failed turn is rolled back by identity), and never write: the review card saves through the same `POST /workflows/custom`, which re-validates, and *Edit details* opens the draft in the advanced editor (`/jobs/new?designer=<id>`) before saving.\n\n**Editing is a conversation too.** *Edit* on a saved workflow (`/jobs/new?edit=<name>`) opens the same designer through `POST /workflows/designer/edit`: the stored definition is the starting draft, so the user's first message (\"have Mark sign off and run it Fridays\") is applied to it like any refinement, and a note in the context block (a user turn, never the cached system block) tells the model it is changing a saved workflow. The draft always keeps the workflow's name — `_check_draft` puts it back and skips the taken-name check — and the route keeps its on/off state, so an edit can neither fork a copy nor switch on a tool workflow waiting for approval. The stored draft is handed to the model marked as data, not instructions. The review card lists *What changes* in plain words against the saved version (`lib/workflowChanges.ts`): every field is compared, with named sentences for the sign-off person and what it asks for — a sign-off moved off approve/reject is called out because the run then continues whatever the answer — the schedule, steps added/removed/reordered, inputs, tools and the tool-call budget, and a catch-all line for any other field. It names only the newly added tools that may change things. *Save changes* calls `POST /workflows/custom/{name}/save-edit`, which saves the session's own draft after re-validating it, only when it is the draft the card showed (the card sends what it displayed; a message that landed since is a 409), and only if the stored workflow has not changed since the conversation opened (409 otherwise), so an edit started before someone switched the workflow off cannot switch it back on. The step-by-step form stays as the advanced editor (`/jobs/new?edit=<name>&mode=advanced`).\n\n**Action steps (workflows that act).** An `action` step (`ActionStepSpec`: `goal`, `tools`, `max_tool_calls` 1–50, default 20) is run by the `workflow_actor` Council agent (`openexecutive/workflows/action_step.py`) as a bounded tool-use loop. Its `tools` list is an allowlist of exact tool names — MCP gateway tools (`server__tool`, e.g. `google_workspace__append_table_rows`) and a small set of built-ins (`oe__read_file`, `oe__message_person`, `oe__create_alert`) — and **clicking Create is the user's approval** of that list: the review card shows every tool, flags the ones that may change things (anything whose name doesn't say it reads, and any tool with a URL/URI/endpoint/webhook parameter anywhere in its schema, or whose schema couldn't be read, since a URL can carry data out), and names them in a \"this workflow can use these without asking again\" notice. Because that card is the approval, a workflow with action steps saved from chat (`draft_workflow`/`save_workflow`, whose confirmation is a token the model holds itself) is stored **switched off** (`is_active=false`) and `save_workflow` returns `status: saved_pending_review` with a `/jobs/{name}` link. An inactive workflow can't run or schedule (`get_workflow` and the scheduler skip it); `/jobs/{name}` shows it on the same review card, where **Turn on workflow** (`POST /workflows/custom/{name}/activate`, which re-validates the tools) is the approval — the click sends the definition the card showed, and the server refuses (409) if the stored one has changed since, so chat overwriting it while the card is open can't get new tools switched on. Chat may revise a switched-off workflow but refuses to replace a switched-on one that has action steps (an inbound email steering the Executive could otherwise switch off an approved job and stage new tools under its name); those edits happen in the advanced editor. The Workflows catalog (`/jobs`) lists switched-off workflows in a compact *Waiting for your approval* strip, and saving edits in the advanced editor keeps one off. `tool_catalog.py` resolves names to schemas through the gateway's `search_tools` (extensible-mcp's Markdown output is parsed; resolving is also the per-session discovery its `call_tool` requires) and validates availability at save time (`POST`/`PUT /workflows/custom`, chat `draft_workflow`/`save_workflow`, designer drafts) — an unknown, misspelled or deny-listed tool is a 422. The gateway meta-tools (`search_tools`, `call_tool`, `load_mcp_server`) can never be step tools. `run()` resolves every action step's tools before any step runs, so a vanished tool stops the run before an earlier step acts; `resume()` checks only the steps still to run, after the gate and decision are validated. The last turn of an action step always switches tools off so the agent can report. At run time only the allowlisted tools are offered (sorted; constant cached system block, per-run goal/inputs/earlier results in the user turn), every `tool_use` is re-checked against the allowlist, MCP calls go through `MCPGateway.call_tool` so the config deny-list and the Gmail/Calendar/Drive recipient gates still apply, the call budget is enforced (then a final tools-off turn asks for the report; running out of turns fails the step and lists what was already done), tool errors (the gateway's `Error:` / `Error calling tool` / `Tool error:` prefixes, or a truthy `error` field) return to the model as `is_error` results, each call emits a `progress` event and an audit row (`workflow_tool_call`: tool and outcome; for tools that may change things also a short digest of target arguments — ids, recipients, URLs, paths — so a redirected write is visible; content arguments are never logged), and the step's output is the agent's report plus an \"Actions taken\" list. Tool output is framed as data, never instructions.\n\n**Scripts in action steps.** Besides the step's own tools, the agent is offered one more, `run_script` (`openexecutive/workflows/step_script.py`), for work that repeats over many items — every file in a folder, every row, every email — or chains calls with simple logic. It writes a short Python script once instead of spending a model turn per call; each of the step's tools is a function in the script (`drive__move_file(file_id=…)`, or `call_tool(name, arguments)` by exact name), and only the script's final value and printed output come back. The script runs in **Monty** (`pydantic-monty`), a sandboxed Python interpreter for model-written code, in a worker process spawned per script: no files, network, environment, subprocesses or third-party imports — the step's tools are the only things it can reach. **A script is a faster way to make the same calls, never a way around them:** every call it makes goes through the same per-call path as a direct `tool_use` (`action_step._StepCalls`) — the allowlist, a call budget (a script's calls have their own: 10× the step's `max_tool_calls`, capped at `WORKFLOW_SCRIPT_MAX_CALLS`, default 500, because they cost no model turn each — so a step can work through a folder of hundreds of files while its direct calls stay capped at 1–50), the first-write target check (a held write comes back to the script as `{\"status\": \"held\", …}`) and one `workflow_tool_call` audit row per call — and the script itself is audited as `run_script`, with how many calls it made and how long it took. The step's \"Actions taken\" list collapses repeats into one line with a count. A step whose direct calls are spent keeps its tools on while the script budget has room, so it can still script the rest (a direct call is refused as over budget). When a tool returns a list of five or more items and the round didn't use `run_script`, the step adds the same `FANOUT_HINT` as chat (once a step), pointing the per-item work at one script. The worker is stepped from the step's own task (each call pauses it and is answered there), so cancelling the run stops the script and its in-flight call together, and each call's progress shows as it happens. At most `SCRIPT_MAX_WORKERS` (default 2) scripts run at once across the server, chat and workflows together; another waits for a slot within its own wall clock, else returns \"the server was busy\". A worker exists only while its script runs (about 8 MB idle). Execution time (excluding time waiting on tools), memory and recursion depth are capped inside the worker, the script has a wall-clock limit, and its text is size-capped; printed output past its cap is dropped. A failing script returns its error to the agent as an `is_error` result, with the calls it already made listed first (ahead of any long result, so a cut never drops them) so a fixed re-run doesn't repeat writes; a call cut off by the time limit is listed as `may_have_run`, and gets a `cancelled (may have run)` audit row. A workflow definition still holds no code; a script lives on only if the agent saves it as a tool (below). Because a step can loop at run time, the workflow designer keeps work that repeats over items (each email, file, row or listed follow-up) in ONE action step whose goal spells out the items, rather than a step per item; `run_script` is never a step tool, and a definition that lists it is rejected (`validate_definition`) so the designer's repair turn removes it. `WORKFLOW_STEP_SCRIPTS=false` turns the tool off, and a step then only calls its tools one by one.\n\n**Python jobs (`run_python_job`, `openexecutive/workflows/python_job.py`).** Monty can't run libraries, so library work on files (split or merge PDFs with pypdf, spreadsheets with openpyxl and pandas, charts with matplotlib, images with Pillow, and documents made from scratch or from files: Word with python-docx, PowerPoint with python-pptx, laid-out PDFs with fpdf2) runs as a Python job: Pyodide (Python in WebAssembly) executed by **Deno** in a process of its own per job. Pyodide alone is not a sandbox — its Python can call into the JavaScript runtime, which under plain Node reaches the server's environment, files and network — so Deno is what isolates it: it is started with read access to the sandbox folder and the worker script only, with no network, environment, writes or processes, and an empty process environment. Files go in on stdin and results come back on stdout. A job gets only what the call names: Knowledge-library files, the files attached to the chat message being answered (`attachment_files`, the uploads as sent, bound for that turn only by `POST /chat/upload` in `workflows/turn_files.py`, so the model never retypes an attachment's text), text the caller passes, and `inputs` (a dict the code reads); result files are kept 7 days under the company folder and served to the principal at `GET /python-jobs/{job}/{file}`, which the Executive links in its reply. A deployment can instead send jobs to a separate runner (`PYTHON_JOB_RUNNER_URL`, with `PYTHON_JOB_RUNNER_KEY` as a bearer token): `run_job` POSTs the same job (code, inputs, files, caps, time and memory limits) and reads back the same result object, at most as large as a local job's output, keeping only its known fields, so kept results, links and metering are unchanged; the local sandbox is then not needed, and the runner is trusted to isolate jobs. The library set is fixed at build time: the API image installs Pyodide and Deno from npm and the compiled libraries from Pyodide's CDN, each checked against the sha256 in Pyodide's lock file, and a build smoke runs a job. Pyodide can't infer the compiled packages the pure-Python libraries import (lxml, Pillow, fonttools), so the worker loads them when a job imports docx, pptx or fpdf. Pyodide, Deno and the pure-Python libraries are pinned by hash too (an npm lock installed with `npm ci`, pip `--require-hashes`). One job runs at a time per process (200–450 MB each), with a `PYTHON_JOB_TIMEOUT_S` limit and a memory cap (`PYTHON_JOB_MEMORY_MB`, default 1536, as the process's data limit, plus V8 heap and WebAssembly memory caps); the worker returns at most 20 result files and 25 MB, and the API stops the job as soon as its reply passes that. The job runs at a lower CPU priority and is first in line if the machine runs out of memory, so a heavy job slows or loses itself rather than the API; Deno's own caches go in a temporary folder removed with the job; and kept results are capped at 200 MB in all, oldest first. The tool is chat-only, offered only on the principal's own turns (`PRINCIPAL_ONLY_TOOLS`), withheld on private turns, unattended passes and under the Act as me lockdown, and not offered at all without the sandbox (`PYTHON_JOBS_ENABLED`, `PYTHON_SANDBOX_DIR`).\n\nA job that worked can be kept the same way as a script (below): `save_as` and `description` keep its code as a Custom tool of kind `python`, and `run_python_job(tool=<name>, inputs={...})` with its files runs it again, recording the run, so a job done before costs no code-writing tokens. Python tools run only on the principal's own chat turns: they are never listed for workflow steps and never turn on for workflows (`PUT /saved-tools/{name}` answers 409), and `run_script` refuses one (and `run_python_job` a script tool). Every job writes one `python_job` audit row (`ok`, `duration_ms`, `cpu_ms` and `peak_mb` read from the process's /proc entry just before it is stopped, files and bytes in and out, attachments, the Custom tool it ran or kept), which `GET /audit/usage` sums into `python_jobs` with the model usage of the turns that ran one.\n\n**Saved tools.** A script that works can be kept to run again by name (`openexecutive/workflows/saved_tools.py`); each tool has a `kind`, `script` (`run_script`) or `python` (`run_python_job`, above). `run_script` takes `save_as` and `description` with a new script: if the script succeeds it is saved, with the set of tools it actually called, and a failed script is never saved. Later, `run_script(tool=<name>, inputs={...})` runs the saved script instead of new code, with `inputs` available to it as a dict. Only the principal's own turns save: a chat turn where the principal is speaking on a verified, interactive surface (the same test as `load_mcp_server`), never an inbound email, a teammate, an unattended run or a workflow step — a step is offered `tool=` but not `save_as`. A saved tool runs with the authority of whoever runs it, so its author must be trusted at least as much as anyone who may run it. For the same reason only a workflow step (its tools owner-approved) or the principal's own turn may run one: anyone else's chat turn is refused `tool=` and not offered `list_saved_tools`, so a stranger can neither list the principal's automations nor run one with inputs they choose. Descriptions are kept to one line of printable text (they are shown to models), and every save writes a `saved_tool_changed` audit row (`actor: executive`) beside the owner's own changes. Saving needs no approval beyond that, because a saved tool can do no more than the context running it: its calls go through the same per-call path as any script (in a step: the allowlist, budget, target check and audit), and a step runs one only when the step already allows every tool it used (otherwise the result names the `missing_tools`) and only at the version the owner turned on for workflows (`workflow_version`, Settings → Advanced → Custom tools → Use in workflows). A tool kept on a turn that read outside content could otherwise carry injected code into an unattended run, so keeping stays automatic and works in chat at once, but workflows wait for the owner; a newer save doesn't change the version workflows run, and pruning never drops it. Each step's `run_script` description lists the saved tools that step may run; chat asks `list_saved_tools`. Every save of changed content is a new version (the newest 20 are kept, plus the one that ran until then, so a rolled-back version the owner trusts survives later saves), every run is recorded (`saved_tool_runs`, the newest 200 per tool), and the owner can turn a tool off or switch it back to an earlier version on Settings → Advanced → Saved tools (`/saved-tools` routes, the owner's alone). A turned-off tool never runs, and a new version saved by the model does not turn it back on. `SAVED_TOOLS_ENABLED=false` stops saving and running saved tools everywhere.\n\n**Targets are chosen at run time — and a new one is confirmed once.** Approving a workflow approves its *tools*; which sheet, doc, recipient or URL each call hits is chosen by the model at run time, by design (a tab per month, a reply to whoever sent the invoice, a doc per client). What bounds that is the allowlist, the gateway's recipient gates, the audit digest, and **first-write confirmation**: a `TargetPolicy` (one per run) checks every call to a tool that may change things. `resource_targets` walks its arguments recursively for resource keys (snake- or camel-case ids, `to`/`cc`/`bcc`, recipients, addresses, attendees, members, users, owners, parents, URLs/links/hrefs, paths, channels, spaces, rooms, phones, hosts, buckets, tables, spreadsheet/document/folder/calendar/file); a matching key marks everything beneath it, even a `name`. Outside a resource, only a new item's own `name`/`title`/`subject`, message/thread/draft/label ids, `range`/`sheet` and workspace-mcp's acting `user_google_email` are skipped — and whatever the key, a bare value shaped like an email address, a URL or an international phone number is a target, except inside what the call writes (`rows`, `values`, `body`, `text` …). Key names are a heuristic, so this is a best-effort guard layered on the allowlist, the gateway gates and the audit. Float ids are compared as integers. Arguments too deep, naming too many targets, or holding an over-long value **fail closed** (the call is refused). A target is trusted only when it was approved for this workflow before (`workflow_approved_targets`, by value) or is an id a write in this run created — a top-level id/link field (`id`, `spreadsheetId`, `webViewLink` …) of the **structured JSON object** result of a tool whose name says it creates (create / insert / upload …) and that takes no URL — a tool whose schema couldn't be read is assumed to (free text, nested records and readers' or updaters' results are never read, so a fetched page, an echoed title, a shared page's links or a stranger in `ccRecipients` can't plant one); those survive a pause in the resume payload (`run_created`). Values written in the workflow's own text are *not* trusted, since anyone who can edit the workflow could change them without review: they are asked once, then remembered. Approvals are by value, for any of the workflow's tools, and are **cleared when the workflow's tool set changes** (a newly added tool must earn them again). Any other target makes the call **held**: it is not run and costs no budget, the model is told it is waiting for approval and carries on, and it is audited `held for approval` (at most 10 held per step, 16 KB each; an identical repeat is held once). Whether a call is checked at all follows the read-only label, which now never marks a tool whose name carries a write verb anywhere (`find_and_replace_doc`). After the step finishes, the run pauses with a `WaitForHumanEvent` to the workflow's **owner** (`owner_person_id`, the person who created it on the Workflows page or in chat; falling back to the principal if unknown or archived), `on_timeout=auto_proceed`, 48 h, and a `WorkflowResumeState` of `kind=\"held_writes\"` carrying the exact held calls. On resume (`_resume_held_writes`, fingerprint-checked like a gate, with the remaining steps' tools checked first), only `approve` runs them — each still has to be in the live step's allowlist and resolve; its targets are remembered first — and anything else (decline, timeout, unreadable) skips them; either way a *Held for approval* section is added to the step's output and the run continues at the next step. With no one to ask, held writes are skipped and noted; a step that fails or crashes after holding, or a resume that stops early (definition changed, a later tool missing), audits its held writes as dropped and says they did not run. The question quotes each target JSON-escaped (a value can't add its own lines) and always shows a URL's full scheme and host, saying how much of the rest is hidden. The owner answers by chat reply or with **Approve / Decline** on the run page (`POST /workflows/runs/{id}/decision`, which the principal may also use); remembered targets are listed and removable on the workflow page.\n\n`oe__read_file` extracts text (PDF, Word, Excel, CSV, Markdown, text) only from files under `WORKFLOW_FILE_DIRS` (default: workspace-mcp's attachment folder), resolving symlinks and `..` before the check (`tool_catalog.resolve_readable_file`, shared with the Executive's `read_document` tool). A scanned PDF is converted by `knowledge.pdf_reader` instead of failing as \"may be a scanned image\"; one that still cannot be read returns the reason. Action steps are allowed in scheduled workflows. Because they have external side effects, a resumed run heartbeats its claim (`touch_resume_claim`, at most once a minute as events flow — each tool call emits one; a busy database is retried on the next heartbeat, never treated as a lost claim; the sweep's requeue re-checks the claim's age in the same UPDATE, so a heartbeat landing between the sweep's read and its write wins), so the stale-claim sweep never requeues and replays a slow but live run, and a worker that finds its claim superseded stops at its next event. A worker that actually crashes mid-resume is still requeued; its action steps then run again, so goals should say how to avoid duplicates (the actor reads current state before writing). `GET /workflows/tools/search?q=` and `GET /workflows/tools/describe?names=` back the advanced editor's tool picker and the review card's labels.\n\nAn optional cadence (`daily@` / `weekly@` / `quarterly@`) re-uses the scheduler via the `dynamic_workflow` action kind, which runs the workflow on schedule and DMs the artifact to a rostered recipient. A gate followed by further steps pauses at `awaiting_human` and then resumes once answered, so gates no longer have to sit immediately before synthesis. They remain forbidden in cadence-enabled definitions: a scheduled fire has no interactive human to ask.\n\nBuilt-ins the system runs on its own (`morning_brief`, `end_of_day_digest`, `executive_reflection`, `executive_research`) set `background = True`, surfaced as `WorkflowMeta.background`; the `/jobs` ready-made list files them under a **System** chip instead of the default view. They stay runnable by hand.\n\n**Workflows vs. playbooks.** `/jobs` has three tabs: **Build** (`?tab=catalog`, the default), **Runs**, and **Playbooks** (`?tab=playbooks`; `/skills` redirects there). A playbook is a skill (`knowledge/skills_repo.py`): *how* the Executive does a piece of work, with no inputs, schedule or side effects. A workflow is the runnable job. Workflows **follow** playbooks: a step that follows one gets its body appended to the step prompt via `workflows/playbooks.py` (`load_playbook` + `playbook_clause`). Six built-ins do — `board_prep` (`board-prep-deck`), `quarterly_plan` (`quarterly-okr-set`, `quarterly-forecast`), `mbr` (`monthly-business-review`, in the financial, KPI and function steps), `competitive_teardown` (`competitive-teardown`, in the four analysis steps), `fundraising_prep` (`fundraise-narrative`, narrative step) and `exec_search_brief` (`role-scorecard`, profile step). Each declares them in `Workflow.playbooks`; a unit test holds that declaration equal to the `load_playbook(\"…\")` calls in the module, and `WorkflowMeta.playbooks` exposes it. A custom workflow's `specialist` step may name a `playbook` (validated to exist at save; appended after `{field}` rendering, so braces in a playbook are never treated as placeholders). The lookup resolves the playbook in effect, so a company's customized copy changes the workflow's output, and a hidden, missing or malformed one falls back to the step's own prompt plus RAG — a workflow never fails over a playbook (`load_playbook` catches everything; a custom step whose playbook is unavailable emits a `progress` event saying so). Because an approved scheduled workflow reads its playbook at each run, the chat tools refuse to edit or delete any playbook a workflow follows, or to create one under a name a workflow still follows after its playbook was deleted; if the workflow list can't be read they refuse (`unverifiable`) rather than assume nothing follows it. Only a person on the Playbooks tab can change what a workflow follows, and the chat summary of a drafted workflow names each step's playbook. `playbook_users()` maps each playbook to the workflows that follow it — built-ins and every custom workflow, switched-off ones included (turning one on approves its definition, so what it follows must not have changed from chat meanwhile); the skills API reports it as `used_by`, and the Executive's `search_skills` hits carry it as `workflows`.\n\nCompleted artifacts are discoverable in the unified artifacts section (`/artifacts`, labelled **Documents** in the UI) alongside their run-detail page at `/jobs/runs/{run_id}`. A run someone starts by hand (`POST /workflows/{name}/runs`, `run_workflow`, research in chat) stores them as `workflow_runs.owner_person_id` and its history and output are theirs alone (plus the person a gate waits on); scheduled and system runs keep no owner and stay the team's. Workflow output stays Markdown — the scheduler DMs it and reflection rereads it as text — and gains other formats only on the way out: any run can be downloaded as Word (`/artifacts/run:<id>/download?as=docx`), and the Executive can repackage one with `get_artifact` → `draft_artifact(format=…, supersedes=\"run:<id>\")`.\n\n## Data-driven retrospectives\n\n`engagement_value_report` introduces a new workflow pattern: instead of the user typing the substance into a form (the `investor_update` / `mbr` shape), the workflow **reads its substance from the database**. A deterministic gatherer (`clients/engagement_activity.py`) pulls the period's decisions, initiatives, advisory consultations, delivered workflow artifacts (excluding prior value reports), follow-through counts, and alerts handled — from the live DB for the active client, or a parked client slot's `state.db` opened read-only. A single board-comms call narrates those facts (the prompt forbids inventing anything not in them; an empty period skips the LLM entirely), and the artifact's lists are assembled deterministically. Built for renewal conversations — the engagement record is the evidence.\n\nWith Take the lead as the Executive on (`orchestrator.take_the_lead`), reflection and research build their toolkit through `unattended_toolkit(..., source=...)`. That keeps solo's booking and `run_workflow` tools and an unrestricted `message_person` (teammates as well as the principal). Every acting tool goes through the gate, and a held action waits on Today as a `take_the_lead_action` decision. Reflection also gets the MCP gateway's `search_tools` and `call_tool` (`_with_lead_tools`; `call_tool` behind the gate, `load_mcp_server` left out), so it can find and use whatever the deployment has connected, such as building a sheet to track something. A note in the user turn (never the cached system prompt) tells it to act rather than only report, and it gets more room: 10 iterations, 4000 tokens, and tool results up to 4000 characters. The grounding check still refuses outward text naming people or figures the pass didn't see. New mail and chat also wake the reflection early (scheduler section). In training (`take_the_lead.training`) every acting call that isn't allowed waits too, as a training card for the principal, and both passes append `take_the_lead.learned_note()` to their user turn: that it's in training, and the principal's own edits on earlier cards as examples of how they want those done (never in the cached system blocks).",
"mermaid": "flowchart TD\n Input[\"Structured input\"]\n Steps[\"Ordered named steps\"]\n Wait[\"wait_for_human pause\"]\n Sched[\"workflow_runs<br/>(awaiting_human)\"]\n Resolver[\"inbound_resolver<br/>+ resumer\"]\n Artifact[\"Markdown artifact<br/>(workflow_runs.artifact)\"]\n Today[\"Today / Documents\"]\n Input --> Steps\n Steps --> Wait\n Wait --> Sched\n Sched -- \"human reply\" --> Resolver\n Resolver -- \"resolved → claim\" --> Steps\n Steps -- \"complete_run\" --> Artifact\n Artifact --> Today\n class Input entry\n class Steps compute\n class Wait compute\n class Sched storage\n class Resolver compute\n class Artifact storage\n class Today external",
"generated_at": "2026-10-08T21:30:00Z"
}
diff --git a/packages/core/openexecutive/config.py b/packages/core/openexecutive/config.py
index a9b8e2f7..c82a51d6 100644
--- a/packages/core/openexecutive/config.py
+++ b/packages/core/openexecutive/config.py
@@ -608,6 +608,36 @@ class Settings(BaseSettings):
# The sandbox process's data limit (RLIMIT_DATA): measured, pandas plus a
# chart needs about 1.5 GB of it; 1 GB is too little.
python_job_memory_mb: int = Field(1536, alias="PYTHON_JOB_MEMORY_MB", ge=512, le=16384)
+ # Run Python jobs on a separate runner instead of the local sandbox: each
+ # job is POSTed there (workflows/python_job.py, _run_remote) with this key
+ # as a bearer token, and the local sandbox isn't needed. https only, or
+ # plain http to localhost or a .internal / .flycast name. The key is
+ # required with the URL.
+ python_job_runner_url: str | None = Field(None, alias="PYTHON_JOB_RUNNER_URL")
+ python_job_runner_key: str | None = Field(None, alias="PYTHON_JOB_RUNNER_KEY")
+
+ @field_validator("python_job_runner_url")
+ @classmethod
+ def _validate_python_job_runner_url(cls, v: str | None) -> str | None:
+ from urllib.parse import urlparse
+ v = (v or "").strip()
+ if not v:
+ return None
+ parsed = urlparse(v)
+ host = (parsed.hostname or "").lower()
+ private = host in {"localhost", "127.0.0.1", "::1"} or host.endswith((".internal", ".flycast"))
+ if not host or not (parsed.scheme == "https" or (parsed.scheme == "http" and private)):
+ raise ValueError(
+ "PYTHON_JOB_RUNNER_URL must be an https URL (plain http only to localhost, .internal or .flycast)"
+ )
+ return v
+
+ @model_validator(mode="after")
+ def _validate_python_job_runner_key(self) -> "Settings":
+ # Without a key every job (code and files) would go out unauthenticated.
+ if self.python_job_runner_url and not (self.python_job_runner_key or "").strip():
+ raise ValueError("PYTHON_JOB_RUNNER_URL requires PYTHON_JOB_RUNNER_KEY")
+ return self
# A script that worked may be saved and run again by name (saved tools,
# workflows/saved_tools.py). Off: run_script ignores save_as/tool and
# list_saved_tools lists nothing; saved tools stay stored.
diff --git a/packages/core/openexecutive/workflows/python_job.py b/packages/core/openexecutive/workflows/python_job.py
index 49807cf7..6ee20830 100644
--- a/packages/core/openexecutive/workflows/python_job.py
+++ b/packages/core/openexecutive/workflows/python_job.py
@@ -29,11 +29,18 @@ memory and bytes, which ``GET /audit/usage`` sums (``python_jobs``).
The sandbox is installed by the API image (``docker/Dockerfile``) under
``PYTHON_SANDBOX_DIR``; ``available()`` is false without it, and the tool
is then not offered.
+
+A deployment can run jobs elsewhere instead: with ``PYTHON_JOB_RUNNER_URL``
+set, ``run_job`` POSTs each job there (``_run_remote``) and reads back the
+same result object the local worker prints, so files, caps, kept results and
+metering work the same. The runner is trusted to isolate jobs as the local
+sandbox does.
"""
from __future__ import annotations
import asyncio
import base64
+import binascii
import contextlib
import json
import logging
@@ -49,6 +56,8 @@ import uuid
from pathlib import Path
from typing import Any
+import httpx
+
logger = logging.getLogger(__name__)
TOOL_NAME = "run_python_job"
@@ -152,11 +161,15 @@ def _sandbox_dir() -> Path:
def available() -> bool:
- """Whether the sandbox is installed and jobs are on."""
+ """Whether jobs are on and something can run them: a remote runner
+ (``PYTHON_JOB_RUNNER_URL``) or the local sandbox."""
from openexecutive.config import get_settings
- if not get_settings().python_jobs_enabled:
+ settings = get_settings()
+ if not settings.python_jobs_enabled:
return False
+ if settings.python_job_runner_url:
+ return True
base = _sandbox_dir()
return (base / "deno").is_file() and (base / "pyodide" / "pyodide.mjs").is_file()
@@ -229,7 +242,16 @@ async def run_job(
"""Run one job in a fresh sandbox process, which exits when the job ends.
Never raises (cancellation aside, which kills the process). The reply
carries ``_usage`` ({peak_mb, cpu_ms}, either None when unknown), read
- from the process itself rather than from anything the job printed."""
+ from the process itself rather than from anything the job printed.
+ With ``PYTHON_JOB_RUNNER_URL`` set the job runs there instead."""
+ from openexecutive.config import get_settings
+
+ settings = get_settings()
+ if settings.python_job_runner_url:
+ return await _run_remote(
+ settings.python_job_runner_url, settings.python_job_runner_key or "",
+ code, files, timeout_s=timeout_s, memory_mb=memory_mb, inputs=inputs,
+ )
base = _sandbox_dir()
wheels = sorted(str(p) for p in (base / "wheels").glob("*.whl"))
job = json.dumps({
@@ -253,6 +275,91 @@ async def run_job(
await asyncio.to_thread(shutil.rmtree, deno_dir, True)
+def _runner_client(timeout_s: float) -> httpx.AsyncClient:
+ # Proxies from the environment are fine; redirects are not followed, so
+ # the job and the key go to the configured URL only.
+ return httpx.AsyncClient(timeout=httpx.Timeout(timeout_s, connect=10.0), follow_redirects=False)
+
+
+def _text(value: Any, limit: int) -> str | None:
+ return value[:limit] if isinstance(value, str) and value else None
+
+
+def _count(value: Any) -> int | None:
+ return value if isinstance(value, int) and not isinstance(value, bool) and 0 <= value < 2**40 else None
+
+
+async def _run_remote(
+ url: str, key: str, code: str, files: dict[str, bytes], *,
+ timeout_s: float, memory_mb: int, inputs: dict[str, Any] | None,
+) -> dict[str, Any]:
+ """Run one job on the remote runner. The request is the job the local
+ worker reads (less its sandbox paths) plus the time and memory limits;
+ the reply is the worker's result object, with ``usage`` ({peak_mb,
+ cpu_ms}) measured by the runner. Never raises (cancellation aside); the
+ reply is read only up to the cap a local job's stdout has, and only its
+ known fields are kept."""
+ job = {
+ "code": code,
+ "inputs": inputs or {},
+ "files": {n: base64.b64encode(b).decode() for n, b in files.items()},
+ "max_file_bytes": _MAX_OUTPUT_BYTES,
+ "max_total_bytes": _MAX_OUTPUT_BYTES,
+ "max_files": _MAX_OUTPUT_FILES,
+ "timeout_s": timeout_s,
+ "memory_mb": memory_mb,
+ }
+ usage: dict[str, int | None] = {"peak_mb": None, "cpu_ms": None}
+ try:
+ # The runner applies timeout_s to the job itself; the margin covers
+ # sending the files and getting the results back.
+ async with _runner_client(timeout_s + 60) as client, client.stream(
+ "POST", url, json=job, headers={"Authorization": f"Bearer {key}"},
+ ) as resp:
+ if resp.status_code != 200:
+ logger.warning("python job: runner answered HTTP %s", resp.status_code)
+ return {"error": f"the job runner could not run the job (HTTP {resp.status_code})", "_usage": usage}
+ raw = bytearray()
+ async for chunk in resp.aiter_bytes():
+ raw += chunk
+ if len(raw) > _MAX_STDOUT_BYTES:
+ return {"error": "the job's output was too large; nothing was kept", "_usage": usage}
+ except httpx.TimeoutException:
+ return {"error": f"the job ran past its {int(timeout_s)} s limit", "_usage": usage}
+ except httpx.HTTPError as exc:
+ logger.warning("python job: runner unreachable: %s", type(exc).__name__)
+ return {"error": "the job runner could not be reached", "_usage": usage}
+ try:
+ body = json.loads(raw)
+ except ValueError:
+ body = None
+ if not isinstance(body, dict):
+ return {"error": "the job runner sent back something unreadable", "_usage": usage}
+ reported = body.get("usage")
+ if not isinstance(reported, dict):
+ reported = {}
+ usage = {"peak_mb": _count(reported.get("peak_mb")), "cpu_ms": _count(reported.get("cpu_ms"))}
+ out: dict[str, Any] = {"_usage": usage}
+ for field, limit in (("result", _MAX_INPUTS_CHARS), ("error", 2000), ("printed", 8000)):
+ if (text := _text(body.get(field), limit)) is not None:
+ out[field] = text
+ sent = body.get("files")
+ if isinstance(sent, dict) and sent:
+ # The runner is another server: check every file is text that decodes
+ # as base64 before the handler decodes it for real.
+ try:
+ ok = all(isinstance(n, str) and isinstance(b, str) and base64.b64decode(b, validate=True) is not None
+ for n, b in sent.items())
+ except (binascii.Error, ValueError):
+ ok = False
+ if not ok:
+ return {"error": "the job runner sent back a file it couldn't read", "_usage": usage}
+ out["files"] = sent
+ if isinstance(body.get("skipped"), list):
+ out["skipped"] = [n for n in body["skipped"] if isinstance(n, str)][:_MAX_OUTPUT_FILES]
+ return out
+
+
def _proc_usage(pid: int) -> dict[str, int | None]:
"""The process's peak memory (VmHWM) and CPU time so far, from /proc;
None for what can't be read (it already exited, or no /proc)."""
diff --git a/packages/core/tests/unit/test_python_job.py b/packages/core/tests/unit/test_python_job.py
index 1a7cd9a3..c0c8c018 100644
--- a/packages/core/tests/unit/test_python_job.py
+++ b/packages/core/tests/unit/test_python_job.py
@@ -398,3 +398,116 @@ def test_the_real_sandbox_reads_inputs_and_reports_its_usage(company: Path) -> N
))
assert body["result"] == "42"
assert body["_usage"]["peak_mb"] > 50 and body["_usage"]["cpu_ms"] > 0
+
+
+# --------------------------------------------------------------------------- #
+# A remote runner (PYTHON_JOB_RUNNER_URL)
+# --------------------------------------------------------------------------- #
+
+
+def _runner(monkeypatch: pytest.MonkeyPatch, handler: Any) -> list[Any]:
+ import httpx
+
+ seen: list[Any] = []
+
+ def record(request: httpx.Request) -> httpx.Response:
+ seen.append(request)
+ return handler(request)
+
+ monkeypatch.setenv("PYTHON_JOB_RUNNER_URL", "https://runner.example.com/run")
+ monkeypatch.setenv("PYTHON_JOB_RUNNER_KEY", "k-123")
+ monkeypatch.setattr(
+ python_job, "_runner_client",
+ lambda timeout_s: httpx.AsyncClient(transport=httpx.MockTransport(record), timeout=timeout_s),
+ )
+ return seen
+
+
+def test_a_runner_makes_jobs_available_without_the_sandbox(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("PYTHON_SANDBOX_DIR", str(tmp_path / "nothing"))
+ monkeypatch.setenv("PYTHON_JOB_RUNNER_URL", "https://runner.example.com/run")
+ monkeypatch.setenv("PYTHON_JOB_RUNNER_KEY", "k-123")
+ assert python_job.available() is True
+ monkeypatch.setenv("PYTHON_JOBS_ENABLED", "false")
+ assert python_job.available() is False
+
+
+def test_the_job_goes_to_the_runner_and_its_result_comes_back(monkeypatch: pytest.MonkeyPatch) -> None:
+ import httpx
+
+ seen = _runner(monkeypatch, lambda r: httpx.Response(200, json={
+ "result": "3", "printed": "hi\n", "files": {"out.txt": "ZG9uZQ=="}, "skipped": ["big.bin"],
+ "usage": {"peak_mb": 120, "cpu_ms": 900}, "extra": "dropped",
+ }))
+ body = asyncio.run(python_job.run_job("1+2", {"a.csv": b"x"}, timeout_s=30, inputs={"n": 1}))
+ [req] = seen
+ assert req.headers["authorization"] == "Bearer k-123"
+ sent = json.loads(req.content)
+ assert sent["code"] == "1+2" and sent["inputs"] == {"n": 1} and sent["timeout_s"] == 30
+ assert base64.b64decode(sent["files"]["a.csv"]) == b"x"
+ assert "pyodide" not in sent and "wheels" not in sent
+ assert body == {
+ "result": "3", "printed": "hi\n", "files": {"out.txt": "ZG9uZQ=="}, "skipped": ["big.bin"],
+ "_usage": {"peak_mb": 120, "cpu_ms": 900},
+ }
+
+
[email protected](("response", "message"), [
+ ("status", "could not run the job (HTTP 503)"),
+ ("garbage", "unreadable"),
+ ("huge", "too large"),
+ ("timeout", "ran past its 30 s limit"),
+ ("down", "could not be reached"),
+ ("bad_file", "a file it couldn't read"),
+ ("bad_name", "a file it couldn't read"),
+])
+def test_a_runner_failure_is_a_job_error(monkeypatch: pytest.MonkeyPatch, response: str, message: str) -> None:
+ import httpx
+
+ def answer(request: httpx.Request) -> httpx.Response:
+ if response == "status":
+ return httpx.Response(503)
+ if response == "garbage":
+ return httpx.Response(200, content=b"[1, 2]")
+ if response == "huge":
+ return httpx.Response(200, content=b"x" * (python_job._MAX_STDOUT_BYTES + 1))
+ if response == "timeout":
+ raise httpx.ReadTimeout("slow", request=request)
+ if response == "bad_file":
+ return httpx.Response(200, json={"files": {"out.csv": "not-base64!"}})
+ if response == "bad_name":
+ return httpx.Response(200, json={"files": {"out.csv": 5}})
+ raise httpx.ConnectError("refused", request=request)
+
+ _runner(monkeypatch, answer)
+ body = asyncio.run(python_job.run_job("1", {}, timeout_s=30))
+ assert message in body["error"]
+ assert body["_usage"] == {"peak_mb": None, "cpu_ms": None}
+
+
+ "http://runner.example.com/run", "ftp://runner.example.com", "https://", "file:///etc/passwd",
+])
+def test_the_runner_url_must_be_https_or_private(monkeypatch: pytest.MonkeyPatch, url: str) -> None:
+ from openexecutive.config import Settings
+
+ monkeypatch.setenv("PYTHON_JOB_RUNNER_URL", url)
+ with pytest.raises(ValueError):
+ Settings() # type: ignore[call-arg]
+
+
+def test_plain_http_is_allowed_on_a_private_network(monkeypatch: pytest.MonkeyPatch) -> None:
+ from openexecutive.config import Settings
+
+ monkeypatch.setenv("PYTHON_JOB_RUNNER_URL", "http://runner.flycast/run")
+ monkeypatch.setenv("PYTHON_JOB_RUNNER_KEY", "k-123")
+ assert Settings().python_job_runner_url == "http://runner.flycast/run" # type: ignore[call-arg]
+
+
+def test_the_runner_url_needs_its_key(monkeypatch: pytest.MonkeyPatch) -> None:
+ from openexecutive.config import Settings
+
+ monkeypatch.setenv("PYTHON_JOB_RUNNER_URL", "https://runner.example.com/run")
+ monkeypatch.delenv("PYTHON_JOB_RUNNER_KEY", raising=False)
+ with pytest.raises(ValueError, match="PYTHON_JOB_RUNNER_KEY"):
+ Settings() # type: ignore[call-arg]