AI-powered virtual executive team — a single coherent executive persona backed by 8 specialist agents (FastAPI + Next.js).
perf(orchestrator): offer less common tools on demand and trim tool results and specialist lookups (#406)
Rufus303d45eaa0b2
diff --git a/.env.example b/.env.exampleindex 526767d5..efb330e1 100644--- a/.env.example+++ b/.env.example@@ -674,3 +674,14 @@ WATCHLIST_PROPOSAL_TTL_DAYS=14# collection shares the value above.KNOWLEDGE_DISTANCE_THRESHOLD=0.55# KNOWLEDGE_BUILTIN_DISTANCE_THRESHOLD=0.50++# How many knowledge chunks each specialist gets per question: built-in+# knowledge, and company documents (the company count also caps each synced+# source: Notion, Drive, OneDrive, Confluence). Every chunk is sent to the+# model, so higher counts cost more per answer.+# SPECIALIST_BUILTIN_N_RESULTS=3+# SPECIALIST_COMPANY_N_RESULTS=2++# Longest single tool result the model sees, in characters (about 4 per+# token). A longer one is cut with a note asking for a narrower request.+# TOOL_RESULT_MAX_CHARS=20000diff --git a/packages/core/openexecutive/architecture/architecture-facts.yaml b/packages/core/openexecutive/architecture/architecture-facts.yamlindex 72de2b27..36ba0e71 100644--- a/packages/core/openexecutive/architecture/architecture-facts.yaml+++ b/packages/core/openexecutive/architecture/architecture-facts.yaml@@ -259,6 +259,33 @@ caching:same prefix from its first round to its last. recall_history (Always inthe loop, orchestrator.history_tools) joins the same way whenhistory_tools.recall_person allows it, before the same sort.+ deferred_tool_groups: |+ The less common skill tools are not in the tool list. orchestrator.+ tool_groups.GROUPS puts them in named groups (calendar, people, goals,+ watch_and_alerts, workflows, documents, messaging, skills), and+ tool_groups.split takes them out of the turn's offered tools after every+ filter (workspace mode, unattended, private turn) and BEFORE the sort,+ adding two small tools in their place: open_tools(group), whose+ description names every offered group and its tools, and use_tool(name, input).+ open_tools is answered in the loop with the group's tools (name,+ description, input_schema) as a tool RESULT, never by changing the tool+ list, because the list is the start of the cached prefix: a list that+ grew mid-turn would miss the cache for the tools, the system blocks and+ the whole history. So the list is the same on every call of every turn,+ and a group's descriptions are paid for only on a turn that opens it.+ open_tools' description (tool_groups.open_tools_tool) and its results+ name only the tools the turn offers, so a private turn never sees a+ withheld tool's name; the description depends only on that set, fixed+ per kind of turn, so each kind keeps one stable list. A use_tool call is rewritten into the+ tool's own call (same tool_use id, the model's assistant block left as+ sent) before dispatch, so it meets the same guards (workspace mode,+ private turn, unattended, Take the lead), handler, progress events and+ audit rows as a direct call. use_tool accepts only names in+ tool_groups.DEFERRED; anything else, or input that is not an object,+ gets an error tool result. The everyday tools (consult_specialist,+ propose_form_values, people and memory lookups, follow-ups, the MCP+ meta-tools) stay direct. This cut the direct list from about 19.5k to+ about 6-7k tokens, which every model call of a turn re-sends.rag_context_placement: User turn (NOT the system prompt) — RAG hits vary per query, would poison the cache if cached.intra_turn_cache: |executive._apply_loop_cache_marker moves a single ephemeral marker onto@@ -277,7 +304,7 @@ caching:a marker nested inside tool_result["content"] would survive the gate andreach a backend that rejects the field.tool_result_bound: |- TOOL_RESULT_MAX_CHARS (default 50,000) caps one tool result before it+ TOOL_RESULT_MAX_CHARS (default 20,000, about 5k tokens) caps one tool result before itenters the prompt. Applied at the single point every result reaches themodel (the tool_results comprehension in _stream_agent_loop), not at eachproducer: that also covers specialist output, tool errors and the@@ -286,7 +313,9 @@ caching:Capping upstream of that parse would feed it truncated JSON and silentlydrop the form_patch SSE event.A circuit breaker, not a routine clipper: the default is set so ordinary- tool output never reaches it. A cut result carries an explicit marker+ tool output (a status, an email thread, a specialist's analysis) never+ reaches it; a long document read does, and the marker asks for a+ narrower request (a page range, a section). A cut result carries an explicit markernaming the tool and stating the result is incomplete — silent truncationis worse than none, because the model reads the cut text as the wholeanswer and confabulates the rest.diff --git a/packages/core/openexecutive/architecture/prebuilt/agents.json b/packages/core/openexecutive/architecture/prebuilt/agents.jsonindex c5fb8742..0d46e9ff 100644--- a/packages/core/openexecutive/architecture/prebuilt/agents.json+++ b/packages/core/openexecutive/architecture/prebuilt/agents.json@@ -1,7 +1,7 @@{"section_id": "agents","title": "Agent Council",- "markdown": "The Executive (`openexecutive.orchestrator.executive.Executive`) is a single coherent persona that fans out to specialist sub-agents via Anthropic tool use. Each specialist subclasses `openexecutive.agents.base.BaseAgent`, carrying a `name`, `domain`, and `model`, and exposes `analyze(...)`, which wraps the question in user-turn context and calls Claude through the per-call provider. That context is per call and never cached: `<company_stage>` (the stage from the company profile — specialists never see the profile itself), in solo mode `<principal_role>` (what kind of principal the advice is for, in plain words, with their title and remit — `router.principal_role_context`), `<department_memory>`, `<past_decisions>`, `<failure_cases>`, `<relevant_knowledge>` and `<conversation_context>`, outermost first. The specialist's own system prompt is a static domain constant under one `cache_control` block, so it stays byte-identical across companies and profile edits. A chat turn takes the stage from its session's profile (the one the Executive reasons over, and the one an eval scenario injects); workflow steps and the MCP server's `consult_specialist` read it fresh from the profile on disk. The role tag follows the turn's mode: a chat turn pins the role on its session with the mode (`pin_turn_principal_role`: the session's `principal_role` override, else the workspace settings) and sends it in solo and nothing in team, so the org block, every specialist in the turn and any workflow the turn starts see the same role; callers with no chat turn read it fresh (`load_principal_role`: the current session's mode and role, else the workspace's), and a failed read sends no tag rather than failing the consult. The CFO, CSO and sales prompts key their advice on it: for a bootstrapped, self-funded or solo company they drop venture fundraising benchmarks and work from cash, and raise only if the founder wants to.\n\nRouting uses one `consult_specialist` tool. Beside it the Executive carries its skill tools (`search_skills`, `load_skill`, `create_skill`, `update_skill`, `delete_skill` — shown to users as **playbooks**; `create_skill` / `update_skill` / `delete_skill` never change the library — they save a draft (`knowledge/skill_drafts.py`) that a person approves on the Playbooks tab, and the reply links to it; `update_skill` / `delete_skill` (and `create_skill`, for a name a workflow still follows) refuse any built-in name (including a company's customized copy) and any company playbook a workflow (even a switched-off custom one) follows — workflows read playbooks at run time, approved scheduled ones included, and this loop also runs on inbound email and chat, so only a person on the Playbooks tab can change what a workflow follows; a hidden built-in is invisible to search and load; each `search_skills` hit lists the `workflows` that follow it, and the persona routes \"the finished deliverable\" to that workflow and quick or partial asks to the playbook inline); Tool calls made while proactive work is tagged (scheduler dispatch, reflection, research, alert review) feed the Attunement outcome ledger. Goals have three tools: `list_department_goals`, `update_department_goal` (progress the user reports) and `create_goal` (a new goal the principal states, filed under an area by slug or title and creating the area when none matches — see Org). `create_goal` runs only for the principal on a verified surface, and no unattended run is offered it: the tools in `schedule_tools.UNATTENDED_WITHHELD_TOOLS` are dropped from reflection's and research's toolkits by `unattended_toolkit`, and from the chat loop when its Session is marked `unattended` — the scheduler's PROACTIVE TRIGGER run, whose prompt quotes stored intent text; a call the model emits anyway gets an error tool_result. `record_decision_outcome` records how a past decision turned out — the weekly review lists decisions older than 30 days with no outcome as `[decision N]` — writing `decisions.outcome` (`episodic.update_decision`) and auditing the rationale; it has `create_goal`'s gate (the principal on a verified surface only; everyone else gets an audited refusal) and is in `UNATTENDED_WITHHELD_TOOLS` too. A missing or unknown id returns the decisions still waiting on an outcome, so the model can retry with the right one. Its schema is static; the solo persona has one line on it, and team mode learns it from the tool description, so the team persona is byte-identical. Attunement adds `list_open_loops` (what a person owes — everyone's for the principal only in a private conversation, otherwise the caller's own) and `close_open_loop`, which only closes a loop when the turn's resolved speaker is the principal or the loop's owner — an unrostered sender, a signed-in web user not on the roster, and unattended sessions are refused. `assign_open_loop` opens one on request (\"ask Ben to send me the Q3 numbers by Friday\") for the principal or a rostered teammate on a verified surface — never an email, a turn private to the principal or an unattended run, which are also withheld the tool — and only when `task` appears verbatim in what the speaker typed this turn, at most 5 per turn; that turn's extraction pass then opens no second loop for the same owner. The roster write tools (`upsert_person`, `archive_person`, `set_department_head`) are the principal's alone, and only on a surface that verified the speaker — the web chat, the principal's own Slack or Discord, or a private Telegram chat when `TELEGRAM_WEBHOOK_SECRET` is set to a value Telegram can send (Telegram identifies the chat rather than the sender, and without that secret its webhook accepts anyone's update). A teammate, an email (even from the principal's address), Google Chat, the CLI, the MCP server and unattended runs get `status: refused` (audited, and no action chip), because a team member's People row is also the web sign-in and outbound-email allow-list. `upsert_person` takes `kind` (`team` / `contact`; a new person defaults to `contact` in solo mode, an update keeps the current kind) and `email_aliases` (other addresses the person writes from — the full list, replacing the current one; they match mail and may be emailed but never sign in, and one already on someone else is refused), and after every write any pending roster request whose sender now matches someone is closed and its held messages replayed. `resolve_roster_request` answers a **roster request** — someone not on the People list who wrote in and is waiting (see Org): `approve` (a new person; `kind` defaults to team for a company-domain sender, else contact, when the principal did not say), `link` (someone already listed, by `person_id`; `replace_channel_id` only when the principal said so) or `decline`. It has the roster tools' gate (the principal on a verified surface; audited refusal otherwise), is in `UNATTENDED_WITHHELD_TOOLS`, and answers only an id in `Session.trusted_roster_request_ids` — the ids the server printed that turn in a `<roster_requests>` block (`briefing.context.format_roster_requests_for_prompt`, up to 10 pending requests), which `render_and_trust` appends to the `<briefing>` digest on the principal's own verified turn only and records on the session in the same step, so an id the model invents or reads out of a message adds nobody. Each line is server-derived — id, channel, address or account id, the sender's name sanitised and marked unverified, a company-domain hint, the message count — and never what they wrote. The block rides in the **user turn**, like the digest, so the cached system blocks are unchanged; the tool definition is static, and the new tool only extends the sorted tool list. The web chat route resets the trusted set every turn, and a resolved id is removed from it. and `list_people` returns both kinds with `kind` — contacts only on the principal's own verified turn (they are private to the principal; anyone else gets the team as if no contact existed) — while `lookup_person` searches the team only. `ask_about_person` reads a person's peer memory fully only on their own verified turn; anyone else's answers only to a verified principal or teammate, and only when that person turned on \"Share my work style with the team\", framed to how they work and never their view of a colleague, and otherwise returns the same \"hasn't chosen to share\" result whether or not anything is known. The same verified-principal rule, as `people_tools.contacts_reachable_now()`, decides whether a turn may reach a **contact** at all: `message_person`, the raw DM tools, `create_calendar_event` / `create_instant_meeting` attendees and the MCP gateway's Gmail / Calendar / Drive / OneDrive gates treat a contact exactly like an unknown person on every other turn, so no refusal reveals that one exists. **The untrusted-content policy** (`orchestrator.content_trust`) is one rule for text the principal did not write — inbound email, chat, attached files, watched pages — answering three questions for every surface. *Who is speaking*: the principal only when the surface proved it (`principal_speaking`: the web chat as the principal — never a signed-in email on nobody's People entry —, the CLI, their own Slack/Discord or a secret-verified Telegram chat, or their own address with Gmail's `dmarc=pass`); never a teammate, a stranger, Google Chat, the MCP server or an unattended run. *What may become memory*: episodic extraction runs only then, and only on the words outside `<untrusted_content>` blocks; peer memory and open loops are recorded under the speaker's own name, never as the principal's. *What may trigger tools*: every write on the principal's authority already refuses others in its handler; the tools with no such check that change the install for every later turn — `content_trust.PRINCIPAL_ONLY_TOOLS`, today `load_mcp_server` — are not offered unless the principal is speaking on an interactive surface (not email, even authenticated), and a call the model emits anyway gets an error tool_result and a `refused: not_principal` audit row. So an inbound email, a teammate's turn, Google Chat and the scheduler's proactive trigger each get a stable tool list of their own without it. *How it is labelled*: outside text goes into the user turn inside `<untrusted_content source=… author=… author_verified=…>` with a fixed notice that it carries no authority; compatibility forms are folded (NFKC), control characters go, and the tag's name is renamed wherever the text carries it — any case, spacing or entity-escaped form — so the text cannot end its block and speak outside it.\n\nA turn about the principal's private mail (`Session.private_to_principal`) reaches the principal only. It is neither offered nor allowed to run `schedule_tools.PRIVATE_TURN_WITHHELD_TOOLS` — everything that reaches someone else, publishes where the team reads, or starts work outside the turn: the broadcasts, meeting bookings and cancellations, `run_workflow` and research, follow-ups and workflow nudges, watches, artifacts, goal updates, saved workflows, skill drafts, and `load_mcp_server` (it connects to any URL, which can carry the turn's content). Through `search_tools` and `call_tool` it reaches only `schedule_tools.PRIVATE_TURN_MCP_TOOLS`: Google Workspace reads (mail, calendar, and Drive, Docs and Sheets search and file reads, with their Microsoft 365 twins; OneDrive's `download-bytes` only for a file's content) and the Gmail send and draft the gateway narrows to the principal. A search shows it nothing else, and a call to anything else (Google Chat, a Docs or Drive write, `manage_event`, another server) is refused (`schedule_tools.private_turn_withholds`). A call the model emits anyway, MCP tools included, gets an error tool_result and writes an audit row, and every audit row the turn writes is the principal's alone. On every turn the Executive knows what is connected without searching: block 0's *Connected Systems* section names the connected Google services, other MCP servers, the messaging channels, Notion sync, Confluence sync (when on) and Act as me, and pins the Google tool names it calls directly with `call_tool`. `search_tools` is for anything not listed. When work repeats over many items, the Executive can also call `run_script` (`workflows/step_script.py`, offered whenever the gateway runs unless `CHAT_SCRIPTS=false`): one short Python script, run in a sandboxed Monty worker with no files, network or environment, whose only reach is the gateway tools `call_tool` could reach (those `search_tools` has returned; the gateway keeps that set for its own process-wide session, not per conversation). Each call it makes is a `call_tool` in every way that matters — the gateway's discovery, deny-list and recipient gates, Take the lead's gate on an unattended run, and the same action chip and `tool_invocation` row (marked `via: run_script`, `ok: false` when it raised), as each call happens; the script itself gets a `kind: script` row with its source and result. Its wall clock is half the turn's deadline, so a long script ends with the list of calls it made rather than being cut off with the turn, and a stopped turn closes it and its worker. A turn private to the principal is not offered it, and the Act as me lockdown withholds it whole. Besides gateway tools, a chat script may call a fixed list of the Executive's own tools (`step_script.CHAT_OWN_TOOLS`: reads, alerts, open loops, goals, the watch list, follow-ups, `message_person`, `upsert_person` — nothing that broadcasts, books calendars, edits memory or skills, or starts workflows or research), each only when the turn is offered it and through the turn's own handler (Take the lead's gate included), with the same chip and a `kind: skill`, `via: run_script` audit row. A direct call needs a `tool_use` each, so a turn's scripts share a budget instead: `CHAT_SCRIPT_MAX_CALLS` (default 200) calls in all, and per-turn caps on own tools that could otherwise act in bulk (`CHAT_OWN_TOOL_CAPS`: `message_person` 10 — messaging the whole roster would be a broadcast — `upsert_person` 25, `create_alert` 50); a refused call tells the model to say what is left. A call stopped mid-flight by the script's clock or the turn still gets a `cancelled` audit row, since it may have run. Because every round re-reads the conversation, the loop also nudges at the moment of choice: after a round whose results include a list of five or more items (`step_script.lists_many`), with `run_script` on offer and not used that round, it adds the constant `FANOUT_HINT` as a text block in that round's user message (never a cached block), once a turn, pointing the per-item work at one built tool. For library work on files (PDFs, spreadsheets, charts, images) the chat also has `run_python_job` (see Workflows → Python jobs): Python with a fixed library set in a Deno-isolated WebAssembly sandbox, fed Knowledge files by name, the message's own attachments as sent, or text it already holds, and able to keep a job that worked as a Custom tool, with result files linked for the principal to download; it is offered only on the principal's own turns and only where the sandbox is installed (not offered otherwise, so the tool prefix stays fixed per process). Agent Activity shows each run as one \"Built tool\" card (debug event `script_run`: ok, the calls it made, how many, how long, and the kept or reused tool's name). The persona's tool-building section is present only when chat scripts can actually run (the setting and Monty installed). `TOOL_BUILDING_ADDENDUM` in the cached persona tells the Executive when to build a tool — a job that looks something up and then checks or acts on each result, paged results, or more than about ten items, while a handful of items already named go as direct calls in one turn — and never to say it has none, and to talk about it as building a tool or doing it in one go. On the principal's own verified, interactive turns, a script that works can be saved as a named tool (`save_as`; elsewhere the script runs but is not saved) and run again with `run_script(tool=…, inputs=…)`; `list_saved_tools` (a constant definition, in `PRINCIPAL_ONLY_TOOLS` with `load_mcp_server`, so offered only while the principal is speaking on a verified, interactive surface; `run_script(tool=…)` is refused on any other turn) lists them, and each call a saved tool makes is a gateway call like any other (see Workflows → Saved tools). The tool's definition is a constant, so the cached tool prefix does not move. Multiple parallel `tool_use` blocks in a single assistant turn fan out concurrently through `router.route_parallel`, each running its own domain-filtered ChromaDB query before the specialist sees the question. The fan-out is failure-isolated per call: a specialist that raises or returns no text becomes an `UNAVAILABLE` tool_result (`router.specialist_unavailable_result`) instead of failing the whole turn, and its area — \"finance\", \"legal\", never the specialist's name — is reported to the user under the web reply, or by the Executive in its reply on other channels, unless another call to it answered that turn. A failed specialist does not count as consulted, so it takes no committee seat. Cancellation (Stop) still propagates. The Executive then synthesizes the returned analyses into one reply.\n\n`remember_fact`, `forget_fact` and `update_company_profile` (`orchestrator/fact_tools.py`) make a correction stick: the first two keep or retire a standing fact in `memory.facts`, and the third changes one field of the company profile. They share `record_decision_outcome`'s gate (the principal on a verified surface, an audited refusal for everyone else) and are in `UNATTENDED_WITHHELD_TOOLS`. `remember_fact` alone also admits a rostered teammate on a verified surface (web signed in, their own Slack or Discord): the fact is stored attributed (`recorded_by_role=\"teammate\"`, rendered `(per <name>)`), and is held as a `proposed` row (`awaiting_approval`) until the principal approves it, unless the principal marked that teammate trusted (\"needs my approval\" off; on by default) — and even then when it would replace the principal's fact. All three are also in `PRIVATE_TURN_WITHHELD_TOOLS`, since what they write or retire is read on everyone's turns. On the principal's own authenticated email (primary address, with Gmail's own `Authentication-Results` reporting `dmarc=pass`, not forwarded) the three tools check the request as usual, then hold it as `awaiting_confirmation`. It applies only when the principal replies to a one-time token emailed to their own address (`integrations.fact_confirmation`); the split into a check step and an `apply_*` step lets chat and the confirmed reply share one apply path. Each write also needs a `source_quote` found in the speaker's own words this turn. The active facts reach every specialist as a `<standing_facts>` user-turn tag: `route_to_specialist` reads them (None → the store, \"\" → none) for chat consults and workflow steps alike, and `BaseAgent.analyze` places the tag next to `<past_decisions>`, never in the cached specialist system prompt.\n\nThe ten entries in `SPECIALIST_REGISTRY` (`openexecutive.orchestrator.router`):\n\n| Key | Agent | Domain |\n| --- | --- | --- |\n| `cso` | `StrategyAgent` | strategy |\n| `cfo` | `FinanceAgent` | finance |\n| `chro` | `HRAgent` | hr |\n| `gc` | `LegalAgent` | legal |\n| `coo` | `OperationsAgent` | operations |\n| `cmo` | `MarketingAgent` | marketing |\n| `cpo` | `ProductAgent` | product |\n| `sales` | `SalesAgent` | sales |\n| `board_comms` | `BoardCommsAgent` | board |\n| `triage` | `TriageAgent` | triage |\n\nModels: specialists default to `claude-sonnet-5-5`; the deep-reasoning tier (CSO, CFO, GC, Board) runs on `DEEP_REASONING_MODEL`, `claude-sonnet-5-5` by default, with adaptive thinking; routing/fast classification uses `claude-haiku-5-5`. Requests are fitted to the model's generation on the way out (`feature_gate.fit_claude_generation`): 5-generation Claude models never receive `temperature` / `top_p` / `top_k`, and Haiku 5.5, which thinks unless told otherwise, gets thinking turned off when the caller asked for none. The Council's **Deep reasoning** toggle is honoured on Claude Opus/Sonnet (direct or via OpenRouter) and on any OpenRouter model whose live catalog entry supports `reasoning` (today all OpenAI, Gemini, DeepSeek and Grok entries; not Llama) — the request translator turns Anthropic adaptive thinking + effort into OpenRouter's `reasoning` parameter. It is a silent no-op on models that can't reason, and disabled for Haiku, which rejects adaptive thinking. A PDF `document` block is translated too: it becomes an OpenAI `file` part, and OpenRouter requests carrying one add the `file-parser` plugin (`native` for models whose catalog entry lists `file` input, else `PDF_OPENROUTER_ENGINE`, default `mistral-ocr`). `FeatureSpec.supports_pdf_input` gates it per model — off for a local server unless `LOCAL_PDF_INPUT` — and a gated-off PDF becomes a text note rather than vanishing. The Council picks a model in two steps: a **Provider** dropdown (Anthropic, OpenAI, Google, Meta, DeepSeek, xAI, Local, …), then a **Model** dropdown limited to that provider, with readable names (\"Claude Opus 5.5\") and the raw id and serving route shown underneath. The Anthropic group combines the direct Claude models (`claude-fable-5-1`, `claude-opus-5-5`, `claude-opus-5`, `claude-sonnet-5-5`, `claude-sonnet-5`, `claude-haiku-5-5`, `claude-haiku-4-5`) with any Claude generation that exists only on OpenRouter, and never lists a model twice. With `OPENROUTER_ENABLED`, the non-Anthropic entries come from OpenRouter's live `/models` catalog (fetched at startup, refreshed in the background, filtered to tool-capable paid text models from an allowlisted vendor set), falling back to a built-in snapshot if the fetch fails. Every Council agent also takes **additional instructions** (`agent_overrides.instructions`): text appended after its prompt by `agents.overrides.append_instructions` inside an `<additional_instructions>` block. `BaseAgent.base_system_prompt()` is the built-in prompt or its full replacement (what the Council's editor shows); `effective_system_prompt()` is that plus the instructions (what every call sends), and the Executive gets the same text through `build_system_blocks(persona_instructions=…)`. Unlike a replaced prompt, instructions leave the agent on its built-in prompt, so prompt improvements keep reaching it. A **Quality** choice above the agents (`agents.presets`) sets every agent's model and deep reasoning at once: **Fast** puts every agent on a small model (`claude-haiku-5-5`, else `claude-haiku-4-5`) with deep reasoning off, **Balanced** returns each agent to its built-in defaults, and **Thorough** puts the Executive and the domain specialists on the largest allowed model (`claude-opus-5-5`, else `claude-opus-5`) with deep reasoning on where the model supports it, leaving triage and the helpers below on their defaults. Presets change quality only; they never cap spend. Each preset resolves against the install's model allowlist at request time and is offered only when one of its models is allowed. Applying one writes ordinary per-agent overrides (model and deep reasoning only), so history, rollback and later per-agent edits keep working; an agent changed on its own afterwards shows as **Custom**. Each Council agent has a **visibility**: `core` for the Executive and the domain specialists (`BaseAgent.visibility = \"core\"`, which a new specialist must declare) and `internal` (the default) for triage and the helper agents below. Thorough moves only core agents. The Council opens in a **simple view**: the Quality choice, the voice picker, and only the core agents, each with its additional instructions and a model badge. **Show all agents** lists the internal ones too, and **Advanced** opens the full page (the browser remembers the choice). Eight Council-only agents are model/prompt-overridable in the UI but NOT callable via `consult_specialist`, because they are deliberately kept out of `SPECIALIST_REGISTRY`: `quality_judge` (committee review), `utility_fast` (gates, titles, decision parsing), `research` (the `executive_research` fan-out), `fixture_generator` (invents demo companies), `engagement_intake` (models a real client from intake material), `onboarding_interviewer` (interviews a new user about their own company and drafts its profile, people and departments), `workflow_designer` (runs the conversational New workflow wizard on `/jobs/new`: searches the available tools, asks clarifying questions, then drafts a validated dynamic-workflow definition for the user to review), and `workflow_actor` (runs a workflow's `action` steps: a bounded tool-use loop limited to the tools the user approved for that step). `fixture_generator`, `engagement_intake` and `onboarding_interviewer` share one shape — a forced tool call producing a whole company, validated for referential integrity, with a single repair retry — but opposite epistemics: the fixture author invents, the other two extract and leave unstated facts empty rather than guessing. When the workspace is in solo mode, `onboarding_interviewer`'s first user turn is prefixed with a short hint (`interview.solo_hint(role_kind)`, one static constant per role kind from the onboarding role step): the draft names only the principal, as principal, and no departments; an owner is asked about the offer, customers, pricing, cash and runway, an in-house executive about their organisation (drafted as the company — their employer), function, remit, manager, team size as context and what they are measured on, an independent about their practice, clients and pricing, and with no role given the role-neutral `SOLO_HINT` asks which they are. The hint rides in the user turn and the tool array is unchanged, so the cached system prompt and tool prefix are the same in both modes.\n\n**Voice.** The Executive answers in one voice persona, substituted into block 0 at `{VOICE_PERSONA}` (`personas/loader.get_active_body`) from the Executive override's `voice_persona_slug`. Three general voices ship in `personas/builtin/`: **Direct** (`default`, the slug used when none is set: answer first, brief and decisive), **Supportive** (a warm coach who explains the why and checks how the plan sits) and **Analytical** (numbers first, options and trade-offs, then one recommendation). They are offered by a shared picker in setup (`/onboard`, after the workspace choice) and in Settings → Executive, which saves through `PATCH /agents/executive`; a built-in voice's frontmatter carries the picker's one-line `description` and `sample` reply. The named-executive voices are marked `legacy: true`: the picker hides them, but they still load, so an install that picked one keeps it, and the Agent Council lists them under \"Legacy voices\". Voice files hold disposition only; the response-length ladder lives in `prompts/executive_persona.py`, and a test keeps numeric length rules out of every voice file. An eval scenario may set `voice_persona` to answer in a voice; it rides on `Session.voice_persona_slug`, which wins over the override for that session only.\n\nOne more structured-output utility agent sits beside `triage`: `openexecutive.agents.alert_review.AlertReviewAgent` (routing model, one forced `emit_alert_reviews` tool call, never raises). The alert review job hands it a batch of open alerts with evidence and gets back, per alert, a verdict (relevant / changed / resolved / stale), a confidence, the evidence it relied on, a one-line note, why-now, an optional deadline, and a recommended move (none / route / nudge / escalate / draft / suggest_workflow / merge / close) with its parameters. Like `triage`, it never mutates state — `alerts.review.apply_verdict` executes the verdicts deterministically within authority, so the model's job is judgment, not action.\n\n**Solo mode.** When the workspace (or the turn's session) is solo — one person using Open Executive for themselves — the Executive works for that one person, the principal, whatever their role: they may run their own business, lead a function inside a larger organisation, or work independently, and the persona reads which from the context instead of assuming it — the role lines under \"Your Principal\" in the org block, when the principal gave their role (workspace settings: kind, title, reports to, remit, measured on). Its persona swaps the team sections (\"You Are a Member of This Executive Team\", \"Choosing Who to Tell\", \"Departments & Org Coordination\") for \"You Work for One Person\", \"The People in Their World\", \"Who Hears From You\" and \"Goals and Areas\". Solo means only the principal uses Open Executive — not that they have no people: their manager, direct reports, peers, clients, investors and board are contacts. The Executive helps the principal work with them (a 1:1 or review prep, an update or budget case for their manager or board, a negotiation) but does not coordinate them, contacts them only when the principal asks and only if they are a contact, and never offers to loop the principal in. Every specialist stays relevant — HR for the principal's own reports, board and investor communications for reporting upward. Goals are grouped by *area* (the tools' `department_slug` is the area's slug). The chat loop withholds `send_department_message`, `send_company_broadcast` and `set_department_head` from its tool list, and a call to one of them anyway gets a JSON error tool_result instead of running. The mode is resolved once per turn and pinned on the session, so the persona, the org block, the tools and every tool handler agree — a switch made mid-turn applies from the next turn. Team mode is unchanged.\n\nEvery specialist *advises* — each returns domain analysis the Executive synthesizes into one voice. None of them act on the company's behalf; the side-effecting tools in the main loop (schedule, people, departments, broadcast, watchlist, workflows) belong to the Executive, not to a specialist.\n\n`recall_history` (Always in the loop, `orchestrator.history_tools`) reads the speaker's own notes, on the same terms as `ghostwrite_email` below: its own registry, offered only on a verified turn in a conversation private to them with their switch on; its `due` flag lists only dated notes, soonest first (see Memory). The ghostwriter below also gets the writer's notes from mail every recipient was on, as a consistency check (`<writer_noted>`).\n\nThe morning reflection, which has no question to look up and no specialist to consult, gets one more read-only tool of its own: `search_knowledge` (`orchestrator.knowledge_tools`), the same knowledge lookup a chat turn makes, for what the company's documents say about a customer, contract, price or policy before it acts.\n\nA fifth, `remind_me` (`orchestrator.reminder_tools`), sets a plain-text reminder sent to the speaker alone when it is due; it opens no mailbox, so it never locks a turn. A sixth, `propose_actions` (`orchestrator.action_card_tools`), does nothing itself: it leaves the speaker a card of up to five actions (a meeting, a message, a new contact) that happen only when they tap Approve, so it stays on after a turn read mail; it is where what an email itself asks for goes, since that is never done on the email's say-so. Four read-only tools beside it read the speaker's own mailbox on the same terms — `search_my_email`, `read_my_email`, `read_my_email_attachment`, `my_email_awaiting_reply` (`orchestrator.mail_read_tools`) — capped per turn, other people's messages and attached files returned as untrusted blocks (see Act as me). One tool writes under someone else's name, and it is fenced in code rather than the prompt. `ghostwrite_email` (**Act as me**, `orchestrator.delegation_tools`) writes an email as the person the Executive is speaking with and saves it as a draft in their own mailbox, Gmail or Outlook (`delegation.gmail.gmail_for` picks the client by the person's credential); chat never sends it (the person sends it themselves, or taps Send on its card). It lives in its own registry, never `_ALL_SKILL_TOOLS`, so reflection, research and workflow toolkits can never carry it. The chat loop adds it only on a turn `delegation.settings.pin_turn_delegation` offered it to — the principal, or a team member once the owner lets them, with it on, verified, in a conversation private to them (the web chat, a Slack or Discord DM, a verified private Telegram chat), on an attended turn — and builds that turn's handler map from both registries, so on any other turn a call to it is an unknown tool. From the round that calls it until the turn ends, the dispatch guard refuses only what opens an outside address, runs a script or workflow, or posts to everyone (`delegation.lockdown`), without changing the offered list; messages and invites to the roster, follow-ups, facts in the person's own words and a contact they name still run, each fenced in its own path. A later turn in the conversation stays private to its owner but holds nothing: the person asking again, having read the reply, is their own request. A contact add the lockdown refuses becomes an approval card in the chat instead (`propose_actions`). Recipients are chosen in code, never by the model (a draft's new email may go to any address, named back to check when it is neither on the roster nor typed), and the text comes from the **ghostwriter** (`delegation.ghostwriter`), a tool-less call with a constant prompt plus the person's \"How I write\" profile. The ghostwriter is deliberately not a Council agent: the Council's edit routes have no principal check. See **Act as Me (Delegation)**. A second tool joins the same way: `recall_history` (`orchestrator.history_tools`, Always in the loop) reads the speaker's own notes of what they said, by email or in chat. It has its own registry and per-turn handler map too, and the loop adds it only when `history_tools.recall_person` finds the speaker verified, in a conversation private to them, on an interactive turn that isn't private to the principal, with their notes switch on. It doesn't need Act as me. After every chat turn the Executive also hands the speaker's own words to `history_chat.schedule_chat_notes`, which takes notes in the background when their switch is on and the surface verified them (see **Memory**).\n\nThe Executive can also launch *any* workflow conversationally: `orchestrator.workflow_run_tools` adds `list_workflows` (the launchable catalog with each workflow's inputs) and `run_workflow` (runs a built-in or custom workflow to completion and returns its artifact, checkpointing approval gates as `awaiting_human`). `run_workflow` runs the weekly review, and in solo mode the morning brief, only for the principal on a verified surface (`Workflow.principal_only_modes`, `create_goal`'s rule); anyone else gets an audited refusal that does not name the rule. A run whose `result` event reports `private_to_principal` (the morning brief on the principal's own verified turn, when it read their private rows) still returns its artifact into that turn, but the stored run keeps `persistence.PRIVATE_RUN_ARTIFACT` instead of the text, since run history is readable by everyone signed in. This makes the `/jobs` catalog reachable from chat — see the Workflows section.\n\nDeliverables are first-class too. `orchestrator.artifact_tools` gives the Executive three tools: `draft_artifact` publishes a memo, web page (`html`), Word document (`docx`), spreadsheet (`xlsx`, from structured `sheets`) or a `link` to something it created in one of the principal's connected apps through the generic MCP gateway (`search_tools` → `call_tool`, any server — no vendor is named), and returns the artifact's id and link; `list_artifacts` and `get_artifact` let it find and reread its own past work (drafts and workflow outputs alike, through the same `artifact_records` read model the API uses). Every artifact belongs to someone: a draft to the person whose turn published it (`artifact_records.current_viewer`; with nobody speaking, the scheduler or the CLI, the principal), and a run someone started by hand to them, while scheduled runs are the team's. The tools, the recall, `message_person`'s link and mail attachments only ever reach the speaker's own (a link is refused when the recipient couldn't open it), not even the principal sees a teammate's, and a speaker who isn't on the People list publishes nothing. A workflow's own session inside someone's turn (the research synthesis) is pinned to that turn's viewer (`Session.documents_viewer`). Revising is `draft_artifact(supersedes=<id>)`, which archives the earlier version. When the principal asks for a deliverable the tool is always used; the \"quiet is the right default\" bar applies only to unprompted drafts. The format enum is a hard-coded sorted tuple so the cached tool schema stays byte-stable. Delivery: `message_person(person_id, text, artifact_id?)` appends the artifact's title and a `UI_BASE_URL/artifacts/<id>` link (through the same outbound guard as any DM; the link needs Open Executive access), and a Gmail send/draft can attach the file itself by `artifact_id` — see Integrations.\n\nDocuments come in the other way through `read_document` (`orchestrator.document_tools`): it reads a PDF, Word, Excel, CSV, Markdown or text file by `path` — one another tool saved, such as a Gmail attachment (`get_gmail_attachment_content`) or a Drive file (`get_drive_file_download_url`) — or a company document by `filename`, with an optional PDF `pages` range. A scanned PDF is converted (`knowledge.pdf_reader`: local OCR by default; with `PDF_PROVIDER_READING=true`, the deployment's own model through its provider's PDF support, falling back to local OCR) rather than returned empty. Reads are confined to the tool download folders — the same resolve-then-contain check as `oe__read_file` — plus the company docs folder, and it is withheld on turns private to the principal (`PRIVATE_TURN_WITHHELD_TOOLS`): those come from a contact's email, whose own attachments the poller has already read into the turn, and must not steer the Executive into company documents.\n\nThe Executive also backs the **Ask OE** panel's form-fill tier via `orchestrator.form_tools.propose_form_values`: when a chat turn arrives with a `<page_context>` block describing a form on the user's screen, the Executive can propose values for it. The proposal is delivered to the UI as a `form_patch` SSE event and placed into the form as highlighted suggestions — the tool never saves or submits anything; the user reviews and saves manually. The tool is **always** in the tool list (tool definitions sit at the top of the cached prompt prefix, so a conditionally-present tool would invalidate the prompt cache whenever panel and non-panel turns alternate); its description restricts usage to turns that carry a form descriptor.\n\n**Take the lead** (`orchestrator.take_the_lead`) is how the Executive acts on its own, behind one gate. It has two switches, both off until someone turns them on. **As the Executive** is the principal's alone (`PUT /take-the-lead`). With it on, the unattended passes (reflection, research, the scheduler's proactive trigger) keep the acting tools solo mode otherwise withholds from them: booking, starting workflows, and messaging teammates rather than only the principal. `schedule_tools.unattended_toolkit` and the chat loop's unattended path then wrap every acting tool (`ACTING_TOOLS`, plus any MCP tool whose name sends, creates, updates, deletes or shares) in `gated_handlers` / `gated_call_tool`. The principal-only tools (`UNATTENDED_WITHHELD_TOOLS`) stay withheld, and every guard inside a handler still applies, including department approval levels (a booking in a department on Proposes is still a proposal) and the contact-egress rule. New mail or chat (any turn with an `origin_channel`) wakes the existing reflection early (`take_the_lead.wake`): one run waiting at most, five minutes after the first message and no more than every 30 minutes, as a one-off `take_the_lead_wake` scheduled action. Pause holds it like any scheduled work. **As you** is the Handle it for me dial's top step (delegation section).\n\n**The gate** is plain code (`take_the_lead.check`). Six kinds of action wait for a yes first, each with an Ask first switch, all on to start. The switch is the decision ledger's own mode for the class `take_the_lead_action:<kind>` (`propose` asks first, the default; `auto_execute` doesn't). The kinds are money, contracts and legal, hiring and people decisions, deleting/cancelling/sharing, someone new (an address on neither the roster nor the contacts) and big sends (a broadcast or department message, or more than five people). On top of these, the company (set by the principal) and each person (for their own As you) add structured rules: a person or address, a domain, words, or an amount at or above a figure. Added rules always hold, and an error reading them holds the action. The one thing the gate lets straight through is a direct message (`message_person` or a Slack, Telegram or Discord DM) whose only recipient is the principal (`_only_to_principal`): they are who it would wait for, so a card would only ask them to approve a message to themselves. Anything that reaches or acts on anyone else is checked, and a roster that can't be read is not the principal. A held action becomes a `take_the_lead_action` decision instance with a companion `DECISION_ALERT_SOURCE` card on Today, routed to the first teammate holding one of its People approval ranges (`take_the_lead.approval_ranges`, plain code, closest fit first: money to `spend_lt_2k`, `spend_lt_10k` or `spend_gt_10k` by its largest amount, with a higher range able to approve a lower one and no amount read as large, or to `customer_credit` for credit and payment terms; vendor contracts to `vendor_onboarding` then `legal_sign`, other contracts to `legal_sign`; hiring to `hiring_signoff`; anything else about the board or investors to `board_comms`; in Personal as in team mode), or else the principal through the authority gate's fallback (`departments.authority._route_proposal`). Deletes and shares always go to the principal. Only the principal and that approver see or resolve it (`DecisionClassSpec.principal_and_approver`). Approving it claims the row and carries out the exact call once (`take_the_lead.carry_out`: the unwrapped handler, or the gateway's `call_tool`, on an unattended session, with contact egress only when the principal approves). Declining drops it. Everything done or held is a row in `take_the_lead_log`. What it did and what was approved show in Recent activity (`today._build_activity`, kind `took_the_lead`), named but never quoted, since that feed is the team's. The held card's own `decision_resolved` row is left out because its payload quotes the message. `GET /today/activity` also adds the caller's own Handle it sends (kind `sent_as_you`, actor `you`), which are never in the shared feed.\n\n**In training.** The As the Executive switch has three positions: Off, **In training** and On (`take_the_lead.training`). In training, an action the gate would let through still waits, as a `training` hold for the principal alone, unless the principal has allowed that action with that person or thing (`take_the_lead.allowance`: \"Message Priya Nair\" covers a message to her on any channel; \"Book meetings with Priya Nair and Sam Lee\" meetings with exactly them; a workflow by id; a connected tool by name and the file it names, never one that reaches people such as a mail send or a share). The card carries Approve, **Approve + allow**, **Edit** (the message, a meeting's title, times or description, a task and its due date: `take_the_lead.apply_edits`) and Decline. Allowing is the principal's alone and needs a request the API can tie to them, as turning it on does; the six kinds and the added rules hold whatever is allowed. An edit approved with \"Do it like this next time\" is kept as an example (only the edited text, the principal's own words), and `take_the_lead.learned_note` hands the examples and the fact it's in training to the reflection and research passes in their user turn. After `SUGGEST_AFTER` (3) unchanged approvals of the same action in 30 days, Settings suggests allowing it. What it's learned is one list keyed by feature (`take_the_lead_allowed.feature`), so other features that learn from approvals can join it; each keeps its own switch. Act as me's Handle it for me is the second (`act_as_me`, per person, `delegation.training`): its rows carry the person's id, are listed and removed by that person alone, count toward their own cap, and never reach `learned_note`.",+ "markdown": "The Executive (`openexecutive.orchestrator.executive.Executive`) is a single coherent persona that fans out to specialist sub-agents via Anthropic tool use. Each specialist subclasses `openexecutive.agents.base.BaseAgent`, carrying a `name`, `domain`, and `model`, and exposes `analyze(...)`, which wraps the question in user-turn context and calls Claude through the per-call provider. That context is per call and never cached: `<company_stage>` (the stage from the company profile — specialists never see the profile itself), in solo mode `<principal_role>` (what kind of principal the advice is for, in plain words, with their title and remit — `router.principal_role_context`), `<department_memory>`, `<past_decisions>`, `<failure_cases>`, `<relevant_knowledge>` and `<conversation_context>`, outermost first. The specialist's own system prompt is a static domain constant under one `cache_control` block, so it stays byte-identical across companies and profile edits. A chat turn takes the stage from its session's profile (the one the Executive reasons over, and the one an eval scenario injects); workflow steps and the MCP server's `consult_specialist` read it fresh from the profile on disk. The role tag follows the turn's mode: a chat turn pins the role on its session with the mode (`pin_turn_principal_role`: the session's `principal_role` override, else the workspace settings) and sends it in solo and nothing in team, so the org block, every specialist in the turn and any workflow the turn starts see the same role; callers with no chat turn read it fresh (`load_principal_role`: the current session's mode and role, else the workspace's), and a failed read sends no tag rather than failing the consult. The CFO, CSO and sales prompts key their advice on it: for a bootstrapped, self-funded or solo company they drop venture fundraising benchmarks and work from cash, and raise only if the founder wants to.\n\nRouting uses one `consult_specialist` tool. Beside it the Executive carries its skill tools (`search_skills`, `load_skill`, `create_skill`, `update_skill`, `delete_skill` — shown to users as **playbooks**; `create_skill` / `update_skill` / `delete_skill` never change the library — they save a draft (`knowledge/skill_drafts.py`) that a person approves on the Playbooks tab, and the reply links to it; `update_skill` / `delete_skill` (and `create_skill`, for a name a workflow still follows) refuse any built-in name (including a company's customized copy) and any company playbook a workflow (even a switched-off custom one) follows — workflows read playbooks at run time, approved scheduled ones included, and this loop also runs on inbound email and chat, so only a person on the Playbooks tab can change what a workflow follows; a hidden built-in is invisible to search and load; each `search_skills` hit lists the `workflows` that follow it, and the persona routes \"the finished deliverable\" to that workflow and quick or partial asks to the playbook inline); Tool calls made while proactive work is tagged (scheduler dispatch, reflection, research, alert review) feed the Attunement outcome ledger. Goals have three tools: `list_department_goals`, `update_department_goal` (progress the user reports) and `create_goal` (a new goal the principal states, filed under an area by slug or title and creating the area when none matches — see Org). `create_goal` runs only for the principal on a verified surface, and no unattended run is offered it: the tools in `schedule_tools.UNATTENDED_WITHHELD_TOOLS` are dropped from reflection's and research's toolkits by `unattended_toolkit`, and from the chat loop when its Session is marked `unattended` — the scheduler's PROACTIVE TRIGGER run, whose prompt quotes stored intent text; a call the model emits anyway gets an error tool_result. `record_decision_outcome` records how a past decision turned out — the weekly review lists decisions older than 30 days with no outcome as `[decision N]` — writing `decisions.outcome` (`episodic.update_decision`) and auditing the rationale; it has `create_goal`'s gate (the principal on a verified surface only; everyone else gets an audited refusal) and is in `UNATTENDED_WITHHELD_TOOLS` too. A missing or unknown id returns the decisions still waiting on an outcome, so the model can retry with the right one. Its schema is static; the solo persona has one line on it, and team mode learns it from the tool description, so the team persona is byte-identical. Attunement adds `list_open_loops` (what a person owes — everyone's for the principal only in a private conversation, otherwise the caller's own) and `close_open_loop`, which only closes a loop when the turn's resolved speaker is the principal or the loop's owner — an unrostered sender, a signed-in web user not on the roster, and unattended sessions are refused. `assign_open_loop` opens one on request (\"ask Ben to send me the Q3 numbers by Friday\") for the principal or a rostered teammate on a verified surface — never an email, a turn private to the principal or an unattended run, which are also withheld the tool — and only when `task` appears verbatim in what the speaker typed this turn, at most 5 per turn; that turn's extraction pass then opens no second loop for the same owner. The roster write tools (`upsert_person`, `archive_person`, `set_department_head`) are the principal's alone, and only on a surface that verified the speaker — the web chat, the principal's own Slack or Discord, or a private Telegram chat when `TELEGRAM_WEBHOOK_SECRET` is set to a value Telegram can send (Telegram identifies the chat rather than the sender, and without that secret its webhook accepts anyone's update). A teammate, an email (even from the principal's address), Google Chat, the CLI, the MCP server and unattended runs get `status: refused` (audited, and no action chip), because a team member's People row is also the web sign-in and outbound-email allow-list. `upsert_person` takes `kind` (`team` / `contact`; a new person defaults to `contact` in solo mode, an update keeps the current kind) and `email_aliases` (other addresses the person writes from — the full list, replacing the current one; they match mail and may be emailed but never sign in, and one already on someone else is refused), and after every write any pending roster request whose sender now matches someone is closed and its held messages replayed. `resolve_roster_request` answers a **roster request** — someone not on the People list who wrote in and is waiting (see Org): `approve` (a new person; `kind` defaults to team for a company-domain sender, else contact, when the principal did not say), `link` (someone already listed, by `person_id`; `replace_channel_id` only when the principal said so) or `decline`. It has the roster tools' gate (the principal on a verified surface; audited refusal otherwise), is in `UNATTENDED_WITHHELD_TOOLS`, and answers only an id in `Session.trusted_roster_request_ids` — the ids the server printed that turn in a `<roster_requests>` block (`briefing.context.format_roster_requests_for_prompt`, up to 10 pending requests), which `render_and_trust` appends to the `<briefing>` digest on the principal's own verified turn only and records on the session in the same step, so an id the model invents or reads out of a message adds nobody. Each line is server-derived — id, channel, address or account id, the sender's name sanitised and marked unverified, a company-domain hint, the message count — and never what they wrote. The block rides in the **user turn**, like the digest, so the cached system blocks are unchanged; the tool definition is static, and the new tool only extends the sorted tool list. The web chat route resets the trusted set every turn, and a resolved id is removed from it. and `list_people` returns both kinds with `kind` — contacts only on the principal's own verified turn (they are private to the principal; anyone else gets the team as if no contact existed) — while `lookup_person` searches the team only. `ask_about_person` reads a person's peer memory fully only on their own verified turn; anyone else's answers only to a verified principal or teammate, and only when that person turned on \"Share my work style with the team\", framed to how they work and never their view of a colleague, and otherwise returns the same \"hasn't chosen to share\" result whether or not anything is known. The same verified-principal rule, as `people_tools.contacts_reachable_now()`, decides whether a turn may reach a **contact** at all: `message_person`, the raw DM tools, `create_calendar_event` / `create_instant_meeting` attendees and the MCP gateway's Gmail / Calendar / Drive / OneDrive gates treat a contact exactly like an unknown person on every other turn, so no refusal reveals that one exists. **The untrusted-content policy** (`orchestrator.content_trust`) is one rule for text the principal did not write — inbound email, chat, attached files, watched pages — answering three questions for every surface. *Who is speaking*: the principal only when the surface proved it (`principal_speaking`: the web chat as the principal — never a signed-in email on nobody's People entry —, the CLI, their own Slack/Discord or a secret-verified Telegram chat, or their own address with Gmail's `dmarc=pass`); never a teammate, a stranger, Google Chat, the MCP server or an unattended run. *What may become memory*: episodic extraction runs only then, and only on the words outside `<untrusted_content>` blocks; peer memory and open loops are recorded under the speaker's own name, never as the principal's. *What may trigger tools*: every write on the principal's authority already refuses others in its handler; the tools with no such check that change the install for every later turn — `content_trust.PRINCIPAL_ONLY_TOOLS`, today `load_mcp_server` — are not offered unless the principal is speaking on an interactive surface (not email, even authenticated), and a call the model emits anyway gets an error tool_result and a `refused: not_principal` audit row. So an inbound email, a teammate's turn, Google Chat and the scheduler's proactive trigger each get a stable tool list of their own without it. *How it is labelled*: outside text goes into the user turn inside `<untrusted_content source=… author=… author_verified=…>` with a fixed notice that it carries no authority; compatibility forms are folded (NFKC), control characters go, and the tag's name is renamed wherever the text carries it — any case, spacing or entity-escaped form — so the text cannot end its block and speak outside it.\n\nA turn about the principal's private mail (`Session.private_to_principal`) reaches the principal only. It is neither offered nor allowed to run `schedule_tools.PRIVATE_TURN_WITHHELD_TOOLS` — everything that reaches someone else, publishes where the team reads, or starts work outside the turn: the broadcasts, meeting bookings and cancellations, `run_workflow` and research, follow-ups and workflow nudges, watches, artifacts, goal updates, saved workflows, skill drafts, and `load_mcp_server` (it connects to any URL, which can carry the turn's content). Through `search_tools` and `call_tool` it reaches only `schedule_tools.PRIVATE_TURN_MCP_TOOLS`: Google Workspace reads (mail, calendar, and Drive, Docs and Sheets search and file reads, with their Microsoft 365 twins; OneDrive's `download-bytes` only for a file's content) and the Gmail send and draft the gateway narrows to the principal. A search shows it nothing else, and a call to anything else (Google Chat, a Docs or Drive write, `manage_event`, another server) is refused (`schedule_tools.private_turn_withholds`). A call the model emits anyway, MCP tools included, gets an error tool_result and writes an audit row, and every audit row the turn writes is the principal's alone. On every turn the Executive knows what is connected without searching: block 0's *Connected Systems* section names the connected Google services, other MCP servers, the messaging channels, Notion sync, Confluence sync (when on) and Act as me, and pins the Google tool names it calls directly with `call_tool`. `search_tools` is for anything not listed. When work repeats over many items, the Executive can also call `run_script` (`workflows/step_script.py`, offered whenever the gateway runs unless `CHAT_SCRIPTS=false`): one short Python script, run in a sandboxed Monty worker with no files, network or environment, whose only reach is the gateway tools `call_tool` could reach (those `search_tools` has returned; the gateway keeps that set for its own process-wide session, not per conversation). Each call it makes is a `call_tool` in every way that matters — the gateway's discovery, deny-list and recipient gates, Take the lead's gate on an unattended run, and the same action chip and `tool_invocation` row (marked `via: run_script`, `ok: false` when it raised), as each call happens; the script itself gets a `kind: script` row with its source and result. Its wall clock is half the turn's deadline, so a long script ends with the list of calls it made rather than being cut off with the turn, and a stopped turn closes it and its worker. A turn private to the principal is not offered it, and the Act as me lockdown withholds it whole. Besides gateway tools, a chat script may call a fixed list of the Executive's own tools (`step_script.CHAT_OWN_TOOLS`: reads, alerts, open loops, goals, the watch list, follow-ups, `message_person`, `upsert_person` — nothing that broadcasts, books calendars, edits memory or skills, or starts workflows or research), each only when the turn is offered it and through the turn's own handler (Take the lead's gate included), with the same chip and a `kind: skill`, `via: run_script` audit row. A direct call needs a `tool_use` each, so a turn's scripts share a budget instead: `CHAT_SCRIPT_MAX_CALLS` (default 200) calls in all, and per-turn caps on own tools that could otherwise act in bulk (`CHAT_OWN_TOOL_CAPS`: `message_person` 10 — messaging the whole roster would be a broadcast — `upsert_person` 25, `create_alert` 50); a refused call tells the model to say what is left. A call stopped mid-flight by the script's clock or the turn still gets a `cancelled` audit row, since it may have run. Because every round re-reads the conversation, the loop also nudges at the moment of choice: after a round whose results include a list of five or more items (`step_script.lists_many`), with `run_script` on offer and not used that round, it adds the constant `FANOUT_HINT` as a text block in that round's user message (never a cached block), once a turn, pointing the per-item work at one built tool. For library work on files (PDFs, spreadsheets, charts, images) the chat also has `run_python_job` (see Workflows → Python jobs): Python with a fixed library set in a Deno-isolated WebAssembly sandbox, fed Knowledge files by name, the message's own attachments as sent, or text it already holds, and able to keep a job that worked as a Custom tool, with result files linked for the principal to download; it is offered only on the principal's own turns and only where the sandbox is installed (not offered otherwise, so the tool prefix stays fixed per process). Agent Activity shows each run as one \"Built tool\" card (debug event `script_run`: ok, the calls it made, how many, how long, and the kept or reused tool's name). The persona's tool-building section is present only when chat scripts can actually run (the setting and Monty installed). `TOOL_BUILDING_ADDENDUM` in the cached persona tells the Executive when to build a tool — a job that looks something up and then checks or acts on each result, paged results, or more than about ten items, while a handful of items already named go as direct calls in one turn — and never to say it has none, and to talk about it as building a tool or doing it in one go. On the principal's own verified, interactive turns, a script that works can be saved as a named tool (`save_as`; elsewhere the script runs but is not saved) and run again with `run_script(tool=…, inputs=…)`; `list_saved_tools` (a constant definition, in `PRINCIPAL_ONLY_TOOLS` with `load_mcp_server`, so offered only while the principal is speaking on a verified, interactive surface; `run_script(tool=…)` is refused on any other turn) lists them, and each call a saved tool makes is a gateway call like any other (see Workflows → Saved tools). The tool's definition is a constant, so the cached tool prefix does not move. Multiple parallel `tool_use` blocks in a single assistant turn fan out concurrently through `router.route_parallel`, each running its own domain-filtered ChromaDB query (3 built-in and 2 company chunks by default, see the RAG section) before the specialist sees the question. The fan-out is failure-isolated per call: a specialist that raises or returns no text becomes an `UNAVAILABLE` tool_result (`router.specialist_unavailable_result`) instead of failing the whole turn, and its area — \"finance\", \"legal\", never the specialist's name — is reported to the user under the web reply, or by the Executive in its reply on other channels, unless another call to it answered that turn. A failed specialist does not count as consulted, so it takes no committee seat. Cancellation (Stop) still propagates. The Executive then synthesizes the returned analyses into one reply.\n\n`remember_fact`, `forget_fact` and `update_company_profile` (`orchestrator/fact_tools.py`) make a correction stick: the first two keep or retire a standing fact in `memory.facts`, and the third changes one field of the company profile. They share `record_decision_outcome`'s gate (the principal on a verified surface, an audited refusal for everyone else) and are in `UNATTENDED_WITHHELD_TOOLS`. `remember_fact` alone also admits a rostered teammate on a verified surface (web signed in, their own Slack or Discord): the fact is stored attributed (`recorded_by_role=\"teammate\"`, rendered `(per <name>)`), and is held as a `proposed` row (`awaiting_approval`) until the principal approves it, unless the principal marked that teammate trusted (\"needs my approval\" off; on by default) — and even then when it would replace the principal's fact. All three are also in `PRIVATE_TURN_WITHHELD_TOOLS`, since what they write or retire is read on everyone's turns. On the principal's own authenticated email (primary address, with Gmail's own `Authentication-Results` reporting `dmarc=pass`, not forwarded) the three tools check the request as usual, then hold it as `awaiting_confirmation`. It applies only when the principal replies to a one-time token emailed to their own address (`integrations.fact_confirmation`); the split into a check step and an `apply_*` step lets chat and the confirmed reply share one apply path. Each write also needs a `source_quote` found in the speaker's own words this turn. The active facts reach every specialist as a `<standing_facts>` user-turn tag: `route_to_specialist` reads them (None → the store, \"\" → none) for chat consults and workflow steps alike, and `BaseAgent.analyze` places the tag next to `<past_decisions>`, never in the cached specialist system prompt.\n\nThe ten entries in `SPECIALIST_REGISTRY` (`openexecutive.orchestrator.router`):\n\n| Key | Agent | Domain |\n| --- | --- | --- |\n| `cso` | `StrategyAgent` | strategy |\n| `cfo` | `FinanceAgent` | finance |\n| `chro` | `HRAgent` | hr |\n| `gc` | `LegalAgent` | legal |\n| `coo` | `OperationsAgent` | operations |\n| `cmo` | `MarketingAgent` | marketing |\n| `cpo` | `ProductAgent` | product |\n| `sales` | `SalesAgent` | sales |\n| `board_comms` | `BoardCommsAgent` | board |\n| `triage` | `TriageAgent` | triage |\n\nModels: specialists default to `claude-sonnet-5-5`; the deep-reasoning tier (CSO, CFO, GC, Board) runs on `DEEP_REASONING_MODEL`, `claude-sonnet-5-5` by default, with adaptive thinking; routing/fast classification uses `claude-haiku-5-5`. Requests are fitted to the model's generation on the way out (`feature_gate.fit_claude_generation`): 5-generation Claude models never receive `temperature` / `top_p` / `top_k`, and Haiku 5.5, which thinks unless told otherwise, gets thinking turned off when the caller asked for none. The Council's **Deep reasoning** toggle is honoured on Claude Opus/Sonnet (direct or via OpenRouter) and on any OpenRouter model whose live catalog entry supports `reasoning` (today all OpenAI, Gemini, DeepSeek and Grok entries; not Llama) — the request translator turns Anthropic adaptive thinking + effort into OpenRouter's `reasoning` parameter. It is a silent no-op on models that can't reason, and disabled for Haiku, which rejects adaptive thinking. A PDF `document` block is translated too: it becomes an OpenAI `file` part, and OpenRouter requests carrying one add the `file-parser` plugin (`native` for models whose catalog entry lists `file` input, else `PDF_OPENROUTER_ENGINE`, default `mistral-ocr`). `FeatureSpec.supports_pdf_input` gates it per model — off for a local server unless `LOCAL_PDF_INPUT` — and a gated-off PDF becomes a text note rather than vanishing. The Council picks a model in two steps: a **Provider** dropdown (Anthropic, OpenAI, Google, Meta, DeepSeek, xAI, Local, …), then a **Model** dropdown limited to that provider, with readable names (\"Claude Opus 5.5\") and the raw id and serving route shown underneath. The Anthropic group combines the direct Claude models (`claude-fable-5-1`, `claude-opus-5-5`, `claude-opus-5`, `claude-sonnet-5-5`, `claude-sonnet-5`, `claude-haiku-5-5`, `claude-haiku-4-5`) with any Claude generation that exists only on OpenRouter, and never lists a model twice. With `OPENROUTER_ENABLED`, the non-Anthropic entries come from OpenRouter's live `/models` catalog (fetched at startup, refreshed in the background, filtered to tool-capable paid text models from an allowlisted vendor set), falling back to a built-in snapshot if the fetch fails. Every Council agent also takes **additional instructions** (`agent_overrides.instructions`): text appended after its prompt by `agents.overrides.append_instructions` inside an `<additional_instructions>` block. `BaseAgent.base_system_prompt()` is the built-in prompt or its full replacement (what the Council's editor shows); `effective_system_prompt()` is that plus the instructions (what every call sends), and the Executive gets the same text through `build_system_blocks(persona_instructions=…)`. Unlike a replaced prompt, instructions leave the agent on its built-in prompt, so prompt improvements keep reaching it. A **Quality** choice above the agents (`agents.presets`) sets every agent's model and deep reasoning at once: **Fast** puts every agent on a small model (`claude-haiku-5-5`, else `claude-haiku-4-5`) with deep reasoning off, **Balanced** returns each agent to its built-in defaults, and **Thorough** puts the Executive and the domain specialists on the largest allowed model (`claude-opus-5-5`, else `claude-opus-5`) with deep reasoning on where the model supports it, leaving triage and the helpers below on their defaults. Presets change quality only; they never cap spend. Each preset resolves against the install's model allowlist at request time and is offered only when one of its models is allowed. Applying one writes ordinary per-agent overrides (model and deep reasoning only), so history, rollback and later per-agent edits keep working; an agent changed on its own afterwards shows as **Custom**. Each Council agent has a **visibility**: `core` for the Executive and the domain specialists (`BaseAgent.visibility = \"core\"`, which a new specialist must declare) and `internal` (the default) for triage and the helper agents below. Thorough moves only core agents. The Council opens in a **simple view**: the Quality choice, the voice picker, and only the core agents, each with its additional instructions and a model badge. **Show all agents** lists the internal ones too, and **Advanced** opens the full page (the browser remembers the choice). Eight Council-only agents are model/prompt-overridable in the UI but NOT callable via `consult_specialist`, because they are deliberately kept out of `SPECIALIST_REGISTRY`: `quality_judge` (committee review), `utility_fast` (gates, titles, decision parsing), `research` (the `executive_research` fan-out), `fixture_generator` (invents demo companies), `engagement_intake` (models a real client from intake material), `onboarding_interviewer` (interviews a new user about their own company and drafts its profile, people and departments), `workflow_designer` (runs the conversational New workflow wizard on `/jobs/new`: searches the available tools, asks clarifying questions, then drafts a validated dynamic-workflow definition for the user to review), and `workflow_actor` (runs a workflow's `action` steps: a bounded tool-use loop limited to the tools the user approved for that step). `fixture_generator`, `engagement_intake` and `onboarding_interviewer` share one shape — a forced tool call producing a whole company, validated for referential integrity, with a single repair retry — but opposite epistemics: the fixture author invents, the other two extract and leave unstated facts empty rather than guessing. When the workspace is in solo mode, `onboarding_interviewer`'s first user turn is prefixed with a short hint (`interview.solo_hint(role_kind)`, one static constant per role kind from the onboarding role step): the draft names only the principal, as principal, and no departments; an owner is asked about the offer, customers, pricing, cash and runway, an in-house executive about their organisation (drafted as the company — their employer), function, remit, manager, team size as context and what they are measured on, an independent about their practice, clients and pricing, and with no role given the role-neutral `SOLO_HINT` asks which they are. The hint rides in the user turn and the tool array is unchanged, so the cached system prompt and tool prefix are the same in both modes.\n\n**Voice.** The Executive answers in one voice persona, substituted into block 0 at `{VOICE_PERSONA}` (`personas/loader.get_active_body`) from the Executive override's `voice_persona_slug`. Three general voices ship in `personas/builtin/`: **Direct** (`default`, the slug used when none is set: answer first, brief and decisive), **Supportive** (a warm coach who explains the why and checks how the plan sits) and **Analytical** (numbers first, options and trade-offs, then one recommendation). They are offered by a shared picker in setup (`/onboard`, after the workspace choice) and in Settings → Executive, which saves through `PATCH /agents/executive`; a built-in voice's frontmatter carries the picker's one-line `description` and `sample` reply. The named-executive voices are marked `legacy: true`: the picker hides them, but they still load, so an install that picked one keeps it, and the Agent Council lists them under \"Legacy voices\". Voice files hold disposition only; the response-length ladder lives in `prompts/executive_persona.py`, and a test keeps numeric length rules out of every voice file. An eval scenario may set `voice_persona` to answer in a voice; it rides on `Session.voice_persona_slug`, which wins over the override for that session only.\n\nOne more structured-output utility agent sits beside `triage`: `openexecutive.agents.alert_review.AlertReviewAgent` (routing model, one forced `emit_alert_reviews` tool call, never raises). The alert review job hands it a batch of open alerts with evidence and gets back, per alert, a verdict (relevant / changed / resolved / stale), a confidence, the evidence it relied on, a one-line note, why-now, an optional deadline, and a recommended move (none / route / nudge / escalate / draft / suggest_workflow / merge / close) with its parameters. Like `triage`, it never mutates state — `alerts.review.apply_verdict` executes the verdicts deterministically within authority, so the model's job is judgment, not action.\n\n**Solo mode.** When the workspace (or the turn's session) is solo — one person using Open Executive for themselves — the Executive works for that one person, the principal, whatever their role: they may run their own business, lead a function inside a larger organisation, or work independently, and the persona reads which from the context instead of assuming it — the role lines under \"Your Principal\" in the org block, when the principal gave their role (workspace settings: kind, title, reports to, remit, measured on). Its persona swaps the team sections (\"You Are a Member of This Executive Team\", \"Choosing Who to Tell\", \"Departments & Org Coordination\") for \"You Work for One Person\", \"The People in Their World\", \"Who Hears From You\" and \"Goals and Areas\". Solo means only the principal uses Open Executive — not that they have no people: their manager, direct reports, peers, clients, investors and board are contacts. The Executive helps the principal work with them (a 1:1 or review prep, an update or budget case for their manager or board, a negotiation) but does not coordinate them, contacts them only when the principal asks and only if they are a contact, and never offers to loop the principal in. Every specialist stays relevant — HR for the principal's own reports, board and investor communications for reporting upward. Goals are grouped by *area* (the tools' `department_slug` is the area's slug). The chat loop withholds `send_department_message`, `send_company_broadcast` and `set_department_head` from its tool list, and a call to one of them anyway gets a JSON error tool_result instead of running. The mode is resolved once per turn and pinned on the session, so the persona, the org block, the tools and every tool handler agree — a switch made mid-turn applies from the next turn. Team mode is unchanged.\n\nEvery specialist *advises* — each returns domain analysis the Executive synthesizes into one voice. None of them act on the company's behalf; the side-effecting tools in the main loop (schedule, people, departments, broadcast, watchlist, workflows) belong to the Executive, not to a specialist.\n\n`recall_history` (Always in the loop, `orchestrator.history_tools`) reads the speaker's own notes, on the same terms as `ghostwrite_email` below: its own registry, offered only on a verified turn in a conversation private to them with their switch on; its `due` flag lists only dated notes, soonest first (see Memory). The ghostwriter below also gets the writer's notes from mail every recipient was on, as a consistency check (`<writer_noted>`).\n\nThe morning reflection, which has no question to look up and no specialist to consult, gets one more read-only tool of its own: `search_knowledge` (`orchestrator.knowledge_tools`), the same knowledge lookup a chat turn makes, for what the company's documents say about a customer, contract, price or policy before it acts.\n\nA fifth, `remind_me` (`orchestrator.reminder_tools`), sets a plain-text reminder sent to the speaker alone when it is due; it opens no mailbox, so it never locks a turn. A sixth, `propose_actions` (`orchestrator.action_card_tools`), does nothing itself: it leaves the speaker a card of up to five actions (a meeting, a message, a new contact) that happen only when they tap Approve, so it stays on after a turn read mail; it is where what an email itself asks for goes, since that is never done on the email's say-so. Four read-only tools beside it read the speaker's own mailbox on the same terms — `search_my_email`, `read_my_email`, `read_my_email_attachment`, `my_email_awaiting_reply` (`orchestrator.mail_read_tools`) — capped per turn, other people's messages and attached files returned as untrusted blocks (see Act as me). One tool writes under someone else's name, and it is fenced in code rather than the prompt. `ghostwrite_email` (**Act as me**, `orchestrator.delegation_tools`) writes an email as the person the Executive is speaking with and saves it as a draft in their own mailbox, Gmail or Outlook (`delegation.gmail.gmail_for` picks the client by the person's credential); chat never sends it (the person sends it themselves, or taps Send on its card). It lives in its own registry, never `_ALL_SKILL_TOOLS`, so reflection, research and workflow toolkits can never carry it. The chat loop adds it only on a turn `delegation.settings.pin_turn_delegation` offered it to — the principal, or a team member once the owner lets them, with it on, verified, in a conversation private to them (the web chat, a Slack or Discord DM, a verified private Telegram chat), on an attended turn — and builds that turn's handler map from both registries, so on any other turn a call to it is an unknown tool. From the round that calls it until the turn ends, the dispatch guard refuses only what opens an outside address, runs a script or workflow, or posts to everyone (`delegation.lockdown`), without changing the offered list; messages and invites to the roster, follow-ups, facts in the person's own words and a contact they name still run, each fenced in its own path. A later turn in the conversation stays private to its owner but holds nothing: the person asking again, having read the reply, is their own request. A contact add the lockdown refuses becomes an approval card in the chat instead (`propose_actions`). Recipients are chosen in code, never by the model (a draft's new email may go to any address, named back to check when it is neither on the roster nor typed), and the text comes from the **ghostwriter** (`delegation.ghostwriter`), a tool-less call with a constant prompt plus the person's \"How I write\" profile. The ghostwriter is deliberately not a Council agent: the Council's edit routes have no principal check. See **Act as Me (Delegation)**. A second tool joins the same way: `recall_history` (`orchestrator.history_tools`, Always in the loop) reads the speaker's own notes of what they said, by email or in chat. It has its own registry and per-turn handler map too, and the loop adds it only when `history_tools.recall_person` finds the speaker verified, in a conversation private to them, on an interactive turn that isn't private to the principal, with their notes switch on. It doesn't need Act as me. After every chat turn the Executive also hands the speaker's own words to `history_chat.schedule_chat_notes`, which takes notes in the background when their switch is on and the surface verified them (see **Memory**).\n\nThe Executive can also launch *any* workflow conversationally: `orchestrator.workflow_run_tools` adds `list_workflows` (the launchable catalog with each workflow's inputs) and `run_workflow` (runs a built-in or custom workflow to completion and returns its artifact, checkpointing approval gates as `awaiting_human`). `run_workflow` runs the weekly review, and in solo mode the morning brief, only for the principal on a verified surface (`Workflow.principal_only_modes`, `create_goal`'s rule); anyone else gets an audited refusal that does not name the rule. A run whose `result` event reports `private_to_principal` (the morning brief on the principal's own verified turn, when it read their private rows) still returns its artifact into that turn, but the stored run keeps `persistence.PRIVATE_RUN_ARTIFACT` instead of the text, since run history is readable by everyone signed in. This makes the `/jobs` catalog reachable from chat — see the Workflows section.\n\nDeliverables are first-class too. `orchestrator.artifact_tools` gives the Executive three tools: `draft_artifact` publishes a memo, web page (`html`), Word document (`docx`), spreadsheet (`xlsx`, from structured `sheets`) or a `link` to something it created in one of the principal's connected apps through the generic MCP gateway (`search_tools` → `call_tool`, any server — no vendor is named), and returns the artifact's id and link; `list_artifacts` and `get_artifact` let it find and reread its own past work (drafts and workflow outputs alike, through the same `artifact_records` read model the API uses). Every artifact belongs to someone: a draft to the person whose turn published it (`artifact_records.current_viewer`; with nobody speaking, the scheduler or the CLI, the principal), and a run someone started by hand to them, while scheduled runs are the team's. The tools, the recall, `message_person`'s link and mail attachments only ever reach the speaker's own (a link is refused when the recipient couldn't open it), not even the principal sees a teammate's, and a speaker who isn't on the People list publishes nothing. A workflow's own session inside someone's turn (the research synthesis) is pinned to that turn's viewer (`Session.documents_viewer`). Revising is `draft_artifact(supersedes=<id>)`, which archives the earlier version. When the principal asks for a deliverable the tool is always used; the \"quiet is the right default\" bar applies only to unprompted drafts. The format enum is a hard-coded sorted tuple so the cached tool schema stays byte-stable. Delivery: `message_person(person_id, text, artifact_id?)` appends the artifact's title and a `UI_BASE_URL/artifacts/<id>` link (through the same outbound guard as any DM; the link needs Open Executive access), and a Gmail send/draft can attach the file itself by `artifact_id` — see Integrations.\n\nDocuments come in the other way through `read_document` (`orchestrator.document_tools`): it reads a PDF, Word, Excel, CSV, Markdown or text file by `path` — one another tool saved, such as a Gmail attachment (`get_gmail_attachment_content`) or a Drive file (`get_drive_file_download_url`) — or a company document by `filename`, with an optional PDF `pages` range. A scanned PDF is converted (`knowledge.pdf_reader`: local OCR by default; with `PDF_PROVIDER_READING=true`, the deployment's own model through its provider's PDF support, falling back to local OCR) rather than returned empty. Reads are confined to the tool download folders — the same resolve-then-contain check as `oe__read_file` — plus the company docs folder, and it is withheld on turns private to the principal (`PRIVATE_TURN_WITHHELD_TOOLS`): those come from a contact's email, whose own attachments the poller has already read into the turn, and must not steer the Executive into company documents.\n\nThe Executive also backs the **Ask OE** panel's form-fill tier via `orchestrator.form_tools.propose_form_values`: when a chat turn arrives with a `<page_context>` block describing a form on the user's screen, the Executive can propose values for it. The proposal is delivered to the UI as a `form_patch` SSE event and placed into the form as highlighted suggestions — the tool never saves or submits anything; the user reviews and saves manually. The tool is **always** in the tool list (tool definitions sit at the top of the cached prompt prefix, so a conditionally-present tool would invalidate the prompt cache whenever panel and non-panel turns alternate); its description restricts usage to turns that carry a form descriptor.\n\n**Take the lead** (`orchestrator.take_the_lead`) is how the Executive acts on its own, behind one gate. It has two switches, both off until someone turns them on. **As the Executive** is the principal's alone (`PUT /take-the-lead`). With it on, the unattended passes (reflection, research, the scheduler's proactive trigger) keep the acting tools solo mode otherwise withholds from them: booking, starting workflows, and messaging teammates rather than only the principal. `schedule_tools.unattended_toolkit` and the chat loop's unattended path then wrap every acting tool (`ACTING_TOOLS`, plus any MCP tool whose name sends, creates, updates, deletes or shares) in `gated_handlers` / `gated_call_tool`. The principal-only tools (`UNATTENDED_WITHHELD_TOOLS`) stay withheld, and every guard inside a handler still applies, including department approval levels (a booking in a department on Proposes is still a proposal) and the contact-egress rule. New mail or chat (any turn with an `origin_channel`) wakes the existing reflection early (`take_the_lead.wake`): one run waiting at most, five minutes after the first message and no more than every 30 minutes, as a one-off `take_the_lead_wake` scheduled action. Pause holds it like any scheduled work. **As you** is the Handle it for me dial's top step (delegation section).\n\n**The gate** is plain code (`take_the_lead.check`). Six kinds of action wait for a yes first, each with an Ask first switch, all on to start. The switch is the decision ledger's own mode for the class `take_the_lead_action:<kind>` (`propose` asks first, the default; `auto_execute` doesn't). The kinds are money, contracts and legal, hiring and people decisions, deleting/cancelling/sharing, someone new (an address on neither the roster nor the contacts) and big sends (a broadcast or department message, or more than five people). On top of these, the company (set by the principal) and each person (for their own As you) add structured rules: a person or address, a domain, words, or an amount at or above a figure. Added rules always hold, and an error reading them holds the action. The one thing the gate lets straight through is a direct message (`message_person` or a Slack, Telegram or Discord DM) whose only recipient is the principal (`_only_to_principal`): they are who it would wait for, so a card would only ask them to approve a message to themselves. Anything that reaches or acts on anyone else is checked, and a roster that can't be read is not the principal. A held action becomes a `take_the_lead_action` decision instance with a companion `DECISION_ALERT_SOURCE` card on Today, routed to the first teammate holding one of its People approval ranges (`take_the_lead.approval_ranges`, plain code, closest fit first: money to `spend_lt_2k`, `spend_lt_10k` or `spend_gt_10k` by its largest amount, with a higher range able to approve a lower one and no amount read as large, or to `customer_credit` for credit and payment terms; vendor contracts to `vendor_onboarding` then `legal_sign`, other contracts to `legal_sign`; hiring to `hiring_signoff`; anything else about the board or investors to `board_comms`; in Personal as in team mode), or else the principal through the authority gate's fallback (`departments.authority._route_proposal`). Deletes and shares always go to the principal. Only the principal and that approver see or resolve it (`DecisionClassSpec.principal_and_approver`). Approving it claims the row and carries out the exact call once (`take_the_lead.carry_out`: the unwrapped handler, or the gateway's `call_tool`, on an unattended session, with contact egress only when the principal approves). Declining drops it. Everything done or held is a row in `take_the_lead_log`. What it did and what was approved show in Recent activity (`today._build_activity`, kind `took_the_lead`), named but never quoted, since that feed is the team's. The held card's own `decision_resolved` row is left out because its payload quotes the message. `GET /today/activity` also adds the caller's own Handle it sends (kind `sent_as_you`, actor `you`), which are never in the shared feed.\n\n**In training.** The As the Executive switch has three positions: Off, **In training** and On (`take_the_lead.training`). In training, an action the gate would let through still waits, as a `training` hold for the principal alone, unless the principal has allowed that action with that person or thing (`take_the_lead.allowance`: \"Message Priya Nair\" covers a message to her on any channel; \"Book meetings with Priya Nair and Sam Lee\" meetings with exactly them; a workflow by id; a connected tool by name and the file it names, never one that reaches people such as a mail send or a share). The card carries Approve, **Approve + allow**, **Edit** (the message, a meeting's title, times or description, a task and its due date: `take_the_lead.apply_edits`) and Decline. Allowing is the principal's alone and needs a request the API can tie to them, as turning it on does; the six kinds and the added rules hold whatever is allowed. An edit approved with \"Do it like this next time\" is kept as an example (only the edited text, the principal's own words), and `take_the_lead.learned_note` hands the examples and the fact it's in training to the reflection and research passes in their user turn. After `SUGGEST_AFTER` (3) unchanged approvals of the same action in 30 days, Settings suggests allowing it. What it's learned is one list keyed by feature (`take_the_lead_allowed.feature`), so other features that learn from approvals can join it; each keeps its own switch. Act as me's Handle it for me is the second (`act_as_me`, per person, `delegation.training`): its rows carry the person's id, are listed and removed by that person alone, count toward their own cap, and never reach `learned_note`.","mermaid": "flowchart TD\n User([User message]) --> Exec[\"Executive<br/>orchestrator\"]\n Exec -->|consult_specialist| Cso[\"cso: StrategyAgent\"]\n Exec -->|consult_specialist| Cfo[\"cfo: FinanceAgent\"]\n Exec -->|consult_specialist| Gc[\"gc: LegalAgent\"]\n Exec -->|consult_specialist| More[\"chro / coo / cmo / cpo / sales<br/>board_comms / triage\"]\n Mode{{\"Team or solo<br/>(persona + tools per turn)\"}} --> Exec\n Exec -->|\"ghostwrite_email (Act as me)\"| Gw[\"ghostwriter<br/>(tool-less, not a Council agent)\"]\n Cso --> Claude[\"Claude (Anthropic)\"]\n Cfo --> Claude\n Gc --> Claude\n More --> Claude\n Claude --> Exec\n class User entry\n class Exec compute\n class Mode compute\n class Cso compute\n class Cfo compute\n class Gc compute\n class More compute\n class Claude external\n class Gw compute",- "generated_at": "2026-10-09T03:42:10.392375+00:00"+ "generated_at": "2026-10-09T05:00:00Z"}diff --git a/packages/core/openexecutive/architecture/prebuilt/caching.json b/packages/core/openexecutive/architecture/prebuilt/caching.jsonindex b35c701a..215fc31d 100644--- a/packages/core/openexecutive/architecture/prebuilt/caching.json+++ b/packages/core/openexecutive/architecture/prebuilt/caching.json@@ -1,7 +1,7 @@{"section_id": "caching","title": "Prompt Caching",- "markdown": "The Executive system prompt is partitioned for Anthropic prompt caching by `openexecutive.prompts.cache_manager.build_system_blocks`. It emits at most two `cache_control` blocks, consolidated down from four to stay within Anthropic's 4-block API limit alongside the tool definitions and the agent loop's intra-turn marker.\n\n- **Block 0 — persona + knowledge (1h TTL, always present):** the executive persona constant — `EXECUTIVE_PERSONA_PROMPT` for a team, `EXECUTIVE_PERSONA_SOLO_PROMPT` in solo mode — or an admin override, which wins as-is in either mode, with the `{VOICE_PERSONA}` substitution, then the admin's Council **additional instructions** for the Executive (`persona_instructions`, wrapped in `<additional_instructions>`; blank leaves the block byte-identical, and like the override it changes only on save), conditional `WEB_SEARCH_ADDENDUM` / `MCP_ADDENDUM`, the constant `TOOL_BUILDING_ADDENDUM` (when to build a tool with `run_script`, check `list_saved_tools` first, keep what will be needed again, and talk about it without saying \"script\"; present with a gateway unless `CHAT_SCRIPTS=false`, a setting, so constant per deployment), the identity addendum (exec name/email, impersonation guard), the *Connected Systems* section, the constant `DELEGATION_ADDENDUM` while anyone on the install has Act as me on, the timezone addendum, and `KNOWLEDGE_INDEX_SUMMARY`. The timezone addendum names the user's zone, read fresh on every build from the workspace settings (`memory.workspace_settings.get_user_timezone`: the workspace's zone, else `USER_TIMEZONE`, else UTC). Every component is a constant or changes only when an admin or the user changes a setting, so the TTL is long. Setting a new time zone changes the addendum, so the 1h block misses once on that change and is warm again from the next turn — accepted, since a stale zone would mis-schedule follow-ups. Act as me works the same way: its addendum is keyed on the install-level setting (`delegation.settings.block0_delegation_on`), never on the turn or the speaker, so switching it on or off misses the 1h block once; off, block 0 is byte-identical to an install without the feature.\n The *Connected Systems* section (`prompts.connected_systems.render_connected_systems`) tells the model what is connected so it never has to discover it: Google Workspace on or off (on only when `google_workspace` is among the servers the running gateway started with, `MCPGateway.server_names`, so a gateway that failed to start never reads as connected), and when on, how to act from its own account plus the pinned Google tool names by service (`GOOGLE_TOOL_MANIFEST`: Gmail, Calendar, Drive, Docs, Sheets, as workspace-mcp 1.29.0 names them) to call directly; any other configured MCP servers by name (`_comment` keys skipped); Slack / Discord / Telegram / Google Chat on or off from their settings (names only, never a token); Notion knowledge sync; a Confluence knowledge sync line only when that sync is on (a setting, so constant per deployment); and one constant Act as me line that points at `DELEGATION_ADDENDUM`, so switching Act as me still changes block 0 by that addendum alone. The \"never ask which address to send from\" guidance lives here now, shown only while Google is connected; the identity addendum keeps the name, address and impersonation guard. Every input is fixed per process and the output is sorted, so block 0 stays warm; it is appended even under a persona override. It replaces what the model used to learn from `search_tools`, an uncached proxy whose results were gone by the next turn.\n- **Block 1 — company + org (5m TTL, only if non-empty):** `company_profile.to_prompt_block()` plus `render_org_block(mode=…)` — for a team: departments, heads, principals, charters and the People roster; in solo: the principal's own line, their role lines when set (a static lead-in marking the role private to the principal, then kind in plain words, title, reports to, remit, measured on — from the role pinned for the turn: the workspace settings, or a session's `principal_role` override), and their goals grouped by area. The role is set once per install and changes only when the principal edits it, so it is as stable as the rest of the block. Shorter TTL so org edits propagate quickly. The principal's contacts are private to them, so `build_system_blocks(include_contacts=...)` adds the `## Contacts` section, after the team or solo block, only on the principal's own verified turn (`executive._contacts_in_prompt`). That gives block 1 exactly two stable variants per mode — with and without contacts — and never anything per-request; every other turn's block is byte-identical to an install with no contacts.\n\n## Solo mode: one prefix per mode\n\nSolo mode — one person using Open Executive for themselves — changes three cached parts, all resolved **once per turn** by `pin_turn_workspace_mode(session)` (the session's override, else the workspace setting) so they always agree: the block-0 persona, the block-1 org block, and the tool list, which drops `send_company_broadcast`, `send_department_message` and `set_department_head` *before* the sort, so it stays sorted and keeps its 1h marker on the last tool. The resolved mode is pinned on the session for the rest of the turn, so the tool handlers use it too and a switch made mid-turn applies from the next turn. Every part is a constant or stable per install, so each mode keeps its own warm cache and switching modes misses once.\n\nThe two personas share most of their text. `prompts/executive_persona.py` holds plain string literals — a shared head, body and tail plus team or solo sections — concatenated at import and never formatted, so both prompts are constants. The team prompt is byte-for-byte what it was before the split; a test pins its sha256.\n\nTool schemas are static: nothing in a definition depends on company data — `create_goal` takes its area as free text rather than an enum of the current departments, which would change the cached tool prefix every time an area was added, and `record_decision_outcome` takes a plain decision id. The solo persona is still one constant: its goal and decision-outcome lines (`_SOLO_CREATE_GOAL`, `_SOLO_DECISION_OUTCOME`) are literals concatenated at import, and the team persona is byte-identical (a test pins its sha256). The scheduler's proactive-trigger run (a Session marked `unattended`) is sent the list without `schedule_tools.UNATTENDED_WITHHELD_TOOLS`, so it has a stable tool prefix of its own, as each mode does. So does a turn private to the principal (`Session.private_to_principal`: mail from one of their contacts, or mail they forwarded): it is sent the list without `schedule_tools.PRIVATE_TURN_WITHHELD_TOOLS`, dropped before the sort, so private turns in a mode share one sorted prefix with its marker on the last tool. Act as me adds tools rather than dropping any: on a turn `delegation.settings.pin_turn_delegation` offered it to (the principal's own verified, attended turn in a private conversation, with it on), `ghostwrite_email`, the four mailbox reads (`search_my_email`, `read_my_email`, `read_my_email_attachment`, `my_email_awaiting_reply`), `remind_me` and `propose_actions` join the pool together before the sort, so those turns share one more sorted prefix and every other turn's list is byte-identical to before. The offer never depends on the Gmail connection, so a lapsed token never flips the prefix. `recall_history` (Always in the loop) joins the same way, before the sort, on a turn whose speaker may read their own notes; its offer reads only their switch and the surface, never the notes themselves. The chat note-taker runs after the turn, outside the cached prompt. Tool definitions are sorted by name so their serialized order is stable across requests; the `web_search` server tool is appended after the sorted list so it never receives a `cache_control` marker. The last client tool carries the third marker (1h TTL).\n\n## The intra-turn marker\n\nThe fourth and final breakpoint belongs to the tool-use loop. After each iteration appends its `tool_result` blocks, `executive._apply_loop_cache_marker` moves a single ephemeral marker onto the **newest non-empty tool result**. Without it the loop caches only the static tools+system prefix, so every iteration re-sends the whole accumulated transcript at full input price — the runtime signature is `cache_read_input_tokens` pinned flat while the prompt grows iteration over iteration.\n\nThe marker **moves rather than accumulates**: each pass first sweeps every `tool_result` clean, then re-marks the newest one. A marker left in place per iteration would reach `max_iterations` breakpoints and blow the 4-block limit. Sweeping first also makes the helper idempotent — there is no bookkeeping index to drift.\n\n## The history marker\n\nOnce a conversation has history, the agent loop also marks the **newest non-empty history turn** (normally the previous reply) with a **1h** breakpoint (`executive._apply_history_cache_marker`), so the next message reads the whole conversation back from cache and writes only what is new: the last exchange, this turn's context and its tool calls. History turns are rebuilt from plain text, so the previous turn's loop entries, which ended inside that turn's retrieved context and tool results, never match; without this marker every message re-wrote the whole conversation into the cache. 1h because people reply in minutes, not seconds.\n\nThe history marker takes the company block's slot: block 1 stays in the prefix, covered by the history marker, and dropping its 5m marker keeps the TTLs in the order the API requires, longer before shorter. On a first message (no history) block 1 keeps its 5m marker as before. `_build_messages` itself never marks history, so the committee revision call, which reuses those messages, keeps the plain layout. `Session.get_recent_history` moves its window's start in steps of 10 turns (keeping 20 to 30), so a long conversation's first history message, and the cached prefix with it, stays put between steps instead of shifting every turn. On the OpenRouter path the translator keeps assistant turns as typed text parts whenever a request carries the history marker (`translator._request_caches_assistant_text`), so the marker reaches Anthropic and every reply has the same shape whether or not it is the one marked. A rolling marker on every history turn used to claim a latent fifth breakpoint, which Anthropic rejects outright.\n\n**The breakpoint budget, in full:** last client tool (1h) + persona+knowledge (1h) + company+org (5m, first message only) or history (1h) + intra-turn tool result (5m) = at most 4, longer TTLs first.\n\nRAG context is **never** placed in a cached system block — retrieved hits vary per query and would poison the cache. It is injected into the **user turn** instead.\n\nSpecialist calls follow the same rule. Each specialist caches only its static domain prompt (`BaseAgent.analyze`, one ephemeral block); everything per call — retrieved knowledge, failure cases, past decisions, department memory, the company's stage (`<company_stage>`) and, in solo mode, the principal's role (`<principal_role>`) — goes in the specialist's user turn, so the cached domain prompt is byte-identical across companies and profile edits.\n\nThe ghostwriter's call (Act as me) carries no cache markers: its system prompt is the constant `GHOSTWRITER_PROMPT` plus that person's voice block — short, and sent only when a draft is written — and the thread and the intent go in its user turn.\n\n## Tool-result bound\n\n`TOOL_RESULT_MAX_CHARS` (default 50,000) caps a single tool result before it enters the prompt, applied at the one point every result reaches the model rather than at each producer — so it also covers specialist output, tool errors and the unknown-tool fallback, while non-model consumers (the `propose_form_values` JSON parse, the audit trail) still read the full text. It is a circuit breaker, not a routine clipper: the default is set so ordinary tool output never reaches it. A cut result carries an explicit marker naming the tool and stating that the result is incomplete — truncating silently is worse than not truncating, because the model reads the cut text as the whole answer.\n\n**The invariant:** never put dynamic content in a `cache_control` block. A single mutating value breaks the cache and drives roughly a 10x cost increase.",- "mermaid": "flowchart TD\n Tools[\"Tool defs (sorted by name)<br/>last one: cache_control 1h\"] --> B0\n Mode{{\"effective_workspace_mode<br/>(team or solo, once per turn)\"}} --> Tools\n Mode --> B0\n Mode --> B1\n Deleg{{\"Act as me on?<br/>(install-level, not per turn)\"}} --> B0\n Offer{{\"ghostwrite_email offered?<br/>(pinned once per turn)\"}} --> Tools\n B0[\"Block 0: persona (team or solo) + knowledge<br/>cache_control 1h\"] --> B1\n B1[\"Block 1: company + org (team or solo render)<br/>cache_control 5m on a first message\"] --> Limit{{\"≤ 4 cache_control blocks\"}}\n Loop[\"Newest tool_result<br/>cache_control 5m (moves each iteration)\"] --> Limit\n Hist[\"Newest history turn<br/>cache_control 1h (replaces block 1 marker)\"] --> Limit\n Limit --> Turn[\"User turn\"]\n RAG[\"RAG hits<br/>(vary per query)\"] --> Turn\n Turn --> API[\"Anthropic API\"]\n class B0 cache\n class B1 cache\n class RAG hot\n class Turn hot\n class Tools compute\n class Mode compute\n class Loop cache\n class API external\n class Deleg compute\n class Offer compute",- "generated_at": "2026-10-08T17:10:00Z"+ "markdown": "The Executive system prompt is partitioned for Anthropic prompt caching by `openexecutive.prompts.cache_manager.build_system_blocks`. It emits at most two `cache_control` blocks, consolidated down from four to stay within Anthropic's 4-block API limit alongside the tool definitions and the agent loop's intra-turn marker.\n\n- **Block 0 — persona + knowledge (1h TTL, always present):** the executive persona constant — `EXECUTIVE_PERSONA_PROMPT` for a team, `EXECUTIVE_PERSONA_SOLO_PROMPT` in solo mode — or an admin override, which wins as-is in either mode, with the `{VOICE_PERSONA}` substitution, then the admin's Council **additional instructions** for the Executive (`persona_instructions`, wrapped in `<additional_instructions>`; blank leaves the block byte-identical, and like the override it changes only on save), conditional `WEB_SEARCH_ADDENDUM` / `MCP_ADDENDUM`, the constant `TOOL_BUILDING_ADDENDUM` (when to build a tool with `run_script`, check `list_saved_tools` first, keep what will be needed again, and talk about it without saying \"script\"; present with a gateway unless `CHAT_SCRIPTS=false`, a setting, so constant per deployment), the identity addendum (exec name/email, impersonation guard), the *Connected Systems* section, the constant `DELEGATION_ADDENDUM` while anyone on the install has Act as me on, the timezone addendum, and `KNOWLEDGE_INDEX_SUMMARY`. The timezone addendum names the user's zone, read fresh on every build from the workspace settings (`memory.workspace_settings.get_user_timezone`: the workspace's zone, else `USER_TIMEZONE`, else UTC). Every component is a constant or changes only when an admin or the user changes a setting, so the TTL is long. Setting a new time zone changes the addendum, so the 1h block misses once on that change and is warm again from the next turn — accepted, since a stale zone would mis-schedule follow-ups. Act as me works the same way: its addendum is keyed on the install-level setting (`delegation.settings.block0_delegation_on`), never on the turn or the speaker, so switching it on or off misses the 1h block once; off, block 0 is byte-identical to an install without the feature.\n The *Connected Systems* section (`prompts.connected_systems.render_connected_systems`) tells the model what is connected so it never has to discover it: Google Workspace on or off (on only when `google_workspace` is among the servers the running gateway started with, `MCPGateway.server_names`, so a gateway that failed to start never reads as connected), and when on, how to act from its own account plus the pinned Google tool names by service (`GOOGLE_TOOL_MANIFEST`: Gmail, Calendar, Drive, Docs, Sheets, as workspace-mcp 1.29.0 names them) to call directly; any other configured MCP servers by name (`_comment` keys skipped); Slack / Discord / Telegram / Google Chat on or off from their settings (names only, never a token); Notion knowledge sync; a Confluence knowledge sync line only when that sync is on (a setting, so constant per deployment); and one constant Act as me line that points at `DELEGATION_ADDENDUM`, so switching Act as me still changes block 0 by that addendum alone. The \"never ask which address to send from\" guidance lives here now, shown only while Google is connected; the identity addendum keeps the name, address and impersonation guard. Every input is fixed per process and the output is sorted, so block 0 stays warm; it is appended even under a persona override. It replaces what the model used to learn from `search_tools`, an uncached proxy whose results were gone by the next turn.\n- **Block 1 — company + org (5m TTL, only if non-empty):** `company_profile.to_prompt_block()` plus `render_org_block(mode=…)` — for a team: departments, heads, principals, charters and the People roster; in solo: the principal's own line, their role lines when set (a static lead-in marking the role private to the principal, then kind in plain words, title, reports to, remit, measured on — from the role pinned for the turn: the workspace settings, or a session's `principal_role` override), and their goals grouped by area. The role is set once per install and changes only when the principal edits it, so it is as stable as the rest of the block. Shorter TTL so org edits propagate quickly. The principal's contacts are private to them, so `build_system_blocks(include_contacts=...)` adds the `## Contacts` section, after the team or solo block, only on the principal's own verified turn (`executive._contacts_in_prompt`). That gives block 1 exactly two stable variants per mode — with and without contacts — and never anything per-request; every other turn's block is byte-identical to an install with no contacts.\n\n## Solo mode: one prefix per mode\n\nSolo mode — one person using Open Executive for themselves — changes three cached parts, all resolved **once per turn** by `pin_turn_workspace_mode(session)` (the session's override, else the workspace setting) so they always agree: the block-0 persona, the block-1 org block, and the tool list, which drops `send_company_broadcast`, `send_department_message` and `set_department_head` *before* the sort, so it stays sorted and keeps its 1h marker on the last tool. The resolved mode is pinned on the session for the rest of the turn, so the tool handlers use it too and a switch made mid-turn applies from the next turn. Every part is a constant or stable per install, so each mode keeps its own warm cache and switching modes misses once.\n\nThe two personas share most of their text. `prompts/executive_persona.py` holds plain string literals — a shared head, body and tail plus team or solo sections — concatenated at import and never formatted, so both prompts are constants. The team prompt is byte-for-byte what it was before the split; a test pins its sha256.\n\nTool schemas are static: nothing in a definition depends on company data — `create_goal` takes its area as free text rather than an enum of the current departments, which would change the cached tool prefix every time an area was added, and `record_decision_outcome` takes a plain decision id. The solo persona is still one constant: its goal and decision-outcome lines (`_SOLO_CREATE_GOAL`, `_SOLO_DECISION_OUTCOME`) are literals concatenated at import, and the team persona is byte-identical (a test pins its sha256). The scheduler's proactive-trigger run (a Session marked `unattended`) is sent the list without `schedule_tools.UNATTENDED_WITHHELD_TOOLS`, so it has a stable tool prefix of its own, as each mode does. So does a turn private to the principal (`Session.private_to_principal`: mail from one of their contacts, or mail they forwarded): it is sent the list without `schedule_tools.PRIVATE_TURN_WITHHELD_TOOLS`, dropped before the sort, so private turns in a mode share one sorted prefix with its marker on the last tool. Act as me adds tools rather than dropping any: on a turn `delegation.settings.pin_turn_delegation` offered it to (the principal's own verified, attended turn in a private conversation, with it on), `ghostwrite_email`, the four mailbox reads (`search_my_email`, `read_my_email`, `read_my_email_attachment`, `my_email_awaiting_reply`), `remind_me` and `propose_actions` join the pool together before the sort, so those turns share one more sorted prefix and every other turn's list is byte-identical to before. The offer never depends on the Gmail connection, so a lapsed token never flips the prefix. `recall_history` (Always in the loop) joins the same way, before the sort, on a turn whose speaker may read their own notes; its offer reads only their switch and the surface, never the notes themselves. The chat note-taker runs after the turn, outside the cached prompt. Tool definitions are sorted by name so their serialized order is stable across requests; the `web_search` server tool is appended after the sorted list so it never receives a `cache_control` marker. The last client tool carries the third marker (1h TTL).\n\n## Tools opened on demand\n\nThe less common tools are not in the list at all. `orchestrator.tool_groups` puts them in named groups — calendar, people, goals, watch_and_alerts, workflows, documents, messaging, skills — and `tool_groups.split` takes them out of the turn's offered tools after every filter above (mode, unattended, private turn) and before the sort, adding two small tools in their place: `open_tools(group)`, whose description names each offered group and its tools, and `use_tool(name, input)`. When the model opens a group, the loop answers with that group's tools (name, description, input schema) as a **tool result**. It never adds them to the list: the list is the start of the cached prefix, so a list that grew mid-turn would miss the cache for the tools, both system blocks and the whole history. The list is the same on every call of every turn, and a group's descriptions are paid for only on a turn that opens it. That took the direct list from about 19,500 tokens to about 6,500, re-sent on every model call.\n\n`open_tools`'s description and its results name only the tools the turn offers (`tool_groups.open_tools_tool`), so a private turn never sees a withheld tool's name. The description depends only on that set, which is fixed for each kind of turn, so each still has one stable list. A `use_tool` call is rewritten into the tool's own call, same id, before dispatch (`tool_groups.unwrap`); the model's assistant block stays as it was sent, so the result pairs with the id it used. The rewritten call meets the same guards, handler, progress events and audit rows as a direct call. `use_tool` takes only names in `tool_groups.DEFERRED`; anything else, or an input that is not an object, gets an error tool result. The everyday tools (`consult_specialist`, `propose_form_values`, people and memory lookups, follow-ups, the MCP meta-tools) stay direct.\n\n## The intra-turn marker\n\nThe fourth and final breakpoint belongs to the tool-use loop. After each iteration appends its `tool_result` blocks, `executive._apply_loop_cache_marker` moves a single ephemeral marker onto the **newest non-empty tool result**. Without it the loop caches only the static tools+system prefix, so every iteration re-sends the whole accumulated transcript at full input price — the runtime signature is `cache_read_input_tokens` pinned flat while the prompt grows iteration over iteration.\n\nThe marker **moves rather than accumulates**: each pass first sweeps every `tool_result` clean, then re-marks the newest one. A marker left in place per iteration would reach `max_iterations` breakpoints and blow the 4-block limit. Sweeping first also makes the helper idempotent — there is no bookkeeping index to drift.\n\n## The history marker\n\nOnce a conversation has history, the agent loop also marks the **newest non-empty history turn** (normally the previous reply) with a **1h** breakpoint (`executive._apply_history_cache_marker`), so the next message reads the whole conversation back from cache and writes only what is new: the last exchange, this turn's context and its tool calls. History turns are rebuilt from plain text, so the previous turn's loop entries, which ended inside that turn's retrieved context and tool results, never match; without this marker every message re-wrote the whole conversation into the cache. 1h because people reply in minutes, not seconds.\n\nThe history marker takes the company block's slot: block 1 stays in the prefix, covered by the history marker, and dropping its 5m marker keeps the TTLs in the order the API requires, longer before shorter. On a first message (no history) block 1 keeps its 5m marker as before. `_build_messages` itself never marks history, so the committee revision call, which reuses those messages, keeps the plain layout. `Session.get_recent_history` moves its window's start in steps of 10 turns (keeping 20 to 30), so a long conversation's first history message, and the cached prefix with it, stays put between steps instead of shifting every turn. On the OpenRouter path the translator keeps assistant turns as typed text parts whenever a request carries the history marker (`translator._request_caches_assistant_text`), so the marker reaches Anthropic and every reply has the same shape whether or not it is the one marked. A rolling marker on every history turn used to claim a latent fifth breakpoint, which Anthropic rejects outright.\n\n**The breakpoint budget, in full:** last client tool (1h) + persona+knowledge (1h) + company+org (5m, first message only) or history (1h) + intra-turn tool result (5m) = at most 4, longer TTLs first.\n\nRAG context is **never** placed in a cached system block — retrieved hits vary per query and would poison the cache. It is injected into the **user turn** instead.\n\nSpecialist calls follow the same rule. Each specialist caches only its static domain prompt (`BaseAgent.analyze`, one ephemeral block); everything per call — retrieved knowledge, failure cases, past decisions, department memory, the company's stage (`<company_stage>`) and, in solo mode, the principal's role (`<principal_role>`) — goes in the specialist's user turn, so the cached domain prompt is byte-identical across companies and profile edits.\n\nThe ghostwriter's call (Act as me) carries no cache markers: its system prompt is the constant `GHOSTWRITER_PROMPT` plus that person's voice block — short, and sent only when a draft is written — and the thread and the intent go in its user turn.\n\n## Tool-result bound\n\n`TOOL_RESULT_MAX_CHARS` (default 20,000 characters, about 5,000 tokens) caps a single tool result before it enters the prompt, applied at the one point every result reaches the model rather than at each producer — so it also covers specialist output, tool errors and the unknown-tool fallback, while non-model consumers (the `propose_form_values` JSON parse, the audit trail) still read the full text. It is a circuit breaker, not a routine clipper: ordinary tool output (a status, an email thread, a specialist's analysis) stays under it. A long document read does not, and the marker asks for a narrower request, such as a page range or a section, because a result stays in the prompt for every remaining call of the turn. A cut result carries an explicit marker naming the tool and stating that the result is incomplete — truncating silently is worse than not truncating, because the model reads the cut text as the whole answer.\n\n**The invariant:** never put dynamic content in a `cache_control` block. A single mutating value breaks the cache and drives roughly a 10x cost increase.",+ "mermaid": "flowchart TD\n Tools[\"Tool defs (sorted by name)<br/>less common ones behind open_tools<br/>last one: cache_control 1h\"] --> B0\n Mode{{\"effective_workspace_mode<br/>(team or solo, once per turn)\"}} --> Tools\n Mode --> B0\n Mode --> B1\n Deleg{{\"Act as me on?<br/>(install-level, not per turn)\"}} --> B0\n Offer{{\"ghostwrite_email offered?<br/>(pinned once per turn)\"}} --> Tools\n B0[\"Block 0: persona (team or solo) + knowledge<br/>cache_control 1h\"] --> B1\n B1[\"Block 1: company + org (team or solo render)<br/>cache_control 5m on a first message\"] --> Limit{{\"≤ 4 cache_control blocks\"}}\n Loop[\"Newest tool_result<br/>cache_control 5m (moves each iteration)\"] --> Limit\n Hist[\"Newest history turn<br/>cache_control 1h (replaces block 1 marker)\"] --> Limit\n Limit --> Turn[\"User turn\"]\n RAG[\"RAG hits<br/>(vary per query)\"] --> Turn\n Turn --> API[\"Anthropic API\"]\n class B0 cache\n class B1 cache\n class RAG hot\n class Turn hot\n class Tools compute\n class Mode compute\n class Loop cache\n class API external\n class Deleg compute\n class Offer compute",+ "generated_at": "2026-10-09T05:00:00Z"}diff --git a/packages/core/openexecutive/architecture/prebuilt/rag.json b/packages/core/openexecutive/architecture/prebuilt/rag.jsonindex e026238f..cc67a0c8 100644--- a/packages/core/openexecutive/architecture/prebuilt/rag.json+++ b/packages/core/openexecutive/architecture/prebuilt/rag.json@@ -1,7 +1,7 @@{"section_id": "rag","title": "Knowledge & RAG",- "markdown": "Retrieval is handled by `openexecutive.knowledge.retriever.retrieve`, querying ChromaDB collections through `openexecutive.knowledge.store.ChromaDBStore`. When the caller passes `record_source`, `retrieve` also names every document its block quotes — company uploads by filename, built-in knowledge by a title made from its filename, Notion pages, synced Drive files by name with their Drive link, synced Confluence pages by title with their page link, research notes by date, and earlier published documents with a link to `/artifacts/<id>` — so the web chat can list what an answer looked at: the chat route's own search, and each specialist's that answered (see the Lifecycle and API sections). The recording never raises, and a retrieval that fails leaves the specialist without context rather than failing the turn. Eight collections back the system:\n\n- **BUILTIN** — curated MBA-style domain knowledge shipped with the repo, including a `sales` domain (founder-led sales, qualification and pipeline, proposals and pricing conversations, follow-up and forecasting) and solo-founder guides in finance, marketing, legal and HR. Seeded at boot by `loader.seed_builtin_knowledge` (FAILURES by `seed_failures`): the whole tree on an empty collection, otherwise only the `SHIPPED_BUILTIN_FILES` entries with no chunks yet — so a doc added in a later release reaches an install seeded by an earlier one, instead of shipping, registering as approved, and never being retrievable. Presence is judged from each row's `source` path relative to `knowledge/builtin/` (not the absolute path, which moves with the install), read in one metadata scan narrowed to seeded rows so the external OER corpus sharing the collection is never pulled. A complete store indexes nothing.\n- **COMPANY** — customer documents ingested via the `/documents` endpoint. Curated uploads only — synced Notion pages, Drive files and Confluence pages do not land here. A document's identity is its **filename**: chunk metadata and the chunk id both derive from it, so re-uploading upserts in place rather than duplicating, and `DELETE /documents/{filename}` matches the rows it claims to remove. Inbound attachments are **not** here — see ATTACHMENTS below; rows left by the earlier scheme are migrated out on boot. A scanned PDF (no text layer) is converted before chunking — see \"Reading scanned PDFs\" below — so it no longer indexes zero chunks while reporting `indexed`.\n- **NOTION_WIKI** — optional Notion sync (`NOTION_SYNC_ENABLED`) incrementally re-indexes pages shared with a Notion internal integration into a separate `notion_wiki` collection (`type=notion`). A Notion share is inherited and multi-writer, so this text is unreviewed relative to curated company docs; the retriever labels it as such and ranks it below COMPANY. Un-shared pages are purged on the next tick.\nSynced Drive files and Notion pages are listed on the knowledge page next to uploads (one Company documents list, each row badged with its source, its sync time and a link to the original), read from the sync state files rather than Chroma; each connected source shows its last run, its last error (a failed tick, or a count of files or pages a tick couldn't index; the next clean tick clears it) and a **Sync now** button (`POST /documents/sources/{id}/sync`). Uploads from the page no longer ask for a domain: they go in as `general`, which every specialist's COMPANY filter already includes.\n\n- **DRIVE** — optional Google Drive folder sync (`DRIVE_SYNC_ENABLED`, `knowledge.drive_sync`). It incrementally re-indexes the files in `DRIVE_SYNC_FOLDER_IDS`, and their subfolders to four levels, into a separate `drive_docs` collection (`type=drive`). It reads the Drive v3 API directly as a service account with `drive.readonly` (`knowledge.drive_client`), not through the MCP gateway, so it sees only folders shared with that account and can change nothing. Google Docs and Slides are exported as text and Sheets as `.xlsx`. PDF, `.docx`, `.xlsx` and plain text files go through the loader's extractors, with no model call, so a scanned PDF yields nothing. Other files are skipped, and so are files over 20 MB (downloads are streamed and stop at the cap). A `.docx` or `.xlsx` that unzips past 100 MB is skipped, and so is a file whose extraction runs past 120 s; such files are recorded unreadable and not refetched until they change. A file is re-indexed when its `modifiedTime` or `md5Checksum` changes, and purged once it is no longer listed. Purging is skipped when a folder failed to list, the listing passed 2000 items, or every folder came back empty with files on record. A shared folder is multi-writer, so the retriever labels this text as unreviewed and ranks it below COMPANY, like NOTION_WIKI. Each chunk is labelled `[drive · file id … · synced … · \"<name>\"]`. The id and time come first because the sync wrote them. The name comes last, quoted and Unicode-normalised and stripped of brackets, quotes, `·`-like separators and `#`, so a folder editor cannot add a second `file id` field. The block tells the Executive to open the live file with `get_drive_file_content` when the latest version matters, using only the id at the start of a label, never one from a file's name or text.\n- **ONEDRIVE** — optional OneDrive / SharePoint folder sync (`ONEDRIVE_SYNC_ENABLED`, `knowledge.onedrive_sync`), DRIVE's twin: the files in `ONEDRIVE_SYNC_FOLDERS` (`<drive id>/<item id>` pairs) go into a separate `onedrive_docs` collection (`type=onedrive`, keyed `<drive id>:<item id>`). It calls Microsoft Graph directly (`knowledge.onedrive_client`, GETs only) as the Executive's own Microsoft 365 sign-in, with a token the Microsoft server's launcher mints for file reads only, because Microsoft has no per-folder service account; the folder list is the scope. PDF, Word, Excel and plain text files go through the same extractors, caps and deadlines as Drive (imported from `drive_sync`); PowerPoint, older Office formats, OpenDocument and RTF are fetched as Graph's PDF rendering first. A file is re-indexed when its `lastModifiedDateTime` or `cTag` changes; purging, its skip rules and the locking are Drive's. Retrieved right after DRIVE, labelled `[onedrive · item <drive id>:<item id> · synced … · \"<name>\"]` with the name defanged the same way, and the block tells the Executive to open the live file with its OneDrive tools using only the ids after `item`.\n- **CONFLUENCE** — optional Confluence space sync (`CONFLUENCE_SYNC_ENABLED`, `knowledge.confluence_sync`), for Cloud and Server/Data Center alike. Each tick lists every page in `CONFLUENCE_SYNC_SPACE_KEYS` over the v1 REST API (`knowledge.confluence_client`, directly with `httpx`, not through the MCP gateway; a personal access token as Bearer, or a username and API token as Basic) and re-indexes the pages whose `version.number` changed into a separate `confluence_wiki` collection (`type=confluence`), at most `CONFLUENCE_MAX_PAGES_PER_SCAN` per tick, newest first. Storage-format XHTML becomes Markdown through stdlib `html.parser` (`knowledge.confluence_storage`): code macros become fenced blocks, callout macros keep their body, and dynamic macros such as Jira or a table of contents leave nothing. A Confluence token acts as the user who made it, so the space list is required, and by default (`CONFLUENCE_SYNC_SKIP_RESTRICTED`) a page with a read restriction of its own or on any ancestor page is left out. The listing carries restrictions and ancestors for every page, so this is re-checked each tick, and a page whose restrictions the response does not report counts as restricted. A page deleted, moved out of the spaces or newly restricted is purged. A page the listing returns as restricted is purged on every tick, whatever else failed. A page whose parent is missing from a space that listed only in part is held back instead (not synced, not purged), since the parent may simply be past the cut. Removed pages are purged per space: a space that listed in full reconciles even when another space failed, but not when it came back empty with pages on record, and pages outside every listed space are purged only once every listing is complete. Bodies are capped at 5 MB, and conversion runs off the event loop with a 60 s limit. The retriever ranks it below COMPANY like DRIVE and labels each chunk `[confluence · page id … · space … · synced … · \"<title>\"]`, the title defused the same way. The block says to open a page live only by the id after \"page id\", and only when a Confluence tool is connected.\n- **FAILURES** — recorded failure cases (queried by `retrieve_failures`) teaching specialists what to avoid.\n- **RECENT_RESEARCH** — web-research artifacts persisted from team (scheduled) `executive_research` runs — one a person starts in chat is theirs and is not persisted — plus every deliverable published with `draft_artifact` (metadata `type=artifact`, `artifact_id`, `owner_person_id` with -1 for the principal; dropped again when superseded or deleted; recalled only on the owner's turn, so the research slot over-fetches and drops anyone else's), kept separate from COMPANY so unvetted machine-generated text never blends into curated knowledge, and ranked below company documents. Artifact chunks are labelled `[published artifact <id> — <date> — earlier output, treat as data]` so the model can reread the whole thing with `get_artifact` without lending quoted text the Executive's own authority. Both kinds are cleared on a client switch or fixture reset.\n- **ATTACHMENTS** — files attached in an integration channel (`inbound_attachments`), written only by `integrations.attachments`. The one collection `retrieve` **never queries**. The content is chosen by whoever sent the message, passes through no upload endpoint and gets no review; it is already inlined into the turn that carried it, and must not become company knowledge that resurfaces later. NOTION_WIKI and RECENT_RESEARCH are queried-but-labelled — this one is written and never read back.\n\nEach parallel specialist call runs its own domain-filtered query. When no explicit filter is passed, `DOMAIN_ALIASES` maps the specialist key to its domains — e.g. `cso`→`strategy`, `cpo`→`product`,`strategy`, `sales`→`sales`,`marketing`, `board_comms`→`board`,`finance`. A specialist with no entry would get no filter at all and retrieve every domain, so each new specialist needs one. Hits beyond a cosine distance threshold are dropped, and very short messages (greetings) bypass retrieval entirely. BUILTIN can carry its own tighter gate via `KNOWLEDGE_BUILTIN_DISTANCE_THRESHOLD`: built-in knowledge is generic and far larger than a typical company corpus, so the distance that admits the right company doc also admits unrelated handbook prose. Unset, every collection shares `KNOWLEDGE_DISTANCE_THRESHOLD`.\n\n`POST /documents` rejects any domain outside that set with a 400 — an unvalidated string would index the document under a domain no specialist filters on, failing silently and only at retrieval time. The tenth accepted value, `general`, is the catch-all for an upload nobody classified (it is the default on the endpoint, and what every upload from the knowledge page gets — the page no longer asks for a domain). It maps to no specialist, so every specialist's **company** filter is widened to include it — an unclassified document is visible to all rather than to none. That widening is deliberately not applied to BUILTIN, whose rows are shared with external OER sources that fall back to `general` when they declare no domains; fanning those into every specialist would blend unvetted third-party material into every answer.\n\nMulti-domain chunks fan out to one row per declared domain, so `_dedupe_by_text` drops duplicate-text hits while preserving relevance order before results are returned.\n\n### Embeddings\n\nEvery collection embeds with one shared all-MiniLM-L6-v2 ONNX session (`knowledge.store._embedding_function`), the same model and vectors as Chroma's default. Chroma's own default reloads the model on every call and keeps ONNX Runtime's memory arena, so a 300-chunk upload peaked near 760 MB and each query spent ~240 ms reloading. The shared session is built once, runs without the arena and embeds four texts at a time (the model pads every text to 256 tokens, so batch size sets the peak): ~300 MB and ~30 ms. It answers to Chroma's name `default` with an empty config, because every existing collection records that name and Chroma refuses a collection opened with a differently named function; and it must not be a `DefaultEmbeddingFunction` instance, which Chroma swaps for a fresh default on every embed.\n\n### Reading scanned PDFs\n\npypdf returns only a PDF's saved text layer, so a scan, a phone photo saved as PDF, or a deck printed to PDF as images used to come back empty everywhere a PDF is read. `knowledge.pdf_reader.read_pdf_text` tries three readers in order and never raises:\n\n1. **Text layer** (pypdf) — used whenever it yields real text (at least ~20 visible characters per page, so a scan carrying only page numbers still falls through).\n2. **The deployment's own model, through its provider's PDF support** — opt-in, `PDF_PROVIDER_READING=true` (off by default: an upgrade never starts sending scanned PDFs off the server; this also means an Anthropic-direct deployment OCRs scans locally until it opts in) — the pages as a base64 `document` block, transcribed to Markdown by `PDF_VISION_MODEL` (unset: `DEFAULT_MODEL`, the model the deployment already runs on), `PDF_VISION_PAGES_PER_CALL` pages per request (default 20), three requests at a time. Each provider carries the block its own way: Anthropic reads it natively; on OpenRouter the translator makes it an OpenAI `file` part and the provider adds the `file-parser` plugin — engine `native` for Claude or any model the catalog lists with `file` input (OpenAI's GPT-4o and later, Gemini…), else `PDF_OPENROUTER_ENGINE` (default `mistral-ocr`, $2 per 1,000 pages); a local OpenAI-compatible server gets the `file` part only with `LOCAL_PDF_INPUT=true` (e.g. `LOCAL_BASE_URL` pointed at OpenAI's own API). `providers.registry.pdf_input_supported` decides; a model that cannot take one never receives it (the feature gate swaps a stray `document` block for a note). Turning it on sends scanned pages off the server — to the provider, and on OpenRouter possibly to its parser (Mistral OCR) — and the destination is logged once per process.\n3. **Local OCR** — pages rendered with pypdfium2 and read by RapidOCR (ONNX, models bundled in the wheel): offline, no key, no per-page cost. The default reader while step 2 is off, what a local model without PDF input gets, and the fallback on every provider when step 2 errors, refuses or returns nothing. `PDF_OCR_ENABLED=false` turns it off.\n\n`PDF_VISION_MAX_PAGES` (default 100) caps the pages read, with a note saying how many were; a file a channel delivered (`inbound=True`) is capped at `PDF_INBOUND_MAX_PAGES` (30) and metered by a rolling `PDF_INBOUND_PAGES_PER_HOUR` (300) budget. OCR is bounded too: a page renders within a 25-megapixel budget whatever its MediaBox claims (a crafted 20000pt page would otherwise allocate gigabytes), at most two documents OCR at once, and each stops after 240 s with what it has. A PDF of more than 2,000 pages is refused before any page is parsed (the caps above bound conversion, not the text-layer pass). A Claude slice that runs into the output limit is split in half until each part fits; a single page that still does not fit keeps what was written, marked `[transcription cut off here]`, and the note says so. A result is cached in process by the file's sha256 (32 entries) so a re-read does not re-bill; a failure is not cached. `loader.read_document_text` / `extract_text_from_file_async` route a PDF through it and every other type through the sync extractor, and are what `ingest_file` (uploads, boot reconcile), the `/documents/{filename}` preview, intake uploads, chat attachments, the email poller, `oe__read_file` and the Executive's `read_document` tool use.\n\n### Parsing in a child process\n\nParsing a large document builds hundreds of MB of small Python objects, and CPython keeps the memory they lived in, so a parse inside the API process left its peak there for good. On a 34 MB, 131-page text-layer PDF uploaded through `POST /documents`, uvicorn went from 272 MB to a 440 MB peak and stayed at 378 MB. On a 30-page scan read by OCR it peaked at 787 MB and stayed at 446 MB. Now the text-layer pass and local OCR in `pdf_reader`, and the PDF, Word and Excel extractors behind `loader.extract_text_from_file`, run in a short-lived child process (`knowledge.isolated.run_isolated`). Everything the child allocated goes back to the OS when it exits. The same uploads now peak at 324 MB and 297 MB and settle at 318 MB and 290 MB, and what remains is embedding. The OCR child itself still peaks near 600 MB while it reads, then exits.\n\nThe call goes to the child pickled on stdin and the answer comes back as JSON, so the API never unpickles anything a hostile file could shape. Both of the child's output streams go to temp files, not the API's memory. The child logs nothing, because pypdf logs text taken from the file and a crafted 9 KB PDF made it write megabytes. Only the last 2,000 characters of its stderr are read back, and an answer over 32 MB is refused. The child inherits only what it needs to start Python (`PATH`, `HOME`, `PYTHONPATH`, the library path, locale, temp dir), not the deployment's keys. That is hygiene rather than a sandbox, since the child runs as the same user. A PDF read by path is cached only if its hash is unchanged after the read, so an upload that replaces the file mid-read cannot store its text under the old file's hash. `read_document_text` hands `pdf_reader` the file's path, so the API process holds a whole PDF only when its pages go to the model (step 2). The contract is unchanged. `read_pdf_text` still never raises: a child that dies or runs past its limit (300 s for the text layer, or OCR's 240 s budget plus 120 s) comes back as `method=\"none\"` with a note. A Word or Excel parse error still raises, as `IsolatedError` naming the original. At most two parser processes run at once. A caller waits at most 30 seconds for a free slot, and that wait counts against its own time limit. If no slot frees up in time, the file is reported as \"busy, try again\" rather than unreadable. An upload, PDF or not, gets a 503 and nothing is stored. Drive sync retries the file on its next tick. A PDF read elsewhere comes back empty with a note saying so, and is not cached. OCR has its own two-slot limit, so a long OCR never holds up a quick text-layer parse. A scan waits up to one OCR run (240 s) for a slot, and if none frees up it gives its pages back to the hourly inbound budget. Only when there is no usable Python to start does the parse run in-process as before, and it does not take a slot. Failing to start a process for lack of memory is reported as unreadable, because that is the worst moment to parse inside the API. A Word or Excel file that cannot be read is a 422 from `POST /documents` and the preview, not a 500. An Excel workbook is read for at most 200,000 rows (blank ones included) and 1,024 columns: openpyxl walks every row a sheet's `<dimension>` claims, so a few-KB file claiming A1:XFD1048576 would otherwise hold a parser slot until the timeout. `POST /documents` accepts `.pdf`, `.docx`, `.doc`, `.xlsx`, `.xlsm`, `.csv`, `.md` and `.txt`; text formats are decoded as UTF-8 with bad bytes replaced.\n\n`POST /documents` streams the upload to its staging file 1 MB at a time and returns 413 as soon as it passes 50 MB, instead of reading the whole body into memory first. The stored copy is copied from that file.\n\n### The SME review gate\n\nBefore the distance threshold applies, both `retrieve` and `retrieve_failures` drop **withheld** items — anything the review store reports as `pending` or `rejected` (`ReviewStore.get_withheld_filenames`). Approved items with a priority set are then sorted ahead of the rest.\n\nKnowledge shipped with the product registers as a trusted default (`approved`, NULL `reviewed_at`), so nothing is withheld until a human deliberately queues a domain for curation or rejects something. See the **SME Knowledge Review** section for the full state model. The filter matches on the `(domain, filename)` pair Chroma carries. A bare filename was ambiguous *and* caller-choosable — `POST /knowledge/builtin` takes any `(domain, filename)`, so uploading a file named after a shipped doc in another domain silently withheld that shipped doc from every specialist. Failure case studies are keyed in their own namespace, so a user-authored one cannot collide with a shipped one.\n\nThe formatted context is injected into the **user turn** (wrapped in `<relevant_knowledge>` by `BaseAgent.analyze`), never into a cached system block — per-query hits would otherwise poison the prompt cache.\n\n### Skills (Playbooks) collection\n\nSkills — called **playbooks** in the UI, where they are a tab on Workflows — are Markdown procedures with YAML frontmatter, indexed one document per skill in their own `skills` collection (name + description + when_to_use; the body is fetched by `load_skill`). Built-ins ship under `knowledge/builtin/skills/`; company skills live in `company/skills/`. Customizing a built-in (a `PUT` from the Playbooks tab; the chat tools refuse built-ins) writes a company skill of the same name flagged `customizes_builtin: true`, which **replaces** the built-in: it shadows the built-in everywhere (chat, and the workflows that follow it — see the Workflows section), and deleting it reverts. A built-in cannot be deleted, so \"delete\" **hides** it via `company/skills/.hidden.yaml`; hidden and shadowed built-ins are dropped from the index. `sync_builtin_skill_index` reconciles the built-in rows at startup, after a client-slot restore, and after a fixture reset — so a built-in added in a release reaches an already-seeded index, and a removed one leaves it.\n\n**Drafts.** The Executive's chat tools never change the library directly: `create_skill`, `update_skill` and `delete_skill` save a draft as JSON in `company/skills/.drafts/` (one per name; a newer proposal replaces the pending one). Drafts are not `*.md`, so they are never listed, indexed, searched, loaded or followed. A person approves a draft on the Playbooks tab (`POST /skill-drafts/{name}/approve`), which applies it through the ordinary skills operations, or discards it. Each save gets a fresh `id`, and approve/discard name the version the person saw: a proposal swapped in after review (say by a crafted inbound message) is never applied unseen, and approving never removes a newer draft that arrived meanwhile. This matches the draft → human-approval shape of custom workflows: a crafted inbound message can propose a playbook, but only a person can put one in effect.",+ "markdown": "Retrieval is handled by `openexecutive.knowledge.retriever.retrieve`, querying ChromaDB collections through `openexecutive.knowledge.store.ChromaDBStore`. When the caller passes `record_source`, `retrieve` also names every document its block quotes — company uploads by filename, built-in knowledge by a title made from its filename, Notion pages, synced Drive files by name with their Drive link, synced Confluence pages by title with their page link, research notes by date, and earlier published documents with a link to `/artifacts/<id>` — so the web chat can list what an answer looked at: the chat route's own search, and each specialist's that answered (see the Lifecycle and API sections). The recording never raises, and a retrieval that fails leaves the specialist without context rather than failing the turn. Eight collections back the system:\n\n- **BUILTIN** — curated MBA-style domain knowledge shipped with the repo, including a `sales` domain (founder-led sales, qualification and pipeline, proposals and pricing conversations, follow-up and forecasting) and solo-founder guides in finance, marketing, legal and HR. Seeded at boot by `loader.seed_builtin_knowledge` (FAILURES by `seed_failures`): the whole tree on an empty collection, otherwise only the `SHIPPED_BUILTIN_FILES` entries with no chunks yet — so a doc added in a later release reaches an install seeded by an earlier one, instead of shipping, registering as approved, and never being retrievable. Presence is judged from each row's `source` path relative to `knowledge/builtin/` (not the absolute path, which moves with the install), read in one metadata scan narrowed to seeded rows so the external OER corpus sharing the collection is never pulled. A complete store indexes nothing.\n- **COMPANY** — customer documents ingested via the `/documents` endpoint. Curated uploads only — synced Notion pages, Drive files and Confluence pages do not land here. A document's identity is its **filename**: chunk metadata and the chunk id both derive from it, so re-uploading upserts in place rather than duplicating, and `DELETE /documents/{filename}` matches the rows it claims to remove. Inbound attachments are **not** here — see ATTACHMENTS below; rows left by the earlier scheme are migrated out on boot. A scanned PDF (no text layer) is converted before chunking — see \"Reading scanned PDFs\" below — so it no longer indexes zero chunks while reporting `indexed`.\n- **NOTION_WIKI** — optional Notion sync (`NOTION_SYNC_ENABLED`) incrementally re-indexes pages shared with a Notion internal integration into a separate `notion_wiki` collection (`type=notion`). A Notion share is inherited and multi-writer, so this text is unreviewed relative to curated company docs; the retriever labels it as such and ranks it below COMPANY. Un-shared pages are purged on the next tick.\nSynced Drive files and Notion pages are listed on the knowledge page next to uploads (one Company documents list, each row badged with its source, its sync time and a link to the original), read from the sync state files rather than Chroma; each connected source shows its last run, its last error (a failed tick, or a count of files or pages a tick couldn't index; the next clean tick clears it) and a **Sync now** button (`POST /documents/sources/{id}/sync`). Uploads from the page no longer ask for a domain: they go in as `general`, which every specialist's COMPANY filter already includes.\n\n- **DRIVE** — optional Google Drive folder sync (`DRIVE_SYNC_ENABLED`, `knowledge.drive_sync`). It incrementally re-indexes the files in `DRIVE_SYNC_FOLDER_IDS`, and their subfolders to four levels, into a separate `drive_docs` collection (`type=drive`). It reads the Drive v3 API directly as a service account with `drive.readonly` (`knowledge.drive_client`), not through the MCP gateway, so it sees only folders shared with that account and can change nothing. Google Docs and Slides are exported as text and Sheets as `.xlsx`. PDF, `.docx`, `.xlsx` and plain text files go through the loader's extractors, with no model call, so a scanned PDF yields nothing. Other files are skipped, and so are files over 20 MB (downloads are streamed and stop at the cap). A `.docx` or `.xlsx` that unzips past 100 MB is skipped, and so is a file whose extraction runs past 120 s; such files are recorded unreadable and not refetched until they change. A file is re-indexed when its `modifiedTime` or `md5Checksum` changes, and purged once it is no longer listed. Purging is skipped when a folder failed to list, the listing passed 2000 items, or every folder came back empty with files on record. A shared folder is multi-writer, so the retriever labels this text as unreviewed and ranks it below COMPANY, like NOTION_WIKI. Each chunk is labelled `[drive · file id … · synced … · \"<name>\"]`. The id and time come first because the sync wrote them. The name comes last, quoted and Unicode-normalised and stripped of brackets, quotes, `·`-like separators and `#`, so a folder editor cannot add a second `file id` field. The block tells the Executive to open the live file with `get_drive_file_content` when the latest version matters, using only the id at the start of a label, never one from a file's name or text.\n- **ONEDRIVE** — optional OneDrive / SharePoint folder sync (`ONEDRIVE_SYNC_ENABLED`, `knowledge.onedrive_sync`), DRIVE's twin: the files in `ONEDRIVE_SYNC_FOLDERS` (`<drive id>/<item id>` pairs) go into a separate `onedrive_docs` collection (`type=onedrive`, keyed `<drive id>:<item id>`). It calls Microsoft Graph directly (`knowledge.onedrive_client`, GETs only) as the Executive's own Microsoft 365 sign-in, with a token the Microsoft server's launcher mints for file reads only, because Microsoft has no per-folder service account; the folder list is the scope. PDF, Word, Excel and plain text files go through the same extractors, caps and deadlines as Drive (imported from `drive_sync`); PowerPoint, older Office formats, OpenDocument and RTF are fetched as Graph's PDF rendering first. A file is re-indexed when its `lastModifiedDateTime` or `cTag` changes; purging, its skip rules and the locking are Drive's. Retrieved right after DRIVE, labelled `[onedrive · item <drive id>:<item id> · synced … · \"<name>\"]` with the name defanged the same way, and the block tells the Executive to open the live file with its OneDrive tools using only the ids after `item`.\n- **CONFLUENCE** — optional Confluence space sync (`CONFLUENCE_SYNC_ENABLED`, `knowledge.confluence_sync`), for Cloud and Server/Data Center alike. Each tick lists every page in `CONFLUENCE_SYNC_SPACE_KEYS` over the v1 REST API (`knowledge.confluence_client`, directly with `httpx`, not through the MCP gateway; a personal access token as Bearer, or a username and API token as Basic) and re-indexes the pages whose `version.number` changed into a separate `confluence_wiki` collection (`type=confluence`), at most `CONFLUENCE_MAX_PAGES_PER_SCAN` per tick, newest first. Storage-format XHTML becomes Markdown through stdlib `html.parser` (`knowledge.confluence_storage`): code macros become fenced blocks, callout macros keep their body, and dynamic macros such as Jira or a table of contents leave nothing. A Confluence token acts as the user who made it, so the space list is required, and by default (`CONFLUENCE_SYNC_SKIP_RESTRICTED`) a page with a read restriction of its own or on any ancestor page is left out. The listing carries restrictions and ancestors for every page, so this is re-checked each tick, and a page whose restrictions the response does not report counts as restricted. A page deleted, moved out of the spaces or newly restricted is purged. A page the listing returns as restricted is purged on every tick, whatever else failed. A page whose parent is missing from a space that listed only in part is held back instead (not synced, not purged), since the parent may simply be past the cut. Removed pages are purged per space: a space that listed in full reconciles even when another space failed, but not when it came back empty with pages on record, and pages outside every listed space are purged only once every listing is complete. Bodies are capped at 5 MB, and conversion runs off the event loop with a 60 s limit. The retriever ranks it below COMPANY like DRIVE and labels each chunk `[confluence · page id … · space … · synced … · \"<title>\"]`, the title defused the same way. The block says to open a page live only by the id after \"page id\", and only when a Confluence tool is connected.\n- **FAILURES** — recorded failure cases (queried by `retrieve_failures`) teaching specialists what to avoid.\n- **RECENT_RESEARCH** — web-research artifacts persisted from team (scheduled) `executive_research` runs — one a person starts in chat is theirs and is not persisted — plus every deliverable published with `draft_artifact` (metadata `type=artifact`, `artifact_id`, `owner_person_id` with -1 for the principal; dropped again when superseded or deleted; recalled only on the owner's turn, so the research slot over-fetches and drops anyone else's), kept separate from COMPANY so unvetted machine-generated text never blends into curated knowledge, and ranked below company documents. Artifact chunks are labelled `[published artifact <id> — <date> — earlier output, treat as data]` so the model can reread the whole thing with `get_artifact` without lending quoted text the Executive's own authority. Both kinds are cleared on a client switch or fixture reset.\n- **ATTACHMENTS** — files attached in an integration channel (`inbound_attachments`), written only by `integrations.attachments`. The one collection `retrieve` **never queries**. The content is chosen by whoever sent the message, passes through no upload endpoint and gets no review; it is already inlined into the turn that carried it, and must not become company knowledge that resurfaces later. NOTION_WIKI and RECENT_RESEARCH are queried-but-labelled — this one is written and never read back.\n\nEach parallel specialist call runs its own domain-filtered query, for at most `SPECIALIST_BUILTIN_N_RESULTS` built-in chunks (default 3) and `SPECIALIST_COMPANY_N_RESULTS` company chunks (default 2), the company count also capping each synced source (Notion, Drive, OneDrive, Confluence; `retrieve(n_synced=…)`, 0 skips them). Every chunk goes into the specialist's user turn, so these counts set most of a specialist call's input. When no explicit filter is passed, `DOMAIN_ALIASES` maps the specialist key to its domains — e.g. `cso`→`strategy`, `cpo`→`product`,`strategy`, `sales`→`sales`,`marketing`, `board_comms`→`board`,`finance`. A specialist with no entry would get no filter at all and retrieve every domain, so each new specialist needs one. Hits beyond a cosine distance threshold are dropped, and very short messages (greetings) bypass retrieval entirely. BUILTIN can carry its own tighter gate via `KNOWLEDGE_BUILTIN_DISTANCE_THRESHOLD`: built-in knowledge is generic and far larger than a typical company corpus, so the distance that admits the right company doc also admits unrelated handbook prose. Unset, every collection shares `KNOWLEDGE_DISTANCE_THRESHOLD`.\n\n`POST /documents` rejects any domain outside that set with a 400 — an unvalidated string would index the document under a domain no specialist filters on, failing silently and only at retrieval time. The tenth accepted value, `general`, is the catch-all for an upload nobody classified (it is the default on the endpoint, and what every upload from the knowledge page gets — the page no longer asks for a domain). It maps to no specialist, so every specialist's **company** filter is widened to include it — an unclassified document is visible to all rather than to none. That widening is deliberately not applied to BUILTIN, whose rows are shared with external OER sources that fall back to `general` when they declare no domains; fanning those into every specialist would blend unvetted third-party material into every answer.\n\nMulti-domain chunks fan out to one row per declared domain, so `_dedupe_by_text` drops duplicate-text hits while preserving relevance order before results are returned.\n\n### Embeddings\n\nEvery collection embeds with one shared all-MiniLM-L6-v2 ONNX session (`knowledge.store._embedding_function`), the same model and vectors as Chroma's default. Chroma's own default reloads the model on every call and keeps ONNX Runtime's memory arena, so a 300-chunk upload peaked near 760 MB and each query spent ~240 ms reloading. The shared session is built once, runs without the arena and embeds four texts at a time (the model pads every text to 256 tokens, so batch size sets the peak): ~300 MB and ~30 ms. It answers to Chroma's name `default` with an empty config, because every existing collection records that name and Chroma refuses a collection opened with a differently named function; and it must not be a `DefaultEmbeddingFunction` instance, which Chroma swaps for a fresh default on every embed.\n\n### Reading scanned PDFs\n\npypdf returns only a PDF's saved text layer, so a scan, a phone photo saved as PDF, or a deck printed to PDF as images used to come back empty everywhere a PDF is read. `knowledge.pdf_reader.read_pdf_text` tries three readers in order and never raises:\n\n1. **Text layer** (pypdf) — used whenever it yields real text (at least ~20 visible characters per page, so a scan carrying only page numbers still falls through).\n2. **The deployment's own model, through its provider's PDF support** — opt-in, `PDF_PROVIDER_READING=true` (off by default: an upgrade never starts sending scanned PDFs off the server; this also means an Anthropic-direct deployment OCRs scans locally until it opts in) — the pages as a base64 `document` block, transcribed to Markdown by `PDF_VISION_MODEL` (unset: `DEFAULT_MODEL`, the model the deployment already runs on), `PDF_VISION_PAGES_PER_CALL` pages per request (default 20), three requests at a time. Each provider carries the block its own way: Anthropic reads it natively; on OpenRouter the translator makes it an OpenAI `file` part and the provider adds the `file-parser` plugin — engine `native` for Claude or any model the catalog lists with `file` input (OpenAI's GPT-4o and later, Gemini…), else `PDF_OPENROUTER_ENGINE` (default `mistral-ocr`, $2 per 1,000 pages); a local OpenAI-compatible server gets the `file` part only with `LOCAL_PDF_INPUT=true` (e.g. `LOCAL_BASE_URL` pointed at OpenAI's own API). `providers.registry.pdf_input_supported` decides; a model that cannot take one never receives it (the feature gate swaps a stray `document` block for a note). Turning it on sends scanned pages off the server — to the provider, and on OpenRouter possibly to its parser (Mistral OCR) — and the destination is logged once per process.\n3. **Local OCR** — pages rendered with pypdfium2 and read by RapidOCR (ONNX, models bundled in the wheel): offline, no key, no per-page cost. The default reader while step 2 is off, what a local model without PDF input gets, and the fallback on every provider when step 2 errors, refuses or returns nothing. `PDF_OCR_ENABLED=false` turns it off.\n\n`PDF_VISION_MAX_PAGES` (default 100) caps the pages read, with a note saying how many were; a file a channel delivered (`inbound=True`) is capped at `PDF_INBOUND_MAX_PAGES` (30) and metered by a rolling `PDF_INBOUND_PAGES_PER_HOUR` (300) budget. OCR is bounded too: a page renders within a 25-megapixel budget whatever its MediaBox claims (a crafted 20000pt page would otherwise allocate gigabytes), at most two documents OCR at once, and each stops after 240 s with what it has. A PDF of more than 2,000 pages is refused before any page is parsed (the caps above bound conversion, not the text-layer pass). A Claude slice that runs into the output limit is split in half until each part fits; a single page that still does not fit keeps what was written, marked `[transcription cut off here]`, and the note says so. A result is cached in process by the file's sha256 (32 entries) so a re-read does not re-bill; a failure is not cached. `loader.read_document_text` / `extract_text_from_file_async` route a PDF through it and every other type through the sync extractor, and are what `ingest_file` (uploads, boot reconcile), the `/documents/{filename}` preview, intake uploads, chat attachments, the email poller, `oe__read_file` and the Executive's `read_document` tool use.\n\n### Parsing in a child process\n\nParsing a large document builds hundreds of MB of small Python objects, and CPython keeps the memory they lived in, so a parse inside the API process left its peak there for good. On a 34 MB, 131-page text-layer PDF uploaded through `POST /documents`, uvicorn went from 272 MB to a 440 MB peak and stayed at 378 MB. On a 30-page scan read by OCR it peaked at 787 MB and stayed at 446 MB. Now the text-layer pass and local OCR in `pdf_reader`, and the PDF, Word and Excel extractors behind `loader.extract_text_from_file`, run in a short-lived child process (`knowledge.isolated.run_isolated`). Everything the child allocated goes back to the OS when it exits. The same uploads now peak at 324 MB and 297 MB and settle at 318 MB and 290 MB, and what remains is embedding. The OCR child itself still peaks near 600 MB while it reads, then exits.\n\nThe call goes to the child pickled on stdin and the answer comes back as JSON, so the API never unpickles anything a hostile file could shape. Both of the child's output streams go to temp files, not the API's memory. The child logs nothing, because pypdf logs text taken from the file and a crafted 9 KB PDF made it write megabytes. Only the last 2,000 characters of its stderr are read back, and an answer over 32 MB is refused. The child inherits only what it needs to start Python (`PATH`, `HOME`, `PYTHONPATH`, the library path, locale, temp dir), not the deployment's keys. That is hygiene rather than a sandbox, since the child runs as the same user. A PDF read by path is cached only if its hash is unchanged after the read, so an upload that replaces the file mid-read cannot store its text under the old file's hash. `read_document_text` hands `pdf_reader` the file's path, so the API process holds a whole PDF only when its pages go to the model (step 2). The contract is unchanged. `read_pdf_text` still never raises: a child that dies or runs past its limit (300 s for the text layer, or OCR's 240 s budget plus 120 s) comes back as `method=\"none\"` with a note. A Word or Excel parse error still raises, as `IsolatedError` naming the original. At most two parser processes run at once. A caller waits at most 30 seconds for a free slot, and that wait counts against its own time limit. If no slot frees up in time, the file is reported as \"busy, try again\" rather than unreadable. An upload, PDF or not, gets a 503 and nothing is stored. Drive sync retries the file on its next tick. A PDF read elsewhere comes back empty with a note saying so, and is not cached. OCR has its own two-slot limit, so a long OCR never holds up a quick text-layer parse. A scan waits up to one OCR run (240 s) for a slot, and if none frees up it gives its pages back to the hourly inbound budget. Only when there is no usable Python to start does the parse run in-process as before, and it does not take a slot. Failing to start a process for lack of memory is reported as unreadable, because that is the worst moment to parse inside the API. A Word or Excel file that cannot be read is a 422 from `POST /documents` and the preview, not a 500. An Excel workbook is read for at most 200,000 rows (blank ones included) and 1,024 columns: openpyxl walks every row a sheet's `<dimension>` claims, so a few-KB file claiming A1:XFD1048576 would otherwise hold a parser slot until the timeout. `POST /documents` accepts `.pdf`, `.docx`, `.doc`, `.xlsx`, `.xlsm`, `.csv`, `.md` and `.txt`; text formats are decoded as UTF-8 with bad bytes replaced.\n\n`POST /documents` streams the upload to its staging file 1 MB at a time and returns 413 as soon as it passes 50 MB, instead of reading the whole body into memory first. The stored copy is copied from that file.\n\n### The SME review gate\n\nBefore the distance threshold applies, both `retrieve` and `retrieve_failures` drop **withheld** items — anything the review store reports as `pending` or `rejected` (`ReviewStore.get_withheld_filenames`). Approved items with a priority set are then sorted ahead of the rest.\n\nKnowledge shipped with the product registers as a trusted default (`approved`, NULL `reviewed_at`), so nothing is withheld until a human deliberately queues a domain for curation or rejects something. See the **SME Knowledge Review** section for the full state model. The filter matches on the `(domain, filename)` pair Chroma carries. A bare filename was ambiguous *and* caller-choosable — `POST /knowledge/builtin` takes any `(domain, filename)`, so uploading a file named after a shipped doc in another domain silently withheld that shipped doc from every specialist. Failure case studies are keyed in their own namespace, so a user-authored one cannot collide with a shipped one.\n\nThe formatted context is injected into the **user turn** (wrapped in `<relevant_knowledge>` by `BaseAgent.analyze`), never into a cached system block — per-query hits would otherwise poison the prompt cache.\n\n### Skills (Playbooks) collection\n\nSkills — called **playbooks** in the UI, where they are a tab on Workflows — are Markdown procedures with YAML frontmatter, indexed one document per skill in their own `skills` collection (name + description + when_to_use; the body is fetched by `load_skill`). Built-ins ship under `knowledge/builtin/skills/`; company skills live in `company/skills/`. Customizing a built-in (a `PUT` from the Playbooks tab; the chat tools refuse built-ins) writes a company skill of the same name flagged `customizes_builtin: true`, which **replaces** the built-in: it shadows the built-in everywhere (chat, and the workflows that follow it — see the Workflows section), and deleting it reverts. A built-in cannot be deleted, so \"delete\" **hides** it via `company/skills/.hidden.yaml`; hidden and shadowed built-ins are dropped from the index. `sync_builtin_skill_index` reconciles the built-in rows at startup, after a client-slot restore, and after a fixture reset — so a built-in added in a release reaches an already-seeded index, and a removed one leaves it.\n\n**Drafts.** The Executive's chat tools never change the library directly: `create_skill`, `update_skill` and `delete_skill` save a draft as JSON in `company/skills/.drafts/` (one per name; a newer proposal replaces the pending one). Drafts are not `*.md`, so they are never listed, indexed, searched, loaded or followed. A person approves a draft on the Playbooks tab (`POST /skill-drafts/{name}/approve`), which applies it through the ordinary skills operations, or discards it. Each save gets a fresh `id`, and approve/discard name the version the person saw: a proposal swapped in after review (say by a crafted inbound message) is never applied unseen, and approving never removes a newer draft that arrived meanwhile. This matches the draft → human-approval shape of custom workflows: a crafted inbound message can propose a playbook, but only a person can put one in effect.","mermaid": "flowchart TD\n Call[\"Specialist call<br/>(domain-filtered)\"] --> Q[\"retrieve(query, domain)\"]\n Q --> Builtin[(BUILTIN)]\n Q --> Company[(COMPANY)]\n Q --> NotionWiki[(NOTION_WIKI)]\n Q --> DriveDocs[(DRIVE)]\n Q --> ConfluenceWiki[(CONFLUENCE)]\n Q --> Failures[(FAILURES)]\n Q --> Research[(RECENT_RESEARCH)]\n Attach[\"integrations.attachments\"] --> Attachments[(ATTACHMENTS<br/>never queried)]\n Builtin --> Dedupe[\"dedupe by text<br/>+ distance gate\"]\n Company --> Dedupe\n NotionWiki --> Dedupe\n DriveDocs --> Dedupe\n ConfluenceWiki --> Dedupe\n Failures --> Dedupe\n Research --> Dedupe\n Dedupe --> Inject[\"Inject into USER turn<br/>(not cached)\"]\n class Call entry\n class Q compute\n class Dedupe compute\n class Builtin storage\n class Company storage\n class NotionWiki storage\n class DriveDocs storage\n class ConfluenceWiki storage\n class Failures storage\n class Research storage\n class Attachments storage\n class Attach compute\n class Inject hot",- "generated_at": "2026-10-03T15:20:51Z"+ "generated_at": "2026-10-09T05:00:00Z"}diff --git a/packages/core/openexecutive/config.py b/packages/core/openexecutive/config.pyindex c82a51d6..307302e1 100644--- a/packages/core/openexecutive/config.py+++ b/packages/core/openexecutive/config.py@@ -151,6 +151,11 @@ class Settings(BaseSettings):)knowledge_builtin_n_results: int = Field(5, alias="KNOWLEDGE_BUILTIN_N_RESULTS")knowledge_company_n_results: int = Field(3, alias="KNOWLEDGE_COMPANY_N_RESULTS")+ # A specialist's own retrieval (router._retrieve_for_call) takes fewer+ # chunks than the Executive's: it answers one narrow question. The company+ # count also caps each synced source (Notion, Drive, OneDrive, Confluence).+ specialist_builtin_n_results: int = Field(3, alias="SPECIALIST_BUILTIN_N_RESULTS", ge=0)+ specialist_company_n_results: int = Field(2, alias="SPECIALIST_COMPANY_N_RESULTS", ge=0)# Max parallel `consult_specialist` calls dispatched in one chat turn.# 0 (default) is inert: resolve_fanout_cap() falls back to the specialist@@ -570,10 +575,11 @@ class Settings(BaseSettings):# Upper bound on a single tool result's characters before it enters the# prompt. A circuit breaker against an unbounded result (a large document# fetch) dominating a turn and then being re-sent on every remaining- # iteration of the tool loop — deliberately set high enough that ordinary- # tool output never reaches it. Applies to every tool, not just MCP.+ # iteration of the tool loop. 20,000 characters (about 5,500 tokens) is+ # well above a typical result; a longer one is cut with a marker that+ # asks for a narrower request. Applies to every tool, not just MCP.tool_result_max_chars: int = Field(- 50_000, alias="TOOL_RESULT_MAX_CHARS", ge=1_000+ 20_000, alias="TOOL_RESULT_MAX_CHARS", ge=1_000)# Workflow action steps may also act through one short sandboxed script# (`run_script`, workflows/step_script.py) that calls the step's own toolsdiff --git a/packages/core/openexecutive/knowledge/retriever.py b/packages/core/openexecutive/knowledge/retriever.pyindex 5c556593..4b717db6 100644--- a/packages/core/openexecutive/knowledge/retriever.py+++ b/packages/core/openexecutive/knowledge/retriever.py@@ -252,9 +252,13 @@ def retrieve(distance_threshold: float | None = None,builtin_distance_threshold: float | None = None,record_source: Callable[..., None] | None = None,+ n_synced: int = 3,) -> str:"""Retrieve knowledge for ``query`` as a prompt-ready block ("" for none).+ ``n_synced`` is how many chunks each synced source (Notion, Drive,+ OneDrive, Confluence) may add.+``record_source(kind, title, url=None)`` is told about each document theblock quotes — the web chat lists them under the answer (see``orchestrator.answer_sources``).@@ -369,8 +373,8 @@ def retrieve(query_text=query,collection=ChromaDBStore.NOTION_COLLECTION,domain_filter=effective_domains,- n_results=3,- )+ n_results=n_synced,+ ) if n_synced > 0 else []notion_results = [r for r in raw_notion if _passes_threshold(r, distance_threshold)]@@ -381,8 +385,8 @@ def retrieve(query_text=query,collection=ChromaDBStore.DRIVE_COLLECTION,domain_filter=effective_domains,- n_results=3,- )+ n_results=n_synced,+ ) if n_synced > 0 else []drive_results = [r for r in raw_drive if _passes_threshold(r, distance_threshold)]# Synced OneDrive folders (knowledge.onedrive_sync) — the same isolation@@ -391,8 +395,8 @@ def retrieve(query_text=query,collection=ChromaDBStore.ONEDRIVE_COLLECTION,domain_filter=effective_domains,- n_results=3,- )+ n_results=n_synced,+ ) if n_synced > 0 else []onedrive_results = [r for r in raw_onedrive if _passes_threshold(r, distance_threshold)]# Synced Confluence spaces (knowledge.confluence_sync) — isolated and@@ -401,8 +405,8 @@ def retrieve(query_text=query,collection=ChromaDBStore.CONFLUENCE_COLLECTION,domain_filter=effective_domains,- n_results=3,- )+ n_results=n_synced,+ ) if n_synced > 0 else []confluence_results = [r for r in raw_confluence if _passes_threshold(r, distance_threshold)]diff --git a/packages/core/openexecutive/orchestrator/executive.py b/packages/core/openexecutive/orchestrator/executive.pyindex 4830f4fc..118788b5 100644--- a/packages/core/openexecutive/orchestrator/executive.py+++ b/packages/core/openexecutive/orchestrator/executive.py@@ -39,7 +39,7 @@ from openexecutive.memory.workspace_settings import (pin_turn_principal_role,pin_turn_workspace_mode,)-from openexecutive.orchestrator import take_the_lead+from openexecutive.orchestrator import take_the_lead, tool_groupsfrom openexecutive.orchestrator.action_chips import summarize_actionfrom openexecutive.orchestrator.activity_labels import (fallback_activity,@@ -1921,19 +1921,20 @@ class Executive:# Solo withholds the team-only tools before the sort, so each mode# has its own stable, sorted tool prefix (as do an unattended run# and a turn private to the principal).- client_tools = sorted(- (- t for t in filter_tools_for_workspace_mode(- [- *SPECIALIST_TOOLS, *_ALL_SKILL_TOOLS, *self._mcp_tools,- *self._script_tools, *delegation_tools,- ],- workspace_mode,- )- if t["name"] not in not_offered- ),- key=lambda t: t["name"],+ # The less common tools are offered through open_tools/use_tool+ # rather than one by one (tool_groups), so the list is short and+ # stays the same whichever groups the turn opens.+ direct_tools, deferred_tools = tool_groups.split(+ t for t in filter_tools_for_workspace_mode(+ [+ *SPECIALIST_TOOLS, *_ALL_SKILL_TOOLS, *self._mcp_tools,+ *self._script_tools, *delegation_tools,+ ],+ workspace_mode,+ )+ if t["name"] not in not_offered)+ client_tools = sorted(direct_tools, key=lambda t: t["name"])tools_with_cache: list[dict[str, Any]] = [*client_tools[:-1],{**client_tools[-1], "cache_control": {"type": "ephemeral", "ttl": "1h"}},@@ -2033,6 +2034,20 @@ class Executive:if final_msg.stop_reason != "tool_use":return+ # open_tools is answered here; a use_tool call becomes the call it+ # stands for (same id), so every guard, handler, chip and audit+ # row below sees the tool's own name, as for a direct call.+ group_results: dict[str, str] = {}+ for i, tu in enumerate(tool_uses):+ if tu["name"] == tool_groups.OPEN_TOOLS:+ group_results[tu["id"]] = tool_groups.open_result(tu["input"], deferred_tools)+ elif tu["name"] == tool_groups.USE_TOOL:+ call, error = tool_groups.unwrap(tu)+ if call is None:+ group_results[tu["id"]] = error or ""+ else:+ tool_uses[i] = call+specialist_tool_uses = [tu for tu in tool_uses if tu["name"] == "consult_specialist"]skill_tool_uses = [tu for tu in tool_uses if tu["name"] in turn_handlers]# Dispatch guard: a tool this mode does not offer never runs, even@@ -2175,7 +2190,7 @@ class Executive:# Signal that tool calls are in flight so the client can show progress.yield self._THINKING- results_by_id: dict[str, str] = {}+ results_by_id: dict[str, str] = dict(group_results)event_cursor = len(debug_collector._events) if debug_collector else 0session_id = getattr(current_session.get(), "session_id", None)diff --git a/packages/core/openexecutive/orchestrator/router.py b/packages/core/openexecutive/orchestrator/router.pyindex f62443a6..7baf2ce1 100644--- a/packages/core/openexecutive/orchestrator/router.py+++ b/packages/core/openexecutive/orchestrator/router.py@@ -284,13 +284,18 @@ async def _retrieve_for_call(rather than failing the turn: every specialist's retrieval is gatheredtogether, so one exception here used to lose the whole answer."""+ from openexecutive.config import get_settingsfrom openexecutive.knowledge.retriever import retrieve+ settings = get_settings()try:return await asyncio.to_thread(retrieve,query=call["query"],specialist_name=call["specialist"],+ n_builtin=settings.specialist_builtin_n_results,+ n_company=settings.specialist_company_n_results,+ n_synced=settings.specialist_company_n_results,record_source=record_source,)except Exception:diff --git a/packages/core/openexecutive/orchestrator/tool_groups.py b/packages/core/openexecutive/orchestrator/tool_groups.pynew file mode 100644index 00000000..46821350--- /dev/null+++ b/packages/core/openexecutive/orchestrator/tool_groups.py@@ -0,0 +1,196 @@+"""Tools the Executive opens when it needs them, instead of carrying them all.++Every model call in a chat turn re-sends the whole tool list, and the list is+the first thing in the cached prefix. Most answers use none of the less+common tools (goals, the watch list, workflows, documents, …), yet their+descriptions were most of that prefix. Here those tools sit in named groups:+the turn offers two small tools in their place,++ * ``open_tools(group)`` returns the group's tools (name, description, input+ schema) as a tool result, and+ * ``use_tool(name, input)`` runs one of them,++so the offered list stays the same on every call of every turn (a group+opening never changes the cached prefix) and a group's descriptions are paid+for only on a turn that needs them. A ``use_tool`` call is rewritten into the+tool's own call before dispatch (``unwrap``), so it meets exactly the guards,+handlers, chips and audit rows a direct call would.+"""++from __future__ import annotations++import json+from collections.abc import Iterable+from typing import Any++OPEN_TOOLS = "open_tools"+USE_TOOL = "use_tool"++# group -> (what it is for, the tools in it). Each tool belongs to one group.+# Everything not listed here stays on the direct list.+GROUPS: dict[str, tuple[str, tuple[str, ...]]] = {+ "calendar": (+ "create, start or cancel calendar events and meetings",+ ("cancel_calendar_event", "create_calendar_event", "create_instant_meeting"),+ ),+ "people": (+ "add, change or archive people on the roster, department heads, roster requests",+ ("archive_person", "resolve_roster_request", "set_department_head", "upsert_person"),+ ),+ "goals": (+ "department goals, decision outcomes, open loops",+ (+ "assign_open_loop",+ "close_open_loop",+ "create_goal",+ "list_department_goals",+ "list_open_loops",+ "record_decision_outcome",+ "update_department_goal",+ ),+ ),+ "watch_and_alerts": (+ "the watch list, alerts, executive research runs",+ (+ "ack_alert",+ "add_watchlist_entry",+ "create_alert",+ "find_alerts",+ "list_watchlist",+ "remove_watchlist_entry",+ "run_executive_research",+ "tune_watchlist_entry",+ ),+ ),+ "workflows": (+ "draft, save, list, suggest or run workflows; Python jobs on files",+ ("draft_workflow", "list_workflows", "run_python_job", "run_workflow", "save_workflow", "suggest_workflow"),+ ),+ "documents": (+ "draft, read back or list documents (artifacts)",+ ("draft_artifact", "get_artifact", "list_artifacts"),+ ),+ "messaging": (+ "Slack, Telegram and Discord messages, department messages, company broadcasts",+ (+ "send_company_broadcast",+ "send_department_message",+ "send_discord_dm",+ "send_slack_dm",+ "send_telegram_message",+ ),+ ),+ "skills": (+ "create, change or delete skills",+ ("create_skill", "delete_skill", "update_skill"),+ ),+}++DEFERRED: dict[str, str] = {tool: group for group, (_, tools) in GROUPS.items() for tool in tools}+++def open_tools_tool(offered: Iterable[str]) -> dict[str, Any]:+ """The open_tools definition for a turn whose deferred tools are+ ``offered``: it names only the groups and tools that turn may use, so a+ tool a guard withholds is never named. The text depends only on that set,+ which is fixed for each kind of turn, so each keeps its one cached list."""+ names = set(offered)+ groups = {+ group: (purpose, [t for t in tools if t in names])+ for group, (purpose, tools) in GROUPS.items()+ }+ groups = {group: entry for group, entry in groups.items() if entry[1]}+ lines = "\n".join(f"- {group}: {purpose} ({', '.join(tools)})" for group, (purpose, tools) in groups.items())+ return {+ "name": OPEN_TOOLS,+ "description": (+ "Open a group of further tools. You have more tools than the ones listed "+ "directly; they sit in the groups below. When a request needs one, open "+ "its group: the result gives each tool's description and input, and you "+ "then call it with use_tool. Open a group only when the request needs it.\n"+ + lines+ ),+ "input_schema": {+ "type": "object",+ "properties": {"group": {"type": "string", "enum": sorted(groups)}},+ "required": ["group"],+ },+ }+++USE_TOOL_TOOL: dict[str, Any] = {+ "name": USE_TOOL,+ "description": (+ "Run a tool from a group opened with open_tools: its name, and its input "+ "as the object its input schema describes."+ ),+ "input_schema": {+ "type": "object",+ "properties": {+ "name": {"type": "string", "description": "The tool's name, as open_tools listed it."},+ "input": {"type": "object", "description": "The tool's input."},+ },+ "required": ["name", "input"],+ },+}+++def split(tools: Iterable[dict[str, Any]]) -> tuple[list[dict[str, Any]], dict[str, dict[str, Any]]]:+ """The turn's offered tools as (the direct list, the deferred tools by+ name). The direct list carries open_tools and use_tool whenever any tool+ is deferred, so it depends only on what the turn offers, never on which+ groups get opened."""+ direct: list[dict[str, Any]] = []+ deferred: dict[str, dict[str, Any]] = {}+ for tool in tools:+ if tool["name"] in DEFERRED:+ deferred[tool["name"]] = tool+ else:+ direct.append(tool)+ if deferred:+ direct += [open_tools_tool(deferred), USE_TOOL_TOOL]+ return direct, deferred+++def _offered_groups(deferred: dict[str, dict[str, Any]]) -> set[str]:+ return {DEFERRED[name] for name in deferred if name in DEFERRED}+++def open_result(tool_input: Any, deferred: dict[str, dict[str, Any]]) -> str:+ """The tool result for an ``open_tools`` call: the group's tools this turn+ offers, for use_tool."""+ group = tool_input.get("group") if isinstance(tool_input, dict) else None+ if not isinstance(group, str) or group not in GROUPS:+ return json.dumps({"error": f"No such group. Groups: {', '.join(sorted(_offered_groups(deferred)))}."})+ tools = [+ {"name": t["name"], "description": t.get("description", ""), "input_schema": t.get("input_schema", {})}+ for name in GROUPS[group][1]+ if (t := deferred.get(name)) is not None+ ]+ if not tools:+ return json.dumps({"error": f"No such group. Groups: {', '.join(sorted(_offered_groups(deferred)))}."})+ return json.dumps({"group": group, "call_with": USE_TOOL, "tools": tools})+++def unwrap(tool_use: dict[str, Any]) -> tuple[dict[str, Any] | None, str | None]:+ """A ``use_tool`` call as the call it stands for: (the tool's own call,+ same id, None), or (None, the error tool result) when it names no+ deferred tool or carries no input object. The rewritten call goes through+ the turn's usual guards, so a tool the turn doesn't offer is refused there+ exactly as a direct call to it is."""+ raw = tool_use.get("input")+ name = raw.get("name") if isinstance(raw, dict) else None+ if not isinstance(name, str) or name not in DEFERRED:+ return None, json.dumps({+ "error": "use_tool runs only the tools open_tools lists. Call the others directly by name."+ })+ args = raw.get("input") if isinstance(raw, dict) else None+ if isinstance(args, str):+ # Some models send the object as JSON text.+ try:+ args = json.loads(args)+ except ValueError:+ args = None+ if not isinstance(args, dict):+ return None, json.dumps({"error": f"use_tool needs {name}'s input as an object."})+ return {**tool_use, "name": name, "input": args}, Nonediff --git a/packages/core/tests/unit/offered_tools.py b/packages/core/tests/unit/offered_tools.pynew file mode 100644index 00000000..41c5e835--- /dev/null+++ b/packages/core/tests/unit/offered_tools.py@@ -0,0 +1,30 @@+"""What a chat turn offers the model, deferred tools included.++The loop sends the less common tools through open_tools/use_tool+(orchestrator.tool_groups), so the tools in a request are not the whole+offer. ``capture_offered`` records, per loop iteration, the sorted names of+every tool the turn offered before they were split.+"""++from __future__ import annotations++from collections.abc import Iterable, Iterator+from contextlib import contextmanager+from typing import Any+from unittest.mock import patch++from openexecutive.orchestrator import tool_groups+++@contextmanager+def capture_offered() -> Iterator[list[list[str]]]:+ seen: list[list[str]] = []+ real = tool_groups.split++ def spy(tools: Iterable[dict[str, Any]]) -> Any:+ listed = list(tools)+ seen.append(sorted(t["name"] for t in listed))+ return real(listed)++ with patch.object(tool_groups, "split", spy):+ yield seendiff --git a/packages/core/tests/unit/test_department_tools.py b/packages/core/tests/unit/test_department_tools.pyindex b4292aa3..42e05480 100644--- a/packages/core/tests/unit/test_department_tools.py+++ b/packages/core/tests/unit/test_department_tools.py@@ -783,6 +783,7 @@ def test_unattended_loop_neither_offers_nor_runs_create_goal() -> None:from openexecutive.orchestrator.executive import Executivefrom openexecutive.orchestrator.schedule_tools import set_sessionfrom openexecutive.orchestrator.session import Session+ from tests.unit.offered_tools import capture_offereddept_store.create_department("Engineering")tool_use = SimpleNamespace(@@ -798,18 +799,22 @@ def test_unattended_loop_neither_offers_nor_runs_create_goal() -> None:async def _go() -> list[Any]:with (+ capture_offered() as seen,patch("openexecutive.orchestrator.executive.get_provider", return_value=provider),set_session(Session(unattended=True)),):- return [+ items = [item async for item in Executive()._stream_agent_loop(system_blocks=[], messages=[{"role": "user", "content": "x"}],model="claude-test", workspace_mode="team",)]+ offered_out.extend(seen[0])+ return items+ offered_out: list[str] = []items = asyncio.run(_go())- offered = {t.get("name") for t in provider.calls[0]["tools"]}+ offered = set(offered_out)assert "create_goal" not in offered and "update_department_goal" in offeredresult = json.loads(provider.calls[1]["messages"][-1]["content"][0]["content"])assert "unattended" in result["error"]diff --git a/packages/core/tests/unit/test_dependable_answers.py b/packages/core/tests/unit/test_dependable_answers.pyindex a2fe6986..84124356 100644--- a/packages/core/tests/unit/test_dependable_answers.py+++ b/packages/core/tests/unit/test_dependable_answers.py@@ -334,7 +334,7 @@ def fake_specialists(monkeypatch: pytest.MonkeyPatch) -> dict[str, Any]:raise answerreturn str(answer)- def retrieve(*, query: str, specialist_name: str, record_source: Any = None) -> str:+ def retrieve(*, query: str, specialist_name: str, record_source: Any = None, **_k: Any) -> str:if record_source is not None:record_source("company", f"{specialist_name} notes.pdf")return f"knowledge for {specialist_name}"diff --git a/packages/core/tests/unit/test_executive_form_patch.py b/packages/core/tests/unit/test_executive_form_patch.pyindex d4528ba2..12fb87d9 100644--- a/packages/core/tests/unit/test_executive_form_patch.py+++ b/packages/core/tests/unit/test_executive_form_patch.py@@ -183,14 +183,16 @@ def test_tool_list_is_cache_stable_and_includes_form_tool() -> None:_run_loop(provider)sent_names = [t["name"] for t in provider.calls[0]["tools"]]+ from openexecutive.orchestrator import tool_groupsfrom openexecutive.workflows import python_job# run_python_job is offered only on the principal's own turns where its# sandbox is installed; this turn isn't the principal's.- expected = sorted(- t["name"] for t in [*SPECIALIST_TOOLS, *_ALL_SKILL_TOOLS]- if t["name"] != python_job.TOOL_NAME+ # The less common tools are behind open_tools/use_tool (tool_groups).+ direct, _ = tool_groups.split(+ t for t in [*SPECIALIST_TOOLS, *_ALL_SKILL_TOOLS] if t["name"] != python_job.TOOL_NAME)+ expected = sorted(t["name"] for t in direct)assert PROPOSE_FORM_VALUES in sent_names# Anthropic server-side tools (web_search) are appended AFTER the cached# client prefix — only the prefix participates in the cache key.diff --git a/packages/core/tests/unit/test_private_turns.py b/packages/core/tests/unit/test_private_turns.pyindex f83d6814..2378561d 100644--- a/packages/core/tests/unit/test_private_turns.py+++ b/packages/core/tests/unit/test_private_turns.py@@ -30,6 +30,7 @@ from openexecutive.departments import store as dept_storefrom openexecutive.memory import episodicfrom openexecutive.orchestrator.schedule_tools import current_sessionfrom openexecutive.orchestrator.session import Session+from tests.unit.offered_tools import capture_offeredfrom openexecutive.people import registry as people_registryfrom openexecutive.people import store as people_store@@ -951,6 +952,8 @@ class _ScriptedStreams:def __init__(self, finals: list[Any]) -> None:self._finals = list(finals)self.calls: list[dict[str, Any]] = []+ # Per call, every tool the turn offered (tests.unit.offered_tools).+ self.offered: list[list[str]] = []def messages_stream(self, **kwargs: Any) -> Any:self.calls.append(kwargs)@@ -997,6 +1000,7 @@ def _run_loop(async def _go() -> None:with (+ capture_offered() as offered,patch("openexecutive.orchestrator.executive.get_provider", return_value=provider),# The loop's module-level alias, bound when executive was first# imported: another module's test that imports it lazily while@@ -1011,15 +1015,16 @@ def _run_loop(turn_id="t-loop",):pass+ provider.offered = offeredasyncio.run(_go())return providerdef _offered(provider: _ScriptedStreams) -> list[str]:- """The client tools offered on the first call (server tools have no- input_schema)."""- return [t["name"] for t in provider.calls[0]["tools"] if "input_schema" in t]+ """Every client tool the first call offered, the ones behind open_tools+ included (sorted names)."""+ return provider.offered[0]def test_every_withheld_name_is_a_real_tool() -> None:@@ -1087,6 +1092,32 @@ def test_a_private_turn_refuses_a_withheld_tool_the_model_emits_anyway(assert all(e.private for e in _audit().query(limit=1000))+def test_a_private_turn_refuses_a_withheld_tool_through_use_tool(roster: SimpleNamespace) -> None:+ """The less common tools are reached through open_tools / use_tool+ (tool_groups). A use_tool call meets the same guard as a direct call, and+ the private turn's open_tools never lists a withheld tool."""+ from openexecutive.orchestrator import executive as executive_module+ from openexecutive.orchestrator.people_tools import PRIVATE_TURN_REFUSAL++ handler = AsyncMock(return_value=json.dumps({"status": "sent"}))+ uses = [+ SimpleNamespace(type="tool_use", id="tu-1", name="use_tool",+ input={"name": "send_company_broadcast", "input": {"text": "Jordan"}}),+ SimpleNamespace(type="tool_use", id="tu-2", name="open_tools", input={"group": "messaging"}),+ ]+ with patch.dict(executive_module._ALL_SKILL_HANDLERS, {"send_company_broadcast": handler}):+ provider = _run_loop(_private_turn(roster), uses)+ handler.assert_not_awaited()+ results = {b["tool_use_id"]: b["content"] for b in provider.calls[1]["messages"][-1]["content"]}+ assert PRIVATE_TURN_REFUSAL in json.loads(results["tu-1"])["error"]+ listed = {t["name"] for t in json.loads(results["tu-2"])["tools"]}+ assert "send_slack_dm" in listed+ assert not listed & {"send_company_broadcast", "send_department_message"}+ # Nor does the open_tools definition name them.+ open_tools = next(t for t in provider.calls[0]["tools"] if t["name"] == "open_tools")+ assert "send_company_broadcast" not in open_tools["description"]++def test_a_private_turn_refuses_load_mcp_server(roster: SimpleNamespace) -> None:from openexecutive.orchestrator.people_tools import PRIVATE_TURN_REFUSALdiff --git a/packages/core/tests/unit/test_solo_executive.py b/packages/core/tests/unit/test_solo_executive.pyindex 3b80b383..b34bc7b0 100644--- a/packages/core/tests/unit/test_solo_executive.py+++ b/packages/core/tests/unit/test_solo_executive.py@@ -29,6 +29,7 @@ from openexecutive.orchestrator.schedule_tools import (tools_withheld_in_mode,)from openexecutive.orchestrator.session import Session+from tests.unit.offered_tools import capture_offeredfrom openexecutive.people import registry as people_registryfrom openexecutive.people import store as people_storefrom openexecutive.prompts.cache_manager import build_system_blocks@@ -330,7 +331,10 @@ def test_team_keeps_every_tool_and_the_same_objects() -> None:def _run_loop(provider: ScriptedProvider, **kw: Any) -> list[Any]:async def _go() -> list[Any]:items: list[Any] = []- with patch("openexecutive.orchestrator.executive.get_provider", return_value=provider):+ with (+ capture_offered() as offered,+ patch("openexecutive.orchestrator.executive.get_provider", return_value=provider),+ ):async for item in Executive()._stream_agent_loop(system_blocks=[],messages=[{"role": "user", "content": "tell everyone"}],@@ -338,6 +342,7 @@ def _run_loop(provider: ScriptedProvider, **kw: Any) -> list[Any]:**kw,):items.append(item)+ provider.offered = offeredreturn itemsreturn asyncio.run(_go())@@ -351,7 +356,8 @@ def _no_loop_audit(monkeypatch: pytest.MonkeyPatch) -> None:def _offered(provider: ScriptedProvider, call: int = 0) -> list[str]:- return [t["name"] for t in provider.calls[call]["tools"] if "name" in t]+ """Every tool the call offered, the ones behind open_tools included."""+ return provider.offered[call]@pytest.mark.usefixtures("_no_loop_audit")diff --git a/packages/core/tests/unit/test_tool_groups.py b/packages/core/tests/unit/test_tool_groups.pynew file mode 100644index 00000000..f0f1424c--- /dev/null+++ b/packages/core/tests/unit/test_tool_groups.py@@ -0,0 +1,303 @@+"""The less common tools sit behind open_tools / use_tool (orchestrator.tool_groups).++The tool list is the start of the cached prefix, so it must be the same on+every call of every turn: a group the model opens comes back as a tool+result, never as a change to the list. A use_tool call is rewritten into the+tool's own call before dispatch, so it runs the same handler, guards and+audit rows as a direct call would.++Also pinned here: the specialist lookups take their result counts from+settings, and a zero synced count skips the synced-source queries.+"""++from __future__ import annotations++import asyncio+import json+from types import SimpleNamespace+from typing import Any+from unittest.mock import AsyncMock, MagicMock, patch++import pytest++from openexecutive.orchestrator import tool_groups+from tests.unit.offered_tools import capture_offered++[email protected](autouse=True)+def _no_audit_writes(monkeypatch: pytest.MonkeyPatch) -> list[tuple[Any, ...]]:+ rows: list[tuple[Any, ...]] = []+ monkeypatch.setattr(+ "openexecutive.orchestrator.executive.audit_log",+ lambda *a, **_k: rows.append(a),+ )+ return rows+++# --------------------------------------------------------------------- #+# The groups+# --------------------------------------------------------------------- #+++def test_every_deferred_name_is_a_real_tool_with_a_handler() -> None:+ from openexecutive.orchestrator.executive import _ALL_SKILL_HANDLERS, _ALL_SKILL_TOOLS++ names = {t["name"] for t in _ALL_SKILL_TOOLS}+ assert set(tool_groups.DEFERRED) <= names+ assert set(tool_groups.DEFERRED) <= set(_ALL_SKILL_HANDLERS)+++def test_each_tool_sits_in_one_group() -> None:+ listed = [t for _, tools in tool_groups.GROUPS.values() for t in tools]+ assert len(listed) == len(set(listed))++[email protected]("name", ["consult_specialist", "propose_form_values", "message_person"])+def test_the_everyday_tools_stay_direct(name: str) -> None:+ assert name not in tool_groups.DEFERRED+++def test_open_tools_lists_every_group() -> None:+ desc = tool_groups.open_tools_tool(tool_groups.DEFERRED)["description"]+ for group, (_, tools) in tool_groups.GROUPS.items():+ assert group in desc+ assert all(t in desc for t in tools)+++def test_open_tools_names_only_the_offered_tools() -> None:+ tool = tool_groups.open_tools_tool(["create_goal", "send_slack_dm"])+ assert "create_goal" in tool["description"] and "send_slack_dm" in tool["description"]+ assert "send_company_broadcast" not in tool["description"]+ assert "calendar" not in tool["description"]+ assert tool["input_schema"]["properties"]["group"]["enum"] == ["goals", "messaging"]+ # The same offer gives the same definition (one cached list per kind of turn).+ assert tool == tool_groups.open_tools_tool(["send_slack_dm", "create_goal"])+++def test_split_keeps_the_direct_list_the_same_whatever_is_opened() -> None:+ tools = [{"name": n, "input_schema": {}} for n in ("consult_specialist", "create_goal", "list_watchlist")]+ direct, deferred = tool_groups.split(tools)+ assert [t["name"] for t in direct] == ["consult_specialist", "open_tools", "use_tool"]+ assert set(deferred) == {"create_goal", "list_watchlist"}+ # Nothing deferred on the turn: no open_tools / use_tool either.+ direct, deferred = tool_groups.split(tools[:1])+ assert [t["name"] for t in direct] == ["consult_specialist"] and deferred == {}+++def test_open_result_gives_only_the_offered_tools() -> None:+ deferred = {"create_goal": {"name": "create_goal", "description": "d", "input_schema": {"type": "object"}}}+ out = json.loads(tool_groups.open_result({"group": "goals"}, deferred))+ assert out["call_with"] == "use_tool"+ assert [t["name"] for t in out["tools"]] == ["create_goal"]+ assert "error" in json.loads(tool_groups.open_result({"group": "calendar"}, deferred))+ assert "error" in json.loads(tool_groups.open_result({"group": "nope"}, deferred))+ assert "error" in json.loads(tool_groups.open_result("goals", deferred))+++def test_unwrap() -> None:+ call, err = tool_groups.unwrap(+ {"id": "tu-1", "name": "use_tool", "input": {"name": "create_goal", "input": {"title": "t"}}}+ )+ assert err is None and call == {"id": "tu-1", "name": "create_goal", "input": {"title": "t"}}+ # JSON text for the input is accepted.+ call, _ = tool_groups.unwrap({"id": "tu-2", "name": "use_tool",+ "input": {"name": "create_goal", "input": '{"title": "t"}'}})+ assert call is not None and call["input"] == {"title": "t"}+ # A direct tool, an unknown name, or a missing input object is refused.+ for bad in (+ {"name": "consult_specialist", "input": {}},+ {"name": "use_tool", "input": {}},+ {"name": "create_goal", "input": "not json"},+ {"name": "create_goal"},+ "create_goal",+ ):+ call, err = tool_groups.unwrap({"id": "tu-3", "name": "use_tool", "input": bad})+ assert call is None and err and "error" in json.loads(err)+++# --------------------------------------------------------------------- #+# In the loop+# --------------------------------------------------------------------- #+++class _Provider:+ def __init__(self, finals: list[Any]) -> None:+ self._finals = list(finals)+ self.calls: list[dict[str, Any]] = []++ def messages_stream(self, **kwargs: Any) -> Any:+ import copy++ self.calls.append(copy.deepcopy(kwargs))+ final = self._finals.pop(0)++ class _Stream:+ async def __aenter__(self) -> _Stream:+ return self++ async def __aexit__(self, *_a: Any) -> None:+ return None++ def __aiter__(self) -> _Stream:+ return self++ async def __anext__(self) -> Any:+ raise StopAsyncIteration++ async def get_final_message(self) -> Any:+ return final++ return _Stream()+++def _use(id_: str, name: str, input_: Any) -> Any:+ return SimpleNamespace(type="tool_use", id=id_, name=name, input=input_)+++def _run(rounds: list[list[Any]]) -> tuple[_Provider, list[Any], list[list[str]]]:+ from openexecutive.orchestrator.executive import Executive+ from openexecutive.orchestrator.schedule_tools import set_session+ from openexecutive.orchestrator.session import Session++ finals = [SimpleNamespace(content=r, stop_reason="tool_use", usage=None) for r in rounds]+ finals.append(SimpleNamespace(content=[SimpleNamespace(type="text", text="ok")],+ stop_reason="end_turn", usage=None))+ provider = _Provider(finals)+ items: list[Any] = []++ async def _go() -> list[list[str]]:+ with (+ capture_offered() as offered,+ patch("openexecutive.orchestrator.executive.get_provider", return_value=provider),+ set_session(Session(session_id="web:groups")),+ ):+ async for item in Executive()._stream_agent_loop(+ system_blocks=[], messages=[{"role": "user", "content": "x"}],+ model="claude-test", workspace_mode="team", principal_role_tag="",+ turn_id="t-groups",+ ):+ items.append(item)+ return offered++ offered = asyncio.run(_go())+ return provider, items, offered+++def _results(provider: _Provider, call: int) -> dict[str, str]:+ return {b["tool_use_id"]: b["content"] for b in provider.calls[call]["messages"][-1]["content"]}+++def test_the_request_carries_open_tools_and_no_deferred_tool() -> None:+ provider, _, offered = _run([])+ names = [t["name"] for t in provider.calls[0]["tools"] if "input_schema" in t]+ assert {"open_tools", "use_tool"} <= set(names)+ assert not set(names) & set(tool_groups.DEFERRED)+ assert names == sorted(names)+ # ...while the turn still offers them behind open_tools.+ assert {"create_goal", "list_watchlist"} <= set(offered[0])+++def test_opening_a_group_returns_its_tools_and_keeps_the_list_unchanged() -> None:+ provider, _, _ = _run([[_use("tu-1", "open_tools", {"group": "goals"})]])+ out = json.loads(_results(provider, 1)["tu-1"])+ assert out["group"] == "goals"+ names = {t["name"] for t in out["tools"]}+ assert names == set(tool_groups.GROUPS["goals"][1])+ assert all(t["input_schema"] for t in out["tools"])+ # The cached prefix is byte-for-byte the same on the next call.+ assert provider.calls[1]["tools"] == provider.calls[0]["tools"]+++def test_use_tool_runs_the_tools_own_handler(_no_audit_writes: list[tuple[Any, ...]]) -> None:+ from openexecutive.orchestrator import executive as executive_module++ handler = AsyncMock(return_value=json.dumps({"status": "ok"}))+ with patch.dict(executive_module._ALL_SKILL_HANDLERS, {"list_watchlist": handler}):+ provider, items, _ = _run(+ [[_use("tu-1", "use_tool", {"name": "list_watchlist", "input": {"limit": 5}})]]+ )+ handler.assert_awaited_once_with({"limit": 5})+ assert json.loads(_results(provider, 1)["tu-1"]) == {"status": "ok"}+ # The assistant turn keeps the model's own use_tool block, so the result+ # pairs with the id it sent.+ assistant = provider.calls[1]["messages"][-2]["content"]+ assert any(b.get("name") == "use_tool" and b.get("id") == "tu-1" for b in assistant)+ # The audit row and progress event name the tool itself.+ assert any("list_watchlist" in str(row[1]) for row in _no_audit_writes if len(row) > 1)+ assert "use_tool" not in json.dumps([i for i in items if isinstance(i, dict)], default=str)+++def test_use_tool_on_a_direct_tool_or_without_input_is_an_error() -> None:+ provider, _, _ = _run([[+ _use("tu-1", "use_tool", {"name": "consult_specialist", "input": {}}),+ _use("tu-2", "use_tool", {"name": "create_goal"}),+ ]])+ results = _results(provider, 1)+ assert "error" in json.loads(results["tu-1"])+ assert "error" in json.loads(results["tu-2"])+++# --------------------------------------------------------------------- #+# Specialist lookups+# --------------------------------------------------------------------- #+++def test_specialist_retrieval_uses_the_settings_counts(monkeypatch: pytest.MonkeyPatch) -> None:+ from openexecutive.orchestrator import router++ monkeypatch.setattr(+ "openexecutive.config.get_settings",+ lambda: SimpleNamespace(specialist_builtin_n_results=4, specialist_company_n_results=1),+ )+ seen: dict[str, Any] = {}++ def fake_retrieve(**kw: Any) -> str:+ seen.update(kw)+ return "ctx"++ monkeypatch.setattr("openexecutive.knowledge.retriever.retrieve", fake_retrieve)+ out = asyncio.run(router._retrieve_for_call({"query": "q", "specialist": "cfo"}))+ assert out == "ctx"+ assert (seen["n_builtin"], seen["n_company"], seen["n_synced"]) == (4, 1, 1)+++def test_the_specialist_defaults_are_small() -> None:+ from openexecutive.config import Settings++ fields = Settings.model_fields+ assert fields["specialist_builtin_n_results"].default == 3+ assert fields["specialist_company_n_results"].default == 2+++def test_no_synced_count_skips_the_synced_sources(monkeypatch: pytest.MonkeyPatch) -> None:+ import openexecutive.knowledge.retriever as retriever_mod+ from openexecutive.knowledge.store import ChromaDBStore++ monkeypatch.setattr(retriever_mod, "_audit_log", lambda *a, **k: None)+ review_store = SimpleNamespace(+ get_withheld_keys=lambda _ct: set(),+ get_withheld_source_ids=lambda: set(),+ get_priority_map=lambda _ct: {},+ list_annotations=lambda domains=None, active_only=True: [],+ )+ store = MagicMock()+ store.query.return_value = []++ def collections() -> set[str]:+ return {c.kwargs["collection"] for c in store.query.call_args_list}++ synced = {+ ChromaDBStore.NOTION_COLLECTION,+ ChromaDBStore.DRIVE_COLLECTION,+ ChromaDBStore.ONEDRIVE_COLLECTION,+ ChromaDBStore.CONFLUENCE_COLLECTION,+ }+ retriever_mod.retrieve(query="How should we price the renewal?", store=store,+ review_store=review_store, n_builtin=1, n_company=1, n_synced=0)+ assert not collections() & synced+ store.query.reset_mock()+ retriever_mod.retrieve(query="How should we price the renewal?", store=store,+ review_store=review_store, n_builtin=1, n_company=1, n_synced=2)+ assert collections() >= synced+ assert all(c.kwargs["n_results"] == 2 for c in store.query.call_args_list+ if c.kwargs["collection"] in synced)diff --git a/packages/core/tests/unit/test_tool_result_cap.py b/packages/core/tests/unit/test_tool_result_cap.pyindex 6143cf70..cb0b5592 100644--- a/packages/core/tests/unit/test_tool_result_cap.py+++ b/packages/core/tests/unit/test_tool_result_cap.py@@ -78,8 +78,8 @@ def test_limit_is_honoured_not_hardcoded() -> None:[500, # a short JSON status result4_000, # a specialist's analysis- 20_000, # a substantial document excerpt- 40_000, # the attachments extractor's own ceiling+ 15_000, # a long email thread+ 20_000, # a substantial document excerpt (the shipped default)],)def test_default_limit_does_not_fire_on_ordinary_output(size: int) -> None: