AI-powered virtual executive team — a single coherent executive persona backed by 8 specialist agents (FastAPI + Next.js).
fix(knowledge): read scanned PDFs from Drive with OCR (#453)
Rufuseb70dfe9c2e4
diff --git a/docs/drive_sync_setup.md b/docs/drive_sync_setup.mdindex 382a03e3..2df66ad1 100644--- a/docs/drive_sync_setup.md+++ b/docs/drive_sync_setup.md@@ -72,7 +72,14 @@ cd packages/core && uv run openexecutive sync-drive- **Google Docs and Slides** are exported as text.- **Google Sheets** are exported as `.xlsx` and read sheet by sheet.- **PDF, Word (`.docx`) and Excel (`.xlsx`)** files have their text- extracted. A scanned PDF with no text layer yields nothing.+ extracted. A scanned PDF with no text layer is read the way an uploaded+ one is: by local OCR on this server (or by the deployment's own model when+ `PDF_PROVIDER_READING=true`), up to `PDF_VISION_MAX_PAGES` pages.+ `PDF_OCR_ENABLED=false` turns that off for Drive as it does for uploads.+- A file with no readable text (a scan OCR could not read, a scan while OCR+ is off, a file over the size cap) is listed on the knowledge page as+ "No readable text" with the reason, and is not fetched again until it+ changes in Drive.- **Plain text, Markdown and CSV** are read as they are.- Anything else (images, video, other binaries) is skipped, and so is anyfile over 20 MB.diff --git a/packages/core/openexecutive/architecture/architecture-facts.yaml b/packages/core/openexecutive/architecture/architecture-facts.yamlindex e6343bff..43787edb 100644--- a/packages/core/openexecutive/architecture/architecture-facts.yaml+++ b/packages/core/openexecutive/architecture/architecture-facts.yaml@@ -1813,15 +1813,27 @@ knowledge:newest first, capped at DRIVE_MAX_FILES_PER_SCAN; the overflow waitsfor the next tick.- Text: Docs / Slides are exported as text/plain and Sheets as .xlsx- (Drive's CSV export holds only the first sheet). PDF / .docx /- .xlsx / txt / md / csv are downloaded and read with- loader.extract_text_from_file, with no model call, so a scanned PDF- yields nothing. Anything else, and anything over 20 MB, is skipped.- Downloads stream and stop at the cap. A .docx / .xlsx whose zip- declares more than 100 MB uncompressed is not parsed. Extraction- has a 120 s deadline (the worker thread cannot be killed, but the- tick moves on). An unreadable file is recorded with no chunks, so- it is not refetched until it changes.+ (Drive's CSV export holds only the first sheet). .docx / .xlsx /+ txt / md / csv are downloaded and read with+ loader.extract_text_from_file. A PDF goes through+ pdf_reader.read_pdf_text, the path uploads take (not inbound, so+ PDF_VISION_MAX_PAGES, not the inbound budget): the text layer, else+ the opt-in model step (PDF_PROVIDER_READING), else local OCR in a+ child process under the two-slot OCR gate, with PDF_OCR_ENABLED+ switching it off for Drive as for uploads. Why: a scanned contract+ in a shared folder used to be skipped silently. Anything else, and+ anything over 20 MB, is skipped. Downloads stream and stop at the+ cap. A .docx / .xlsx whose zip declares more than 100 MB+ uncompressed is not parsed. Non-PDF extraction has a 120 s deadline+ (the worker thread cannot be killed, but the tick moves on); a PDF+ read keeps pdf_reader's own child limits, under a backstop+ (_PDF_READ_TIMEOUT_S, 1200 s: text layer + a model allowance + OCR+ slot wait, budget and slack). Busy parsers / OCR slots are a retry+ next tick, not a verdict. An unreadable file is recorded with no+ chunks and its reason (`unreadable` in the state record, e.g. the+ reader's "it looks scanned and local OCR is turned off"), logged,+ and shown on the knowledge page through `unreadable_reason` in+ `GET /documents/drive`; it is not refetched until it changes.- A file with no text is recorded with no chunks, so it is notrefetched until it changes.- Text is written under <company_profile_dir>/docs/drive/@@ -1879,7 +1891,8 @@ knowledge:.xls / .rtf / .odt / .odp / .ods are fetched as Graph's PDF rendering(`?format=pdf`) and read as PDF. The extractor, size caps, depth 4,2000-item cap, 120 s deadline and request pause are imported from- drive_sync, so the two stay in step. Text is written under+ drive_sync, so the two stay in step. Unlike Drive, a PDF here is read+ by that extractor (text layer only), so a scanned PDF yields nothing. Text is written under<company_profile_dir>/docs/onedrive/ (onedrive-<id>-<slug>.md, withan onedrive_key comment). Reconcile, skip rules, locking and resets(fixture load/reset, client-slot switch) are Drive's. A refuseddiff --git a/packages/core/openexecutive/architecture/prebuilt/api.json b/packages/core/openexecutive/architecture/prebuilt/api.jsonindex af719d56..65e95795 100644--- a/packages/core/openexecutive/architecture/prebuilt/api.json+++ b/packages/core/openexecutive/architecture/prebuilt/api.json@@ -1,7 +1,7 @@{"section_id": "api","title": "API Reference",- "markdown": "The FastAPI surface lives in `openexecutive.api.main`, which mounts one router per topic. A shared-secret middleware gates every request via an `x-api-key` header, except `/health` and the `/webhook/*` callbacks (which carry their own verification). With local login (`OE_LOCAL_LOGIN=1`, which only `make dev` sets, and only while neither Google nor SSO sign-in is set up) a second middleware answers 403 to any request whose raw `Host` is not `localhost`, `127.0.0.1` or `[::1]` (except a webhook that verifies its caller: Google Chat and Slack always, Telegram only with a `TELEGRAM_WEBHOOK_SECRET` Telegram can send), and to any write whose `Sec-Fetch-Site` is present and not `same-origin`/`none`. The UI signs that owner in with no password and this API usually has no secret, so without it a web page in the owner's browser could drive the API as the principal — by DNS-rebinding its own hostname to 127.0.0.1, or by posting a plain form from another site or localhost port. CORS is restricted to the UI origins in `BACKEND_ALLOWED_ORIGINS`, plus the `make dev` UI on `localhost:3000` only when `OE_PUBLIC_DEPLOYMENT` is unset, so a public instance never answers a localhost page with credentials. The gate is off when `BACKEND_SHARED_SECRET` is unset — intended for local development only — so any internet-reachable deployment sets `OE_PUBLIC_DEPLOYMENT`, and `create_app()` then refuses to boot without the secret rather than serving an unauthenticated API.\n\n**Who is calling** is read in one place, `api/caller.py`, which answers with a `Caller` kind and email; a test fails on any other read of an `x-caller-*` header. By default (`open`) it is the `x-caller-email` the UI proxy stamps from the sign-in, trusted because whoever holds the shared secret is trusted as the proxy, and a request with none runs as the principal. **Signed callers** remove that trust. With `CALLER_ASSERTION_PUBLIC_KEYS` set, the proxy signs every request with an Ed25519 key only the UI holds, and a third middleware checks the `x-caller-assertion` header:\n\n- **What it covers.** It is good for one method, the exact path and query, at most 120 seconds, and one use. The API keeps used ids in memory.\n- **What it names.** A `user`, by email, or the `operator`: local login, which runs as the principal.\n- **No caller header at all** is a `service`, for scripts or MCP clients holding only the secret. It is never the principal, not even on an install with no owner yet, so it can't save a first setup.\n- **Anything else is a 401.** That includes an `x-caller-email` without an assertion (`caller_assertion_required`), and a bad, expired, reused or mismatched assertion (`caller_assertion_invalid`).\n\nPublic keys that can't be read stop the boot, rather than fall back to trusting the header. `scripts/make-caller-keys.py` makes the pair, and `scripts/mint-caller-assertion.py` signs one request for curl. The API holds only public keys. Setup's API-protection check warns while a shared secret is set without them.\n\nEndpoints grouped by router:\n\n| Router | Key endpoints | Purpose |\n|---|---|---|\n| `chat` | `POST /chat`, `POST /chat/stop`, `POST /chat/add`, `POST /chat/background` | SSE stream of a live Executive turn. `POST /chat/background` starts a new chat that the server answers to the end with nobody connected (a phone approving a card, then closing the app): it answers `202` with the new `session_id` once the same caller, access and quota checks as `/chat` pass, the turn saves itself as a `/chat` turn does, and one caller may have three running at once (`429` beyond that). `POST /chat/add` hands the running turn a message the person typed while it works; the turn takes it at its next step and the stream names it in a `message_added` event, and a second `POST /chat` on the same conversation waits for the first to finish. The request body optionally carries `page_context` (route, page title, guide section id, and a form descriptor with current values) — sent only by the Ask OE side panel so the Executive can explain the current page or propose form values. When the Executive calls its `propose_form_values` tool, the stream emits a `form_patch` event (`{type, form_id, fields, rationale, iteration}`) that the panel applies into the on-screen form as highlighted suggestions; the user reviews and saves through the form's normal submit path. Each round of tool calls also emits an `activity` event (`{type, label, tool, iteration}`) naming what is running — progress only, forwarded verbatim and never persisted with the message, unlike an `action_taken` chip. A `session_id` in the body must be a chat the caller may use (the same ownership rule as `sessions`). If it isn't, the turn runs in a fresh chat instead, so nothing is written into someone else's session and the reply looks the same as for an unknown id. An unknown namespaced id (`slack:…`, `telegram:…` and so on) is ignored and a fresh chat is started, because those ids belong to the channel adapters and claiming one first would make the caller its owner. The client mints a `client_turn_id` and sends it with the turn; `POST /chat/stop` takes that id and halts the turn mid-flight, so the button is live from the moment Send is pressed rather than only once the first SSE byte arrives. A stopped turn emits a `stopped` event and then the usual `done` over the same still-open stream — never an `error` — and whatever streamed so far is persisted and flagged. The endpoint returns 404 both for an unknown id and for another caller's turn, so it cannot be used to probe which turns are live. An optional `memory_text` (1–2000 chars) is what peer memory records as the caller's words instead of `message` — the briefing handoffs send it because their seed quotes the Executive's own card; `POST /chat/upload` derives it from the typed text and filenames. Since it never reaches the transcript, the `chat_turn` audit row keeps it beside `message`. After the reply — a stopped or timed-out one too, before the `stopped` / `error` / `done` frames — a turn that looked at anything sends one `sources` event, `{type, session_id, sources: [{kind, title, url}], unavailable: [area]}`: `kind` is `company`, `knowledge`, `notion`, `research`, `document` or `web`, `url` is only ever an http(s) link or, for an earlier document, `/artifacts/<id>`, at most 12 sources (6 from the web, pages the reply cites first), and `unavailable` names parts of the analysis (\"finance\") whose specialist failed or said nothing every time it was asked. It is persisted with the reply as `chat_messages.sources`. A request that carries `x-caller-email` marks its turn signed in (`Session.web_caller_signed_in`), which Act as me needs on a deployment without local login. |\n| `sessions` | `GET /sessions`, `GET /sessions/{id}`, `GET /sessions/{id}/messages`, `DELETE /sessions/{id}`, `POST /sessions/{id}/messages/{message_id}/feedback`, `GET /sessions/{id}/followup` | List, read and delete saved chat sessions (assistant messages carry their row `id` and any `feedback`). The list is scoped to the caller, and each chat carries `preview`: its last message as one plain line, Markdown marks stripped and cut at a word near 120 characters, with `You:` before the person's own words. With `?q=` it is a search: the caller's own chats whose title or message text contains the words (case-insensitive, `LIKE` with wildcards escaped), each with `match_count` (messages that contain them; 0 for a title-only match) and `snippet` (the newest matching message, around the match, prefixed `You:` or `Executive:`). Scoped exactly like the list, so nobody else's conversation can match. The Chats page searches this way and opens a result with `?find=`, so the chat starts on that message and steps through the others. A chat-app conversation started with /new is left off the list until its first turn (`memory.conversations`). Every per-session route checks ownership, because session ids are guessable (`slack:dm:<user id>`, `telegram:<chat id>`): only the session's owner or the principal gets through, and a legacy row with no owner is the principal's alone. A teammate's own chat (a web chat, a DM, a private Telegram chat; not a channel or group conversation, `chat._shared_space`) is theirs alone, the principal included, as their mail with the Executive is. Anyone else gets the same 404 as for an unknown id, so the routes can't be used to find out which chats exist. The one other way in is having started the chat in this process (matched by Person id or verified email, so it survives the caller being added to the roster), which is how a caller the roster can't resolve keeps using their own chat. That record is in memory, so after a restart such an ownerless chat is refused for reads. The principal can continue someone else's ownerless chat without becoming its owner. Deleting a session also drops its in-memory copy, so a later turn naming that id starts fresh. The feedback route records 👍/👎 (`{\"feedback\": \"up\"|\"down\"|null}`) on one assistant reply; only the session's own caller or the principal may rate it. The rater is stored (`feedback_by_person_id`), and a 👎 on a reply to the rater's own message triggers a working-style pass for them (paced; see attunement). The chat stream's `done` event carries `message_id` for the persisted reply. The followup route returns `{\"suggestion\": string|null}` — one suggested next user message, written by the utility-fast model from the last few turns, that the chat composer shows as its greyed placeholder (Tab / → or the inline chip fills it in without sending). It is null unless the session ends on a persisted assistant reply, and null (uncached) when the model fails. Same ownership check as feedback; successful suggestions are cached in-process per (session, reply id). Messages from `GET /sessions/{id}/messages` carry each assistant reply's saved `sources` (the same shape as the chat stream's `sources` event, without `type`), so a reloaded chat shows them again. |\n| `today` | `GET /today`, `GET /today/activity`, `GET /today/activity/daily`, `GET /today/brief-delivery`, `GET /today/top-three`, `GET /today/weekly-review` | Morning-brief aggregation, recent-action rail, and Pulse heartbeat heatmap (per-day activity counts). `GET /today` carries the viewer's cached \"What's going on\" `narrative` with `narrative_generated_at` (when it was written) and `narrative_stale` (a fresher one is being written — the page re-polls; see Today). A proposal backed by a roster request carries `roster_request` (the new sender's server-derived details and held-message previews; the principal's only) and is answered through the `people` routes below. `brief-delivery` returns the latest morning brief or end-of-day digest that didn't reach the owner, as `{brief, at, problem, fix, readable}` in plain language (`readable` is false when it couldn't be written, so there is nothing to read), or null — null while briefs go out, once the cause is fixed, while the scheduler is off, and for anyone but the owner (`chat._caller_is_principal_or_unclaimed`); it carries no address or channel id. `top-three` and `weekly-review` feed the solo Briefing's own cards and answer only in solo mode and only to the principal (`workflows._caller_is_the_principal`: a principal match or no caller header, never an unclaimed roster); anyone else, and team mode, get null. `top-three` returns `{items: [{key, kind, text, why, slot}]}` — the morning brief's top three from the same code (`build_top_three(principal_due_soon())`), `slot` a free block like `10:00–11:00`, `\"\"` when none is left, null when no calendar was read — and is kept off `GET /today` because its calendar read may take up to 4 s. `weekly-review` returns the latest completed `weekly_review` run as `{run_id, completed_at, period, top_three, excerpt}` (next week's top three as plain text; `excerpt` when that list is empty), or null when none has completed. |\n| `people` | `GET\\|POST\\|PATCH /people`, `POST /people/{id}/archive`, `GET /people/requests`, `GET /people/requests/{id}`, `POST /people/requests/{id}/approve`, `POST /people/requests/{id}/decline`, `GET\\|POST /people/{id}/open-loops`, `POST /open-loops/{loop_id}/close`, `GET /people/{id}/outreach`, `GET\\|PUT\\|DELETE /people/{id}/attunement` | Roster CRUD and scope lookups. Every `Person` carries `kind` (`team` / `contact`). `GET /people` lists team members only unless `?include_contacts=true` (the People page asks for both; the department-head and workflow pickers keep the default); `POST` / `PATCH` accept `kind`, and a principal contact is a 422 either way. They also take `email_aliases` (other addresses the person writes from, at most 10; on `PATCH` the full list replaces the current one, omitted leaves it): an alias matches their mail and may be emailed but never signs in; a malformed one is a 422, and an alias — or a new primary `email` — that is already someone else's address or alias is a 409. Every `POST` / `PATCH` then closes any pending roster request whose sender now matches someone and replays its held messages. Adding, editing and archiving (`POST`, `PATCH`, `/archive`) are the principal's only, or anyone's while no one is principal yet so a first setup can add its owner (`chat._caller_is_principal_or_unclaimed`, the check behind `PUT /workspace`; a request with no `x-caller-email`, such as the CLI or local login, counts as the principal). Anyone else, a person editing their own row included, gets 403 before the person lookup, so it can't probe which ids exist. Contacts are private to the principal (`caller_is_principal`: the signed-in email is the principal's, or no `x-caller-email`; stricter than the owner rule above, so while no one is principal nobody is): for anyone else `include_contacts` is ignored, a contact's id is a 404 on `GET /people/{id}` and open-loops (exactly as for a missing id; `PATCH` and archive are already a 403 for them), and creating a contact or turning someone into one is a 403, even on a first setup with no principal yet. `GET /people/me` returns `{person_id, is_principal}` so the UI offers contacts to the principal only. `GET /people/by-scope/{token}` never returns a contact. Archiving a person closes their open loops. The open-loop routes list what a person owes (soonest due first) and close one (`reason`: done / not_needed / cancelled) — both the principal's or the owner's only (403 otherwise, via `people.store.is_principal_or_self`). `POST /people/{id}/open-loops` `{task, due_date?}` assigns that person a task (201 with the loop; `due_date` is a local date, due 17:00 in the user's timezone, clamped to 60 days, default 2 days): any active team member may assign anyone on the team, 403 for anyone else; 409 for an archived person, a contact (as seen by the principal), a person at the open-loop cap, a duplicate or open loops turned off; 422 for an empty task. Nobody is messaged at assignment — the owner is chased once it is due. `/outreach` returns, per kind of proactive outreach over 30 days, `sent` / `replied` / `acted` / `ignored` / `pending` (voided DMs are excluded) — same access rule, checked before the person lookup so it can't probe which ids exist. `/attunement` reads, replaces (`rules`: up to 4 strings, or omitted to only set `locked`) or resets the person's working-style rules — same access rule, 404 for an archived person; `PUT` rejects a rule that is not purely about style with 422. Archiving a person also drops their working style and frees their aliases. The **roster-request** routes (declared before `/people/{person_id}`) are the principal's alone (`caller_is_principal`; anyone else gets 404, as for a missing id): `GET /people/requests?status=pending|approved|linked|declined|expired|superseded|all` lists up to 100 (`id`, `channel`, `channel_ref`, sanitised `display_name`, `profile_email`, `on_company_domain`, `suggested_kind`, `suggested_person_id` / `_name`, `status`, `resolved_person_id` / `_kind`, `message_count`, timestamps, `ack_sent`, and `previews` of the held messages while pending — shown to the principal, never to a model); `approve` takes either `link_person_id` (+ `replace_channel_id`) or `full_name` + `kind` (no default: 422 without both) + optional `role`; `decline` takes no body. An answered request is a 409, a chat id or address already on someone else is a 409; the held messages are replayed (or dropped) in the background. |\n| `departments` | `GET\\|POST\\|PATCH\\|DELETE /departments`, `.../goals` | Department state and per-department goals. `PATCH /departments/{slug}` also takes `watched_entities` (≤50 names, `[]` clears) — the entities the research watch policy may add watches for and route to this department. A `cadences` spec that does not parse is rejected with a 422 (empty means no cadence). A new `head_person_id` must be a team member: an unknown id and one of the principal's contacts get the same name-free 422 (`person_id N is not on the team.`), so the route — open to any signed-in user — never reveals that a contact exists; an unchanged head is not re-checked. `POST .../goals` requires only `key_result`: a missing or blank `period_value` is filled with the current period for `period_type` in the user's zone (the same label `create_goal` uses), and `target` defaults to empty. `PATCH .../goals/{id}` rejects a blank `key_result` or `period_value` with a 422; `target` may be cleared. |\n| `agents` | `GET /agents`, `GET /agents/models`, `GET /agents/models/options`, `GET /agents/presets`, `POST /agents/presets/{id}`, `PATCH /agents/{id}`, `POST /agents/{id}/test` | Specialist metadata and Council overrides. `/agents/models` is the flat model allowlist the PATCH validator uses; `/agents/models/options` returns the same ids in the same order, each tagged with `provider`, `provider_label`, `route` (`direct` / `openrouter` / `local`) and a display `label`, for the Council's Provider → Model picker. `AgentDetail` carries `instructions` (additional instructions, appended after `prompt`, which stays the built-in prompt or its replacement); `PATCH` and `/test` accept `instructions` (max 4,000 chars, blank clears it). `GET /agents/presets` lists the Fast, Balanced and Thorough quality presets (each with `available` and the resolved `model` for this install) plus `active` (null when the council is Custom), `base` and `custom_agents`; `POST /agents/presets/{id}` applies one by writing per-agent model and deep-reasoning overrides and returns the same shape (404 unknown preset, 409 when the install allows none of its models). Each `AgentMeta` carries `visibility` (`core` or `internal`) so the Council's simple view can list core agents first. |\n| `personas` | `GET\\|PUT\\|POST\\|DELETE /personas` | Executive persona definitions. |\n| `alerts` | `POST /alerts/{id}/ack`, `POST /alerts/bulk-ack`, `POST /alerts/{id}/reopen`, `POST /alerts/review` | Groom the briefing queue. `ack` (`read`/`ack`/`dismissed`, optional `mute_topic`) also teaches a watch-sourced alert's watch (`trust_score`); `bulk-ack` takes explicit `alert_ids` and/or `older_than_days` + `category` (exempt sources skipped); `reopen` is the Undo for a `resolved` / `expired` / `dismissed` alert (never `ack` or an artifact / decision row; back to `unread` with every review stamp cleared and its TTL anchor refreshed, so it is live again even if older than its TTL); `review` runs the Executive's relevance review on demand (ignores the interval; still honours `ALERT_REVIEW_ENABLED`, the per-pass caps, the single-flight lock and per-alert idempotency) and returns the move counts. `bulk-ack` caps ids at 500, needs `older_than_days >= 1`, and applies watch feedback per row; `mute_topic` must be one of the alert's tags or a scoped `external:` / `department:` pattern. Alerts still surface only through the briefing (`/today` reads the live view) — there is no list, prefs, mutes, or live-stream endpoint. |\n| `artifacts` | `GET /artifacts`, `GET /artifacts/{id}`, `GET /artifacts/{id}/download?as=`, `POST /artifacts/{id}/archive`, `POST /artifacts/{id}/restore`, `DELETE /artifacts/{id}` | Browsable deliverables in any format — Markdown, HTML, Word, Excel, or a link into a connected app — with `format`, `downloads`, `external_url`, `supersedes_id` and, for an extension's kind of document, `collection` / `collection_label` on every summary (`orchestrator.extensions`; archive, restore and delete tell the collection first and answer 502 if it refuses). `download` renders the file (Word/Excel built at request time from the stored text; any Markdown artifact, workflow output included, can export `?as=docx`) and is always an attachment with `nosniff` and a sandbox CSP. Every route answers for the signed-in caller (`artifact_records.Viewer`): a drafted artifact is its owner's alone (`alerts.owner_person_id`, the person whose conversation published it; none = the principal's), a workflow output follows its run (`persistence.run_visible_to`), and anything the caller may not see is a 404 like a missing id, the principal included. `POST /alerts/{id}/ack` and `/reopen` answer 404 for someone else's drafted artifact the same way. |\n| `decisions` | `GET /decisions`, `POST /decisions/{id}/{approve,reject,cancel}`, `GET`/`PUT /decisions/classes/meeting_scheduling`, `GET /audit/reliability` | Approval-gated decision instances (e.g. calendar bookings). What approve/reject/cancel do is per class (`DECISION_CLASSES`, one `DecisionClassSpec` each; meeting booking books on approve and deletes on cancel; `delegation_reply`, an inbox reply card, is its approver's alone (`approver_only`) — hidden from everyone else, the principal included when it is a team member's — sends its Gmail draft on approve and, after a reject, deletes the draft unless it was edited), and an instance of a class with no spec gets 409 instead of being carried out as another. Only the principal, or the person the decision went to for approval (`approver_person_id`), may approve, reject or cancel it. Anyone else gets 403, and a decision nobody was asked about, or one private to the principal, is the principal's alone. The resolver is recorded as `resolver_person_id`, and a reject that loses the race to another answer is a 409. `GET /audit/reliability` is the principal's alone (403 otherwise), because it counts private decisions too. A proposed booking is surfaced on the briefing as a companion alert; approve/reject/cancel here also clear that alert. An approval whose caller resolves to the principal runs the booking inside `grant_contact_egress()`, so a meeting the principal proposed with a contact can be created. Such a booking's payload is marked `private`: it is proposed to the principal, left out of anyone else's `GET /decisions`, and a 404 for them on `GET /decisions/{id}` and every action route. Reviewed in the briefing — there is no separate proposals page. The class routes read and set how meetings are handled: both return `{decision_class: \"meeting_scheduling\", mode: \"propose\" \\| \"auto_execute\"}` (default `propose`); the `PUT` body is `{mode}`, principal-only (403 otherwise, open before a principal exists — but then only `propose` can be set: `auto_execute` is a 409 until a non-archived principal is on the roster), 422 on any other mode, and a change writes a `decision_class_mode_changed` audit row. In solo mode the class mode decides a booking, and `auto_execute` applies only to a turn the principal started on a verified surface — any other turn gets a proposal. |\n| `delegation` | `GET`/`PUT /delegation`, `GET`/`PUT`/`DELETE /delegation/voice`, `POST /delegation/voice/learn`, `POST /delegation/voice/describe`, `POST /delegation/voice/signature`, `PUT /delegation/inbox`, `POST /delegation/inbox/check`, `GET /delegation/replies`, `PUT /delegation/replies/{id}/text`, `GET /delegation/actions`, `PUT /delegation/handle-it`, `PUT /delegation/disclosure`, `PUT /delegation/training`, `DELETE /delegation/learned/{id}`, `GET /delegation/handled`, `PUT /delegation/team` | Act as me — the **caller's own** delegation only; nobody sets it for anyone else. It answers the principal, and a team member once the owner lets them; anyone else gets 403 `not_available_yet`. `GET /delegation` returns `{enabled, gmail: {status, message, email, connect_command, outlook_connect_command, provider}, inbox: {enabled, status, message, watch_since, last_poll_at, checking}, handle_it: {enabled, mode, available, sent_today, lead, lead_available}, training: {replies, follow_ups, actions, drafts, learned}, disclosure: {enabled, handle_it, approved_sends, line}, team}`, where `team` is the owner's switch for the principal while `DELEGATION_TEAM_MEMBERS` allows it — `{enabled, members: [{person_id, name, enabled, inbox, drafts_30d, sent_30d}]}`, counts only — and `null` otherwise. `PUT /delegation/handle-it` takes `{enabled?, mode?}` (`mode` is `careful`, `balanced` or `bold`; anything else is a 422 `bad_mode`) from a caller the API knows is that person — 403 `not_yours`, 409 `caller_signing_required` with only the shared secret — and turning it on needs Act as me and Draft replies to my inbox on (409 `act_as_me_off` / `inbox_off`); `GET /delegation/handled` returns `{replies: [{decision_id, sent_at, to_name, to_email, subject, body, open_questions, gmail_link, source}]}`, the caller's replies and follow-ups (`source` `follow_up`) sent on its own in the last 7 days; a card on `GET /delegation/replies` carries `source` the same way, `can_allow` while Replies (Follow-ups, on a follow-up) is in training, and `learns_style` while Drafts is. `PUT /delegation/disclosure` takes `{handle_it?, approved_sends?}` (any other field is a 422), the caller's own switches for the AI line on replies sent as them (`disclosure.enabled` is the install's `AI_DISCLOSURE`); turning one on needs no signed sign-in, turning one off does (409 `caller_signing_required`), and a change is audited as `delegation_disclosure_changed`. `PUT /delegation/training` takes `{replies?, follow_ups?, actions?, drafts?}` (any other field is a 422) and returns the `GET /delegation` shape; putting Replies or Follow-ups in, or Suggested actions out, needs no signed sign-in, anything else does (409 `caller_signing_required`), and putting Replies or Follow-ups in turns Take the lead as you off (`PUT /delegation/take-the-lead` on answers 409 `in_training` while they are). `training.learned` is `[{id, label, setting, example, uses, created_at}]`, what the caller allowed or kept in training (`setting` `replies`, `follow_ups`, `actions` or `drafts`; the kept example, or \"\"); `DELETE /delegation/learned/{id}` removes one of the caller's own (404 for anyone else's). `PUT /delegation/team` takes `{enabled}` from the principal alone (403 `principal_only`; 409 `not_available` when the install doesn't allow it). The Gmail `status` is `connected`, `not_configured`, `needs_reconnect`, `mismatch`, `no_email`, `shared_mailbox` or `error`, checked live with Google on each call. `PUT` takes `{enabled}` (any other field is a 422); turning it on needs the caller's own Gmail connected, otherwise a 409 whose code says why (`gmail_not_connected`, `gmail_needs_reconnect`, `gmail_mismatch`, `no_email`, `shared_mailbox`, `gmail_error`). The voice routes show, learn (from the caller's sent mail, one pass at a time — a 409 `in_progress`, `locked`, `too_soon`, `not_enough_mail`, `no_profile` or `changed` when it was locked or edited while learning, or a Gmail code), write it from the caller's own description (`POST /delegation/voice/describe` with `{description}`: one model call applies their words to the profile they have and returns it with a short `sample_reply`, saving nothing, so they review it and save it with `PUT`; a 409 `empty`, `too_long`, `in_progress`, `too_many` past 10 an hour or `no_profile`; no mailbox needed), edit and lock, and reset \"How I write\": a typed field that fails validation is a 422 `invalid_voice` listing each `rejected` field, and `clear_exemplars` / `clear_signature` drop those. `POST /delegation/voice/signature` reads the signature from the caller's Gmail settings again and changes nothing else (a locked profile stays locked), with the same Gmail codes as learning: a 409 when their mailbox isn't connected, it refuses the sign-in or the check itself fails (`gmail_error`), and a 502 `gmail_error` when reading the signature fails. On an Outlook mailbox it is a 409 `signature_unavailable`, since Outlook gives apps no signature to read, and the stored one is left alone. A caller `can_delegate` refuses — anyone but the principal, for now — gets 403 `not_available_yet`, which hides the Settings card. A request with no `x-caller-email` is a 403 `sign_in_required` unless the API runs under local login — a header-less call is not a sign-in (the caller is read through `api/caller.py`, as on every principal-only route). `PUT /delegation/inbox` takes `{enabled}` and turns **Draft replies to my inbox** on or off, returning the same shape as `GET /delegation`: on needs Act as me on (409 `act_as_me_off`) and the mailbox connected (the Gmail codes above, which keep their names for an Outlook mailbox too), and turning Act as me off turns it off too. `POST /delegation/inbox/check` starts a scan now in the background and answers 202 with the `inbox` state (409 `inbox_off` or `in_progress`). `GET /delegation/replies` returns `{cards: [...]}`, the caller's open reply cards from the database alone (no Gmail call): the decision id, the sender's name, address, relation and whether it was verified, the subject, when it arrived, what they wrote, the draft's recipients, subject and body, `open_questions`, `flags` and a Gmail link built from a fixed prefix. **Send** is `POST /decisions/{id}/approve`: it sends the card's draft exactly as it is in Gmail, after checking again that the caller is its owner — which on a server needs signed callers (409 `caller_signing_required` otherwise) — that both switches are on (`inbox_off`), no client slot is active (`client_slot`), the Gmail is connected (the Gmail codes), the draft is still there and unanswered (`draft_gone`, `you_replied`) and its recipients are allowed (`no_recipients`, `too_many_recipients`, `executive_recipient`, `not_from_you`). Recipients that changed, or a newer message in the thread, answer 409 `{code: \"confirm\", message, reasons, recipients}`; approving again with `edits: {recipients: <the list shown>, thread_moved_on: true}` is the second yes. **Send + allow** adds `edits.allow: true`: 422 `cant_allow` unless Replies (Follow-ups, on a follow-up card) is in training, 409 `learned_full` when their list is full, both before anything is sent; `edits.example: true` keeps a draft edited in the mailbox as how they write to that person while Drafts is in training. A draft edited in Gmail between those checks and the send is not sent (`draft_changed`). A double tap sends once (`already_handled`); Gmail refusing answers 429 `rate_limited` or 502 `gmail_error` with nothing sent, and a send Gmail didn't confirm answers 502 `send_unconfirmed` and is settled by the next inbox scan. Dismiss is `POST /decisions/{id}/reject`. `GET /delegation/actions` returns `{cards: [...]}`, the caller's own approval cards (`delegation_actions`): the decision id, when it was made, a one-line `why`, and each action's `index`, `kind` (`invite`, `message`, `add_contact`), `summary` and `text` (a message as it will be sent, or a meeting's description). **Approve** is `POST /decisions/{id}/approve` with `edits: {only: [indexes]}` (and `allow: true` for Approve + allow while Suggested actions is in training: 422 `cant_allow` otherwise or when nothing on it can be allowed, 409 `learned_full`; a card carries `can_allow`); it needs the card's own person signed in (403 `not_yours`, 409 `caller_signing_required`), Act as me on (409 `act_as_me_off`) and a card under a week old (409 `expired`), runs once (409 `already_handled`), and returns the decision with `final_payload_json.results`: per action `done`, `waiting` (sent to whoever approves meetings), `failed` with why, or `skipped`. Every change writes a private audit row. |\n| `history` | `GET /memories/history` (`?q=`), `PUT /memories/history/settings`, `PATCH`/`DELETE /memories/history/{id}`, `POST /memories/history/{id}/forget-conversation` | Always in the loop: the **caller's own** notes and switches only, resolved as for Act as me (a signed-in person on the roster, or the principal under local login; otherwise 403 `sign_in_required` / `not_on_roster`). `GET` returns `{notes, reply_notes, retention_days, effective_retention_days, company_retention_days, retention_choices, can_keep_notes, can_note_replies, can_set_company_retention, share_work_style, can_share_work_style}` (`can_keep_notes`: a team member on the People list may turn notes on; `can_note_replies`: they can use Act as me, so email replies are noted too; `share_work_style`: their own \"Share my work style with the team\" switch, `can_share_work_style`: a team member in a team workspace). `PUT` turns notes on only for someone who may keep them, and `share_work_style` on only for someone who may share it (403 `not_available` for either; turning either off always works), takes a person's retention only when shorter than the company's (422 `invalid_setting`), and the company retention only from the principal (403 `principal_only`). `PATCH` pins or corrects a note; `DELETE` forgets it; `forget-conversation` is \"Don't remember this\". A note that isn't the caller's is 404. Every change is audited privately to its person. |\n| `audit` | `GET /audit/logs`, `GET /audit/logs/{id}`, `GET /audit/sessions/{id}`, `GET /audit/usage`, `POST /audit/log` | Durable audit trail of side effects. `/audit/sessions/{id}` adds a per-session token cost summary; `/audit/usage` aggregates token usage and OpenRouter cost across all sessions (totals, by day, by model, and by source — the part of the system that made the calls — including server-side web-search counts), plus `scripts`: how many `run_script` scripts ran (and worked, and in workflows), the tool calls they made, their time, and `turns_avoided` — an upper bound on the model turns those calls would otherwise have needed (each call past a script's first) — and `python_jobs`: how many Python jobs ran (and worked, ran a kept Custom tool, took a chat attachment), their time, CPU, largest peak memory and bytes in and out, and the tokens and charge of the turns that ran one. The Token usage page shows it as a Scripts section. Rows private to the principal (`private_to_principal`: a turn about their private mail, or a row naming one of their contacts on their own turn) are theirs alone: for any other caller, resolved as for `/alerts`, the usage totals leave them out. Once a principal is on the roster, anyone else reads only their own rows (a teammate's own turns, their Act as me work) in the list, by id and in a session, and the principal reads those as actions only — what was done, when and in whose conversation, with the text, details and `full` left out, and no text search over them; any other row's id is a 404, and a session shows only the rows the caller reads. |\n| `architecture` | `GET /architecture/sections`, `.../sections/{id}` | This documentation surface. |\n| `guide` | `GET /guide/sections`, `.../sections/{id}` | Plain-language user-guide content surface — mirrors `architecture` (static, version-controlled files; no LLM on the serving path). |\n| `health` | `GET /health`, `GET /health/honcho` | Liveness probes. Only `/health` is unauthenticated; `/health/honcho` sits behind the gate like everything else, because the exemption is an exact-path set. `/health` also carries `google_tools_missing`, the pinned Google tools startup priming could not find: `[]` once it has checked and they are all there, a list of names when a workspace-mcp rename or the configured services and tool tier left one out, and `null` for no answer — Google Workspace is not configured, or the probe has not finished or it failed. Only `[]` asserts they are present. It also carries `gateway`, the MCP gateway's state from its supervisor (`orchestrator.gateway_boot`): `state` is `off` (MCP not enabled), `no_servers` (enabled, the config names none), `starting` (an attempt is running), `up` or `down` (every attempt so far failed; `next_retry_at` is the next one), with `attempts`, `up_since` and `email_poller` (whether the poller that rides on the gateway runs); the failure's own text stays out, since the route is open to anyone, and goes to the log and the owner's notice on Home. A hosted instance whose gateway never came up used to answer `ok` here with no email going out or in. Only `[]` asserts the tools are present, so a probe that died cannot read as healthy; Setup status says which of the three `null` means. |\n| `version` | `GET /version` | The running version, the latest release on GitHub and `update_available`, for Settings → About. The GitHub answer is cached 6h, only a semver `tag_name` is used and the release link is built from it; `UPDATE_CHECK_ENABLED=false` skips the call and still returns the running version. No in-app updater: upgrading is host-side (`docs/deployment.md#upgrading`). |\n| `setup_status` | `GET /setup/status` | Settings → Setup status: one light per part of the install — AI model, company setup, owner, the Executive's address, API protection, Gmail, your own mailbox (Act as me), Slack, Discord, Telegram, Google Chat, the scheduler, the daily brief (when and where it goes) and Honcho — as `{checked_at, checks: [{id, label, state, summary, fix, link, last_activity}]}`, where `state` is `ok`, `warn`, `error` or `off` (optional and not set up), `fix` says what to do in plain language, `link` is an in-app page that helps (`/people`, `/onboard`) and `last_activity` is the channel's latest `integration_inbound` audit time. It tests live, with at most one free read-only call per service, each bounded to 5 s: Anthropic `models.list` (so an organisation key missing `ANTHROPIC_WORKSPACE_ID` shows up), Slack `auth.test`, Telegram `getMe` and `getWebhookInfo`, Discord `users/@me` only while the bot is still connecting, and — once the owner's own Gmail credential exists — Google's token refresh and Gmail `users/me/profile`. It also reads the listeners the lifespan leaves on `app.state` (Slack's socket, the Discord bot and its task, the MCP gateway), the scheduler heartbeat and the pause switch. When `google_workspace` is configured, the lifespan also starts `MCPGateway.prime_pinned_tools()` in the background (held on `app.state.mcp_prime_task`, cancelled at shutdown), so the pinned Google tools named in the prompt are callable without a search. With `DRIVE_SYNC_ENABLED` the lifespan also bootstraps the `drive_sync_scan` heartbeat beside Notion's (see Scheduler); the Drive folder sync has no HTTP route, only the `openexecutive sync-drive` / `purge-drive` CLI. With `CONFLUENCE_SYNC_ENABLED` it bootstraps the `confluence_sync_scan` heartbeat the same way; that sync is run by hand with `openexecutive sync-confluence` / `purge-confluence`. Sample values copied from `.env.example` are named as such without being sent anywhere. No token reaches a response or a log line: probes report a status code, a service's short error code or an exception's class name. Requests within 5 s of a run get that run's answer and concurrent requests share one run (kept per app on `app.state`), so the button can't become a stream of calls to those services. Gated like every other route; the UI adds a sign-in light of its own, from its own settings. |\n| `documents` | `POST\\|GET\\|DELETE /documents`, `GET /documents/{filename}`, `GET /documents/sources`, `POST /documents/sources/{drive\\|onedrive\\|notion}/sync`, `GET /documents/{drive\\|onedrive\\|notion}`, `GET /documents/{drive\\|onedrive\\|notion}/{id}` | Upload, list, view extracted text, and delete indexed company docs. `GET /documents` rows carry `source: \"upload\"` and the `domain` the upload was indexed under. The connected-source routes back the knowledge page's single Company documents list: `/documents/sources` gives each source's `{id, label, enabled, syncing, last_run, last_error, interval_minutes, file_count}`; `/documents/drive`, `/documents/onedrive` and `/documents/notion` list synced files from the sync state (`{enabled, last_run, last_error, files: [{id, name, url, modified_at, synced_at, indexed}]}`); `/documents/{drive\\|onedrive\\|notion}/{id}` returns the stored text (a OneDrive id is `<drive id>:<item id>`) for an id on record (the path comes from the state, never the caller). They are read-only — a synced file is removed where it lives. `POST …/sync` starts one tick of the same sync the scheduler runs as a background task (202); 409 when the source is not connected or a tick is already running (every sync holds a per-process run lock, so a manual run never overlaps the heartbeat; the route also keeps its own started task per source and re-checks just before starting one, so two simultaneous requests start one tick, not two), 429 with `Retry-After` within 60 s of the last successful run or of the end of the last tick in this process, failed ones included, so a failing or slow source can't be re-run back-to-back. The page's text view never loads images from synced content (a remote image would be a read beacon for whoever edits the shared folder). Upload is keyed on the filename, so re-uploading a document replaces its chunks rather than duplicating them, and `DELETE` removes the rows as well as the file. `domain` must be one of the nine specialist domains (including `sales`) or `general`; anything else is a 400 rather than a 200 that indexes where no specialist can retrieve. A scanned PDF is converted (`knowledge.pdf_reader`) before it is indexed and in the text view, as are scanned PDFs in `POST /chat/upload` and the client / onboarding intake uploads. The upload is streamed to disk 1 MB at a time and refused with a 413 once it passes 50 MB, and the file is parsed in a short-lived child process (see the RAG section), so a large document does not stay in the API process's memory. A Word or Excel file that cannot be parsed, or whose parse runs out of time, is a 422 with a readable reason. When every parser is busy, the response is a 503 with `Retry-After`. |\n| `executive` | `GET /executive/status`, `POST /executive/pause`, `POST /executive/resume` | Operator pause switch for all autonomous work (scheduler, Gmail poller, workflow resumer); inbound chat and DMs stay live. Status returns `{paused, paused_at, paused_by, reason, held_actions, can_resume}` — `held_actions` counts due `pending` rows that fire on resume. `pause` takes an optional `reason` (≤200 chars) and is idempotent (re-pausing keeps the original start and reason); `paused_by` is the UI proxy's `x-caller-email`, else `api`. Behind the shared-secret middleware — not `SCHEDULED_ADMIN_TOKEN`, which the UI cannot send. Any caller may pause; `resume` is principal-only (`is_principal_or_self`, 403 otherwise; open while no principal is rostered; a header-less caller resolves to the principal as in `chat`), and status carries `can_resume` for the viewer so the UI hides Resume from everyone else. |\n| `workspace` | `GET /workspace`, `PUT /workspace` | Solo/team mode, the user's time zone and the principal's role (`memory.workspace_settings`). Both return `{mode, timezone, effective_timezone, role_kind, role_title, reports_to, remit, measured_on, company_domains, company_domains_custom}` — `timezone` is what the user set (null when unset), `effective_timezone` the zone in effect (`timezone`, else `USER_TIMEZONE`, else UTC), and each role field is null until set. `PUT` is a partial update: only the fields sent change; `mode` is `solo` or `team` (not null), `timezone` an IANA zone, or null/blank to clear it; `role_kind` is `owner`, `in_house`, `independent` or `other`, and `role_title` / `reports_to` / `remit` / `measured_on` are free text (trimmed, at most 120 / 120 / 500 / 300 characters), each null or blank to clear it; an unknown zone, a bad mode or role kind, an over-long role field or an unknown field is a 422 and changes nothing. Switching to solo cancels every pending department check-in, switching to team re-schedules them, and a new zone re-times the pending briefs and reflection; a role change has no side effects (the next turn reads it). `company_domains` is the company's own email domains (a teammate matches by local part on them — see Org): `GET` returns the ones in effect (the stored list, or the domain of the principal's primary address, with `company_domains_custom` false) only to the same caller the role is shown to, `[]` for anyone else; `PUT` takes a list (at most 10 plain domain names, never a free-mail provider's — 422 otherwise, the value never quoted) or null / `[]` to go back to deriving them, and a change is audited with the new list. Each change writes a `workspace_settings_changed` audit row (a role change adds `details.role_fields_changed`, the names of the fields that changed — never their text). The role is the principal's: `GET` returns it only to a caller `PUT` would accept and reads it as nulls for anyone else. `PUT` is the principal's alone (`is_principal_or_self`, 403 otherwise; a header-less caller resolves to the principal as in `chat`) and open while no principal is rostered, so first-run setup can pick a mode. Both work before onboarding — unlike `company-profile`, nothing here needs a profile. |\n| `scheduled` | `GET\\|DELETE` scheduled actions (`?status=…&order=asc\\|desc`) | Pending and fired scheduled actions; `order` flips the `run_at` sort (asc=soonest-first for the upcoming queue, desc=most-recent-first for terminal-status history). `DELETE /scheduled/{id}` cancels a `pending` row (409 otherwise). Like pause it only stops work, so any signed-in user may — the UI proxy's `x-caller-email` is trusted when the request also carries the `BACKEND_SHARED_SECRET` `x-api-key` — as may a matching `X-Admin-Token` (`SCHEDULED_ADMIN_TOKEN`) or a loopback caller with no token set; otherwise 401 (token set) or 503. Audited as `scheduled_action_cancelled`. |\n| `watchlist` | `GET\\|POST\\|PATCH\\|DELETE /watchlist`, `POST /watchlist/{slug}/approve`, `POST /watchlist/{slug}/decline` | External-monitoring watchlist + signals. Rows carry `origin` (`manual` \\| `executive` \\| `research` \\| `research_proposed`); `approve` turns a research suggestion (`research_proposed`, `dry_run`) into a live watch, `decline` `{reason: not_relevant \\| too_noisy \\| wrong_source}` removes it and records the decline (or, for `too_noisy`, keeps it live at a high floor), both 409 unless the row is a pending suggestion; `DELETE ?reason=` on a research-origin row records the decline too. |\n| `workflows` | `GET /workflows`, `POST /workflows/{name}/runs`, `GET\\|POST\\|PUT\\|DELETE /workflows/custom`, `GET\\|DELETE /workflows/custom/{name}/targets`, `POST /workflows/runs/{run_id}/decision`, `GET /workflows/tools/search?q=`, `GET /workflows/tools/describe?names=` | Workflow catalog, run management, and user-created (dynamic) workflow CRUD. Custom definitions come back with a server-managed `owner_person_id` (the creator — the web caller on `POST`, set on insert only; a client-sent value is ignored), who is asked to approve a workflow's first write to a new target. `GET /workflows/custom/{name}/targets` lists the targets its tool steps may write to without asking and `DELETE …/targets?value=` forgets one. `POST /workflows/{name}/runs` is a 403 (\"Only the principal can run this workflow.\"), audited, before any run row exists, when the workflow is one only the principal may start in the workspace's mode (`weekly_review` always, `morning_brief` in solo) and the caller is not the principal. A request with no `x-caller-email` counts as the principal; with no principal on the roster, no signed-in caller does. This governs starting a run only. A run started by hand (this endpoint, or `run_workflow` / research in chat) records its starter as `owner_person_id` and is theirs alone: `GET /workflows/runs` lists the team's (scheduled, no owner) runs plus the caller's own and the ones waiting on them, and `GET` / `DELETE /workflows/runs/{run_id}` and `POST /workflows/runs/{run_id}/decision` are a 404 for anyone else's, the principal included. A signed-in email that is on nobody's People entry cannot start one (403), and a run whose starter can't be read is refused (503); its steps read and draft documents as the starter only. `POST /workflows/runs/{run_id}/decision` with `{decision: \"approve\"|\"reject\"}` answers a run awaiting a yes/no sign-off from the web app — 403 unless the caller is the person it waits on or the principal, 409 if it isn't awaiting or needs a written reply — through the same `apply_resolution` a chat reply uses. Custom create/update also rejects (422) any action-step tool that does not resolve, and so does `POST /workflows/custom/{name}/activate` when turning a workflow **on** — that click is the approval for a tool workflow chat saved switched off (turning off is never checked). Turning on a workflow with action steps requires the body `{is_active: true, definition}` carrying the definition the user reviewed; if the stored one differs (ignoring `is_active` and timestamps) it returns 409, and the switch itself is a compare-and-set on the validated revision (`dynamic_store.activate_if_unchanged`, keyed on `updated_at`), so a save that lands while the tool check awaits the gateway — even from another process — is a 409 too, so only what was shown gets switched on. `GET /workflows/custom` lists inactive definitions too; `GET /workflows/{name}` 404s for them. The two `tools` endpoints return `{tools: [{name, description, read_only, source}]}` for the advanced editor's tool picker and the review card; `read_only` is true only when the name says it reads (a label, not a control). The run stream adds a `progress` event (`step_id`, `summary`) while an action step uses tools. Each catalog entry (`WorkflowMeta`) carries `is_custom`, `background` (true for the system-run built-ins the UI files under “System”) and `quick` (true for a Quick workflow, one per playbook nothing else follows, named `quick_<playbook>`). The run endpoint streams SSE; its `awaiting_human` event carries `delivery` (`self` \\| `sent` \\| `suppressed` \\| `alerted` \\| `failed`) alongside `person_id` / `question` / `awaiting_until`, because the outbound guard or an unreachable approver means the question may never have been asked — a client must not render “waiting on them” for those. A paused run ends the stream with neither `done` nor `error`, so that `awaiting_human` frame carries `terminal: true` to say so explicitly, plus `resumable` — whether the run continues by itself once answered or stops at the gate. `GET /workflows/runs/{run_id}` returns the run row with the resume payload replaced by a small `resume_progress` summary (`gate_step_id`, `completed_step_ids`, or `null`): the payload holds every completed step's full text, and the run-detail page polls this endpoint while a run is unfinished. |\n| `knowledge` | `/knowledge/builtin`, `/external`, `/search` | MBA knowledge base and external sources. `/search` is the introspection parallel of `retriever.retrieve` and mirrors its filters — including the `general` catch-all on the company collection — so the panel cannot report a specialist seeing nothing that it does retrieve in chat. It accepts `general` in `domain_filter` for the same reason. `/search` applies the same review gate as `retrieve()` — withheld (pending or rejected) built-in, external and failure rows are dropped, so the panel never advertises knowledge the Executive cannot use; it degrades to unfiltered rather than failing if review state is unreadable. `POST /knowledge/builtin` refuses a name already held by shipped content, and `POST|DELETE /knowledge/failures` now register and deregister for review — without that a user-authored failure case study had no review row, so no SME decision could block it. |\n| `skills` | `/skills`, `/skills/search`, `/skills/{name}/restore` | Skill registry (the UI's **Playbooks**) CRUD and search. `PUT` on a built-in saves a company customization that shadows it; `DELETE` returns `{name, outcome}` — `deleted`, `reverted` (customization removed, built-in back) or `hidden` (built-in hidden for this company); `restore` un-hides; `GET /skills?include_hidden=true` lists hidden built-ins. List and detail carry `used_by` — `[{name, title, is_custom}]` of the workflows that follow the playbook (switched-off custom ones included) (`WorkflowMeta.playbooks` is the other direction). |\n| `evals` | `/evals/scenarios`, `/evals/runs` | Eval scenarios and run results. `POST /evals/runs` is a 403 when a workflow scenario names a workflow that only the principal may start in the mode it runs in (the weekly review; the morning brief in solo) and the caller is not the principal — the Jobs page's rule for starting a run; stored results are read under the existing `/evals/runs` rules. |\n| `memories` | `/memories/{decisions,initiatives,advice}`, `GET /memories/facts`, `POST /memories/facts/{fact_id}/{retire,approve,decline}`, `GET /memories/facts/approval`, `PUT /memories/facts/approval/{person_id}`, `GET /memories/people`, `GET /memories/people/{person_id}/conclusions` | Episodic-memory record management; the standing facts the Pulse Corrections tab reads (`{facts: [...], can_retire, retirable_ids, can_review}`; each fact carries `recorded_by_role` / `recorded_by_name`; `?include_inactive=` default true, honoured for the principal only; anyone else gets the active rows plus their own `proposed` ones, with the provenance (`session_id`, `turn_id`, `recorded_by_person_id`, `retired_reason`, and `source_quote` except on their own facts) blanked, never retired, replaced or declined text; `?limit=` 1–500) and their retire (the principal any fact, a teammate only one they recorded that the principal has not approved — else 403; the audit row names the teammate; 404 for an unknown or `profile` row, 409 when no longer active; writes a private `fact_retired` audit row); approve / decline a teammate's `proposed` fact (principal only, else 403; 409 when it is not proposed; private `fact_reviewed` row); and the per-teammate \"needs my approval\" switch (on by default; `GET` lists `[{person_id, full_name, needs_approval}]` for every active team member but the principal, `true` for anyone not yet set; `PUT` takes `{needs_approval}`, 404 for anyone else; both principal only; private `fact_approval_changed` row); plus the read-only peer-memory overview the Pulse People tab reads (`{status: ok|disabled|error, people: [...], conclusion_total}`; `?recent=` 1–50, default 5) and one person's full conclusion list, paged newest first (`{status, person_id, items, page, size, total, has_more}`; `?page=` ≥1, `?size=` 1–100, default 50; 404 when the person has no peer). Peer memory is each person's own: the overview holds only the caller's entry (none for a caller off the roster; `conclusion_total` counts only theirs), the principal included, and `/conclusions` for anyone but the caller is a 404. |\n| `review` | `/review/items`, `/review/stats`, `/review/trusted-defaults`, `POST /review/curate`, `POST /review/bulk-approve` | Knowledge review queue and annotations. Shipped knowledge registers as a trusted default (`approved` with a NULL `reviewed_at`), so the queue starts empty; `curate` (`{domain, action: start|stop}`, domain required) opts one domain in or out, and `trusted-defaults` returns domain → unreviewed count. `bulk-approve` takes a non-empty `domain`, `item_ids` (max 500) or `all_pending: true`, and 400s with no selector; it returns `{approved_count, item_ids}`. |\n| `onboarding` | `/onboard/interview/understand`, `/onboard/interview/start`, `/onboard/interview/message`, `/onboard/interview/draft`, `/onboard/interview/{id}`, `/onboard/interview/commit` | Conversational company setup. The `onboarding_interviewer` agent asks clarifying questions, then emits a draft profile + people + departments. **Nothing is persisted until `POST /onboard/interview/commit`** — the interview itself never writes. `start` is multipart (description + optional deck/one-pager uploads). `commit` answers 403 when the draft would make a different person the owner and the caller is not the current owner (setup demotes the old principal, and any signed-in user can open it). It also takes an optional `owner_email` — the principal's sign-in email, which the review screen pre-fills from the signed-in user — checked before anything is written (422 if it is malformed, already another person's, or would replace the principal's existing email — setup only fills in a missing one; 403 if the caller is not the owner and it is not the address they signed in with, since the roster is the sign-in allow-list) and saved on the principal's People row, so the owner's chats resolve to them from the first turn. |\n| `workflow_designer` | `POST /workflows/designer/start`, `POST /workflows/designer/edit`, `POST /workflows/designer/message`, `POST /workflows/designer/draft`, `GET /workflows/designer/{session_id}` | The conversational New workflow wizard, and editing a saved workflow by conversation. `/edit` (`{name}`) opens a session on a saved custom workflow with no model call: the stored definition is the starting draft, and every response then carries `editing` (its name) and `original` (the stored definition as the session opened on it); drafts keep that name and its on/off state whatever the model writes (404 for an unknown workflow). The UI saves one with `POST /workflows/custom/{name}/save-edit` (`{session_id, definition}`): the server saves the session's own draft, never a client-sent body, and only when it equals the `definition` the user reviewed (a later message is a 409, so the click saves only what was shown); it re-validates it, keeps the stored on/off state, and writes only if the stored workflow is still the version the conversation opened on — otherwise 409, as it is for a session about another workflow or one with nothing drafted yet. It sits under `/workflows/custom` so anything that limits writes to custom workflows covers it too. The `workflow_designer` agent asks clarifying questions (`phase: \"question\"`, with `hint` and up to four suggested `options`), then returns `phase: \"draft\"` with `draft.definition` (exactly the body `POST /workflows/custom` takes), `summary`, and `assumptions`; every response carries the `transcript`. **Nothing is persisted here** — sessions are in memory, and the UI saves through `POST /workflows/custom`. Errors are fixed strings (422 over the 8,000-char message cap or the transcript cap, 404 for an expired session, 409 while another turn on the same session is still running, 502/504 for model failure or timeout). Mounted ahead of the `workflows` router. |\n| `python_jobs` | `GET /python-jobs/{job_id}/{name}` | A result file a Python job (`run_python_job`) wrote, the principal's alone (403 otherwise), served as a download (`Content-Disposition: attachment`, `Content-Security-Policy: sandbox`, `nosniff`, `Cache-Control: no-store`); 404 for a bad id or name or a file past its 7 days. |\n| `saved_tools` | `GET /saved-tools`, `GET\\|PUT /saved-tools/{name}`, `POST /saved-tools/{name}/rollback` | The scripts the Executive saved as tools (Settings → Advanced → Saved tools), the owner's alone (403 otherwise). The list gives each tool's name, description, on/off, current version, `kind` (`script` or `python`, a kept Python job) and the tools it calls, plus whether `SAVED_TOOLS_ENABLED` is on; the detail adds the script, every version and the last 20 runs. `PUT {enabled?, workflows?, version?}` turns one on or off (off also turns it off for workflows, so turning it back on never re-arms unattended runs), and turns the named `version` — the one the owner looked at, never whatever is current by then — on for workflows (or workflows off); a Python tool never turns on for workflows (409) — turning workflows on lets more run unattended, so it needs a request tied to the owner (signed sign-ins or local login, as Take the lead's switches do; 409 otherwise); `rollback {version}` makes that version the one that runs. Both are audited as `saved_tool_changed`. The Executive saves tools itself through `run_script`; there is no create route. |\n| `onboarding` (fallback) | `/onboard/start`, `/onboard/answer`, `/onboard/status/{id}` | The original 12-step form wizard, kept as a fallback (UI at `/onboard?mode=form`) and used by the `openexecutive onboard` CLI. Its final answer saves the profile and adds people with their emails and a principal, so once there is an owner only they can send it (403, and the session rolls back so the owner can still finish it). `/onboard/start` reads the workspace mode once for the session: solo skips the Team Size, Team Members and Fractional Executives steps. Each status response's `current_step` / `total_steps` count only the steps that session asks, and `optional` says whether the current step can be skipped. |\n| `company-profile` | `GET\\|PATCH /company-profile` | Structured company profile. |\n| `alerts` (private cards) | `POST /alerts/{id}/ack`, `/reopen`, `/alerts/bulk-ack` | An alert tagged `private:principal` (see Today) is a 404 on ack and reopen for anyone but the principal, and their bulk sweep skips it. A roster-request card (source `roster_request`) is a 409 on `ack` even for the principal — it is answered at `/people/requests/{id}`, since acking would leave the sender waiting with nothing to show it — and, as an exempt source, `bulk-ack` and `reopen` skip it. |\n| `auth` | `GET /auth/allowed-emails` | Non-archived **team members** with an email, offered to the UI's sign-in gate — a contact is never on it, so adding a client to reach them by email no longer lets them sign in. The gate takes the **union** of this roster and its own `ALLOWED_EMAILS` env list, so env entries always work (and are checked first, short-circuiting the fetch) and roster entries are additive — a roster change can never revoke a configured operator. With local login (`make dev` with Google sign-in not set up — the UI then listens on 127.0.0.1 only) the UI signs in without an email, sends no `x-caller-email`, and the backend resolves the caller to the principal, as it does for the CLI. |\n| `fixtures` | `/fixtures`, `/fixtures/{name}/load` | Demo-company fixture load/reset/snapshot. `POST /fixtures/reset` leaves no departments, like a new install (`departments_seeded` is always 0). |\n| `clients` | `/clients`, `/clients/generate`, `/clients/cockpit`, `/clients/save`, `/clients/{slug}/activate` | Client-company slots — multi-client mode for fractional use; create from current/blank/AI-generated intake draft (pasted notes and/or file attachments, multipart on `/clients/generate`), save-back switching, engagement metadata (PATCH), cross-client cockpit, delete. See the Client Companies section. |\n| webhooks | `POST /webhook/telegram`, `POST /webhook/google-chat`, `POST /webhook/slack` | Inbound integration callbacks (self-verified). `/webhook/slack` answers 503 unless `SLACK_SIGNING_SECRET` is set. |\n| MCP server | `POST\\|GET /mcp` | Open Executive exposed as an MCP server (Streamable-HTTP). Not a router — a mounted FastMCP sub-app. Same shared-secret gate (`x-api-key`); exposes company context as resources and the specialist council as tools. See the MCP Server section. |\n\n**Take the lead** (`api/routes/take_the_lead.py`): `GET /take-the-lead` returns the As the Executive switch, its six Ask first switches (`{kind, label, hint, on}`, the hint a plain line on what each one catches), the company's rules, whether this server can tie a change to the owner (`available`), and whether the Executive is paused. `PUT /take-the-lead` takes `{enabled?, ask_first?}`, and `POST /take-the-lead/rules` / `DELETE /take-the-lead/rules/{id}` add or remove a company rule (`{kind: person|domain|words|amount, value}`, 422 on a bad one). All four are the principal's alone (403). Anything that lets more happen on its own (turning it on, turning an Ask first off, removing a rule) needs a caller the API can tie to them (`delegation.verified.caller_refusal`, else 409). `GET /today/activity` adds `took_the_lead` rows (what Take the lead did, named, not quoted) and, for the caller alone, `sent_as_you` rows for what Handle it for me sent as them. Under `/delegation`: `PUT /delegation/take-the-lead {enabled}` is Take the lead as you (the owner's only for now; turning it on turns Handle it for me on), and `GET|POST /delegation/take-the-lead/rules` and `DELETE …/rules/{id}` manage the caller's own rules. `HandleItOut` gains `lead` and `lead_available`. A `take_the_lead_action` decision is approved or rejected through the existing `/decisions/{id}/approve|reject`. Training: the response also carries `training` (In training: everything waits unless allowed), `learned` (`{id, label, feature, example, uses, created_at}`: what it was allowed, the edit kept as an example by field) and `suggested` (`{key, label, approvals}`); `PUT` takes `training?` too, and leaving training while on needs the same provable owner as turning it on. `POST /take-the-lead/learned {key}` allows a current suggestion (404 for anything else; provable owner), `DELETE /take-the-lead/learned/{id}` makes it ask first again. `learned` also lists the owner's own Act as me senders (`feature` `act_as_me`), never anyone else's, and Remove works on those too. On a `take_the_lead_action` card, `POST /decisions/{id}/approve` takes `edits.input` (the card's editable fields, 422 for any other; the principal's alone, 403 for an area approver) and `edits.allow: true` (a training card only, 422 otherwise; the principal from a provable request, 403 otherwise), finishing `approved_with_edit` when a field changed.",+ "markdown": "The FastAPI surface lives in `openexecutive.api.main`, which mounts one router per topic. A shared-secret middleware gates every request via an `x-api-key` header, except `/health` and the `/webhook/*` callbacks (which carry their own verification). With local login (`OE_LOCAL_LOGIN=1`, which only `make dev` sets, and only while neither Google nor SSO sign-in is set up) a second middleware answers 403 to any request whose raw `Host` is not `localhost`, `127.0.0.1` or `[::1]` (except a webhook that verifies its caller: Google Chat and Slack always, Telegram only with a `TELEGRAM_WEBHOOK_SECRET` Telegram can send), and to any write whose `Sec-Fetch-Site` is present and not `same-origin`/`none`. The UI signs that owner in with no password and this API usually has no secret, so without it a web page in the owner's browser could drive the API as the principal — by DNS-rebinding its own hostname to 127.0.0.1, or by posting a plain form from another site or localhost port. CORS is restricted to the UI origins in `BACKEND_ALLOWED_ORIGINS`, plus the `make dev` UI on `localhost:3000` only when `OE_PUBLIC_DEPLOYMENT` is unset, so a public instance never answers a localhost page with credentials. The gate is off when `BACKEND_SHARED_SECRET` is unset — intended for local development only — so any internet-reachable deployment sets `OE_PUBLIC_DEPLOYMENT`, and `create_app()` then refuses to boot without the secret rather than serving an unauthenticated API.\n\n**Who is calling** is read in one place, `api/caller.py`, which answers with a `Caller` kind and email; a test fails on any other read of an `x-caller-*` header. By default (`open`) it is the `x-caller-email` the UI proxy stamps from the sign-in, trusted because whoever holds the shared secret is trusted as the proxy, and a request with none runs as the principal. **Signed callers** remove that trust. With `CALLER_ASSERTION_PUBLIC_KEYS` set, the proxy signs every request with an Ed25519 key only the UI holds, and a third middleware checks the `x-caller-assertion` header:\n\n- **What it covers.** It is good for one method, the exact path and query, at most 120 seconds, and one use. The API keeps used ids in memory.\n- **What it names.** A `user`, by email, or the `operator`: local login, which runs as the principal.\n- **No caller header at all** is a `service`, for scripts or MCP clients holding only the secret. It is never the principal, not even on an install with no owner yet, so it can't save a first setup.\n- **Anything else is a 401.** That includes an `x-caller-email` without an assertion (`caller_assertion_required`), and a bad, expired, reused or mismatched assertion (`caller_assertion_invalid`).\n\nPublic keys that can't be read stop the boot, rather than fall back to trusting the header. `scripts/make-caller-keys.py` makes the pair, and `scripts/mint-caller-assertion.py` signs one request for curl. The API holds only public keys. Setup's API-protection check warns while a shared secret is set without them.\n\nEndpoints grouped by router:\n\n| Router | Key endpoints | Purpose |\n|---|---|---|\n| `chat` | `POST /chat`, `POST /chat/stop`, `POST /chat/add`, `POST /chat/background` | SSE stream of a live Executive turn. `POST /chat/background` starts a new chat that the server answers to the end with nobody connected (a phone approving a card, then closing the app): it answers `202` with the new `session_id` once the same caller, access and quota checks as `/chat` pass, the turn saves itself as a `/chat` turn does, and one caller may have three running at once (`429` beyond that). `POST /chat/add` hands the running turn a message the person typed while it works; the turn takes it at its next step and the stream names it in a `message_added` event, and a second `POST /chat` on the same conversation waits for the first to finish. The request body optionally carries `page_context` (route, page title, guide section id, and a form descriptor with current values) — sent only by the Ask OE side panel so the Executive can explain the current page or propose form values. When the Executive calls its `propose_form_values` tool, the stream emits a `form_patch` event (`{type, form_id, fields, rationale, iteration}`) that the panel applies into the on-screen form as highlighted suggestions; the user reviews and saves through the form's normal submit path. Each round of tool calls also emits an `activity` event (`{type, label, tool, iteration}`) naming what is running — progress only, forwarded verbatim and never persisted with the message, unlike an `action_taken` chip. A `session_id` in the body must be a chat the caller may use (the same ownership rule as `sessions`). If it isn't, the turn runs in a fresh chat instead, so nothing is written into someone else's session and the reply looks the same as for an unknown id. An unknown namespaced id (`slack:…`, `telegram:…` and so on) is ignored and a fresh chat is started, because those ids belong to the channel adapters and claiming one first would make the caller its owner. The client mints a `client_turn_id` and sends it with the turn; `POST /chat/stop` takes that id and halts the turn mid-flight, so the button is live from the moment Send is pressed rather than only once the first SSE byte arrives. A stopped turn emits a `stopped` event and then the usual `done` over the same still-open stream — never an `error` — and whatever streamed so far is persisted and flagged. The endpoint returns 404 both for an unknown id and for another caller's turn, so it cannot be used to probe which turns are live. An optional `memory_text` (1–2000 chars) is what peer memory records as the caller's words instead of `message` — the briefing handoffs send it because their seed quotes the Executive's own card; `POST /chat/upload` derives it from the typed text and filenames. Since it never reaches the transcript, the `chat_turn` audit row keeps it beside `message`. After the reply — a stopped or timed-out one too, before the `stopped` / `error` / `done` frames — a turn that looked at anything sends one `sources` event, `{type, session_id, sources: [{kind, title, url}], unavailable: [area]}`: `kind` is `company`, `knowledge`, `notion`, `research`, `document` or `web`, `url` is only ever an http(s) link or, for an earlier document, `/artifacts/<id>`, at most 12 sources (6 from the web, pages the reply cites first), and `unavailable` names parts of the analysis (\"finance\") whose specialist failed or said nothing every time it was asked. It is persisted with the reply as `chat_messages.sources`. A request that carries `x-caller-email` marks its turn signed in (`Session.web_caller_signed_in`), which Act as me needs on a deployment without local login. |\n| `sessions` | `GET /sessions`, `GET /sessions/{id}`, `GET /sessions/{id}/messages`, `DELETE /sessions/{id}`, `POST /sessions/{id}/messages/{message_id}/feedback`, `GET /sessions/{id}/followup` | List, read and delete saved chat sessions (assistant messages carry their row `id` and any `feedback`). The list is scoped to the caller, and each chat carries `preview`: its last message as one plain line, Markdown marks stripped and cut at a word near 120 characters, with `You:` before the person's own words. With `?q=` it is a search: the caller's own chats whose title or message text contains the words (case-insensitive, `LIKE` with wildcards escaped), each with `match_count` (messages that contain them; 0 for a title-only match) and `snippet` (the newest matching message, around the match, prefixed `You:` or `Executive:`). Scoped exactly like the list, so nobody else's conversation can match. The Chats page searches this way and opens a result with `?find=`, so the chat starts on that message and steps through the others. A chat-app conversation started with /new is left off the list until its first turn (`memory.conversations`). Every per-session route checks ownership, because session ids are guessable (`slack:dm:<user id>`, `telegram:<chat id>`): only the session's owner or the principal gets through, and a legacy row with no owner is the principal's alone. A teammate's own chat (a web chat, a DM, a private Telegram chat; not a channel or group conversation, `chat._shared_space`) is theirs alone, the principal included, as their mail with the Executive is. Anyone else gets the same 404 as for an unknown id, so the routes can't be used to find out which chats exist. The one other way in is having started the chat in this process (matched by Person id or verified email, so it survives the caller being added to the roster), which is how a caller the roster can't resolve keeps using their own chat. That record is in memory, so after a restart such an ownerless chat is refused for reads. The principal can continue someone else's ownerless chat without becoming its owner. Deleting a session also drops its in-memory copy, so a later turn naming that id starts fresh. The feedback route records 👍/👎 (`{\"feedback\": \"up\"|\"down\"|null}`) on one assistant reply; only the session's own caller or the principal may rate it. The rater is stored (`feedback_by_person_id`), and a 👎 on a reply to the rater's own message triggers a working-style pass for them (paced; see attunement). The chat stream's `done` event carries `message_id` for the persisted reply. The followup route returns `{\"suggestion\": string|null}` — one suggested next user message, written by the utility-fast model from the last few turns, that the chat composer shows as its greyed placeholder (Tab / → or the inline chip fills it in without sending). It is null unless the session ends on a persisted assistant reply, and null (uncached) when the model fails. Same ownership check as feedback; successful suggestions are cached in-process per (session, reply id). Messages from `GET /sessions/{id}/messages` carry each assistant reply's saved `sources` (the same shape as the chat stream's `sources` event, without `type`), so a reloaded chat shows them again. |\n| `today` | `GET /today`, `GET /today/activity`, `GET /today/activity/daily`, `GET /today/brief-delivery`, `GET /today/top-three`, `GET /today/weekly-review` | Morning-brief aggregation, recent-action rail, and Pulse heartbeat heatmap (per-day activity counts). `GET /today` carries the viewer's cached \"What's going on\" `narrative` with `narrative_generated_at` (when it was written) and `narrative_stale` (a fresher one is being written — the page re-polls; see Today). A proposal backed by a roster request carries `roster_request` (the new sender's server-derived details and held-message previews; the principal's only) and is answered through the `people` routes below. `brief-delivery` returns the latest morning brief or end-of-day digest that didn't reach the owner, as `{brief, at, problem, fix, readable}` in plain language (`readable` is false when it couldn't be written, so there is nothing to read), or null — null while briefs go out, once the cause is fixed, while the scheduler is off, and for anyone but the owner (`chat._caller_is_principal_or_unclaimed`); it carries no address or channel id. `top-three` and `weekly-review` feed the solo Briefing's own cards and answer only in solo mode and only to the principal (`workflows._caller_is_the_principal`: a principal match or no caller header, never an unclaimed roster); anyone else, and team mode, get null. `top-three` returns `{items: [{key, kind, text, why, slot}]}` — the morning brief's top three from the same code (`build_top_three(principal_due_soon())`), `slot` a free block like `10:00–11:00`, `\"\"` when none is left, null when no calendar was read — and is kept off `GET /today` because its calendar read may take up to 4 s. `weekly-review` returns the latest completed `weekly_review` run as `{run_id, completed_at, period, top_three, excerpt}` (next week's top three as plain text; `excerpt` when that list is empty), or null when none has completed. |\n| `people` | `GET\\|POST\\|PATCH /people`, `POST /people/{id}/archive`, `GET /people/requests`, `GET /people/requests/{id}`, `POST /people/requests/{id}/approve`, `POST /people/requests/{id}/decline`, `GET\\|POST /people/{id}/open-loops`, `POST /open-loops/{loop_id}/close`, `GET /people/{id}/outreach`, `GET\\|PUT\\|DELETE /people/{id}/attunement` | Roster CRUD and scope lookups. Every `Person` carries `kind` (`team` / `contact`). `GET /people` lists team members only unless `?include_contacts=true` (the People page asks for both; the department-head and workflow pickers keep the default); `POST` / `PATCH` accept `kind`, and a principal contact is a 422 either way. They also take `email_aliases` (other addresses the person writes from, at most 10; on `PATCH` the full list replaces the current one, omitted leaves it): an alias matches their mail and may be emailed but never signs in; a malformed one is a 422, and an alias — or a new primary `email` — that is already someone else's address or alias is a 409. Every `POST` / `PATCH` then closes any pending roster request whose sender now matches someone and replays its held messages. Adding, editing and archiving (`POST`, `PATCH`, `/archive`) are the principal's only, or anyone's while no one is principal yet so a first setup can add its owner (`chat._caller_is_principal_or_unclaimed`, the check behind `PUT /workspace`; a request with no `x-caller-email`, such as the CLI or local login, counts as the principal). Anyone else, a person editing their own row included, gets 403 before the person lookup, so it can't probe which ids exist. Contacts are private to the principal (`caller_is_principal`: the signed-in email is the principal's, or no `x-caller-email`; stricter than the owner rule above, so while no one is principal nobody is): for anyone else `include_contacts` is ignored, a contact's id is a 404 on `GET /people/{id}` and open-loops (exactly as for a missing id; `PATCH` and archive are already a 403 for them), and creating a contact or turning someone into one is a 403, even on a first setup with no principal yet. `GET /people/me` returns `{person_id, is_principal}` so the UI offers contacts to the principal only. `GET /people/by-scope/{token}` never returns a contact. Archiving a person closes their open loops. The open-loop routes list what a person owes (soonest due first) and close one (`reason`: done / not_needed / cancelled) — both the principal's or the owner's only (403 otherwise, via `people.store.is_principal_or_self`). `POST /people/{id}/open-loops` `{task, due_date?}` assigns that person a task (201 with the loop; `due_date` is a local date, due 17:00 in the user's timezone, clamped to 60 days, default 2 days): any active team member may assign anyone on the team, 403 for anyone else; 409 for an archived person, a contact (as seen by the principal), a person at the open-loop cap, a duplicate or open loops turned off; 422 for an empty task. Nobody is messaged at assignment — the owner is chased once it is due. `/outreach` returns, per kind of proactive outreach over 30 days, `sent` / `replied` / `acted` / `ignored` / `pending` (voided DMs are excluded) — same access rule, checked before the person lookup so it can't probe which ids exist. `/attunement` reads, replaces (`rules`: up to 4 strings, or omitted to only set `locked`) or resets the person's working-style rules — same access rule, 404 for an archived person; `PUT` rejects a rule that is not purely about style with 422. Archiving a person also drops their working style and frees their aliases. The **roster-request** routes (declared before `/people/{person_id}`) are the principal's alone (`caller_is_principal`; anyone else gets 404, as for a missing id): `GET /people/requests?status=pending|approved|linked|declined|expired|superseded|all` lists up to 100 (`id`, `channel`, `channel_ref`, sanitised `display_name`, `profile_email`, `on_company_domain`, `suggested_kind`, `suggested_person_id` / `_name`, `status`, `resolved_person_id` / `_kind`, `message_count`, timestamps, `ack_sent`, and `previews` of the held messages while pending — shown to the principal, never to a model); `approve` takes either `link_person_id` (+ `replace_channel_id`) or `full_name` + `kind` (no default: 422 without both) + optional `role`; `decline` takes no body. An answered request is a 409, a chat id or address already on someone else is a 409; the held messages are replayed (or dropped) in the background. |\n| `departments` | `GET\\|POST\\|PATCH\\|DELETE /departments`, `.../goals` | Department state and per-department goals. `PATCH /departments/{slug}` also takes `watched_entities` (≤50 names, `[]` clears) — the entities the research watch policy may add watches for and route to this department. A `cadences` spec that does not parse is rejected with a 422 (empty means no cadence). A new `head_person_id` must be a team member: an unknown id and one of the principal's contacts get the same name-free 422 (`person_id N is not on the team.`), so the route — open to any signed-in user — never reveals that a contact exists; an unchanged head is not re-checked. `POST .../goals` requires only `key_result`: a missing or blank `period_value` is filled with the current period for `period_type` in the user's zone (the same label `create_goal` uses), and `target` defaults to empty. `PATCH .../goals/{id}` rejects a blank `key_result` or `period_value` with a 422; `target` may be cleared. |\n| `agents` | `GET /agents`, `GET /agents/models`, `GET /agents/models/options`, `GET /agents/presets`, `POST /agents/presets/{id}`, `PATCH /agents/{id}`, `POST /agents/{id}/test` | Specialist metadata and Council overrides. `/agents/models` is the flat model allowlist the PATCH validator uses; `/agents/models/options` returns the same ids in the same order, each tagged with `provider`, `provider_label`, `route` (`direct` / `openrouter` / `local`) and a display `label`, for the Council's Provider → Model picker. `AgentDetail` carries `instructions` (additional instructions, appended after `prompt`, which stays the built-in prompt or its replacement); `PATCH` and `/test` accept `instructions` (max 4,000 chars, blank clears it). `GET /agents/presets` lists the Fast, Balanced and Thorough quality presets (each with `available` and the resolved `model` for this install) plus `active` (null when the council is Custom), `base` and `custom_agents`; `POST /agents/presets/{id}` applies one by writing per-agent model and deep-reasoning overrides and returns the same shape (404 unknown preset, 409 when the install allows none of its models). Each `AgentMeta` carries `visibility` (`core` or `internal`) so the Council's simple view can list core agents first. |\n| `personas` | `GET\\|PUT\\|POST\\|DELETE /personas` | Executive persona definitions. |\n| `alerts` | `POST /alerts/{id}/ack`, `POST /alerts/bulk-ack`, `POST /alerts/{id}/reopen`, `POST /alerts/review` | Groom the briefing queue. `ack` (`read`/`ack`/`dismissed`, optional `mute_topic`) also teaches a watch-sourced alert's watch (`trust_score`); `bulk-ack` takes explicit `alert_ids` and/or `older_than_days` + `category` (exempt sources skipped); `reopen` is the Undo for a `resolved` / `expired` / `dismissed` alert (never `ack` or an artifact / decision row; back to `unread` with every review stamp cleared and its TTL anchor refreshed, so it is live again even if older than its TTL); `review` runs the Executive's relevance review on demand (ignores the interval; still honours `ALERT_REVIEW_ENABLED`, the per-pass caps, the single-flight lock and per-alert idempotency) and returns the move counts. `bulk-ack` caps ids at 500, needs `older_than_days >= 1`, and applies watch feedback per row; `mute_topic` must be one of the alert's tags or a scoped `external:` / `department:` pattern. Alerts still surface only through the briefing (`/today` reads the live view) — there is no list, prefs, mutes, or live-stream endpoint. |\n| `artifacts` | `GET /artifacts`, `GET /artifacts/{id}`, `GET /artifacts/{id}/download?as=`, `POST /artifacts/{id}/archive`, `POST /artifacts/{id}/restore`, `DELETE /artifacts/{id}` | Browsable deliverables in any format — Markdown, HTML, Word, Excel, or a link into a connected app — with `format`, `downloads`, `external_url`, `supersedes_id` and, for an extension's kind of document, `collection` / `collection_label` on every summary (`orchestrator.extensions`; archive, restore and delete tell the collection first and answer 502 if it refuses). `download` renders the file (Word/Excel built at request time from the stored text; any Markdown artifact, workflow output included, can export `?as=docx`) and is always an attachment with `nosniff` and a sandbox CSP. Every route answers for the signed-in caller (`artifact_records.Viewer`): a drafted artifact is its owner's alone (`alerts.owner_person_id`, the person whose conversation published it; none = the principal's), a workflow output follows its run (`persistence.run_visible_to`), and anything the caller may not see is a 404 like a missing id, the principal included. `POST /alerts/{id}/ack` and `/reopen` answer 404 for someone else's drafted artifact the same way. |\n| `decisions` | `GET /decisions`, `POST /decisions/{id}/{approve,reject,cancel}`, `GET`/`PUT /decisions/classes/meeting_scheduling`, `GET /audit/reliability` | Approval-gated decision instances (e.g. calendar bookings). What approve/reject/cancel do is per class (`DECISION_CLASSES`, one `DecisionClassSpec` each; meeting booking books on approve and deletes on cancel; `delegation_reply`, an inbox reply card, is its approver's alone (`approver_only`) — hidden from everyone else, the principal included when it is a team member's — sends its Gmail draft on approve and, after a reject, deletes the draft unless it was edited), and an instance of a class with no spec gets 409 instead of being carried out as another. Only the principal, or the person the decision went to for approval (`approver_person_id`), may approve, reject or cancel it. Anyone else gets 403, and a decision nobody was asked about, or one private to the principal, is the principal's alone. The resolver is recorded as `resolver_person_id`, and a reject that loses the race to another answer is a 409. `GET /audit/reliability` is the principal's alone (403 otherwise), because it counts private decisions too. A proposed booking is surfaced on the briefing as a companion alert; approve/reject/cancel here also clear that alert. An approval whose caller resolves to the principal runs the booking inside `grant_contact_egress()`, so a meeting the principal proposed with a contact can be created. Such a booking's payload is marked `private`: it is proposed to the principal, left out of anyone else's `GET /decisions`, and a 404 for them on `GET /decisions/{id}` and every action route. Reviewed in the briefing — there is no separate proposals page. The class routes read and set how meetings are handled: both return `{decision_class: \"meeting_scheduling\", mode: \"propose\" \\| \"auto_execute\"}` (default `propose`); the `PUT` body is `{mode}`, principal-only (403 otherwise, open before a principal exists — but then only `propose` can be set: `auto_execute` is a 409 until a non-archived principal is on the roster), 422 on any other mode, and a change writes a `decision_class_mode_changed` audit row. In solo mode the class mode decides a booking, and `auto_execute` applies only to a turn the principal started on a verified surface — any other turn gets a proposal. |\n| `delegation` | `GET`/`PUT /delegation`, `GET`/`PUT`/`DELETE /delegation/voice`, `POST /delegation/voice/learn`, `POST /delegation/voice/describe`, `POST /delegation/voice/signature`, `PUT /delegation/inbox`, `POST /delegation/inbox/check`, `GET /delegation/replies`, `PUT /delegation/replies/{id}/text`, `GET /delegation/actions`, `PUT /delegation/handle-it`, `PUT /delegation/disclosure`, `PUT /delegation/training`, `DELETE /delegation/learned/{id}`, `GET /delegation/handled`, `PUT /delegation/team` | Act as me — the **caller's own** delegation only; nobody sets it for anyone else. It answers the principal, and a team member once the owner lets them; anyone else gets 403 `not_available_yet`. `GET /delegation` returns `{enabled, gmail: {status, message, email, connect_command, outlook_connect_command, provider}, inbox: {enabled, status, message, watch_since, last_poll_at, checking}, handle_it: {enabled, mode, available, sent_today, lead, lead_available}, training: {replies, follow_ups, actions, drafts, learned}, disclosure: {enabled, handle_it, approved_sends, line}, team}`, where `team` is the owner's switch for the principal while `DELEGATION_TEAM_MEMBERS` allows it — `{enabled, members: [{person_id, name, enabled, inbox, drafts_30d, sent_30d}]}`, counts only — and `null` otherwise. `PUT /delegation/handle-it` takes `{enabled?, mode?}` (`mode` is `careful`, `balanced` or `bold`; anything else is a 422 `bad_mode`) from a caller the API knows is that person — 403 `not_yours`, 409 `caller_signing_required` with only the shared secret — and turning it on needs Act as me and Draft replies to my inbox on (409 `act_as_me_off` / `inbox_off`); `GET /delegation/handled` returns `{replies: [{decision_id, sent_at, to_name, to_email, subject, body, open_questions, gmail_link, source}]}`, the caller's replies and follow-ups (`source` `follow_up`) sent on its own in the last 7 days; a card on `GET /delegation/replies` carries `source` the same way, `can_allow` while Replies (Follow-ups, on a follow-up) is in training, and `learns_style` while Drafts is. `PUT /delegation/disclosure` takes `{handle_it?, approved_sends?}` (any other field is a 422), the caller's own switches for the AI line on replies sent as them (`disclosure.enabled` is the install's `AI_DISCLOSURE`); turning one on needs no signed sign-in, turning one off does (409 `caller_signing_required`), and a change is audited as `delegation_disclosure_changed`. `PUT /delegation/training` takes `{replies?, follow_ups?, actions?, drafts?}` (any other field is a 422) and returns the `GET /delegation` shape; putting Replies or Follow-ups in, or Suggested actions out, needs no signed sign-in, anything else does (409 `caller_signing_required`), and putting Replies or Follow-ups in turns Take the lead as you off (`PUT /delegation/take-the-lead` on answers 409 `in_training` while they are). `training.learned` is `[{id, label, setting, example, uses, created_at}]`, what the caller allowed or kept in training (`setting` `replies`, `follow_ups`, `actions` or `drafts`; the kept example, or \"\"); `DELETE /delegation/learned/{id}` removes one of the caller's own (404 for anyone else's). `PUT /delegation/team` takes `{enabled}` from the principal alone (403 `principal_only`; 409 `not_available` when the install doesn't allow it). The Gmail `status` is `connected`, `not_configured`, `needs_reconnect`, `mismatch`, `no_email`, `shared_mailbox` or `error`, checked live with Google on each call. `PUT` takes `{enabled}` (any other field is a 422); turning it on needs the caller's own Gmail connected, otherwise a 409 whose code says why (`gmail_not_connected`, `gmail_needs_reconnect`, `gmail_mismatch`, `no_email`, `shared_mailbox`, `gmail_error`). The voice routes show, learn (from the caller's sent mail, one pass at a time — a 409 `in_progress`, `locked`, `too_soon`, `not_enough_mail`, `no_profile` or `changed` when it was locked or edited while learning, or a Gmail code), write it from the caller's own description (`POST /delegation/voice/describe` with `{description}`: one model call applies their words to the profile they have and returns it with a short `sample_reply`, saving nothing, so they review it and save it with `PUT`; a 409 `empty`, `too_long`, `in_progress`, `too_many` past 10 an hour or `no_profile`; no mailbox needed), edit and lock, and reset \"How I write\": a typed field that fails validation is a 422 `invalid_voice` listing each `rejected` field, and `clear_exemplars` / `clear_signature` drop those. `POST /delegation/voice/signature` reads the signature from the caller's Gmail settings again and changes nothing else (a locked profile stays locked), with the same Gmail codes as learning: a 409 when their mailbox isn't connected, it refuses the sign-in or the check itself fails (`gmail_error`), and a 502 `gmail_error` when reading the signature fails. On an Outlook mailbox it is a 409 `signature_unavailable`, since Outlook gives apps no signature to read, and the stored one is left alone. A caller `can_delegate` refuses — anyone but the principal, for now — gets 403 `not_available_yet`, which hides the Settings card. A request with no `x-caller-email` is a 403 `sign_in_required` unless the API runs under local login — a header-less call is not a sign-in (the caller is read through `api/caller.py`, as on every principal-only route). `PUT /delegation/inbox` takes `{enabled}` and turns **Draft replies to my inbox** on or off, returning the same shape as `GET /delegation`: on needs Act as me on (409 `act_as_me_off`) and the mailbox connected (the Gmail codes above, which keep their names for an Outlook mailbox too), and turning Act as me off turns it off too. `POST /delegation/inbox/check` starts a scan now in the background and answers 202 with the `inbox` state (409 `inbox_off` or `in_progress`). `GET /delegation/replies` returns `{cards: [...]}`, the caller's open reply cards from the database alone (no Gmail call): the decision id, the sender's name, address, relation and whether it was verified, the subject, when it arrived, what they wrote, the draft's recipients, subject and body, `open_questions`, `flags` and a Gmail link built from a fixed prefix. **Send** is `POST /decisions/{id}/approve`: it sends the card's draft exactly as it is in Gmail, after checking again that the caller is its owner — which on a server needs signed callers (409 `caller_signing_required` otherwise) — that both switches are on (`inbox_off`), no client slot is active (`client_slot`), the Gmail is connected (the Gmail codes), the draft is still there and unanswered (`draft_gone`, `you_replied`) and its recipients are allowed (`no_recipients`, `too_many_recipients`, `executive_recipient`, `not_from_you`). Recipients that changed, or a newer message in the thread, answer 409 `{code: \"confirm\", message, reasons, recipients}`; approving again with `edits: {recipients: <the list shown>, thread_moved_on: true}` is the second yes. **Send + allow** adds `edits.allow: true`: 422 `cant_allow` unless Replies (Follow-ups, on a follow-up card) is in training, 409 `learned_full` when their list is full, both before anything is sent; `edits.example: true` keeps a draft edited in the mailbox as how they write to that person while Drafts is in training. A draft edited in Gmail between those checks and the send is not sent (`draft_changed`). A double tap sends once (`already_handled`); Gmail refusing answers 429 `rate_limited` or 502 `gmail_error` with nothing sent, and a send Gmail didn't confirm answers 502 `send_unconfirmed` and is settled by the next inbox scan. Dismiss is `POST /decisions/{id}/reject`. `GET /delegation/actions` returns `{cards: [...]}`, the caller's own approval cards (`delegation_actions`): the decision id, when it was made, a one-line `why`, and each action's `index`, `kind` (`invite`, `message`, `add_contact`), `summary` and `text` (a message as it will be sent, or a meeting's description). **Approve** is `POST /decisions/{id}/approve` with `edits: {only: [indexes]}` (and `allow: true` for Approve + allow while Suggested actions is in training: 422 `cant_allow` otherwise or when nothing on it can be allowed, 409 `learned_full`; a card carries `can_allow`); it needs the card's own person signed in (403 `not_yours`, 409 `caller_signing_required`), Act as me on (409 `act_as_me_off`) and a card under a week old (409 `expired`), runs once (409 `already_handled`), and returns the decision with `final_payload_json.results`: per action `done`, `waiting` (sent to whoever approves meetings), `failed` with why, or `skipped`. Every change writes a private audit row. |\n| `history` | `GET /memories/history` (`?q=`), `PUT /memories/history/settings`, `PATCH`/`DELETE /memories/history/{id}`, `POST /memories/history/{id}/forget-conversation` | Always in the loop: the **caller's own** notes and switches only, resolved as for Act as me (a signed-in person on the roster, or the principal under local login; otherwise 403 `sign_in_required` / `not_on_roster`). `GET` returns `{notes, reply_notes, retention_days, effective_retention_days, company_retention_days, retention_choices, can_keep_notes, can_note_replies, can_set_company_retention, share_work_style, can_share_work_style}` (`can_keep_notes`: a team member on the People list may turn notes on; `can_note_replies`: they can use Act as me, so email replies are noted too; `share_work_style`: their own \"Share my work style with the team\" switch, `can_share_work_style`: a team member in a team workspace). `PUT` turns notes on only for someone who may keep them, and `share_work_style` on only for someone who may share it (403 `not_available` for either; turning either off always works), takes a person's retention only when shorter than the company's (422 `invalid_setting`), and the company retention only from the principal (403 `principal_only`). `PATCH` pins or corrects a note; `DELETE` forgets it; `forget-conversation` is \"Don't remember this\". A note that isn't the caller's is 404. Every change is audited privately to its person. |\n| `audit` | `GET /audit/logs`, `GET /audit/logs/{id}`, `GET /audit/sessions/{id}`, `GET /audit/usage`, `POST /audit/log` | Durable audit trail of side effects. `/audit/sessions/{id}` adds a per-session token cost summary; `/audit/usage` aggregates token usage and OpenRouter cost across all sessions (totals, by day, by model, and by source — the part of the system that made the calls — including server-side web-search counts), plus `scripts`: how many `run_script` scripts ran (and worked, and in workflows), the tool calls they made, their time, and `turns_avoided` — an upper bound on the model turns those calls would otherwise have needed (each call past a script's first) — and `python_jobs`: how many Python jobs ran (and worked, ran a kept Custom tool, took a chat attachment), their time, CPU, largest peak memory and bytes in and out, and the tokens and charge of the turns that ran one. The Token usage page shows it as a Scripts section. Rows private to the principal (`private_to_principal`: a turn about their private mail, or a row naming one of their contacts on their own turn) are theirs alone: for any other caller, resolved as for `/alerts`, the usage totals leave them out. Once a principal is on the roster, anyone else reads only their own rows (a teammate's own turns, their Act as me work) in the list, by id and in a session, and the principal reads those as actions only — what was done, when and in whose conversation, with the text, details and `full` left out, and no text search over them; any other row's id is a 404, and a session shows only the rows the caller reads. |\n| `architecture` | `GET /architecture/sections`, `.../sections/{id}` | This documentation surface. |\n| `guide` | `GET /guide/sections`, `.../sections/{id}` | Plain-language user-guide content surface — mirrors `architecture` (static, version-controlled files; no LLM on the serving path). |\n| `health` | `GET /health`, `GET /health/honcho` | Liveness probes. Only `/health` is unauthenticated; `/health/honcho` sits behind the gate like everything else, because the exemption is an exact-path set. `/health` also carries `google_tools_missing`, the pinned Google tools startup priming could not find: `[]` once it has checked and they are all there, a list of names when a workspace-mcp rename or the configured services and tool tier left one out, and `null` for no answer — Google Workspace is not configured, or the probe has not finished or it failed. Only `[]` asserts they are present. It also carries `gateway`, the MCP gateway's state from its supervisor (`orchestrator.gateway_boot`): `state` is `off` (MCP not enabled), `no_servers` (enabled, the config names none), `starting` (an attempt is running), `up` or `down` (every attempt so far failed; `next_retry_at` is the next one), with `attempts`, `up_since` and `email_poller` (whether the poller that rides on the gateway runs); the failure's own text stays out, since the route is open to anyone, and goes to the log and the owner's notice on Home. A hosted instance whose gateway never came up used to answer `ok` here with no email going out or in. Only `[]` asserts the tools are present, so a probe that died cannot read as healthy; Setup status says which of the three `null` means. |\n| `version` | `GET /version` | The running version, the latest release on GitHub and `update_available`, for Settings → About. The GitHub answer is cached 6h, only a semver `tag_name` is used and the release link is built from it; `UPDATE_CHECK_ENABLED=false` skips the call and still returns the running version. No in-app updater: upgrading is host-side (`docs/deployment.md#upgrading`). |\n| `setup_status` | `GET /setup/status` | Settings → Setup status: one light per part of the install — AI model, company setup, owner, the Executive's address, API protection, Gmail, your own mailbox (Act as me), Slack, Discord, Telegram, Google Chat, the scheduler, the daily brief (when and where it goes) and Honcho — as `{checked_at, checks: [{id, label, state, summary, fix, link, last_activity}]}`, where `state` is `ok`, `warn`, `error` or `off` (optional and not set up), `fix` says what to do in plain language, `link` is an in-app page that helps (`/people`, `/onboard`) and `last_activity` is the channel's latest `integration_inbound` audit time. It tests live, with at most one free read-only call per service, each bounded to 5 s: Anthropic `models.list` (so an organisation key missing `ANTHROPIC_WORKSPACE_ID` shows up), Slack `auth.test`, Telegram `getMe` and `getWebhookInfo`, Discord `users/@me` only while the bot is still connecting, and — once the owner's own Gmail credential exists — Google's token refresh and Gmail `users/me/profile`. It also reads the listeners the lifespan leaves on `app.state` (Slack's socket, the Discord bot and its task, the MCP gateway), the scheduler heartbeat and the pause switch. When `google_workspace` is configured, the lifespan also starts `MCPGateway.prime_pinned_tools()` in the background (held on `app.state.mcp_prime_task`, cancelled at shutdown), so the pinned Google tools named in the prompt are callable without a search. With `DRIVE_SYNC_ENABLED` the lifespan also bootstraps the `drive_sync_scan` heartbeat beside Notion's (see Scheduler); the Drive folder sync has no HTTP route, only the `openexecutive sync-drive` / `purge-drive` CLI. With `CONFLUENCE_SYNC_ENABLED` it bootstraps the `confluence_sync_scan` heartbeat the same way; that sync is run by hand with `openexecutive sync-confluence` / `purge-confluence`. Sample values copied from `.env.example` are named as such without being sent anywhere. No token reaches a response or a log line: probes report a status code, a service's short error code or an exception's class name. Requests within 5 s of a run get that run's answer and concurrent requests share one run (kept per app on `app.state`), so the button can't become a stream of calls to those services. Gated like every other route; the UI adds a sign-in light of its own, from its own settings. |\n| `documents` | `POST\\|GET\\|DELETE /documents`, `GET /documents/{filename}`, `GET /documents/sources`, `POST /documents/sources/{drive\\|onedrive\\|notion}/sync`, `GET /documents/{drive\\|onedrive\\|notion}`, `GET /documents/{drive\\|onedrive\\|notion}/{id}` | Upload, list, view extracted text, and delete indexed company docs. `GET /documents` rows carry `source: \"upload\"` and the `domain` the upload was indexed under. The connected-source routes back the knowledge page's single Company documents list: `/documents/sources` gives each source's `{id, label, enabled, syncing, last_run, last_error, interval_minutes, file_count}`; `/documents/drive`, `/documents/onedrive` and `/documents/notion` list synced files from the sync state (`{enabled, last_run, last_error, files: [{id, name, url, modified_at, synced_at, indexed}]}`; a Drive row also carries `unreadable_reason`, why a file with `indexed: false` has no text — e.g. a scan OCR could not read — or null); `/documents/{drive\\|onedrive\\|notion}/{id}` returns the stored text (a OneDrive id is `<drive id>:<item id>`) for an id on record (the path comes from the state, never the caller). They are read-only — a synced file is removed where it lives. `POST …/sync` starts one tick of the same sync the scheduler runs as a background task (202); 409 when the source is not connected or a tick is already running (every sync holds a per-process run lock, so a manual run never overlaps the heartbeat; the route also keeps its own started task per source and re-checks just before starting one, so two simultaneous requests start one tick, not two), 429 with `Retry-After` within 60 s of the last successful run or of the end of the last tick in this process, failed ones included, so a failing or slow source can't be re-run back-to-back. The page's text view never loads images from synced content (a remote image would be a read beacon for whoever edits the shared folder). Upload is keyed on the filename, so re-uploading a document replaces its chunks rather than duplicating them, and `DELETE` removes the rows as well as the file. `domain` must be one of the nine specialist domains (including `sales`) or `general`; anything else is a 400 rather than a 200 that indexes where no specialist can retrieve. A scanned PDF is converted (`knowledge.pdf_reader`) before it is indexed and in the text view, as are scanned PDFs in `POST /chat/upload` and the client / onboarding intake uploads. The upload is streamed to disk 1 MB at a time and refused with a 413 once it passes 50 MB, and the file is parsed in a short-lived child process (see the RAG section), so a large document does not stay in the API process's memory. A Word or Excel file that cannot be parsed, or whose parse runs out of time, is a 422 with a readable reason. When every parser is busy, the response is a 503 with `Retry-After`. |\n| `executive` | `GET /executive/status`, `POST /executive/pause`, `POST /executive/resume` | Operator pause switch for all autonomous work (scheduler, Gmail poller, workflow resumer); inbound chat and DMs stay live. Status returns `{paused, paused_at, paused_by, reason, held_actions, can_resume}` — `held_actions` counts due `pending` rows that fire on resume. `pause` takes an optional `reason` (≤200 chars) and is idempotent (re-pausing keeps the original start and reason); `paused_by` is the UI proxy's `x-caller-email`, else `api`. Behind the shared-secret middleware — not `SCHEDULED_ADMIN_TOKEN`, which the UI cannot send. Any caller may pause; `resume` is principal-only (`is_principal_or_self`, 403 otherwise; open while no principal is rostered; a header-less caller resolves to the principal as in `chat`), and status carries `can_resume` for the viewer so the UI hides Resume from everyone else. |\n| `workspace` | `GET /workspace`, `PUT /workspace` | Solo/team mode, the user's time zone and the principal's role (`memory.workspace_settings`). Both return `{mode, timezone, effective_timezone, role_kind, role_title, reports_to, remit, measured_on, company_domains, company_domains_custom}` — `timezone` is what the user set (null when unset), `effective_timezone` the zone in effect (`timezone`, else `USER_TIMEZONE`, else UTC), and each role field is null until set. `PUT` is a partial update: only the fields sent change; `mode` is `solo` or `team` (not null), `timezone` an IANA zone, or null/blank to clear it; `role_kind` is `owner`, `in_house`, `independent` or `other`, and `role_title` / `reports_to` / `remit` / `measured_on` are free text (trimmed, at most 120 / 120 / 500 / 300 characters), each null or blank to clear it; an unknown zone, a bad mode or role kind, an over-long role field or an unknown field is a 422 and changes nothing. Switching to solo cancels every pending department check-in, switching to team re-schedules them, and a new zone re-times the pending briefs and reflection; a role change has no side effects (the next turn reads it). `company_domains` is the company's own email domains (a teammate matches by local part on them — see Org): `GET` returns the ones in effect (the stored list, or the domain of the principal's primary address, with `company_domains_custom` false) only to the same caller the role is shown to, `[]` for anyone else; `PUT` takes a list (at most 10 plain domain names, never a free-mail provider's — 422 otherwise, the value never quoted) or null / `[]` to go back to deriving them, and a change is audited with the new list. Each change writes a `workspace_settings_changed` audit row (a role change adds `details.role_fields_changed`, the names of the fields that changed — never their text). The role is the principal's: `GET` returns it only to a caller `PUT` would accept and reads it as nulls for anyone else. `PUT` is the principal's alone (`is_principal_or_self`, 403 otherwise; a header-less caller resolves to the principal as in `chat`) and open while no principal is rostered, so first-run setup can pick a mode. Both work before onboarding — unlike `company-profile`, nothing here needs a profile. |\n| `scheduled` | `GET\\|DELETE` scheduled actions (`?status=…&order=asc\\|desc`) | Pending and fired scheduled actions; `order` flips the `run_at` sort (asc=soonest-first for the upcoming queue, desc=most-recent-first for terminal-status history). `DELETE /scheduled/{id}` cancels a `pending` row (409 otherwise). Like pause it only stops work, so any signed-in user may — the UI proxy's `x-caller-email` is trusted when the request also carries the `BACKEND_SHARED_SECRET` `x-api-key` — as may a matching `X-Admin-Token` (`SCHEDULED_ADMIN_TOKEN`) or a loopback caller with no token set; otherwise 401 (token set) or 503. Audited as `scheduled_action_cancelled`. |\n| `watchlist` | `GET\\|POST\\|PATCH\\|DELETE /watchlist`, `POST /watchlist/{slug}/approve`, `POST /watchlist/{slug}/decline` | External-monitoring watchlist + signals. Rows carry `origin` (`manual` \\| `executive` \\| `research` \\| `research_proposed`); `approve` turns a research suggestion (`research_proposed`, `dry_run`) into a live watch, `decline` `{reason: not_relevant \\| too_noisy \\| wrong_source}` removes it and records the decline (or, for `too_noisy`, keeps it live at a high floor), both 409 unless the row is a pending suggestion; `DELETE ?reason=` on a research-origin row records the decline too. |\n| `workflows` | `GET /workflows`, `POST /workflows/{name}/runs`, `GET\\|POST\\|PUT\\|DELETE /workflows/custom`, `GET\\|DELETE /workflows/custom/{name}/targets`, `POST /workflows/runs/{run_id}/decision`, `GET /workflows/tools/search?q=`, `GET /workflows/tools/describe?names=` | Workflow catalog, run management, and user-created (dynamic) workflow CRUD. Custom definitions come back with a server-managed `owner_person_id` (the creator — the web caller on `POST`, set on insert only; a client-sent value is ignored), who is asked to approve a workflow's first write to a new target. `GET /workflows/custom/{name}/targets` lists the targets its tool steps may write to without asking and `DELETE …/targets?value=` forgets one. `POST /workflows/{name}/runs` is a 403 (\"Only the principal can run this workflow.\"), audited, before any run row exists, when the workflow is one only the principal may start in the workspace's mode (`weekly_review` always, `morning_brief` in solo) and the caller is not the principal. A request with no `x-caller-email` counts as the principal; with no principal on the roster, no signed-in caller does. This governs starting a run only. A run started by hand (this endpoint, or `run_workflow` / research in chat) records its starter as `owner_person_id` and is theirs alone: `GET /workflows/runs` lists the team's (scheduled, no owner) runs plus the caller's own and the ones waiting on them, and `GET` / `DELETE /workflows/runs/{run_id}` and `POST /workflows/runs/{run_id}/decision` are a 404 for anyone else's, the principal included. A signed-in email that is on nobody's People entry cannot start one (403), and a run whose starter can't be read is refused (503); its steps read and draft documents as the starter only. `POST /workflows/runs/{run_id}/decision` with `{decision: \"approve\"|\"reject\"}` answers a run awaiting a yes/no sign-off from the web app — 403 unless the caller is the person it waits on or the principal, 409 if it isn't awaiting or needs a written reply — through the same `apply_resolution` a chat reply uses. Custom create/update also rejects (422) any action-step tool that does not resolve, and so does `POST /workflows/custom/{name}/activate` when turning a workflow **on** — that click is the approval for a tool workflow chat saved switched off (turning off is never checked). Turning on a workflow with action steps requires the body `{is_active: true, definition}` carrying the definition the user reviewed; if the stored one differs (ignoring `is_active` and timestamps) it returns 409, and the switch itself is a compare-and-set on the validated revision (`dynamic_store.activate_if_unchanged`, keyed on `updated_at`), so a save that lands while the tool check awaits the gateway — even from another process — is a 409 too, so only what was shown gets switched on. `GET /workflows/custom` lists inactive definitions too; `GET /workflows/{name}` 404s for them. The two `tools` endpoints return `{tools: [{name, description, read_only, source}]}` for the advanced editor's tool picker and the review card; `read_only` is true only when the name says it reads (a label, not a control). The run stream adds a `progress` event (`step_id`, `summary`) while an action step uses tools. Each catalog entry (`WorkflowMeta`) carries `is_custom`, `background` (true for the system-run built-ins the UI files under “System”) and `quick` (true for a Quick workflow, one per playbook nothing else follows, named `quick_<playbook>`). The run endpoint streams SSE; its `awaiting_human` event carries `delivery` (`self` \\| `sent` \\| `suppressed` \\| `alerted` \\| `failed`) alongside `person_id` / `question` / `awaiting_until`, because the outbound guard or an unreachable approver means the question may never have been asked — a client must not render “waiting on them” for those. A paused run ends the stream with neither `done` nor `error`, so that `awaiting_human` frame carries `terminal: true` to say so explicitly, plus `resumable` — whether the run continues by itself once answered or stops at the gate. `GET /workflows/runs/{run_id}` returns the run row with the resume payload replaced by a small `resume_progress` summary (`gate_step_id`, `completed_step_ids`, or `null`): the payload holds every completed step's full text, and the run-detail page polls this endpoint while a run is unfinished. |\n| `knowledge` | `/knowledge/builtin`, `/external`, `/search` | MBA knowledge base and external sources. `/search` is the introspection parallel of `retriever.retrieve` and mirrors its filters — including the `general` catch-all on the company collection — so the panel cannot report a specialist seeing nothing that it does retrieve in chat. It accepts `general` in `domain_filter` for the same reason. `/search` applies the same review gate as `retrieve()` — withheld (pending or rejected) built-in, external and failure rows are dropped, so the panel never advertises knowledge the Executive cannot use; it degrades to unfiltered rather than failing if review state is unreadable. `POST /knowledge/builtin` refuses a name already held by shipped content, and `POST|DELETE /knowledge/failures` now register and deregister for review — without that a user-authored failure case study had no review row, so no SME decision could block it. |\n| `skills` | `/skills`, `/skills/search`, `/skills/{name}/restore` | Skill registry (the UI's **Playbooks**) CRUD and search. `PUT` on a built-in saves a company customization that shadows it; `DELETE` returns `{name, outcome}` — `deleted`, `reverted` (customization removed, built-in back) or `hidden` (built-in hidden for this company); `restore` un-hides; `GET /skills?include_hidden=true` lists hidden built-ins. List and detail carry `used_by` — `[{name, title, is_custom}]` of the workflows that follow the playbook (switched-off custom ones included) (`WorkflowMeta.playbooks` is the other direction). |\n| `evals` | `/evals/scenarios`, `/evals/runs` | Eval scenarios and run results. `POST /evals/runs` is a 403 when a workflow scenario names a workflow that only the principal may start in the mode it runs in (the weekly review; the morning brief in solo) and the caller is not the principal — the Jobs page's rule for starting a run; stored results are read under the existing `/evals/runs` rules. |\n| `memories` | `/memories/{decisions,initiatives,advice}`, `GET /memories/facts`, `POST /memories/facts/{fact_id}/{retire,approve,decline}`, `GET /memories/facts/approval`, `PUT /memories/facts/approval/{person_id}`, `GET /memories/people`, `GET /memories/people/{person_id}/conclusions` | Episodic-memory record management; the standing facts the Pulse Corrections tab reads (`{facts: [...], can_retire, retirable_ids, can_review}`; each fact carries `recorded_by_role` / `recorded_by_name`; `?include_inactive=` default true, honoured for the principal only; anyone else gets the active rows plus their own `proposed` ones, with the provenance (`session_id`, `turn_id`, `recorded_by_person_id`, `retired_reason`, and `source_quote` except on their own facts) blanked, never retired, replaced or declined text; `?limit=` 1–500) and their retire (the principal any fact, a teammate only one they recorded that the principal has not approved — else 403; the audit row names the teammate; 404 for an unknown or `profile` row, 409 when no longer active; writes a private `fact_retired` audit row); approve / decline a teammate's `proposed` fact (principal only, else 403; 409 when it is not proposed; private `fact_reviewed` row); and the per-teammate \"needs my approval\" switch (on by default; `GET` lists `[{person_id, full_name, needs_approval}]` for every active team member but the principal, `true` for anyone not yet set; `PUT` takes `{needs_approval}`, 404 for anyone else; both principal only; private `fact_approval_changed` row); plus the read-only peer-memory overview the Pulse People tab reads (`{status: ok|disabled|error, people: [...], conclusion_total}`; `?recent=` 1–50, default 5) and one person's full conclusion list, paged newest first (`{status, person_id, items, page, size, total, has_more}`; `?page=` ≥1, `?size=` 1–100, default 50; 404 when the person has no peer). Peer memory is each person's own: the overview holds only the caller's entry (none for a caller off the roster; `conclusion_total` counts only theirs), the principal included, and `/conclusions` for anyone but the caller is a 404. |\n| `review` | `/review/items`, `/review/stats`, `/review/trusted-defaults`, `POST /review/curate`, `POST /review/bulk-approve` | Knowledge review queue and annotations. Shipped knowledge registers as a trusted default (`approved` with a NULL `reviewed_at`), so the queue starts empty; `curate` (`{domain, action: start|stop}`, domain required) opts one domain in or out, and `trusted-defaults` returns domain → unreviewed count. `bulk-approve` takes a non-empty `domain`, `item_ids` (max 500) or `all_pending: true`, and 400s with no selector; it returns `{approved_count, item_ids}`. |\n| `onboarding` | `/onboard/interview/understand`, `/onboard/interview/start`, `/onboard/interview/message`, `/onboard/interview/draft`, `/onboard/interview/{id}`, `/onboard/interview/commit` | Conversational company setup. The `onboarding_interviewer` agent asks clarifying questions, then emits a draft profile + people + departments. **Nothing is persisted until `POST /onboard/interview/commit`** — the interview itself never writes. `start` is multipart (description + optional deck/one-pager uploads). `commit` answers 403 when the draft would make a different person the owner and the caller is not the current owner (setup demotes the old principal, and any signed-in user can open it). It also takes an optional `owner_email` — the principal's sign-in email, which the review screen pre-fills from the signed-in user — checked before anything is written (422 if it is malformed, already another person's, or would replace the principal's existing email — setup only fills in a missing one; 403 if the caller is not the owner and it is not the address they signed in with, since the roster is the sign-in allow-list) and saved on the principal's People row, so the owner's chats resolve to them from the first turn. |\n| `workflow_designer` | `POST /workflows/designer/start`, `POST /workflows/designer/edit`, `POST /workflows/designer/message`, `POST /workflows/designer/draft`, `GET /workflows/designer/{session_id}` | The conversational New workflow wizard, and editing a saved workflow by conversation. `/edit` (`{name}`) opens a session on a saved custom workflow with no model call: the stored definition is the starting draft, and every response then carries `editing` (its name) and `original` (the stored definition as the session opened on it); drafts keep that name and its on/off state whatever the model writes (404 for an unknown workflow). The UI saves one with `POST /workflows/custom/{name}/save-edit` (`{session_id, definition}`): the server saves the session's own draft, never a client-sent body, and only when it equals the `definition` the user reviewed (a later message is a 409, so the click saves only what was shown); it re-validates it, keeps the stored on/off state, and writes only if the stored workflow is still the version the conversation opened on — otherwise 409, as it is for a session about another workflow or one with nothing drafted yet. It sits under `/workflows/custom` so anything that limits writes to custom workflows covers it too. The `workflow_designer` agent asks clarifying questions (`phase: \"question\"`, with `hint` and up to four suggested `options`), then returns `phase: \"draft\"` with `draft.definition` (exactly the body `POST /workflows/custom` takes), `summary`, and `assumptions`; every response carries the `transcript`. **Nothing is persisted here** — sessions are in memory, and the UI saves through `POST /workflows/custom`. Errors are fixed strings (422 over the 8,000-char message cap or the transcript cap, 404 for an expired session, 409 while another turn on the same session is still running, 502/504 for model failure or timeout). Mounted ahead of the `workflows` router. |\n| `python_jobs` | `GET /python-jobs/{job_id}/{name}` | A result file a Python job (`run_python_job`) wrote, the principal's alone (403 otherwise), served as a download (`Content-Disposition: attachment`, `Content-Security-Policy: sandbox`, `nosniff`, `Cache-Control: no-store`); 404 for a bad id or name or a file past its 7 days. |\n| `saved_tools` | `GET /saved-tools`, `GET\\|PUT /saved-tools/{name}`, `POST /saved-tools/{name}/rollback` | The scripts the Executive saved as tools (Settings → Advanced → Saved tools), the owner's alone (403 otherwise). The list gives each tool's name, description, on/off, current version, `kind` (`script` or `python`, a kept Python job) and the tools it calls, plus whether `SAVED_TOOLS_ENABLED` is on; the detail adds the script, every version and the last 20 runs. `PUT {enabled?, workflows?, version?}` turns one on or off (off also turns it off for workflows, so turning it back on never re-arms unattended runs), and turns the named `version` — the one the owner looked at, never whatever is current by then — on for workflows (or workflows off); a Python tool never turns on for workflows (409) — turning workflows on lets more run unattended, so it needs a request tied to the owner (signed sign-ins or local login, as Take the lead's switches do; 409 otherwise); `rollback {version}` makes that version the one that runs. Both are audited as `saved_tool_changed`. The Executive saves tools itself through `run_script`; there is no create route. |\n| `onboarding` (fallback) | `/onboard/start`, `/onboard/answer`, `/onboard/status/{id}` | The original 12-step form wizard, kept as a fallback (UI at `/onboard?mode=form`) and used by the `openexecutive onboard` CLI. Its final answer saves the profile and adds people with their emails and a principal, so once there is an owner only they can send it (403, and the session rolls back so the owner can still finish it). `/onboard/start` reads the workspace mode once for the session: solo skips the Team Size, Team Members and Fractional Executives steps. Each status response's `current_step` / `total_steps` count only the steps that session asks, and `optional` says whether the current step can be skipped. |\n| `company-profile` | `GET\\|PATCH /company-profile` | Structured company profile. |\n| `alerts` (private cards) | `POST /alerts/{id}/ack`, `/reopen`, `/alerts/bulk-ack` | An alert tagged `private:principal` (see Today) is a 404 on ack and reopen for anyone but the principal, and their bulk sweep skips it. A roster-request card (source `roster_request`) is a 409 on `ack` even for the principal — it is answered at `/people/requests/{id}`, since acking would leave the sender waiting with nothing to show it — and, as an exempt source, `bulk-ack` and `reopen` skip it. |\n| `auth` | `GET /auth/allowed-emails` | Non-archived **team members** with an email, offered to the UI's sign-in gate — a contact is never on it, so adding a client to reach them by email no longer lets them sign in. The gate takes the **union** of this roster and its own `ALLOWED_EMAILS` env list, so env entries always work (and are checked first, short-circuiting the fetch) and roster entries are additive — a roster change can never revoke a configured operator. With local login (`make dev` with Google sign-in not set up — the UI then listens on 127.0.0.1 only) the UI signs in without an email, sends no `x-caller-email`, and the backend resolves the caller to the principal, as it does for the CLI. |\n| `fixtures` | `/fixtures`, `/fixtures/{name}/load` | Demo-company fixture load/reset/snapshot. `POST /fixtures/reset` leaves no departments, like a new install (`departments_seeded` is always 0). |\n| `clients` | `/clients`, `/clients/generate`, `/clients/cockpit`, `/clients/save`, `/clients/{slug}/activate` | Client-company slots — multi-client mode for fractional use; create from current/blank/AI-generated intake draft (pasted notes and/or file attachments, multipart on `/clients/generate`), save-back switching, engagement metadata (PATCH), cross-client cockpit, delete. See the Client Companies section. |\n| webhooks | `POST /webhook/telegram`, `POST /webhook/google-chat`, `POST /webhook/slack` | Inbound integration callbacks (self-verified). `/webhook/slack` answers 503 unless `SLACK_SIGNING_SECRET` is set. |\n| MCP server | `POST\\|GET /mcp` | Open Executive exposed as an MCP server (Streamable-HTTP). Not a router — a mounted FastMCP sub-app. Same shared-secret gate (`x-api-key`); exposes company context as resources and the specialist council as tools. See the MCP Server section. |\n\n**Take the lead** (`api/routes/take_the_lead.py`): `GET /take-the-lead` returns the As the Executive switch, its six Ask first switches (`{kind, label, hint, on}`, the hint a plain line on what each one catches), the company's rules, whether this server can tie a change to the owner (`available`), and whether the Executive is paused. `PUT /take-the-lead` takes `{enabled?, ask_first?}`, and `POST /take-the-lead/rules` / `DELETE /take-the-lead/rules/{id}` add or remove a company rule (`{kind: person|domain|words|amount, value}`, 422 on a bad one). All four are the principal's alone (403). Anything that lets more happen on its own (turning it on, turning an Ask first off, removing a rule) needs a caller the API can tie to them (`delegation.verified.caller_refusal`, else 409). `GET /today/activity` adds `took_the_lead` rows (what Take the lead did, named, not quoted) and, for the caller alone, `sent_as_you` rows for what Handle it for me sent as them. Under `/delegation`: `PUT /delegation/take-the-lead {enabled}` is Take the lead as you (the owner's only for now; turning it on turns Handle it for me on), and `GET|POST /delegation/take-the-lead/rules` and `DELETE …/rules/{id}` manage the caller's own rules. `HandleItOut` gains `lead` and `lead_available`. A `take_the_lead_action` decision is approved or rejected through the existing `/decisions/{id}/approve|reject`. Training: the response also carries `training` (In training: everything waits unless allowed), `learned` (`{id, label, feature, example, uses, created_at}`: what it was allowed, the edit kept as an example by field) and `suggested` (`{key, label, approvals}`); `PUT` takes `training?` too, and leaving training while on needs the same provable owner as turning it on. `POST /take-the-lead/learned {key}` allows a current suggestion (404 for anything else; provable owner), `DELETE /take-the-lead/learned/{id}` makes it ask first again. `learned` also lists the owner's own Act as me senders (`feature` `act_as_me`), never anyone else's, and Remove works on those too. On a `take_the_lead_action` card, `POST /decisions/{id}/approve` takes `edits.input` (the card's editable fields, 422 for any other; the principal's alone, 403 for an area approver) and `edits.allow: true` (a training card only, 422 otherwise; the principal from a provable request, 403 otherwise), finishing `approved_with_edit` when a field changed.","mermaid": null,"generated_at": "2026-10-13T15:30:00Z"}diff --git a/packages/core/openexecutive/architecture/prebuilt/rag.json b/packages/core/openexecutive/architecture/prebuilt/rag.jsonindex 9a04ca3e..d0fb505e 100644--- a/packages/core/openexecutive/architecture/prebuilt/rag.json+++ b/packages/core/openexecutive/architecture/prebuilt/rag.json@@ -1,7 +1,7 @@{"section_id": "rag","title": "Knowledge & RAG",- "markdown": "Retrieval is handled by `openexecutive.knowledge.retriever.retrieve`, querying ChromaDB collections through `openexecutive.knowledge.store.ChromaDBStore`. When the caller passes `record_source`, `retrieve` also names every document its block quotes — company uploads by filename, built-in knowledge by a title made from its filename, Notion pages, synced Drive files by name with their Drive link, synced Confluence pages by title with their page link, research notes by date, and earlier published documents with a link to `/artifacts/<id>` — so the web chat can list what an answer looked at: the chat route's own search, and each specialist's that answered (see the Lifecycle and API sections). The recording never raises, and a retrieval that fails leaves the specialist without context rather than failing the turn. Eight collections back the system:\n\n- **BUILTIN** — curated MBA-style domain knowledge shipped with the repo, including a `sales` domain (founder-led sales, qualification and pipeline, proposals and pricing conversations, follow-up and forecasting) and solo-founder guides in finance, marketing, legal and HR. Seeded at boot by `loader.seed_builtin_knowledge` (FAILURES by `seed_failures`): the whole tree on an empty collection, otherwise only the `SHIPPED_BUILTIN_FILES` entries with no chunks yet — so a doc added in a later release reaches an install seeded by an earlier one, instead of shipping, registering as approved, and never being retrievable. Presence is judged from each row's `source` path relative to `knowledge/builtin/` (not the absolute path, which moves with the install), read in one metadata scan narrowed to seeded rows so the external OER corpus sharing the collection is never pulled. A complete store indexes nothing.\n- **COMPANY** — customer documents ingested via the `/documents` endpoint. Curated uploads only — synced Notion pages, Drive files and Confluence pages do not land here. A document's identity is its **filename**: chunk metadata and the chunk id both derive from it, so re-uploading upserts in place rather than duplicating, and `DELETE /documents/{filename}` matches the rows it claims to remove. Inbound attachments are **not** here — see ATTACHMENTS below; rows left by the earlier scheme are migrated out on boot. A scanned PDF (no text layer) is converted before chunking — see \"Reading scanned PDFs\" below — so it no longer indexes zero chunks while reporting `indexed`.\n- **NOTION_WIKI** — optional Notion sync (`NOTION_SYNC_ENABLED`) incrementally re-indexes pages shared with a Notion internal integration into a separate `notion_wiki` collection (`type=notion`). A Notion share is inherited and multi-writer, so this text is unreviewed relative to curated company docs; the retriever labels it as such and ranks it below COMPANY. Un-shared pages are purged on the next tick.\nSynced Drive files and Notion pages are listed on the knowledge page next to uploads (one Company documents list, each row badged with its source, its sync time and a link to the original), read from the sync state files rather than Chroma; each connected source shows its last run, its last error (a failed tick, or a count of files or pages a tick couldn't index; the next clean tick clears it) and a **Sync now** button (`POST /documents/sources/{id}/sync`). Uploads from the page no longer ask for a domain: they go in as `general`, which every specialist's COMPANY filter already includes.\n\n- **DRIVE** — optional Google Drive folder sync (`DRIVE_SYNC_ENABLED`, `knowledge.drive_sync`). It incrementally re-indexes the files in `DRIVE_SYNC_FOLDER_IDS`, and their subfolders to four levels, into a separate `drive_docs` collection (`type=drive`). It reads the Drive v3 API directly as a service account with `drive.readonly` (`knowledge.drive_client`), not through the MCP gateway, so it sees only folders shared with that account and can change nothing. Google Docs and Slides are exported as text and Sheets as `.xlsx`. PDF, `.docx`, `.xlsx` and plain text files go through the loader's extractors, with no model call, so a scanned PDF yields nothing. Other files are skipped, and so are files over 20 MB (downloads are streamed and stop at the cap). A `.docx` or `.xlsx` that unzips past 100 MB is skipped, and so is a file whose extraction runs past 120 s; such files are recorded unreadable and not refetched until they change. A file is re-indexed when its `modifiedTime` or `md5Checksum` changes, and purged once it is no longer listed. Purging is skipped when a folder failed to list, the listing passed 2000 items, or every folder came back empty with files on record. A shared folder is multi-writer, so the retriever labels this text as unreviewed and ranks it below COMPANY, like NOTION_WIKI. Each chunk is labelled `[drive · file id … · synced … · \"<name>\"]`. The id and time come first because the sync wrote them. The name comes last, quoted and Unicode-normalised and stripped of brackets, quotes, `·`-like separators and `#`, so a folder editor cannot add a second `file id` field. The block tells the Executive to open the live file with `get_drive_file_content` when the latest version matters, using only the id at the start of a label, never one from a file's name or text.\n- **ONEDRIVE** — optional OneDrive / SharePoint folder sync (`ONEDRIVE_SYNC_ENABLED`, `knowledge.onedrive_sync`), DRIVE's twin: the files in `ONEDRIVE_SYNC_FOLDERS` (`<drive id>/<item id>` pairs) go into a separate `onedrive_docs` collection (`type=onedrive`, keyed `<drive id>:<item id>`). It calls Microsoft Graph directly (`knowledge.onedrive_client`, GETs only) as the Executive's own Microsoft 365 sign-in, with a token the Microsoft server's launcher mints for file reads only, because Microsoft has no per-folder service account; the folder list is the scope. PDF, Word, Excel and plain text files go through the same extractors, caps and deadlines as Drive (imported from `drive_sync`); PowerPoint, older Office formats, OpenDocument and RTF are fetched as Graph's PDF rendering first. A file is re-indexed when its `lastModifiedDateTime` or `cTag` changes; purging, its skip rules and the locking are Drive's. Retrieved right after DRIVE, labelled `[onedrive · item <drive id>:<item id> · synced … · \"<name>\"]` with the name defanged the same way, and the block tells the Executive to open the live file with its OneDrive tools using only the ids after `item`.\n- **CONFLUENCE** — optional Confluence space sync (`CONFLUENCE_SYNC_ENABLED`, `knowledge.confluence_sync`), for Cloud and Server/Data Center alike. Each tick lists every page in `CONFLUENCE_SYNC_SPACE_KEYS` over the v1 REST API (`knowledge.confluence_client`, directly with `httpx`, not through the MCP gateway; a personal access token as Bearer, or a username and API token as Basic) and re-indexes the pages whose `version.number` changed into a separate `confluence_wiki` collection (`type=confluence`), at most `CONFLUENCE_MAX_PAGES_PER_SCAN` per tick, newest first. Storage-format XHTML becomes Markdown through stdlib `html.parser` (`knowledge.confluence_storage`): code macros become fenced blocks, callout macros keep their body, and dynamic macros such as Jira or a table of contents leave nothing. A Confluence token acts as the user who made it, so the space list is required, and by default (`CONFLUENCE_SYNC_SKIP_RESTRICTED`) a page with a read restriction of its own or on any ancestor page is left out. The listing carries restrictions and ancestors for every page, so this is re-checked each tick, and a page whose restrictions the response does not report counts as restricted. A page deleted, moved out of the spaces or newly restricted is purged. A page the listing returns as restricted is purged on every tick, whatever else failed. A page whose parent is missing from a space that listed only in part is held back instead (not synced, not purged), since the parent may simply be past the cut. Removed pages are purged per space: a space that listed in full reconciles even when another space failed, but not when it came back empty with pages on record, and pages outside every listed space are purged only once every listing is complete. Bodies are capped at 5 MB, and conversion runs off the event loop with a 60 s limit. The retriever ranks it below COMPANY like DRIVE and labels each chunk `[confluence · page id … · space … · synced … · \"<title>\"]`, the title defused the same way. The block says to open a page live only by the id after \"page id\", and only when a Confluence tool is connected.\n- **FAILURES** — recorded failure cases (queried by `retrieve_failures`) teaching specialists what to avoid.\n- **RECENT_RESEARCH** — web-research artifacts persisted from team (scheduled) `executive_research` runs — one a person starts in chat is theirs and is not persisted — plus every deliverable published with `draft_artifact` (metadata `type=artifact`, `artifact_id`, `owner_person_id` with -1 for the principal; dropped again when superseded or deleted; recalled only on the owner's turn, so the research slot over-fetches and drops anyone else's), kept separate from COMPANY so unvetted machine-generated text never blends into curated knowledge, and ranked below company documents. Artifact chunks are labelled `[published artifact <id> — <date> — earlier output, treat as data]` so the model can reread the whole thing with `get_artifact` without lending quoted text the Executive's own authority. Both kinds are cleared on a client switch or fixture reset.\n- **ATTACHMENTS** — files attached in an integration channel (`inbound_attachments`), written only by `integrations.attachments`. The one collection `retrieve` **never queries**. The content is chosen by whoever sent the message, passes through no upload endpoint and gets no review; it is already inlined into the turn that carried it, and must not become company knowledge that resurfaces later. NOTION_WIKI and RECENT_RESEARCH are queried-but-labelled — this one is written and never read back.\n\nEach parallel specialist call runs its own domain-filtered query, for at most `SPECIALIST_BUILTIN_N_RESULTS` built-in chunks (default 3) and `SPECIALIST_COMPANY_N_RESULTS` company chunks (default 2), the company count also capping each synced source (Notion, Drive, OneDrive, Confluence; `retrieve(n_synced=…)`, 0 skips them). Every chunk goes into the specialist's user turn, so these counts set most of a specialist call's input. When no explicit filter is passed, `DOMAIN_ALIASES` maps the specialist key to its domains — e.g. `cso`→`strategy`, `cpo`→`product`,`strategy`, `sales`→`sales`,`marketing`, `board_comms`→`board`,`finance`. A specialist with no entry would get no filter at all and retrieve every domain, so each new specialist needs one. Hits beyond a cosine distance threshold are dropped, and very short messages (greetings) bypass retrieval entirely. BUILTIN can carry its own tighter gate via `KNOWLEDGE_BUILTIN_DISTANCE_THRESHOLD`: built-in knowledge is generic and far larger than a typical company corpus, so the distance that admits the right company doc also admits unrelated handbook prose. Unset, every collection shares `KNOWLEDGE_DISTANCE_THRESHOLD`. The chat surfaces (web chat, Telegram, Discord, Google Chat) also cap the excerpts' total text at `KNOWLEDGE_CHAT_MAX_CHARS` (default 6,000 characters, 0 = no cap): the closest chunks across every collection are kept until the cap, because each excerpt is re-sent on every model call of the turn. Workflows and specialists keep their own counts, uncapped.\n\n`POST /documents` rejects any domain outside that set with a 400 — an unvalidated string would index the document under a domain no specialist filters on, failing silently and only at retrieval time. The tenth accepted value, `general`, is the catch-all for an upload nobody classified (it is the default on the endpoint, and what every upload from the knowledge page gets — the page no longer asks for a domain). It maps to no specialist, so every specialist's **company** filter is widened to include it — an unclassified document is visible to all rather than to none. That widening is deliberately not applied to BUILTIN, whose rows are shared with external OER sources that fall back to `general` when they declare no domains; fanning those into every specialist would blend unvetted third-party material into every answer.\n\nMulti-domain chunks fan out to one row per declared domain, so `_dedupe_by_text` drops duplicate-text hits while preserving relevance order before results are returned.\n\n### Embeddings\n\nEvery collection embeds with one shared all-MiniLM-L6-v2 ONNX session (`knowledge.store._embedding_function`), the same model and vectors as Chroma's default. Chroma's own default reloads the model on every call and keeps ONNX Runtime's memory arena, so a 300-chunk upload peaked near 760 MB and each query spent ~240 ms reloading. The shared session is built once, runs without the arena and embeds four texts at a time (the model pads every text to 256 tokens, so batch size sets the peak): ~300 MB and ~30 ms. It answers to Chroma's name `default` with an empty config, because every existing collection records that name and Chroma refuses a collection opened with a differently named function; and it must not be a `DefaultEmbeddingFunction` instance, which Chroma swaps for a fresh default on every embed.\n\n### Reading scanned PDFs\n\npypdf returns only a PDF's saved text layer, so a scan, a phone photo saved as PDF, or a deck printed to PDF as images used to come back empty everywhere a PDF is read. `knowledge.pdf_reader.read_pdf_text` tries three readers in order and never raises:\n\n1. **Text layer** (pypdf) — used whenever it yields real text (at least ~20 visible characters per page, so a scan carrying only page numbers still falls through).\n2. **The deployment's own model, through its provider's PDF support** — opt-in, `PDF_PROVIDER_READING=true` (off by default: an upgrade never starts sending scanned PDFs off the server; this also means an Anthropic-direct deployment OCRs scans locally until it opts in) — the pages as a base64 `document` block, transcribed to Markdown by `PDF_VISION_MODEL` (unset: `DEFAULT_MODEL`, the model the deployment already runs on), `PDF_VISION_PAGES_PER_CALL` pages per request (default 20), three requests at a time. Each provider carries the block its own way: Anthropic reads it natively; on OpenRouter the translator makes it an OpenAI `file` part and the provider adds the `file-parser` plugin — engine `native` for Claude or any model the catalog lists with `file` input (OpenAI's GPT-4o and later, Gemini…), else `PDF_OPENROUTER_ENGINE` (default `mistral-ocr`, $2 per 1,000 pages); a local OpenAI-compatible server gets the `file` part only with `LOCAL_PDF_INPUT=true` (e.g. `LOCAL_BASE_URL` pointed at OpenAI's own API). `providers.registry.pdf_input_supported` decides; a model that cannot take one never receives it (the feature gate swaps a stray `document` block for a note). Turning it on sends scanned pages off the server — to the provider, and on OpenRouter possibly to its parser (Mistral OCR) — and the destination is logged once per process.\n3. **Local OCR** — pages rendered with pypdfium2 and read by RapidOCR (ONNX, models bundled in the wheel): offline, no key, no per-page cost. The default reader while step 2 is off, what a local model without PDF input gets, and the fallback on every provider when step 2 errors, refuses or returns nothing. `PDF_OCR_ENABLED=false` turns it off.\n\n`PDF_VISION_MAX_PAGES` (default 100) caps the pages read, with a note saying how many were; a file a channel delivered (`inbound=True`) is capped at `PDF_INBOUND_MAX_PAGES` (30) and metered by a rolling `PDF_INBOUND_PAGES_PER_HOUR` (300) budget. OCR is bounded too: a page renders within a 25-megapixel budget whatever its MediaBox claims (a crafted 20000pt page would otherwise allocate gigabytes), at most two documents OCR at once, and each stops after 240 s with what it has. A PDF of more than 2,000 pages is refused before any page is parsed (the caps above bound conversion, not the text-layer pass). A Claude slice that runs into the output limit is split in half until each part fits; a single page that still does not fit keeps what was written, marked `[transcription cut off here]`, and the note says so. A result is cached in process by the file's sha256 (32 entries) so a re-read does not re-bill; a failure is not cached. `loader.read_document_text` / `extract_text_from_file_async` route a PDF through it and every other type through the sync extractor, and are what `ingest_file` (uploads, boot reconcile), the `/documents/{filename}` preview, intake uploads, chat attachments, the email poller, `oe__read_file` and the Executive's `read_document` tool use.\n\n### Parsing in a child process\n\nParsing a large document builds hundreds of MB of small Python objects, and CPython keeps the memory they lived in, so a parse inside the API process left its peak there for good. On a 34 MB, 131-page text-layer PDF uploaded through `POST /documents`, uvicorn went from 272 MB to a 440 MB peak and stayed at 378 MB. On a 30-page scan read by OCR it peaked at 787 MB and stayed at 446 MB. Now the text-layer pass and local OCR in `pdf_reader`, and the PDF, Word and Excel extractors behind `loader.extract_text_from_file`, run in a short-lived child process (`knowledge.isolated.run_isolated`). Everything the child allocated goes back to the OS when it exits. The same uploads now peak at 324 MB and 297 MB and settle at 318 MB and 290 MB, and what remains is embedding. The OCR child itself still peaks near 600 MB while it reads, then exits.\n\nThe call goes to the child pickled on stdin and the answer comes back as JSON, so the API never unpickles anything a hostile file could shape. Both of the child's output streams go to temp files, not the API's memory. The child logs nothing, because pypdf logs text taken from the file and a crafted 9 KB PDF made it write megabytes. Only the last 2,000 characters of its stderr are read back, and an answer over 32 MB is refused. The child inherits only what it needs to start Python (`PATH`, `HOME`, `PYTHONPATH`, the library path, locale, temp dir), not the deployment's keys. That is hygiene rather than a sandbox, since the child runs as the same user. A PDF read by path is cached only if its hash is unchanged after the read, so an upload that replaces the file mid-read cannot store its text under the old file's hash. `read_document_text` hands `pdf_reader` the file's path, so the API process holds a whole PDF only when its pages go to the model (step 2). The contract is unchanged. `read_pdf_text` still never raises: a child that dies or runs past its limit (300 s for the text layer, or OCR's 240 s budget plus 120 s) comes back as `method=\"none\"` with a note. A Word or Excel parse error still raises, as `IsolatedError` naming the original. At most two parser processes run at once. A caller waits at most 30 seconds for a free slot, and that wait counts against its own time limit. If no slot frees up in time, the file is reported as \"busy, try again\" rather than unreadable. An upload, PDF or not, gets a 503 and nothing is stored. Drive sync retries the file on its next tick. A PDF read elsewhere comes back empty with a note saying so, and is not cached. OCR has its own two-slot limit, so a long OCR never holds up a quick text-layer parse. A scan waits up to one OCR run (240 s) for a slot, and if none frees up it gives its pages back to the hourly inbound budget. Only when there is no usable Python to start does the parse run in-process as before, and it does not take a slot. Failing to start a process for lack of memory is reported as unreadable, because that is the worst moment to parse inside the API. A Word or Excel file that cannot be read is a 422 from `POST /documents` and the preview, not a 500. An Excel workbook is read for at most 200,000 rows (blank ones included) and 1,024 columns: openpyxl walks every row a sheet's `<dimension>` claims, so a few-KB file claiming A1:XFD1048576 would otherwise hold a parser slot until the timeout. `POST /documents` accepts `.pdf`, `.docx`, `.doc`, `.xlsx`, `.xlsm`, `.csv`, `.md` and `.txt`; text formats are decoded as UTF-8 with bad bytes replaced.\n\n`POST /documents` streams the upload to its staging file 1 MB at a time and returns 413 as soon as it passes 50 MB, instead of reading the whole body into memory first. The stored copy is copied from that file.\n\n### The SME review gate\n\nBefore the distance threshold applies, both `retrieve` and `retrieve_failures` drop **withheld** items — anything the review store reports as `pending` or `rejected` (`ReviewStore.get_withheld_filenames`). Approved items with a priority set are then sorted ahead of the rest.\n\nKnowledge shipped with the product registers as a trusted default (`approved`, NULL `reviewed_at`), so nothing is withheld until a human deliberately queues a domain for curation or rejects something. See the **SME Knowledge Review** section for the full state model. The filter matches on the `(domain, filename)` pair Chroma carries. A bare filename was ambiguous *and* caller-choosable — `POST /knowledge/builtin` takes any `(domain, filename)`, so uploading a file named after a shipped doc in another domain silently withheld that shipped doc from every specialist. Failure case studies are keyed in their own namespace, so a user-authored one cannot collide with a shipped one.\n\nThe formatted context is injected into the **user turn** (wrapped in `<relevant_knowledge>` by `BaseAgent.analyze`), never into a cached system block — per-query hits would otherwise poison the prompt cache.\n\n### Skills (Playbooks) collection\n\nSkills — called **playbooks**, and shown in the UI as **How it's done** on the workflow that follows each one — are Markdown procedures with YAML frontmatter, indexed one document per skill in their own `skills` collection (name + description + when_to_use; the body is fetched by `load_skill`). Built-ins ship under `knowledge/builtin/skills/`; company skills live in `company/skills/`. Customizing a built-in (a `PUT` from a workflow page's **How it's done**; chat never changes playbooks) writes a company skill of the same name flagged `customizes_builtin: true`, which **replaces** the built-in: it shadows the built-in everywhere (chat, and the workflows that follow it — see the Workflows section), and deleting it reverts. A built-in cannot be deleted, so \"delete\" **hides** it via `company/skills/.hidden.yaml`; hidden and shadowed built-ins are dropped from the index. `sync_builtin_skill_index` reconciles the built-in rows at startup, after a client-slot restore, and after a fixture reset — so a built-in added in a release reaches an already-seeded index, and a removed one leaves it.\n\n**Chat reads, people edit.** Chat has only `search_skills` and `load_skill`; it never creates or changes a playbook, and saves repeatable work as a workflow instead. A person edits a playbook as **How it's done** on the workflow page that follows it (`PUT /skills/{name}`, which on a built-in saves the customized copy; `DELETE` on a customized copy reverts it).",+ "markdown": "Retrieval is handled by `openexecutive.knowledge.retriever.retrieve`, querying ChromaDB collections through `openexecutive.knowledge.store.ChromaDBStore`. When the caller passes `record_source`, `retrieve` also names every document its block quotes — company uploads by filename, built-in knowledge by a title made from its filename, Notion pages, synced Drive files by name with their Drive link, synced Confluence pages by title with their page link, research notes by date, and earlier published documents with a link to `/artifacts/<id>` — so the web chat can list what an answer looked at: the chat route's own search, and each specialist's that answered (see the Lifecycle and API sections). The recording never raises, and a retrieval that fails leaves the specialist without context rather than failing the turn. Eight collections back the system:\n\n- **BUILTIN** — curated MBA-style domain knowledge shipped with the repo, including a `sales` domain (founder-led sales, qualification and pipeline, proposals and pricing conversations, follow-up and forecasting) and solo-founder guides in finance, marketing, legal and HR. Seeded at boot by `loader.seed_builtin_knowledge` (FAILURES by `seed_failures`): the whole tree on an empty collection, otherwise only the `SHIPPED_BUILTIN_FILES` entries with no chunks yet — so a doc added in a later release reaches an install seeded by an earlier one, instead of shipping, registering as approved, and never being retrievable. Presence is judged from each row's `source` path relative to `knowledge/builtin/` (not the absolute path, which moves with the install), read in one metadata scan narrowed to seeded rows so the external OER corpus sharing the collection is never pulled. A complete store indexes nothing.\n- **COMPANY** — customer documents ingested via the `/documents` endpoint. Curated uploads only — synced Notion pages, Drive files and Confluence pages do not land here. A document's identity is its **filename**: chunk metadata and the chunk id both derive from it, so re-uploading upserts in place rather than duplicating, and `DELETE /documents/{filename}` matches the rows it claims to remove. Inbound attachments are **not** here — see ATTACHMENTS below; rows left by the earlier scheme are migrated out on boot. A scanned PDF (no text layer) is converted before chunking — see \"Reading scanned PDFs\" below — so it no longer indexes zero chunks while reporting `indexed`.\n- **NOTION_WIKI** — optional Notion sync (`NOTION_SYNC_ENABLED`) incrementally re-indexes pages shared with a Notion internal integration into a separate `notion_wiki` collection (`type=notion`). A Notion share is inherited and multi-writer, so this text is unreviewed relative to curated company docs; the retriever labels it as such and ranks it below COMPANY. Un-shared pages are purged on the next tick.\nSynced Drive files and Notion pages are listed on the knowledge page next to uploads (one Company documents list, each row badged with its source, its sync time and a link to the original), read from the sync state files rather than Chroma; each connected source shows its last run, its last error (a failed tick, or a count of files or pages a tick couldn't index; the next clean tick clears it) and a **Sync now** button (`POST /documents/sources/{id}/sync`). Uploads from the page no longer ask for a domain: they go in as `general`, which every specialist's COMPANY filter already includes.\n\n- **DRIVE** — optional Google Drive folder sync (`DRIVE_SYNC_ENABLED`, `knowledge.drive_sync`). It incrementally re-indexes the files in `DRIVE_SYNC_FOLDER_IDS`, and their subfolders to four levels, into a separate `drive_docs` collection (`type=drive`). It reads the Drive v3 API directly as a service account with `drive.readonly` (`knowledge.drive_client`), not through the MCP gateway, so it sees only folders shared with that account and can change nothing. Google Docs and Slides are exported as text and Sheets as `.xlsx`. `.docx`, `.xlsx` and plain text files go through the loader's extractors. A PDF goes through `pdf_reader.read_pdf_text` the way an upload does (see *Reading scanned PDFs* below): a scan with no text layer is read by local OCR in a child process under the same OCR gate, or by the deployment's model when `PDF_PROVIDER_READING` is on, capped at `PDF_VISION_MAX_PAGES`, and `PDF_OCR_ENABLED=false` turns it off here too. Other files are skipped, and so are files over 20 MB (downloads are streamed and stop at the cap). A `.docx` or `.xlsx` that unzips past 100 MB is skipped, and so is a non-PDF file whose extraction runs past 120 s (a PDF read keeps the reader's own child limits under a 1200 s backstop); a scan OCR could not read, or met with OCR off, is unreadable too. Such files are recorded with no chunks and their reason, which the knowledge page shows (`unreadable_reason`), and are not refetched until they change; busy parsers are a retry next tick. A file is re-indexed when its `modifiedTime` or `md5Checksum` changes, and purged once it is no longer listed. Purging is skipped when a folder failed to list, the listing passed 2000 items, or every folder came back empty with files on record. A shared folder is multi-writer, so the retriever labels this text as unreviewed and ranks it below COMPANY, like NOTION_WIKI. Each chunk is labelled `[drive · file id … · synced … · \"<name>\"]`. The id and time come first because the sync wrote them. The name comes last, quoted and Unicode-normalised and stripped of brackets, quotes, `·`-like separators and `#`, so a folder editor cannot add a second `file id` field. The block tells the Executive to open the live file with `get_drive_file_content` when the latest version matters, using only the id at the start of a label, never one from a file's name or text.\n- **ONEDRIVE** — optional OneDrive / SharePoint folder sync (`ONEDRIVE_SYNC_ENABLED`, `knowledge.onedrive_sync`), DRIVE's twin: the files in `ONEDRIVE_SYNC_FOLDERS` (`<drive id>/<item id>` pairs) go into a separate `onedrive_docs` collection (`type=onedrive`, keyed `<drive id>:<item id>`). It calls Microsoft Graph directly (`knowledge.onedrive_client`, GETs only) as the Executive's own Microsoft 365 sign-in, with a token the Microsoft server's launcher mints for file reads only, because Microsoft has no per-folder service account; the folder list is the scope. PDF, Word, Excel and plain text files go through the same extractors, caps and deadlines as Drive (imported from `drive_sync`), but a PDF gets the text layer only, so a scanned one yields nothing; PowerPoint, older Office formats, OpenDocument and RTF are fetched as Graph's PDF rendering first. A file is re-indexed when its `lastModifiedDateTime` or `cTag` changes; purging, its skip rules and the locking are Drive's. Retrieved right after DRIVE, labelled `[onedrive · item <drive id>:<item id> · synced … · \"<name>\"]` with the name defanged the same way, and the block tells the Executive to open the live file with its OneDrive tools using only the ids after `item`.\n- **CONFLUENCE** — optional Confluence space sync (`CONFLUENCE_SYNC_ENABLED`, `knowledge.confluence_sync`), for Cloud and Server/Data Center alike. Each tick lists every page in `CONFLUENCE_SYNC_SPACE_KEYS` over the v1 REST API (`knowledge.confluence_client`, directly with `httpx`, not through the MCP gateway; a personal access token as Bearer, or a username and API token as Basic) and re-indexes the pages whose `version.number` changed into a separate `confluence_wiki` collection (`type=confluence`), at most `CONFLUENCE_MAX_PAGES_PER_SCAN` per tick, newest first. Storage-format XHTML becomes Markdown through stdlib `html.parser` (`knowledge.confluence_storage`): code macros become fenced blocks, callout macros keep their body, and dynamic macros such as Jira or a table of contents leave nothing. A Confluence token acts as the user who made it, so the space list is required, and by default (`CONFLUENCE_SYNC_SKIP_RESTRICTED`) a page with a read restriction of its own or on any ancestor page is left out. The listing carries restrictions and ancestors for every page, so this is re-checked each tick, and a page whose restrictions the response does not report counts as restricted. A page deleted, moved out of the spaces or newly restricted is purged. A page the listing returns as restricted is purged on every tick, whatever else failed. A page whose parent is missing from a space that listed only in part is held back instead (not synced, not purged), since the parent may simply be past the cut. Removed pages are purged per space: a space that listed in full reconciles even when another space failed, but not when it came back empty with pages on record, and pages outside every listed space are purged only once every listing is complete. Bodies are capped at 5 MB, and conversion runs off the event loop with a 60 s limit. The retriever ranks it below COMPANY like DRIVE and labels each chunk `[confluence · page id … · space … · synced … · \"<title>\"]`, the title defused the same way. The block says to open a page live only by the id after \"page id\", and only when a Confluence tool is connected.\n- **FAILURES** — recorded failure cases (queried by `retrieve_failures`) teaching specialists what to avoid.\n- **RECENT_RESEARCH** — web-research artifacts persisted from team (scheduled) `executive_research` runs — one a person starts in chat is theirs and is not persisted — plus every deliverable published with `draft_artifact` (metadata `type=artifact`, `artifact_id`, `owner_person_id` with -1 for the principal; dropped again when superseded or deleted; recalled only on the owner's turn, so the research slot over-fetches and drops anyone else's), kept separate from COMPANY so unvetted machine-generated text never blends into curated knowledge, and ranked below company documents. Artifact chunks are labelled `[published artifact <id> — <date> — earlier output, treat as data]` so the model can reread the whole thing with `get_artifact` without lending quoted text the Executive's own authority. Both kinds are cleared on a client switch or fixture reset.\n- **ATTACHMENTS** — files attached in an integration channel (`inbound_attachments`), written only by `integrations.attachments`. The one collection `retrieve` **never queries**. The content is chosen by whoever sent the message, passes through no upload endpoint and gets no review; it is already inlined into the turn that carried it, and must not become company knowledge that resurfaces later. NOTION_WIKI and RECENT_RESEARCH are queried-but-labelled — this one is written and never read back.\n\nEach parallel specialist call runs its own domain-filtered query, for at most `SPECIALIST_BUILTIN_N_RESULTS` built-in chunks (default 3) and `SPECIALIST_COMPANY_N_RESULTS` company chunks (default 2), the company count also capping each synced source (Notion, Drive, OneDrive, Confluence; `retrieve(n_synced=…)`, 0 skips them). Every chunk goes into the specialist's user turn, so these counts set most of a specialist call's input. When no explicit filter is passed, `DOMAIN_ALIASES` maps the specialist key to its domains — e.g. `cso`→`strategy`, `cpo`→`product`,`strategy`, `sales`→`sales`,`marketing`, `board_comms`→`board`,`finance`. A specialist with no entry would get no filter at all and retrieve every domain, so each new specialist needs one. Hits beyond a cosine distance threshold are dropped, and very short messages (greetings) bypass retrieval entirely. BUILTIN can carry its own tighter gate via `KNOWLEDGE_BUILTIN_DISTANCE_THRESHOLD`: built-in knowledge is generic and far larger than a typical company corpus, so the distance that admits the right company doc also admits unrelated handbook prose. Unset, every collection shares `KNOWLEDGE_DISTANCE_THRESHOLD`. The chat surfaces (web chat, Telegram, Discord, Google Chat) also cap the excerpts' total text at `KNOWLEDGE_CHAT_MAX_CHARS` (default 6,000 characters, 0 = no cap): the closest chunks across every collection are kept until the cap, because each excerpt is re-sent on every model call of the turn. Workflows and specialists keep their own counts, uncapped.\n\n`POST /documents` rejects any domain outside that set with a 400 — an unvalidated string would index the document under a domain no specialist filters on, failing silently and only at retrieval time. The tenth accepted value, `general`, is the catch-all for an upload nobody classified (it is the default on the endpoint, and what every upload from the knowledge page gets — the page no longer asks for a domain). It maps to no specialist, so every specialist's **company** filter is widened to include it — an unclassified document is visible to all rather than to none. That widening is deliberately not applied to BUILTIN, whose rows are shared with external OER sources that fall back to `general` when they declare no domains; fanning those into every specialist would blend unvetted third-party material into every answer.\n\nMulti-domain chunks fan out to one row per declared domain, so `_dedupe_by_text` drops duplicate-text hits while preserving relevance order before results are returned.\n\n### Embeddings\n\nEvery collection embeds with one shared all-MiniLM-L6-v2 ONNX session (`knowledge.store._embedding_function`), the same model and vectors as Chroma's default. Chroma's own default reloads the model on every call and keeps ONNX Runtime's memory arena, so a 300-chunk upload peaked near 760 MB and each query spent ~240 ms reloading. The shared session is built once, runs without the arena and embeds four texts at a time (the model pads every text to 256 tokens, so batch size sets the peak): ~300 MB and ~30 ms. It answers to Chroma's name `default` with an empty config, because every existing collection records that name and Chroma refuses a collection opened with a differently named function; and it must not be a `DefaultEmbeddingFunction` instance, which Chroma swaps for a fresh default on every embed.\n\n### Reading scanned PDFs\n\npypdf returns only a PDF's saved text layer, so a scan, a phone photo saved as PDF, or a deck printed to PDF as images used to come back empty everywhere a PDF is read. `knowledge.pdf_reader.read_pdf_text` tries three readers in order and never raises:\n\n1. **Text layer** (pypdf) — used whenever it yields real text (at least ~20 visible characters per page, so a scan carrying only page numbers still falls through).\n2. **The deployment's own model, through its provider's PDF support** — opt-in, `PDF_PROVIDER_READING=true` (off by default: an upgrade never starts sending scanned PDFs off the server; this also means an Anthropic-direct deployment OCRs scans locally until it opts in) — the pages as a base64 `document` block, transcribed to Markdown by `PDF_VISION_MODEL` (unset: `DEFAULT_MODEL`, the model the deployment already runs on), `PDF_VISION_PAGES_PER_CALL` pages per request (default 20), three requests at a time. Each provider carries the block its own way: Anthropic reads it natively; on OpenRouter the translator makes it an OpenAI `file` part and the provider adds the `file-parser` plugin — engine `native` for Claude or any model the catalog lists with `file` input (OpenAI's GPT-4o and later, Gemini…), else `PDF_OPENROUTER_ENGINE` (default `mistral-ocr`, $2 per 1,000 pages); a local OpenAI-compatible server gets the `file` part only with `LOCAL_PDF_INPUT=true` (e.g. `LOCAL_BASE_URL` pointed at OpenAI's own API). `providers.registry.pdf_input_supported` decides; a model that cannot take one never receives it (the feature gate swaps a stray `document` block for a note). Turning it on sends scanned pages off the server — to the provider, and on OpenRouter possibly to its parser (Mistral OCR) — and the destination is logged once per process.\n3. **Local OCR** — pages rendered with pypdfium2 and read by RapidOCR (ONNX, models bundled in the wheel): offline, no key, no per-page cost. The default reader while step 2 is off, what a local model without PDF input gets, and the fallback on every provider when step 2 errors, refuses or returns nothing. `PDF_OCR_ENABLED=false` turns it off.\n\n`PDF_VISION_MAX_PAGES` (default 100) caps the pages read, with a note saying how many were; a file a channel delivered (`inbound=True`) is capped at `PDF_INBOUND_MAX_PAGES` (30) and metered by a rolling `PDF_INBOUND_PAGES_PER_HOUR` (300) budget. OCR is bounded too: a page renders within a 25-megapixel budget whatever its MediaBox claims (a crafted 20000pt page would otherwise allocate gigabytes), at most two documents OCR at once, and each stops after 240 s with what it has. A PDF of more than 2,000 pages is refused before any page is parsed (the caps above bound conversion, not the text-layer pass). A Claude slice that runs into the output limit is split in half until each part fits; a single page that still does not fit keeps what was written, marked `[transcription cut off here]`, and the note says so. A result is cached in process by the file's sha256 (32 entries) so a re-read does not re-bill; a failure is not cached. `loader.read_document_text` / `extract_text_from_file_async` route a PDF through it and every other type through the sync extractor, and are what `ingest_file` (uploads, boot reconcile), the `/documents/{filename}` preview, intake uploads, chat attachments, the email poller, `oe__read_file` and the Executive's `read_document` tool use. The Drive sync calls `read_pdf_text` directly on the downloaded bytes, with the same settings as an upload.\n\n### Parsing in a child process\n\nParsing a large document builds hundreds of MB of small Python objects, and CPython keeps the memory they lived in, so a parse inside the API process left its peak there for good. On a 34 MB, 131-page text-layer PDF uploaded through `POST /documents`, uvicorn went from 272 MB to a 440 MB peak and stayed at 378 MB. On a 30-page scan read by OCR it peaked at 787 MB and stayed at 446 MB. Now the text-layer pass and local OCR in `pdf_reader`, and the PDF, Word and Excel extractors behind `loader.extract_text_from_file`, run in a short-lived child process (`knowledge.isolated.run_isolated`). Everything the child allocated goes back to the OS when it exits. The same uploads now peak at 324 MB and 297 MB and settle at 318 MB and 290 MB, and what remains is embedding. The OCR child itself still peaks near 600 MB while it reads, then exits.\n\nThe call goes to the child pickled on stdin and the answer comes back as JSON, so the API never unpickles anything a hostile file could shape. Both of the child's output streams go to temp files, not the API's memory. The child logs nothing, because pypdf logs text taken from the file and a crafted 9 KB PDF made it write megabytes. Only the last 2,000 characters of its stderr are read back, and an answer over 32 MB is refused. The child inherits only what it needs to start Python (`PATH`, `HOME`, `PYTHONPATH`, the library path, locale, temp dir), not the deployment's keys. That is hygiene rather than a sandbox, since the child runs as the same user. A PDF read by path is cached only if its hash is unchanged after the read, so an upload that replaces the file mid-read cannot store its text under the old file's hash. `read_document_text` hands `pdf_reader` the file's path, so the API process holds a whole PDF only when its pages go to the model (step 2). The contract is unchanged. `read_pdf_text` still never raises: a child that dies or runs past its limit (300 s for the text layer, or OCR's 240 s budget plus 120 s) comes back as `method=\"none\"` with a note. A Word or Excel parse error still raises, as `IsolatedError` naming the original. At most two parser processes run at once. A caller waits at most 30 seconds for a free slot, and that wait counts against its own time limit. If no slot frees up in time, the file is reported as \"busy, try again\" rather than unreadable. An upload, PDF or not, gets a 503 and nothing is stored. Drive sync retries the file on its next tick. A PDF read elsewhere comes back empty with a note saying so, and is not cached. OCR has its own two-slot limit, so a long OCR never holds up a quick text-layer parse. A scan waits up to one OCR run (240 s) for a slot, and if none frees up it gives its pages back to the hourly inbound budget. Only when there is no usable Python to start does the parse run in-process as before, and it does not take a slot. Failing to start a process for lack of memory is reported as unreadable, because that is the worst moment to parse inside the API. A Word or Excel file that cannot be read is a 422 from `POST /documents` and the preview, not a 500. An Excel workbook is read for at most 200,000 rows (blank ones included) and 1,024 columns: openpyxl walks every row a sheet's `<dimension>` claims, so a few-KB file claiming A1:XFD1048576 would otherwise hold a parser slot until the timeout. `POST /documents` accepts `.pdf`, `.docx`, `.doc`, `.xlsx`, `.xlsm`, `.csv`, `.md` and `.txt`; text formats are decoded as UTF-8 with bad bytes replaced.\n\n`POST /documents` streams the upload to its staging file 1 MB at a time and returns 413 as soon as it passes 50 MB, instead of reading the whole body into memory first. The stored copy is copied from that file.\n\n### The SME review gate\n\nBefore the distance threshold applies, both `retrieve` and `retrieve_failures` drop **withheld** items — anything the review store reports as `pending` or `rejected` (`ReviewStore.get_withheld_filenames`). Approved items with a priority set are then sorted ahead of the rest.\n\nKnowledge shipped with the product registers as a trusted default (`approved`, NULL `reviewed_at`), so nothing is withheld until a human deliberately queues a domain for curation or rejects something. See the **SME Knowledge Review** section for the full state model. The filter matches on the `(domain, filename)` pair Chroma carries. A bare filename was ambiguous *and* caller-choosable — `POST /knowledge/builtin` takes any `(domain, filename)`, so uploading a file named after a shipped doc in another domain silently withheld that shipped doc from every specialist. Failure case studies are keyed in their own namespace, so a user-authored one cannot collide with a shipped one.\n\nThe formatted context is injected into the **user turn** (wrapped in `<relevant_knowledge>` by `BaseAgent.analyze`), never into a cached system block — per-query hits would otherwise poison the prompt cache.\n\n### Skills (Playbooks) collection\n\nSkills — called **playbooks**, and shown in the UI as **How it's done** on the workflow that follows each one — are Markdown procedures with YAML frontmatter, indexed one document per skill in their own `skills` collection (name + description + when_to_use; the body is fetched by `load_skill`). Built-ins ship under `knowledge/builtin/skills/`; company skills live in `company/skills/`. Customizing a built-in (a `PUT` from a workflow page's **How it's done**; chat never changes playbooks) writes a company skill of the same name flagged `customizes_builtin: true`, which **replaces** the built-in: it shadows the built-in everywhere (chat, and the workflows that follow it — see the Workflows section), and deleting it reverts. A built-in cannot be deleted, so \"delete\" **hides** it via `company/skills/.hidden.yaml`; hidden and shadowed built-ins are dropped from the index. `sync_builtin_skill_index` reconciles the built-in rows at startup, after a client-slot restore, and after a fixture reset — so a built-in added in a release reaches an already-seeded index, and a removed one leaves it.\n\n**Chat reads, people edit.** Chat has only `search_skills` and `load_skill`; it never creates or changes a playbook, and saves repeatable work as a workflow instead. A person edits a playbook as **How it's done** on the workflow page that follows it (`PUT /skills/{name}`, which on a built-in saves the customized copy; `DELETE` on a customized copy reverts it).","mermaid": "flowchart TD\n Call[\"Specialist call<br/>(domain-filtered)\"] --> Q[\"retrieve(query, domain)\"]\n Q --> Builtin[(BUILTIN)]\n Q --> Company[(COMPANY)]\n Q --> NotionWiki[(NOTION_WIKI)]\n Q --> DriveDocs[(DRIVE)]\n Q --> ConfluenceWiki[(CONFLUENCE)]\n Q --> Failures[(FAILURES)]\n Q --> Research[(RECENT_RESEARCH)]\n Attach[\"integrations.attachments\"] --> Attachments[(ATTACHMENTS<br/>never queried)]\n Builtin --> Dedupe[\"dedupe by text<br/>+ distance gate\"]\n Company --> Dedupe\n NotionWiki --> Dedupe\n DriveDocs --> Dedupe\n ConfluenceWiki --> Dedupe\n Failures --> Dedupe\n Research --> Dedupe\n Dedupe --> Inject[\"Inject into USER turn<br/>(not cached)\"]\n class Call entry\n class Q compute\n class Dedupe compute\n class Builtin storage\n class Company storage\n class NotionWiki storage\n class DriveDocs storage\n class ConfluenceWiki storage\n class Failures storage\n class Research storage\n class Attachments storage\n class Attach compute\n class Inject hot",- "generated_at": "2026-10-11T16:38:14Z"+ "generated_at": "2026-10-13T00:00:00Z"}diff --git a/packages/core/openexecutive/knowledge/drive_sync.py b/packages/core/openexecutive/knowledge/drive_sync.pyindex dff9b699..0ab036f1 100644--- a/packages/core/openexecutive/knowledge/drive_sync.py+++ b/packages/core/openexecutive/knowledge/drive_sync.py@@ -45,6 +45,13 @@ from openexecutive.knowledge.drive_client import (from openexecutive.knowledge.isolated import ParserBusyfrom openexecutive.knowledge.loader import extract_text_from_file, ingest_text_syncfrom openexecutive.knowledge.notion_sync import infer_domain+from openexecutive.knowledge.pdf_reader import (+ _OCR_MAX_SLOT_WAIT_S,+ _OCR_PROCESS_SLACK_S,+ _OCR_TIME_BUDGET_S,+ _TEXT_LAYER_TIMEOUT_S,+ read_pdf_text,+)from openexecutive.knowledge.store import ChromaDBStorefrom openexecutive.memory.episodic import insert_scheduled_action@@ -64,6 +71,18 @@ _MAX_UNZIPPED_BYTES = 100 * 1024 * 1024# worker thread cannot be killed, but the tick moves on and the file is# recorded as unreadable, so it is not retried until it changes._EXTRACT_TIMEOUT_S = 120.0+# A PDF is read by ``pdf_reader.read_pdf_text``, the path uploads take, which+# kills its own children (text layer, then OCR) at their own limits. This is+# only a backstop over all of it, with room for the opt-in model step+# (PDF_PROVIDER_READING), so one stuck read cannot hold the tick forever.+_PDF_MODEL_ALLOWANCE_S = 300.0+_PDF_READ_TIMEOUT_S = (+ _TEXT_LAYER_TIMEOUT_S+ + _PDF_MODEL_ALLOWANCE_S+ + _OCR_MAX_SLOT_WAIT_S+ + _OCR_TIME_BUDGET_S+ + _OCR_PROCESS_SLACK_S+)_MAX_VISIBLE_ITEMS = 2000 # files + folders listed per tick, all folders together_MAX_DEPTH = 4 # subfolder levels below a configured folder_REQUEST_PAUSE_S = 0.2@@ -77,8 +96,9 @@ _EXPORTS: dict[str, tuple[str, str]] = {"application/vnd.google-apps.presentation": ("text/plain", ".txt"),"application/vnd.google-apps.spreadsheet": (_XLSX, ".xlsx"),}-# Stored files, downloaded and read with the knowledge loader's extractors-# (no model call: a scanned PDF with no text layer yields nothing).+# Stored files, downloaded and read with the knowledge loader's extractors.+# A PDF goes through ``pdf_reader`` as an upload does, so a scan with no text+# layer is read by local OCR (or the model, when PDF_PROVIDER_READING is on)._DOWNLOADS: dict[str, str] = {"application/pdf": ".pdf","application/vnd.openxmlformats-officedocument.wordprocessingml.document": ".docx",@@ -187,6 +207,8 @@ class _TickFetch:incomplete: str | None = None# (item, text) per file fetched; "" when it had no readable text.fetched: list[tuple[DriveItem, str]] = field(default_factory=list)+ # Why a fetched file has no text, by file id, for the knowledge page.+ unreadable: dict[str, str] = field(default_factory=dict)async def _list_tree(@@ -262,6 +284,26 @@ def _extract(data: bytes, suffix: str) -> str:return extract_text_from_file(path, timeout=_EXTRACT_TIMEOUT_S)+async def _pdf_text(item: DriveItem, data: bytes) -> str:+ """A PDF's text the way an upload reads it: the text layer, else OCR in a+ child process under the OCR gate, with the same PDF_* switches. Raises+ ``_Unreadable`` with the reader's reason when nothing could be read, and+ ``ParserBusy`` when every parser was busy (a retry, not a verdict)."""+ try:+ result = await asyncio.wait_for(+ read_pdf_text(data, filename=_safe_name(item.name)), timeout=_PDF_READ_TIMEOUT_S+ )+ except TimeoutError as exc:+ raise _Unreadable(f"reading the PDF took over {_PDF_READ_TIMEOUT_S:.0f}s") from exc+ if result.busy:+ raise ParserBusy(result.note)+ if not result.text.strip():+ raise _Unreadable(result.note or "no text could be read from the PDF")+ if result.note:+ logger.info("drive_sync: %s read by %s (%s)", item.id, result.method, result.note)+ return result.text++async def _file_text(client: DriveClient, item: DriveItem) -> str:export = _EXPORTS.get(item.mime_type)if export is not None:@@ -272,6 +314,8 @@ async def _file_text(client: DriveClient, item: DriveItem) -> str:if item.size is not None and item.size > _MAX_FILE_BYTES:raise DriveFileTooLarge(item.id)data = await client.download(item.id, max_bytes=_MAX_FILE_BYTES)+ if suffix == ".pdf":+ return await _pdf_text(item, data)try:return await asyncio.wait_for(asyncio.to_thread(_extract, data, suffix), timeout=_EXTRACT_TIMEOUT_S@@ -326,9 +370,11 @@ async def _fetch_tick(text = await _file_text(client, item)except DriveFileTooLarge:logger.info("drive_sync: %s is over %d bytes — not synced", item.id, _MAX_FILE_BYTES)+ fetch.unreadable[item.id] = f"larger than {_MAX_FILE_BYTES // (1024 * 1024)} MB"text = ""except _Unreadable as exc:logger.warning("drive_sync: %s is unreadable (%s) — not synced", item.id, exc)+ fetch.unreadable[item.id] = str(exc)text = ""except Exception:stats["failed"] += 1@@ -470,7 +516,7 @@ async def _apply_tick(logger.exception("drive_sync: failed to index file %s", item.id)continuename = _safe_name(item.name)- state["files"][item.id] = {+ record: dict[str, Any] = {"modified": item.modified,"md5": item.md5,"name": name,@@ -478,6 +524,10 @@ async def _apply_tick("url": item.link,"synced_at": synced_at,}+ reason = fetch.unreadable.get(item.id)+ if not chunks and reason:+ record["unreadable"] = reason[:300]+ state["files"][item.id] = recordstats["updated"] += 1logger.info("drive_sync: indexed %s (%d chunks)", item.id, chunks)@@ -638,7 +688,9 @@ def list_synced_files() -> dict[str, Any]:"""What the knowledge page shows for Drive: one row per file on record.Read from the sync state (not Chroma), so it costs one small JSON read.- ``indexed`` is false for a file with no readable text. Entries written+ ``indexed`` is false for a file with no readable text, and+ ``unreadable_reason`` says why when the sync knows (a scan OCR could not+ read, OCR turned off, a file over the size cap). Entries writtenbefore ``synced_at`` was recorded per file fall back to the tick time."""state = load_state()@@ -659,6 +711,9 @@ def list_synced_files() -> dict[str, Any]:if isinstance(rec.get("synced_at"), str)else last_run,"indexed": bool(filename),+ "unreadable_reason": rec.get("unreadable")+ if not filename and isinstance(rec.get("unreadable"), str)+ else None,})files.sort(key=lambda f: f["name"].lower())diff --git a/packages/core/tests/unit/test_document_sources_routes.py b/packages/core/tests/unit/test_document_sources_routes.pyindex c1f9a331..f694644e 100644--- a/packages/core/tests/unit/test_document_sources_routes.py+++ b/packages/core/tests/unit/test_document_sources_routes.py@@ -75,7 +75,11 @@ def _seed_drive(company: Path, **state: Any) -> None:"modified": "2026-09-29T08:00:00Z","synced_at": "2026-09-30T10:00:00+00:00",},- "1emptyAAAAAA": {"name": "Scan", "filename": ""},+ "1emptyAAAAAA": {+ "name": "Scan",+ "filename": "",+ "unreadable": "no text could be read: it looks scanned and local OCR is turned off",+ },},**state,}@@ -95,7 +99,10 @@ def test_drive_files_list_and_open(client: TestClient, company: Path) -> None:"modified_at": "2026-09-29T08:00:00Z","synced_at": "2026-09-30T10:00:00+00:00","indexed": True,+ "unreadable_reason": None,}+ scan = next(f for f in body["files"] if f["id"] == "1emptyAAAAAA")+ assert scan["indexed"] is False and "local OCR is turned off" in scan["unreadable_reason"]doc = client.get(f"/documents/drive/{DRIVE_ID}").json()assert doc["name"] == "Plan" and doc["content"].startswith("# Plan")assert client.get("/documents/drive/1emptyAAAAAA").status_code == 404diff --git a/packages/core/tests/unit/test_drive_sync.py b/packages/core/tests/unit/test_drive_sync.pyindex 9df4b8a0..4f91be76 100644--- a/packages/core/tests/unit/test_drive_sync.py+++ b/packages/core/tests/unit/test_drive_sync.py@@ -288,6 +288,7 @@ async def test_oversize_and_unreadable_files_are_recorded_without_chunks(tmp_patassert DOCX not in store.file_ids() and TXT not in store.file_ids()files = _state(tmp_path)["files"]assert files[DOCX]["filename"] == "" and files[TXT]["filename"] == ""+ assert files[DOCX]["unreadable"] == "larger than 20 MB" and "unreadable" not in files[TXT]drive.requests.clear()await _sync(drive, store) # unchanged: not fetched againassert not [r for r in drive.requests if r.url.params.get("alt")]@@ -666,3 +667,150 @@ async def test_a_file_that_fails_to_index_is_reported_and_cleared_once_it_syncs(stats = await _sync(_drive(), FakeStore())assert stats["failed"] == 0assert drive_sync.list_synced_files()["last_error"] is None+++# ---------------------------------------------------------------------------+# Scanned PDFs: read with OCR the way uploads are, or recorded with a reason+# ---------------------------------------------------------------------------++PDF = "6pdfFFFFFFFF"+PDF_MIME = "application/pdf"+++def _blank_pdf() -> bytes:+ """A PDF with no text layer, as a scan has."""+ from pypdf import PdfWriter++ writer = PdfWriter()+ writer.add_blank_page(width=612, height=792)+ buf = io.BytesIO()+ writer.write(buf)+ return buf.getvalue()+++def _drive_with_scan() -> FakeDrive:+ drive = _drive()+ drive.folders[ROOT].append(_item(PDF, "Signed contract", PDF_MIME, md5="ccc", size=900))+ drive.content[PDF] = _blank_pdf()+ return drive+++def _fresh_pdf_reader(monkeypatch: pytest.MonkeyPatch) -> Any:+ from openexecutive.knowledge import pdf_reader++ pdf_reader._cache.clear()+ yield pdf_reader+ pdf_reader._cache.clear()+++async def test_a_scanned_pdf_is_read_with_ocr_like_an_upload(+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, _fresh_pdf_reader: Any+) -> None:+ """The real pdf_reader path: the text layer is empty, so the pages go to+ local OCR (stubbed here: no model, no network)."""+ pdf_reader = _fresh_pdf_reader+ ocr_calls: list[int] = []++ def fake_ocr(source: Any, max_pages: int) -> tuple[str, int]:+ ocr_calls.append(max_pages)+ return "--- page 1 ---\nThe term is five years.", 1++ monkeypatch.setattr(pdf_reader, "_ocr_isolated", fake_ocr)+ # The extractor that skips scans must not be what reads a Drive PDF.+ monkeypatch.setattr(+ drive_sync, "extract_text_from_file",+ lambda p, **_: (_ for _ in ()).throw(AssertionError(p)) if p.suffix == ".pdf" else "x",+ )+ store = FakeStore()+ stats = await _sync(_drive_with_scan(), store)++ assert ocr_calls == [1] and stats["failed"] == 0+ assert PDF in store.file_ids()+ assert "five years" in " ".join(+ r["text"] for r in store.collections[DRIVE] if r["metadata"]["drive_file_id"] == PDF+ )+ row = {f["id"]: f for f in drive_sync.list_synced_files()["files"]}[PDF]+ assert row["indexed"] and row["unreadable_reason"] is None+++async def test_a_scan_with_ocr_turned_off_is_recorded_with_the_reason(+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, _fresh_pdf_reader: Any+) -> None:+ """PDF_OCR_ENABLED governs Drive as it does uploads, and the skip is not+ silent: the knowledge page gets the reason, and the file is not refetched+ every tick."""+ pdf_reader = _fresh_pdf_reader+ monkeypatch.setenv("PDF_OCR_ENABLED", "false")+ monkeypatch.setattr(+ pdf_reader, "_ocr_isolated", lambda *a: (_ for _ in ()).throw(AssertionError("OCR ran"))+ )+ drive, store = _drive_with_scan(), FakeStore()+ stats = await _sync(drive, store)++ assert stats["failed"] == 0 and PDF not in store.file_ids()+ assert "local OCR is turned off" in _state(tmp_path)["files"][PDF]["unreadable"]+ row = {f["id"]: f for f in drive_sync.list_synced_files()["files"]}[PDF]+ assert row["indexed"] is False and "local OCR is turned off" in row["unreadable_reason"]++ drive.requests.clear()+ await _sync(drive, store) # unchanged: not fetched again+ assert not [r for r in drive.requests if r.url.params.get("alt")]+++async def test_a_scan_ocr_reads_nothing_from_is_recorded_unreadable(+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, _fresh_pdf_reader: Any+) -> None:+ monkeypatch.setattr(_fresh_pdf_reader, "_ocr_isolated", lambda *a: (" ", 1))+ store = FakeStore()+ await _sync(_drive_with_scan(), store)+ assert PDF not in store.file_ids()+ assert "no text could be read from its pages" in _state(tmp_path)["files"][PDF]["unreadable"]+++async def test_a_pdf_met_by_busy_ocr_is_retried_not_recorded(+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch+) -> None:+ from openexecutive.knowledge.pdf_reader import PdfReadResult++ async def busy(data: Any, **kw: Any) -> PdfReadResult:+ return PdfReadResult("", "none", 1, "the PDF was not read: busy", busy=True)++ monkeypatch.setattr(drive_sync, "read_pdf_text", busy)+ store = FakeStore()+ stats = await _sync(_drive_with_scan(), store)+ assert stats["failed"] == 1+ assert PDF not in _state(tmp_path)["files"]+++async def test_a_pdf_read_is_read_as_an_upload_and_bounded_by_a_backstop(+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch+) -> None:+ """Not metered as an inbound attachment (uploads aren't), and a read that+ never returns is given up on and recorded, not left holding the tick."""+ import openexecutive.knowledge.pdf_reader as pdf_reader++ # The backstop sits above every limit the reader enforces itself.+ assert pytest.approx(+ pdf_reader._TEXT_LAYER_TIMEOUT_S + drive_sync._PDF_MODEL_ALLOWANCE_S+ + pdf_reader._OCR_MAX_SLOT_WAIT_S + pdf_reader._OCR_TIME_BUDGET_S+ + pdf_reader._OCR_PROCESS_SLACK_S+ ) == drive_sync._PDF_READ_TIMEOUT_S++ seen: list[dict[str, Any]] = []++ async def hang(data: Any, **kw: Any) -> Any:+ seen.append(kw)+ await asyncio.sleep(5)++ monkeypatch.setattr(drive_sync, "read_pdf_text", hang)+ monkeypatch.setattr(drive_sync, "_PDF_READ_TIMEOUT_S", 0.05)+ store = FakeStore()+ await _sync(_drive_with_scan(), store)+ assert seen and not seen[0].get("inbound")+ assert "took over" in _state(tmp_path)["files"][PDF]["unreadable"]diff --git a/packages/ui/scripts/companyDocs.test.mjs b/packages/ui/scripts/companyDocs.test.mjsindex 060e969c..3355d773 100644--- a/packages/ui/scripts/companyDocs.test.mjs+++ b/packages/ui/scripts/companyDocs.test.mjs@@ -57,6 +57,19 @@ test("rows without dates sort by name and unreadable files stay listed", () => {assert.equal(rows[1].indexed, false);});+test("an unreadable synced file carries the sync's reason; a readable one never does", () => {+ const rows = mergeDocs(+ [],+ [+ { ...synced("d1", "Scan", null, null, false), unreadable_reason: "OCR is turned off" },+ { ...synced("d2", "Notes", null), unreadable_reason: "stale" },+ ],+ [synced("n1", "Page", null, null, false)],+ );+ const by = Object.fromEntries(rows.map((r) => [r.name, r.unreadableReason]));+ assert.deepEqual(by, { Scan: "OCR is turned off", Notes: null, Page: null });+});+test("filter by source and by name", () => {const rows = mergeDocs([upload("Board deck.pdf", 1)], [synced("d1", "Budget", null)], []);assert.deepEqual(filterDocs(rows, "drive", "").map((r) => r.name), ["Budget"]);diff --git a/packages/ui/src/components/knowledge/CompanyPanel.tsx b/packages/ui/src/components/knowledge/CompanyPanel.tsxindex a4321f34..0f57ad28 100644--- a/packages/ui/src/components/knowledge/CompanyPanel.tsx+++ b/packages/ui/src/components/knowledge/CompanyPanel.tsx@@ -428,7 +428,8 @@ function DocListRow({if (row.addedAt) details.push(`Added ${new Date(row.addedAt).toLocaleDateString()}`);if (row.sizeBytes !== null) details.push(formatSize(row.sizeBytes));} else {- if (!row.indexed) details.push("No readable text");+ if (!row.indexed)+ details.push(row.unreadableReason ? `No readable text: ${row.unreadableReason}` : "No readable text");if (row.addedAt) details.push(`Synced ${formatRelativeTime(row.addedAt)}`);if (row.editedAt)details.push(diff --git a/packages/ui/src/lib/api.ts b/packages/ui/src/lib/api.tsindex 11c95208..8ee248a6 100644--- a/packages/ui/src/lib/api.ts+++ b/packages/ui/src/lib/api.ts@@ -583,6 +583,8 @@ export interface SyncedDoc {modified_at: string | null;synced_at: string | null;indexed: boolean;+ /** Why a file has no readable text, when the sync knows (Drive only). */+ unreadable_reason?: string | null;}export interface SyncedDocList {diff --git a/packages/ui/src/lib/companyDocs.ts b/packages/ui/src/lib/companyDocs.tsindex a1b3233e..25edd0c4 100644--- a/packages/ui/src/lib/companyDocs.ts+++ b/packages/ui/src/lib/companyDocs.ts@@ -24,6 +24,8 @@ export interface SyncedDocIn {synced_at: string | null;/** False when the file had no readable text, so nothing was indexed. */indexed: boolean;+ /** Why it had no readable text, when the sync knows. */+ unreadable_reason?: string | null;}export interface DocRow {@@ -39,6 +41,8 @@ export interface DocRow {editedAt: string | null;domain: string | null;indexed: boolean;+ /** Why a synced file has no readable text, when known. */+ unreadableReason: string | null;}export const SOURCE_LABELS: Record<DocSource, string> = {@@ -71,6 +75,7 @@ export function mergeDocs(editedAt: null,domain: shownDomain(d.domain),indexed: true,+ unreadableReason: null,})),...[["drive", drive] as const,@@ -88,6 +93,7 @@ export function mergeDocs(editedAt: f.modified_at,domain: null,indexed: f.indexed,+ unreadableReason: f.indexed ? null : (f.unreadable_reason ?? null),})),),];