Skip to content

AI-powered virtual executive team — a single coherent executive persona backed by 8 specialist agents (FastAPI + Next.js).

feat(chat): check a reply's claims against the tools it ran (#487)

Rufusdcde468d0e2e
diff --git a/.env.example b/.env.example
index 9cf065da..7291ed28 100644
--- a/.env.example
+++ b/.env.example
@@ -691,6 +691,9 @@ NUDGE_MAX_PER_SCOPE=3
# ATTUNEMENT_IGNORE_AFTER_HOURS=72
# ATTUNEMENT_MUTE_MIN_SENDS=5
# ATTUNEMENT_MUTE_COOLDOWN_MULTIPLIER=3
+# Do what it says: a reply that claims a change no tool made ("Done") gets one
+# more round to make it or say it can't. One small ROUTING_MODEL call per such reply.
+# CLAIM_CHECK_ENABLED=true
# Working style: up to 4 short "how they like replies" rules per person, learned
# from their own thumbs up/down and requests. Re-learned after TRIGGER_TURNS new
# messages (or right after a thumbs-down), at most once per MIN_INTERVAL_HOURS
diff --git a/packages/core/openexecutive/architecture/architecture-facts.yaml b/packages/core/openexecutive/architecture/architecture-facts.yaml
index b3c16f10..4f23cfc9 100644
--- a/packages/core/openexecutive/architecture/architecture-facts.yaml
+++ b/packages/core/openexecutive/architecture/architecture-facts.yaml
@@ -4923,6 +4923,17 @@ chat:
debounces it and opens a message match with /?session=…&find=…, and
the chat starts on that message with previous/next and highlights
the words (CSS Custom Highlight API where the browser has it).
+ claim_check: |
+ Do what it says (orchestrator/claim_check.py). At the end of the
+ agent loop, a reply that uses a word able to claim a change (done,
+ removed, closed, I'll remind you…) is checked by one forced-tool call
+ on ROUTING_MODEL against the turn's tool calls (name + first 160 chars
+ of each result). A claim no call made, copied word for word from the
+ reply (code-checked), gets one more round with a user-turn note: make
+ the change or say it isn't done. Once a turn, never on the last
+ iteration, fails open. Why: replies said "Done" or "I'll treat it as
+ closed" with no tool call, and the brief kept the item.
+ CLAIM_CHECK_ENABLED (default true).
module: openexecutive.api.routes.chat
endpoint: POST /chat — server-sent events stream of the live turn. POST /chat/stop halts one mid-flight. POST /chat/add hands it a message the person typed while it works. POST /chat/background starts a new chat the server answers to the end with no reader (202 + session_id; the caller request is wrapped so is_disconnected is always false, the stream is drained by a task kept in _background_tasks; at most 3 per owner key in _background_counts, claimed before the first await so a burst cannot pass together, else 429).
sse_event_types:
diff --git a/packages/core/openexecutive/architecture/prebuilt/lifecycle.json b/packages/core/openexecutive/architecture/prebuilt/lifecycle.json
index e053ed43..93b00896 100644
--- a/packages/core/openexecutive/architecture/prebuilt/lifecycle.json
+++ b/packages/core/openexecutive/architecture/prebuilt/lifecycle.json
@@ -1,7 +1,7 @@
{
"section_id": "lifecycle",
"title": "Request Lifecycle",
- "markdown": "A single chat message makes a full round-trip through the API, the orchestrator's tool-use loop, a parallel specialist fan-out, and synthesis — all streamed back as server-sent events.\n\n### Steps\n\n1. **Arrival** — the client mints a `client_turn_id` and posts to `POST /chat` (`openexecutive.api.routes.chat`). The turn carries a `<current_speaker>` hint resolved from the caller's `person_id` (followed by that person's `<working_style>` rules when they have any — see the attunement section), plus a `<briefing>` digest of current open alerts (built in parallel with RAG/episodic/peer-memory fetches) so the Executive can discuss any item shown on the `/today` briefing page. A turn arriving from a chat adapter instead of the web app also carries a `<channel>` block naming the surface and stating which approvals can be completed there; on Slack the `<briefing>` digest is gated to the principal's DMs. All three ride in the **user turn**, never a cached system block, as does a `<drive_memory>` block naming the Drive files this conversation already found or opened and the Drive searches that matched nothing (see the memory section). The server registers a stop switch under that id *before* the context fetch, so a Stop pressed during those first seconds halts the turn before any model call is made.\n2. **Tool-use loop** — the `Executive` calls Claude. For a cross-domain question the assistant turn emits multiple `consult_specialist` tool-use blocks at once. The call carries the conversation so far (`Session.get_recent_history`: 20 to 30 turns, its start moving in steps of 10) with a 1h cache marker on the previous reply, so each message reads the earlier conversation from the prompt cache and writes only what is new (see Caching); on the OpenRouter path the translator keeps that marker on the reply's text. The history is re-read from the stored transcript at the start of every web turn, so a chat continued in a chat app and then reopened on the web or phone includes the turns the app added.\n The person can keep typing while it works. A message sent then (`POST /chat/add`) joins the running turn: the loop adds it after the next round's tool results, so the answer follows it, and it is saved as its own message before the reply. One that arrives while the final answer is being written becomes the next turn instead. A conversation runs one turn at a time.\n3. **Parallel fan-out** — `openexecutive.orchestrator.router.route_parallel` dispatches those calls concurrently, bounded by `MAX_PARALLEL_SPECIALISTS`; calls past the cap get a skip tool_result. A specialist that raises (a timeout, an overloaded model) or returns no text no longer fails the batch: its tool_result says it is unavailable and tells the Executive not to invent its view, the others' results stand, and the turn goes on. It does not count as consulted, so it takes no committee seat and no department memory records the turn, and its documents are left out of the answer's sources. On the web chat the reply need not mention the gap, because the app shows it under the answer; on every other channel the tool_result asks the Executive to say in one sentence that part of the analysis is missing. A knowledge retrieval that fails likewise leaves that specialist without context rather than failing the turn.\n4. **Per-specialist RAG** — each specialist runs its own domain-filtered ChromaDB query, then reasons over the retrieved chunks and returns analysis as a tool_result. Its user turn also carries a `<company_stage>` tag from the session's company profile, so stage-sensitive advice (venture benchmarks vs. cash-first for a bootstrapped company) does not depend on the Executive remembering to mention it. Once the batch is done, the documents each answering specialist's retrieval returned are named, in call order, into the turn's `TurnSources` (`orchestrator/answer_sources.py`), which the web chat route owns and which also holds its own search's documents. Synced Google Drive files are recorded as `drive` sources with their Drive link, and the web chat lists them under **Google Drive**.\n5. **Synthesis** — the Executive feeds the tool_results back to Claude and streams one coherent reply. Every tool round — not just a specialist one — emits an `activity` event naming what is running, immediately followed by a `thinking` keepalive. The label names the *activity*, never the internal agent: \"Putting time on the calendar…\", \"Looking up people…\", or, for a connected (MCP) tool, plain words for what it does (\"Searching Drive…\"), from the same table (`orchestrator/tool_labels.py`) that names the finished chip (\"Searched Drive\"). The raw `server__tool` name never reaches the user. Under the reply, repeats of the same action collapse into one chip with a count, and tapping it lists what each run looked at. A round that really does fan out collapses to the generic \"Consulting specialists…\" — the one voice holds even in the progress line. The keepalive carries no tool information of its own; it used to drive a hardcoded \"Consulting specialists…\" string, so an MCP or calendar round misreported itself as a fan-out.\n6. **Optional committee review** — when enabled, `Executive.stream_chat_with_committee()` runs a silent draft, parallel reviewer critiques, then a streamed revision pass.\n7. **Side effects** — after the reply, every rostered speaker's turn (not only the principal's) schedules a fire-and-forget **open-loop pass** (Attunement) beside the episodic extraction: a keyword prefilter, then one forced-tool call that may open or close loops, every item quote-gated against the speaker's own words. The user message is persisted with `sender_person_id` — the resolved speaker, never the principal fallback a Discord / Telegram session uses for ownership — and the session records the speaker for the turn (`Session.caller_person_id`), which `close_open_loop` reads. A DM sent while proactive work is tagged (`attunement.outcomes.tag_proactive` — scheduler dispatch, reflection, research, alert review) also opens an outcome-ledger row at `_record_outbound_context`; a user's own turn is never tagged. A tagged DM is also subject to the recipient's own notification settings: in their quiet hours, or when they chose *In my briefs*, the send handler stores it (`outbound_guard.check_hold`, `people.held_messages`) and returns `{\"status\": \"held\"}` instead of sending (see Org). Any side-effecting tool call surfaces an `action_taken` chip, which unlike the ephemeral `activity` label is persisted with the assistant message; once the reply is over — also when it was stopped or timed out — the web chat route sends one `sources` event: the documents and web pages the turn looked at (its own search, each specialist that answered, web search results, and the pages the reply cites, which are always kept) and the areas (\"finance\", never a specialist) it had to go without. The web chat shows it under the reply, and it is persisted with the assistant message as `chat_messages.sources`; other channels don't collect it. The stream closes with a `done` event — preceded by a `stopped` event when the user halted the turn, in which case the partial reply is still persisted and flagged so it reads as cut short rather than complete, and is mirrored into the live in-memory session so the next turn's prompt matches what was saved. The stream still ends with `done` event. While nothing else is going out — a slow tool call, a long specialist round — the route sends an SSE comment (`: keepalive`) every `CHAT_STREAM_KEEPALIVE_S` (15 s by default, 0 for none) so a browser or proxy does not drop the connection as idle; the step it waits on keeps running across it, and the whole-turn deadline is unchanged. Every call through the MCP gateway (Gmail, Calendar, Drive, Outlook, OneDrive, tool search) has its own ceiling, `MCP_TOOL_TIMEOUT_S` (120 s): past it the model gets an error saying the tool did not answer and may still have run, so it checks before calling again, and no outbound linkage is recorded for a send that timed out. One side-effecting tool is gated server-side rather than by its description: `ack_alert` will only accept an `alert_id` from the live alert board the server itself derived this turn — the whole live set, not only the rows the `<briefing>` digest printed, since `/today` renders more cards than the digest lists and every one of them is Discuss-able — and it fails closed, so an id it cannot even parse is refused rather than waved through to the handler's own parse. That check runs on **every** session. It used to run only on turns that arrived from a chat channel, keyed on `origin_channel`, which the browser never sets — so the web, the surface the principal actually uses, was unchecked and the model could ack any id, including one quoted out of an alert body by the briefing page's Discuss handoff. What that buys is bounded and worth stating plainly: an id that is invented, closed, snoozed or expired is refused, including one quoted out of an attacker-written alert body; the model being argued into acking the wrong *live* card is not prevented. The one way to widen that set is `find_alerts`, for an item the principal names that has left the live board (already opened, snoozed, older than the board): a keyword search over alerts of every status whose own results — never an id the model supplies — join the trusted set. It runs only in a turn where `render_and_trust` showed the principal their own board (`Session.principal_board_shown`: the verified principal, and on a channel only in their DM), never merely because the session has a set — every `Session` starts with an empty one, and that empty set is how a Google Chat turn, someone else's DM or an unattended pass refuses every ack — and not on the principal's turn in a shared Slack or Discord thread either, where others read the reply and write into what the model reads. It trusts only rows still open (`unread`, `read`), never a roster-request card, and at most 25 ids per turn across calls, and refuses a query with no word of three or more letters. It also moves the admitted limit outward: a model argued into the wrong search can now reach up to 25 open alerts that are off the board, not only live ones. The board is recorded on the turn's `Session`, and the route binds that session for the **whole** SSE body (`schedule_tools.set_session`, alongside `set_turn`) rather than letting `stream_chat` bind it from inside its own generator. `_sse_body` drives the executive with `asyncio.wait_for(stream.__anext__())`, which runs each step in a fresh Task holding a *copy* of the context, so a binding made inside the generator is discarded with the step that made it: step one sees the session and every later step sees `None`. The tool-call loop runs on those later steps, so the whole guard read an empty set and refused every ack on the web — on live, open cards — until the binding moved out to the route. Anything reading `current_session` mid-turn has the same requirement, and two of them needed more than the binding to behave. `schedule_followup`'s seen-`channel_ref` anti-spam gate refuses a send to an address the session has not seen; with no session it skipped itself, so a web turn could schedule to **any** address, and with an empty one it refused even the principal's own. The route now seeds the caller's own addresses onto the session, so on web the gate means what it means on every other surface. Reply linkage (`_record_outbound_context`, read back by `inbound_hydration` to quote the originating conversation into the turn handling a recipient's reply) is the other: it was never reachable on web, and rather than switch that on for the principal's broadest surface as a side effect it is held back, keyed on an explicit `Session.from_web_chat` flag. Not on an empty `origin_channel` — alert review, the CLI and the unattended workflows all leave it empty while legitimately recording linkage, as does the email poller under its own `email` tag. `scheduled_actions` rows also take their `session_id` from the session. A session that was shown no board has an empty trusted set and can ack nothing, so the unattended passes (the nightly reflection, the research synthesis) are refused by default — they also still withhold the tool outright, which is belt-and-braces rather than the only thing stopping them. After the stream closes, the turn is also handed to the episodic memory extractor (`memory.episodic.schedule_extraction`, fire-and-forget) — on every turn where the principal said anything at all, and only the principal: a turn naming an `origin_channel` must resolve to a person marked `is_principal`, because the quote validator downstream cannot tell whose words it is checking. Both this path and the committee path call the same `should_extract` predicate, and it applies no length floor: the gate used to sum in the assistant's reply, which let a verbose answer drag a trivial turn in and discarded the short, decisive instructions that are the whole point of extracting. A user-side floor would fare no better — \"Do B.\" and \"Done\" are one character apart. See **Memory** for what that cost on a live tenant.\n8. **Tool failures** — a handler that raises instead of returning its JSON error is isolated: both tool gathers run with `return_exceptions=True`, the model gets an error `tool_result` naming only the exception's TYPE — a message can carry a filesystem path, a validation error echoing the input, or a third-party HTTP body, and anything in model context can be quoted back to the user, so the full repr stays in the log and the audit row — and the failure is audited with `ok=false`. Only `CancelledError` is re-raised, so turn-timeout and client-disconnect handling still work. Before this one crashing tool aborted the whole turn, which reached the user as an unexplained apology with no audit row (#136).\n\nWeb search, when invoked on the Anthropic path, executes inside the Anthropic API itself (`web_search_20250305`) and returns within the same generation; for any model routed via OpenRouter the same tool becomes OpenRouter's `openrouter:web_search` server tool, which also runs inside the generation with the same search cap and domain lists.\n\nThe install's workspace mode (solo or team, see Org) is read fresh at the start of each turn by `memory.workspace_settings.pin_turn_workspace_mode(session)` and pinned on the session for that turn only (`Session.turn_workspace_mode`): the persona, the org block, the tool list and every tool handler in the turn then agree, and a switch made mid-turn applies from the next turn. A session can instead carry an explicit override, `Session.workspace_mode` (`None` by default), which wins over both — so a caller that runs many conversations concurrently on one Executive, such as evals, can pick the mode per session rather than flipping the install-wide setting. Everything reads it through `effective_workspace_mode(session)` (override, then the turn's pin, then the setting). The principal's role is pinned the same way (`pin_turn_principal_role(session, mode)` → `Session.turn_principal_role`: in solo the `Session.principal_role` override, else the workspace settings read fresh; in team an empty role). The org block renders it, each specialist batch gets it as a `<principal_role>` user-turn tag, and a workflow the turn starts reads the same pin through `effective_principal_role` (override, then the pin, then the setting), so a role edited mid-turn applies from the next turn; a team turn sends none.\n\nAct as me is pinned beside them: `delegation.settings.pin_turn_delegation(session, speaker_text)` records on `Session.turn_delegation` whether the speaker has it on, whether `ghostwrite_email` is offered this turn (the principal, or a team member once the owner lets them, on a verified surface, in a conversation private to them, attended), and the speaker's own words. The pin is also held in a context variable for the turn's task, so two turns running at once on one session each read their own. The tool list, the tool handler and the turn's audit privacy all read that one pin, so a switch flipped mid-turn applies from the next turn. Once the tool touches the speaker's mailbox, the conversation is the speaker's alone (`session_store.mark_mail_private`, the principal included when it isn't theirs), the rest of the turn's audit rows are private (a team member's theirs alone) and the turn teaches no memory: no extraction, open-loop, working-style or Honcho pass runs on it. Later turns of that conversation stay private the same way; the pin also records whether the reading turn is still in the history the model is shown (`mail_in_view`, from `sessions.mail_read_at` against `session.history_window_start`), and links, scripts, workflows and posts to everyone stay withheld only while it is.",
+ "markdown": "A single chat message makes a full round-trip through the API, the orchestrator's tool-use loop, a parallel specialist fan-out, and synthesis — all streamed back as server-sent events.\n\n### Steps\n\n1. **Arrival** — the client mints a `client_turn_id` and posts to `POST /chat` (`openexecutive.api.routes.chat`). The turn carries a `<current_speaker>` hint resolved from the caller's `person_id` (followed by that person's `<working_style>` rules when they have any — see the attunement section), plus a `<briefing>` digest of current open alerts (built in parallel with RAG/episodic/peer-memory fetches) so the Executive can discuss any item shown on the `/today` briefing page. A turn arriving from a chat adapter instead of the web app also carries a `<channel>` block naming the surface and stating which approvals can be completed there; on Slack the `<briefing>` digest is gated to the principal's DMs. All three ride in the **user turn**, never a cached system block, as does a `<drive_memory>` block naming the Drive files this conversation already found or opened and the Drive searches that matched nothing (see the memory section). The server registers a stop switch under that id *before* the context fetch, so a Stop pressed during those first seconds halts the turn before any model call is made.\n2. **Tool-use loop** — the `Executive` calls Claude. For a cross-domain question the assistant turn emits multiple `consult_specialist` tool-use blocks at once. The call carries the conversation so far (`Session.get_recent_history`: 20 to 30 turns, its start moving in steps of 10) with a 1h cache marker on the previous reply, so each message reads the earlier conversation from the prompt cache and writes only what is new (see Caching); on the OpenRouter path the translator keeps that marker on the reply's text. The history is re-read from the stored transcript at the start of every web turn, so a chat continued in a chat app and then reopened on the web or phone includes the turns the app added.\n The person can keep typing while it works. A message sent then (`POST /chat/add`) joins the running turn: the loop adds it after the next round's tool results, so the answer follows it, and it is saved as its own message before the reply. One that arrives while the final answer is being written becomes the next turn instead. A conversation runs one turn at a time.\n3. **Parallel fan-out** — `openexecutive.orchestrator.router.route_parallel` dispatches those calls concurrently, bounded by `MAX_PARALLEL_SPECIALISTS`; calls past the cap get a skip tool_result. A specialist that raises (a timeout, an overloaded model) or returns no text no longer fails the batch: its tool_result says it is unavailable and tells the Executive not to invent its view, the others' results stand, and the turn goes on. It does not count as consulted, so it takes no committee seat and no department memory records the turn, and its documents are left out of the answer's sources. On the web chat the reply need not mention the gap, because the app shows it under the answer; on every other channel the tool_result asks the Executive to say in one sentence that part of the analysis is missing. A knowledge retrieval that fails likewise leaves that specialist without context rather than failing the turn.\n4. **Per-specialist RAG** — each specialist runs its own domain-filtered ChromaDB query, then reasons over the retrieved chunks and returns analysis as a tool_result. Its user turn also carries a `<company_stage>` tag from the session's company profile, so stage-sensitive advice (venture benchmarks vs. cash-first for a bootstrapped company) does not depend on the Executive remembering to mention it. Once the batch is done, the documents each answering specialist's retrieval returned are named, in call order, into the turn's `TurnSources` (`orchestrator/answer_sources.py`), which the web chat route owns and which also holds its own search's documents. Synced Google Drive files are recorded as `drive` sources with their Drive link, and the web chat lists them under **Google Drive**.\n5. **Synthesis** — the Executive feeds the tool_results back to Claude and streams one coherent reply. Every tool round — not just a specialist one — emits an `activity` event naming what is running, immediately followed by a `thinking` keepalive. The label names the *activity*, never the internal agent: \"Putting time on the calendar…\", \"Looking up people…\", or, for a connected (MCP) tool, plain words for what it does (\"Searching Drive…\"), from the same table (`orchestrator/tool_labels.py`) that names the finished chip (\"Searched Drive\"). The raw `server__tool` name never reaches the user. Under the reply, repeats of the same action collapse into one chip with a count, and tapping it lists what each run looked at. A round that really does fan out collapses to the generic \"Consulting specialists…\" — the one voice holds even in the progress line. The keepalive carries no tool information of its own; it used to drive a hardcoded \"Consulting specialists…\" string, so an MCP or calendar round misreported itself as a fan-out.\n **Do what it says.** Before the turn ends, its last reply is checked against the tool calls the turn made (`orchestrator/claim_check.py`). A reply with no word that can claim a change (\"done\", \"removed\", \"I'll remind you\"…) ends the turn as before. Otherwise one small forced-tool call on `ROUTING_MODEL` reads the person's message, the reply and each call with the start of its result, and reports a claim of a change no call made (or put up for approval). The claim must appear word for word in the reply, or it is ignored. The loop then gets one more round with a note in the user turn (never a cached block): make the change with the right tool, or say plainly it isn't done. Once a turn; a failed check lets the reply through. On the web chat the extra round streams after the first reply, separated by a blank line; chat apps send the whole reply at once. `CLAIM_CHECK_ENABLED` turns it off.\n6. **Optional committee review** — when enabled, `Executive.stream_chat_with_committee()` runs a silent draft, parallel reviewer critiques, then a streamed revision pass.\n7. **Side effects** — after the reply, every rostered speaker's turn (not only the principal's) schedules a fire-and-forget **open-loop pass** (Attunement) beside the episodic extraction: a keyword prefilter, then one forced-tool call that may open or close loops, every item quote-gated against the speaker's own words. The user message is persisted with `sender_person_id` — the resolved speaker, never the principal fallback a Discord / Telegram session uses for ownership — and the session records the speaker for the turn (`Session.caller_person_id`), which `close_open_loop` reads. A DM sent while proactive work is tagged (`attunement.outcomes.tag_proactive` — scheduler dispatch, reflection, research, alert review) also opens an outcome-ledger row at `_record_outbound_context`; a user's own turn is never tagged. A tagged DM is also subject to the recipient's own notification settings: in their quiet hours, or when they chose *In my briefs*, the send handler stores it (`outbound_guard.check_hold`, `people.held_messages`) and returns `{\"status\": \"held\"}` instead of sending (see Org). Any side-effecting tool call surfaces an `action_taken` chip, which unlike the ephemeral `activity` label is persisted with the assistant message; once the reply is over — also when it was stopped or timed out — the web chat route sends one `sources` event: the documents and web pages the turn looked at (its own search, each specialist that answered, web search results, and the pages the reply cites, which are always kept) and the areas (\"finance\", never a specialist) it had to go without. The web chat shows it under the reply, and it is persisted with the assistant message as `chat_messages.sources`; other channels don't collect it. The stream closes with a `done` event — preceded by a `stopped` event when the user halted the turn, in which case the partial reply is still persisted and flagged so it reads as cut short rather than complete, and is mirrored into the live in-memory session so the next turn's prompt matches what was saved. The stream still ends with `done` event. While nothing else is going out — a slow tool call, a long specialist round — the route sends an SSE comment (`: keepalive`) every `CHAT_STREAM_KEEPALIVE_S` (15 s by default, 0 for none) so a browser or proxy does not drop the connection as idle; the step it waits on keeps running across it, and the whole-turn deadline is unchanged. Every call through the MCP gateway (Gmail, Calendar, Drive, Outlook, OneDrive, tool search) has its own ceiling, `MCP_TOOL_TIMEOUT_S` (120 s): past it the model gets an error saying the tool did not answer and may still have run, so it checks before calling again, and no outbound linkage is recorded for a send that timed out. One side-effecting tool is gated server-side rather than by its description: `ack_alert` will only accept an `alert_id` from the live alert board the server itself derived this turn — the whole live set, not only the rows the `<briefing>` digest printed, since `/today` renders more cards than the digest lists and every one of them is Discuss-able — and it fails closed, so an id it cannot even parse is refused rather than waved through to the handler's own parse. That check runs on **every** session. It used to run only on turns that arrived from a chat channel, keyed on `origin_channel`, which the browser never sets — so the web, the surface the principal actually uses, was unchecked and the model could ack any id, including one quoted out of an alert body by the briefing page's Discuss handoff. What that buys is bounded and worth stating plainly: an id that is invented, closed, snoozed or expired is refused, including one quoted out of an attacker-written alert body; the model being argued into acking the wrong *live* card is not prevented. The one way to widen that set is `find_alerts`, for an item the principal names that has left the live board (already opened, snoozed, older than the board): a keyword search over alerts of every status whose own results — never an id the model supplies — join the trusted set. It runs only in a turn where `render_and_trust` showed the principal their own board (`Session.principal_board_shown`: the verified principal, and on a channel only in their DM), never merely because the session has a set — every `Session` starts with an empty one, and that empty set is how a Google Chat turn, someone else's DM or an unattended pass refuses every ack — and not on the principal's turn in a shared Slack or Discord thread either, where others read the reply and write into what the model reads. It trusts only rows still open (`unread`, `read`), never a roster-request card, and at most 25 ids per turn across calls, and refuses a query with no word of three or more letters. It also moves the admitted limit outward: a model argued into the wrong search can now reach up to 25 open alerts that are off the board, not only live ones. The board is recorded on the turn's `Session`, and the route binds that session for the **whole** SSE body (`schedule_tools.set_session`, alongside `set_turn`) rather than letting `stream_chat` bind it from inside its own generator. `_sse_body` drives the executive with `asyncio.wait_for(stream.__anext__())`, which runs each step in a fresh Task holding a *copy* of the context, so a binding made inside the generator is discarded with the step that made it: step one sees the session and every later step sees `None`. The tool-call loop runs on those later steps, so the whole guard read an empty set and refused every ack on the web — on live, open cards — until the binding moved out to the route. Anything reading `current_session` mid-turn has the same requirement, and two of them needed more than the binding to behave. `schedule_followup`'s seen-`channel_ref` anti-spam gate refuses a send to an address the session has not seen; with no session it skipped itself, so a web turn could schedule to **any** address, and with an empty one it refused even the principal's own. The route now seeds the caller's own addresses onto the session, so on web the gate means what it means on every other surface. Reply linkage (`_record_outbound_context`, read back by `inbound_hydration` to quote the originating conversation into the turn handling a recipient's reply) is the other: it was never reachable on web, and rather than switch that on for the principal's broadest surface as a side effect it is held back, keyed on an explicit `Session.from_web_chat` flag. Not on an empty `origin_channel` — alert review, the CLI and the unattended workflows all leave it empty while legitimately recording linkage, as does the email poller under its own `email` tag. `scheduled_actions` rows also take their `session_id` from the session. A session that was shown no board has an empty trusted set and can ack nothing, so the unattended passes (the nightly reflection, the research synthesis) are refused by default — they also still withhold the tool outright, which is belt-and-braces rather than the only thing stopping them. After the stream closes, the turn is also handed to the episodic memory extractor (`memory.episodic.schedule_extraction`, fire-and-forget) — on every turn where the principal said anything at all, and only the principal: a turn naming an `origin_channel` must resolve to a person marked `is_principal`, because the quote validator downstream cannot tell whose words it is checking. Both this path and the committee path call the same `should_extract` predicate, and it applies no length floor: the gate used to sum in the assistant's reply, which let a verbose answer drag a trivial turn in and discarded the short, decisive instructions that are the whole point of extracting. A user-side floor would fare no better — \"Do B.\" and \"Done\" are one character apart. See **Memory** for what that cost on a live tenant.\n8. **Tool failures** — a handler that raises instead of returning its JSON error is isolated: both tool gathers run with `return_exceptions=True`, the model gets an error `tool_result` naming only the exception's TYPE — a message can carry a filesystem path, a validation error echoing the input, or a third-party HTTP body, and anything in model context can be quoted back to the user, so the full repr stays in the log and the audit row — and the failure is audited with `ok=false`. Only `CancelledError` is re-raised, so turn-timeout and client-disconnect handling still work. Before this one crashing tool aborted the whole turn, which reached the user as an unexplained apology with no audit row (#136).\n\nWeb search, when invoked on the Anthropic path, executes inside the Anthropic API itself (`web_search_20250305`) and returns within the same generation; for any model routed via OpenRouter the same tool becomes OpenRouter's `openrouter:web_search` server tool, which also runs inside the generation with the same search cap and domain lists.\n\nThe install's workspace mode (solo or team, see Org) is read fresh at the start of each turn by `memory.workspace_settings.pin_turn_workspace_mode(session)` and pinned on the session for that turn only (`Session.turn_workspace_mode`): the persona, the org block, the tool list and every tool handler in the turn then agree, and a switch made mid-turn applies from the next turn. A session can instead carry an explicit override, `Session.workspace_mode` (`None` by default), which wins over both — so a caller that runs many conversations concurrently on one Executive, such as evals, can pick the mode per session rather than flipping the install-wide setting. Everything reads it through `effective_workspace_mode(session)` (override, then the turn's pin, then the setting). The principal's role is pinned the same way (`pin_turn_principal_role(session, mode)` → `Session.turn_principal_role`: in solo the `Session.principal_role` override, else the workspace settings read fresh; in team an empty role). The org block renders it, each specialist batch gets it as a `<principal_role>` user-turn tag, and a workflow the turn starts reads the same pin through `effective_principal_role` (override, then the pin, then the setting), so a role edited mid-turn applies from the next turn; a team turn sends none.\n\nAct as me is pinned beside them: `delegation.settings.pin_turn_delegation(session, speaker_text)` records on `Session.turn_delegation` whether the speaker has it on, whether `ghostwrite_email` is offered this turn (the principal, or a team member once the owner lets them, on a verified surface, in a conversation private to them, attended), and the speaker's own words. The pin is also held in a context variable for the turn's task, so two turns running at once on one session each read their own. The tool list, the tool handler and the turn's audit privacy all read that one pin, so a switch flipped mid-turn applies from the next turn. Once the tool touches the speaker's mailbox, the conversation is the speaker's alone (`session_store.mark_mail_private`, the principal included when it isn't theirs), the rest of the turn's audit rows are private (a team member's theirs alone) and the turn teaches no memory: no extraction, open-loop, working-style or Honcho pass runs on it. Later turns of that conversation stay private the same way; the pin also records whether the reading turn is still in the history the model is shown (`mail_in_view`, from `sessions.mail_read_at` against `session.history_window_start`), and links, scripts, workflows and posts to everyone stay withheld only while it is.",
"mermaid": "sequenceDiagram\n participant C as Client\n participant API as FastAPI /chat\n participant E as Executive\n participant CL as Claude\n participant S as Specialists\n participant KB as ChromaDB\n C->>API: POST /chat message\n API->>E: start turn\n E->>CL: messages (tool-use loop)\n CL-->>E: consult_specialist calls\n E->>S: route_parallel fan-out\n S->>KB: domain-filtered RAG\n KB-->>S: chunks\n S-->>E: analysis tool_results\n E->>CL: synthesize\n CL-->>E: streamed reply\n E-->>API: SSE chunks then done\n API-->>C: streamed response",
- "generated_at": "2026-10-14T06:00:00Z"
+ "generated_at": "2026-10-15T15:46:12.213383+00:00"
}
diff --git a/packages/core/openexecutive/config.py b/packages/core/openexecutive/config.py
index 4f0369aa..ed6732d2 100644
--- a/packages/core/openexecutive/config.py
+++ b/packages/core/openexecutive/config.py
@@ -1101,6 +1101,11 @@ class Settings(BaseSettings):
# loop the nudge engine's commitment source chases once it is due, and a
# later "sent it" from the same person closes it. Rows live in
# scheduled_actions (kind="open_loop"); see attunement/open_loops.py.
+ # Do what it says (orchestrator/claim_check.py): a reply that claims a
+ # change no tool call made gets one more round to make it or say it
+ # can't. One small ROUTING_MODEL call, only when the reply uses a word
+ # that can claim a change.
+ claim_check_enabled: bool = Field(True, alias="CLAIM_CHECK_ENABLED")
attunement_enabled: bool = Field(True, alias="ATTUNEMENT_ENABLED")
attunement_open_loops_enabled: bool = Field(True, alias="ATTUNEMENT_OPEN_LOOPS_ENABLED")
# When no due date is stated, the loop is due this many days after it opens.
diff --git a/packages/core/openexecutive/orchestrator/claim_check.py b/packages/core/openexecutive/orchestrator/claim_check.py
new file mode 100644
index 00000000..d9309771
--- /dev/null
+++ b/packages/core/openexecutive/orchestrator/claim_check.py
@@ -0,0 +1,212 @@
+"""Do what it says: a reply may not claim a change this turn didn't make.
+
+The Executive sometimes answers "Done", "Removed it" or "I'll treat it as
+closed" without calling the tool that would make it so, and the person only
+finds out later, when the brief still lists the thing. So before a turn ends,
+its last reply is checked against the tool calls the turn made:
+
+1. A cheap word check (``CLAIM_RE``): a reply that claims no change ("Here are
+ the numbers…") ends the turn as before, with no model call.
+2. Otherwise one small forced-tool call on ``ROUTING_MODEL`` reads the
+ person's message, the reply and the turn's tool calls (name and the start
+ of each result) and says whether the reply claims a change this turn that
+ none of those calls made. A claim about an earlier turn, a plan for later
+ that the reply doesn't present as set up, or a change a call did make (or
+ put up for approval) is not one.
+3. When it finds one, the claim must appear word for word in the reply (code
+ check), or it is ignored. The loop then gets one more round with
+ ``nudge(claim)``: do it now with the right tool, or tell the person
+ plainly it couldn't be done. Once a turn, so a loop can't form.
+
+Never raises and never blocks a reply: a failure, a timeout or a missing
+setting ends the turn as before. ``CLAIM_CHECK_ENABLED`` turns it off.
+"""
+from __future__ import annotations
+
+import asyncio
+import logging
+import re
+from typing import Any
+
+logger = logging.getLogger(__name__)
+
+# Words a reply uses to say something changed. Lexical on purpose: it only
+# decides whether the model call runs, so a miss costs a check, not a reply.
+CLAIM_RE = re.compile(
+ r"\b(?:done|did it|removed|deleted|dropped|closed|cancel+ed|marked|"
+ r"updated|changed|saved|noted|added|set (?:up|it|a)|scheduled|booked|"
+ r"sent|moved|archived|cleared|fixed|taken care of|handled|treat(?:ed|ing)? "
+ r"(?:it|this|that|the)|i'?ve|i have|i'?ll remind|will remind|reminder (?:is )?set|"
+ r"won'?t (?:show|appear|include)|no longer|from now on|going forward)\b",
+ re.IGNORECASE,
+)
+
+MAX_REPLY_CHARS = 3000
+MAX_MESSAGE_CHARS = 2000
+MAX_CALLS = 20
+CALL_RESULT_CHARS = 160
+CLAIM_MAX_CHARS = 200
+_MAX_TOKENS = 300
+_TIMEOUT_S = 20.0
+
+_SYSTEM = (
+ "You check an assistant's reply against what it actually did. You get the "
+ "person's message, the assistant's reply, and every tool call the "
+ "assistant made while writing that reply, with the start of each result.\n\n"
+ "Report an unbacked claim when the reply says or implies that it made a "
+ "change during this reply (removed, closed, saved, scheduled, sent, "
+ "updated, cancelled, set a reminder, will stop doing something, will "
+ "treat something as done) and no tool call in the list made that change "
+ "or put it up for approval. A tool call that failed or was refused does "
+ "not back a claim.\n\n"
+ "Not unbacked: a change one of the calls made; something it says was "
+ "done earlier or by someone else; an offer or a question ('want me to "
+ "remove it?'); advice; a plain answer to a question; saying it can't do "
+ "something; a promise about how it will write or talk (shorter, more "
+ "formal), which needs no tool.\n\n"
+ "When there is one, copy the claim word for word from the reply (one "
+ "short phrase). Record your verdict with the tool."
+)
+
+_TOOL: dict[str, Any] = {
+ "name": "record_claim_check",
+ "description": "Record whether the reply claims a change no tool call made.",
+ "input_schema": {
+ "type": "object",
+ "properties": {
+ "unbacked": {
+ "type": "boolean",
+ "description": "True when the reply claims a change this turn that no listed call made.",
+ },
+ "claim": {
+ "type": "string",
+ "description": "The claim, copied word for word from the reply. Empty when unbacked is false.",
+ },
+ },
+ "required": ["unbacked", "claim"],
+ },
+}
+
+
+def enabled() -> bool:
+ try:
+ from openexecutive.config import get_settings
+
+ return bool(get_settings().claim_check_enabled)
+ except Exception:
+ return False
+
+
+def may_claim(reply: str) -> bool:
+ """Whether the reply uses a word that can claim a change."""
+ return bool(reply.strip()) and CLAIM_RE.search(reply) is not None
+
+
+def render_calls(calls: list[tuple[str, str]]) -> str:
+ if not calls:
+ return "(no tool calls)"
+ lines = []
+ for name, result in calls[-MAX_CALLS:]:
+ preview = " ".join(str(result or "").split())[:CALL_RESULT_CHARS]
+ lines.append(f"- {name}: {preview}")
+ return "\n".join(lines)
+
+
+def _render_turn(message: str, reply: str, calls: list[tuple[str, str]]) -> str:
+ return (
+ f"<person_message>\n{message[-MAX_MESSAGE_CHARS:]}\n</person_message>\n\n"
+ f"<assistant_reply>\n{reply[:MAX_REPLY_CHARS]}\n</assistant_reply>\n\n"
+ f"<tool_calls>\n{render_calls(calls)}\n</tool_calls>"
+ )
+
+
+def _normalize(text: str) -> str:
+ return " ".join(text.replace("’", "'").split()).lower()
+
+
+def accept_claim(payload: dict[str, Any], reply: str) -> str | None:
+ """The model's claim when it reported one and it is in the reply word
+ for word (whitespace, case and curly quotes aside); None otherwise."""
+ if payload.get("unbacked") is not True:
+ return None
+ claim = " ".join(str(payload.get("claim") or "").split())
+ if not claim or len(claim) > CLAIM_MAX_CHARS:
+ return None
+ if _normalize(claim) not in _normalize(reply):
+ return None
+ return claim
+
+
+async def _call_model(model: str, turn: str) -> dict[str, Any]:
+ from openexecutive.audit.usage import log_model_usage
+ from openexecutive.providers import get_provider
+
+ response = await get_provider(model).messages_create(
+ model=model,
+ max_tokens=_MAX_TOKENS,
+ system=_SYSTEM,
+ tools=[_TOOL],
+ tool_choice={"type": "tool", "name": _TOOL["name"]},
+ messages=[{"role": "user", "content": turn}],
+ )
+ log_model_usage(response, model=model, actor="claim_check")
+ for block in response.content:
+ if getattr(block, "type", "") == "tool_use" and getattr(block, "name", "") == _TOOL["name"]:
+ return block.input if isinstance(block.input, dict) else {}
+ return {}
+
+
+async def unbacked_claim(message: str, reply: str, calls: list[tuple[str, str]]) -> str | None:
+ """The reply's claim of a change no call in ``calls`` made, or None.
+ Never raises."""
+ if not enabled() or not may_claim(reply):
+ return None
+ try:
+ from openexecutive.config import get_settings
+
+ model = get_settings().routing_model
+ payload = await asyncio.wait_for(
+ _call_model(model, _render_turn(message, reply, calls)), timeout=_TIMEOUT_S
+ )
+ except Exception:
+ logger.warning("claim_check: check failed, reply goes out as is", exc_info=True)
+ return None
+ return accept_claim(payload, reply)
+
+
+def nudge(claim: str) -> str:
+ """The user-turn note for the extra round (never a cached block)."""
+ return (
+ "[Check before you finish] Your reply says “" + claim + "”, but no "
+ "tool call in this reply made that change. Make it now with the right "
+ "tool and say so in one short line, or, if you can't, tell the person "
+ "plainly that it isn't done and why. Don't repeat the rest of your reply."
+ )
+
+
+def message_text(content: Any) -> str:
+ """The text of a user message's content (a string, or text blocks)."""
+ if isinstance(content, str):
+ return content
+ if isinstance(content, list):
+ return "\n".join(
+ str(b.get("text", "")) for b in content
+ if isinstance(b, dict) and b.get("type") == "text"
+ )
+ return ""
+
+
+def last_person_message(messages: list[dict[str, Any]]) -> str:
+ """The last user message that the person wrote (not a tool-results turn)."""
+ for msg in reversed(messages):
+ if msg.get("role") != "user":
+ continue
+ content = msg.get("content")
+ if isinstance(content, list) and any(
+ isinstance(b, dict) and b.get("type") == "tool_result" for b in content
+ ):
+ continue
+ text = message_text(content)
+ if text.strip():
+ return text
+ return ""
diff --git a/packages/core/openexecutive/orchestrator/executive.py b/packages/core/openexecutive/orchestrator/executive.py
index 58032dd2..7cc51a87 100644
--- a/packages/core/openexecutive/orchestrator/executive.py
+++ b/packages/core/openexecutive/orchestrator/executive.py
@@ -39,7 +39,13 @@ from openexecutive.memory.workspace_settings import (
pin_turn_principal_role,
pin_turn_workspace_mode,
)
-from openexecutive.orchestrator import department_gate, extensions, take_the_lead, tool_groups
+from openexecutive.orchestrator import (
+ claim_check,
+ department_gate,
+ extensions,
+ take_the_lead,
+ tool_groups,
+)
from openexecutive.orchestrator.action_chips import summarize_action
from openexecutive.orchestrator.activity_labels import (
fallback_activity,
@@ -2141,6 +2147,12 @@ class Executive:
turn_budget=self._settings.tool_results_turn_max_chars,
)
+ # Every tool call this turn with the start of its result, for the
+ # end-of-turn check that the reply claims nothing it didn't do
+ # (claim_check); checked once a turn.
+ turn_calls: list[tuple[str, str]] = []
+ claim_checked = False
+
for iteration in range(1, max_iterations + 1):
logger.info(
"iter %d/%d", iteration, max_iterations,
@@ -2277,7 +2289,35 @@ class Executive:
)
if final_msg.stop_reason != "tool_use":
- return
+ # Do what it says: a reply that claims a change no call this
+ # turn made gets one more round to make it or say it can't.
+ if claim_checked or iteration >= max_iterations:
+ return
+ claim_checked = True
+ reply_text = "\n".join(
+ str(b.get("text", "")) for b in response_content if b.get("type") == "text"
+ )
+ claim = await claim_check.unbacked_claim(
+ claim_check.last_person_message(messages), reply_text, turn_calls,
+ )
+ if claim is None:
+ return
+ audit_log(
+ "claim_check",
+ "Reply claimed a change no tool made; one more round",
+ session_id=getattr(current_session.get(), "session_id", None),
+ turn_id=turn_id,
+ actor="executive",
+ details={"iteration": iteration, "calls": len(turn_calls)},
+ full={"claim": claim},
+ private=private_turn,
+ )
+ current_messages.append({"role": "assistant", "content": response_content})
+ current_messages.append(
+ {"role": "user", "content": [{"type": "text", "text": claim_check.nudge(claim)}]}
+ )
+ yield "\n\n"
+ continue
# open_tools is answered here; a use_tool call becomes the call it
# stands for (same id), so every guard, handler, chip and audit
@@ -3143,6 +3183,8 @@ class Executive:
# leaves non-model consumers (the propose_form_values JSON
# parse above, the audit trail) reading the full text.
tool_results: list[dict[str, Any]] = []
+ for tu in tool_uses:
+ turn_calls.append((tu["name"], str(results_by_id.get(tu["id"], ""))))
for tu in tool_uses:
content = reading_budget.cap(
tu["name"],
diff --git a/packages/core/tests/conftest.py b/packages/core/tests/conftest.py
index 17cb6d7b..eb5f8340 100644
--- a/packages/core/tests/conftest.py
+++ b/packages/core/tests/conftest.py
@@ -59,6 +59,14 @@ def _no_ai_disclosure(monkeypatch: pytest.MonkeyPatch):
disclosure.reset()
[email protected](autouse=True)
+def _no_claim_check(monkeypatch: pytest.MonkeyPatch):
+ """Keep the end-of-turn claim check (orchestrator.claim_check) from making
+ a provider call in tests that script a turn; test_claim_check.py turns it
+ back on."""
+ monkeypatch.setenv("CLAIM_CHECK_ENABLED", "false")
+
+
@pytest.fixture(autouse=True)
def reset_active_gateway():
"""Ensure the module-level MCP gateway singleton is cleared between tests."""
diff --git a/packages/core/tests/unit/test_claim_check.py b/packages/core/tests/unit/test_claim_check.py
new file mode 100644
index 00000000..5808558a
--- /dev/null
+++ b/packages/core/tests/unit/test_claim_check.py
@@ -0,0 +1,161 @@
+"""Do what it says (orchestrator.claim_check): a reply that claims a change no
+tool call made gets one more round."""
+from __future__ import annotations
+
+import asyncio
+from typing import Any
+from unittest.mock import patch
+
+import pytest
+
+from openexecutive.orchestrator import claim_check
+from openexecutive.orchestrator.executive import Executive
+from tests.unit._agent_loop_fakes import FinalMsg, ScriptedProvider, TextBlock, ToolUseBlock
+
+
[email protected](autouse=True)
+def _claim_check_on(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("CLAIM_CHECK_ENABLED", "true")
+ monkeypatch.setattr("openexecutive.orchestrator.executive.audit_log", lambda *a, **k: None)
+
+
+def _run(provider: ScriptedProvider, message: str = "Remove it from the brief") -> list[Any]:
+ async def _go() -> list[Any]:
+ items: list[Any] = []
+ with patch("openexecutive.orchestrator.executive.get_provider", return_value=provider):
+ async for item in Executive()._stream_agent_loop(
+ system_blocks=[],
+ messages=[{"role": "user", "content": message}],
+ model="claude-test",
+ ):
+ items.append(item)
+ return items
+
+ return asyncio.run(_go())
+
+
+def _verdict(payload: dict[str, Any]):
+ calls: list[str] = []
+
+ async def fake(model: str, turn: str) -> dict[str, Any]:
+ calls.append(turn)
+ return payload
+
+ return fake, calls
+
+
+# --- the pure parts ----------------------------------------------------------
+
+
+def test_word_check_skips_replies_that_claim_nothing() -> None:
+ assert not claim_check.may_claim("Revenue was up 4% on last quarter.")
+ assert not claim_check.may_claim(" ")
+ assert claim_check.may_claim("Done.")
+ assert claim_check.may_claim("I'll treat the draft as closed.")
+ assert claim_check.may_claim("Removed it from the brief")
+
+
+def test_claim_must_be_in_the_reply_word_for_word() -> None:
+ reply = "Done — I’ll treat the draft as closed."
+ assert claim_check.accept_claim({"unbacked": True, "claim": "I'll treat the draft as closed"}, reply)
+ assert claim_check.accept_claim({"unbacked": True, "claim": "I sent the email"}, reply) is None
+ assert claim_check.accept_claim({"unbacked": False, "claim": "Done"}, reply) is None
+ assert claim_check.accept_claim({"unbacked": True, "claim": ""}, reply) is None
+ assert claim_check.accept_claim({"unbacked": True, "claim": "x" * 300}, "x" * 300) is None
+
+
+def test_last_person_message_skips_tool_result_turns() -> None:
+ messages = [
+ {"role": "user", "content": "I sent it"},
+ {"role": "assistant", "content": [{"type": "tool_use", "id": "t", "name": "x", "input": {}}]},
+ {"role": "user", "content": [{"type": "tool_result", "tool_use_id": "t", "content": "ok"}]},
+ ]
+ assert claim_check.last_person_message(messages) == "I sent it"
+
+
+def test_check_is_off_without_the_setting(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("CLAIM_CHECK_ENABLED", "false")
+ fake, calls = _verdict({"unbacked": True, "claim": "Done"})
+ with patch.object(claim_check, "_call_model", fake):
+ assert asyncio.run(claim_check.unbacked_claim("x", "Done", [])) is None
+ assert calls == []
+
+
+def test_a_failed_check_lets_the_reply_through() -> None:
+ async def boom(model: str, turn: str) -> dict[str, Any]:
+ raise RuntimeError("provider down")
+
+ with patch.object(claim_check, "_call_model", boom):
+ assert asyncio.run(claim_check.unbacked_claim("x", "Done", [])) is None
+
+
+def test_the_check_sees_the_turns_calls() -> None:
+ fake, calls = _verdict({"unbacked": False, "claim": ""})
+ with patch.object(claim_check, "_call_model", fake):
+ asyncio.run(claim_check.unbacked_claim(
+ "close it", "Closed it.", [("close_open_loop", '{"status": "closed"}')],
+ ))
+ assert "close_open_loop" in calls[0]
+ assert "<assistant_reply>\nClosed it." in calls[0]
+
+
+# --- in the agent loop -------------------------------------------------------
+
+
+def test_unbacked_claim_gets_one_more_round_with_the_nudge() -> None:
+ provider = ScriptedProvider([
+ FinalMsg([TextBlock("Done.")], "end_turn"),
+ FinalMsg([TextBlock("I can't remove items from the brief yet.")], "end_turn"),
+ ])
+ fake, calls = _verdict({"unbacked": True, "claim": "Done"})
+ with patch.object(claim_check, "_call_model", fake):
+ items = _run(provider)
+
+ assert len(provider.calls) == 2
+ second = provider.calls[1]["messages"]
+ assert second[-2] == {"role": "assistant", "content": [{"type": "text", "text": "Done."}]}
+ nudge = second[-1]["content"][0]["text"]
+ assert "“Done”" in nudge and "no tool call" in nudge
+ assert "\n\n" in items
+ # Checked once a turn: the second reply ends it without another check.
+ assert len(calls) == 1
+
+
+def test_backed_or_plain_reply_ends_the_turn() -> None:
+ provider = ScriptedProvider([FinalMsg([TextBlock("Done.")], "end_turn")])
+ fake, calls = _verdict({"unbacked": False, "claim": ""})
+ with patch.object(claim_check, "_call_model", fake):
+ _run(provider)
+ assert len(provider.calls) == 1
+ assert len(calls) == 1
+
+
+def test_reply_without_claim_words_makes_no_check_call() -> None:
+ provider = ScriptedProvider([FinalMsg([TextBlock("Revenue was up 4%.")], "end_turn")])
+ fake, calls = _verdict({"unbacked": True, "claim": "Revenue"})
+ with patch.object(claim_check, "_call_model", fake):
+ _run(provider, "How did revenue do?")
+ assert calls == []
+ assert len(provider.calls) == 1
+
+
+def test_tool_calls_this_turn_reach_the_check() -> None:
+ provider = ScriptedProvider([
+ FinalMsg([ToolUseBlock("t1", "list_open_loops", {})], "tool_use"),
+ FinalMsg([TextBlock("Closed it.")], "end_turn"),
+ ])
+ fake, calls = _verdict({"unbacked": False, "claim": ""})
+
+ async def handler(_input: dict[str, Any]) -> str:
+ return '{"loops": []}'
+
+ with (
+ patch.object(claim_check, "_call_model", fake),
+ patch.dict(
+ "openexecutive.orchestrator.executive._ALL_SKILL_HANDLERS",
+ {"list_open_loops": handler},
+ ),
+ ):
+ _run(provider, "I sent it")
+ assert len(calls) == 1
+ assert "- list_open_loops:" in calls[0]