Skip to content

Dodaios is an open-source agent OS for your desktop: it runs your coding agents, browser and apps by voice or chat, on your own machine. Formerly Personal Jarvis.

feat(realtime): the user keeps talking while the tool model works (ADR-0034)

Harald2ca61307ec85
diff --git a/docs/adr/0034-non-blocking-heavy-turns.md b/docs/adr/0034-non-blocking-heavy-turns.md
new file mode 100644
index 0000000000..8f2026391f
--- /dev/null
+++ b/docs/adr/0034-non-blocking-heavy-turns.md
@@ -0,0 +1,234 @@
+# ADR-0034 — Non-blocking heavy turns: the user keeps talking while the tool model works
+
+**Status:** Accepted · **Date:** 2026-08-18 · **Phase:** Voice UX + text surfaces (every engine, every provider)
+
+## Context
+
+A heavy turn — the tool model, a router round trip, a background agent —
+takes 5–60 s. Until now the conversation stopped for that time: in the
+realtime engine the live model was muted while its delegate ran (and, on a
+provider-requested `jarvis_action`, the transport itself waited for the
+function response); in the classic pipeline the turn `await`ed the brain
+inline and any speech into the wait was treated as "cancel and recombine";
+on the text surfaces the reply simply arrived when it arrived. ADR-0033
+gave the wait a first sign of life within 2 s. This ADR removes the wait
+from the conversation altogether.
+
+The maintainer's vision (2026-08-18): hand the heavy work to the background
+at once, keep conversing, and have the result come back later — without the
+result talking over the user, without it landing in the middle of another
+topic unannounced, and without the fast model inventing or denying it in
+the meantime.
+
+What already existed and is kept (measured in `jarvis/realtime/session.py`
+and `jarvis/speech/pipeline.py` on 2026-08-18):
+
+- Realtime: the delegate is a background `asyncio` task; a real new
+ utterance splits the turn; a result whose turn closed is queued as a
+ *late result* and spoken once the session is at rest
+ (`_queue_late_delegate_result` / `_flush_late_delegate_results`); the
+ live model carries `_DELEGATE_PENDING_DIRECTIVE`; presence probes get a
+ deterministic status line (BUG-070); stop words cancel (`interrupt_intent`).
+- Pipeline: `AnnouncementRequested(kind="completion")` is a proven
+ proactive-speech channel — deferred while the user holds the floor,
+ flushed at the next turn boundary, punching through a hang-up
+ (AD-OE5/OE6); the instant ack + one grounded progress line (ADR-0033).
+- Both: the deterministic planner (`plan_turn(...).requires_orchestrator`)
+ decides "heavy" at dispatch with zero latency.
+
+What blocked the vision (log evidence 2026-08-18 15:34 and code):
+
+1. A provider-requested `jarvis_action` is answered on the wire only when the
+ delegate finishes; a transport that waits for the function response
+ (Gemini Live on Vertex — `behavior` is unsupported there; the Developer
+ API only on models that support asynchronous calls) answers nothing new
+ until then.
+2. `_LATE_DELEGATE_DELIVERY_TIMEOUT_S = 30`: a result that finds no 30 s of
+ rest is dropped with a log warning — the user talks, the answer is gone.
+3. `_session_is_at_rest()` refuses every delivery while *any* delegate task
+ exists, so a second background order blocks the first result.
+4. A probe or a continuation of the running order gets one canned pool line
+ through the surface TTS — a second voice, no real progress.
+5. The pipeline has no background turn at all; the continuation window is
+ armed for the whole thinking phase, so every utterance recombines.
+
+## Decision
+
+The rule of CLAUDE.md §3 applies from the first sentence: this is ONE
+provider-neutral design. Every voice engine, every realtime transport,
+every brain family (cloud or local), and the text surfaces get the same
+user-visible behaviour; a transport-specific mechanism is a declared
+capability, never the design.
+
+### 1. One vocabulary: the parked result
+
+A heavy turn that outlives the conversation's attention becomes a **parked
+result** — `jarvis/voice/parked_results.py` (stdlib only, engine-neutral):
+
+- `ParkedResult(text, language, request_text, success, request_turn_id,
+ delivery_id, queued_at, intervening_turns)`; `ParkedResultLedger` with
+ `park()`, `peek()`, `pop(delivery_id)`, `note_turn_completed()`,
+ `retrieve_for(query)`, `supersede(request_text)`, `cancel_all()`,
+ `drain()`; the closed vocabulary `classify_wait_query()`
+ (PROGRESS / RESULT); `requested_result(items, query)`; and `reanchor()`
+ with `topic_of()`. The realtime session keeps its own list of
+ `ParkedResult`s (its `_LateDelegateResult` alias) and the pipeline uses
+ the ledger; both read the same vocabulary.
+- **No time-based expiry.** A parked result leaves the ledger only when it
+ was spoken/sent, when the user cancelled the order (stop word / explicit),
+ when the same order was re-issued (superseded), or at session end — and
+ the last case logs the request it drops by name (BUG-070 lesson).
+- **Delivery gate = a predicate the engine supplies**, evaluated per item at
+ every rest: user not speaking, no output active, no delivery in flight,
+ no interim line in the last `INTERIM_GAP_S`. A running background task of
+ ANOTHER order is not a reason to wait (gate per task, not per session).
+- **Re-anchoring is mandatory when anything happened in between.** The
+ spoken result opens with a short reference to the request it answers
+ (realtime: the `late=True` rendering order carries the request text; the
+ pipeline and the text surfaces use a closed de/en/es prefix pool with the
+ request's own words). Straight after the ack with no turn in between,
+ no prefix.
+- **On demand.** A user question that names the parked work ("was kam bei
+ X raus?", "hast du das Ergebnis?") delivers the ready result now — after
+ the user stops speaking, in the same voice — instead of waiting for rest.
+ A closed multilingual vocabulary decides it (regex, AP-9/11); a miss
+ simply stays native.
+
+### 2. Realtime engine
+
+1. **The transport is freed the moment the user moves on.** A fast order
+ that finishes inside its own turn still answers the provider's function
+ call directly (unchanged wire, unchanged tests). When the user opens a
+ NEW turn while an earlier order's `jarvis_action` call is still open on
+ the wire, the session answers that call at once with the closed interim
+ payload (`_PENDING_TOOL_CALL_INTERIM_RESULT`: "still executing, do not
+ invent an outcome, do not call again, answer the new request"), so a
+ transport with blocking function calls (Gemini Live — `NON_BLOCKING` is
+ unsupported on Vertex and on the 3.1 Live model, and blocking is the
+ default everywhere) can answer the new turn. The order keeps running;
+ its result is parked and delivered through the late-result path — a
+ developer text turn at rest — never as a late tool result against the
+ answered call id. A stop word closes open calls the same way ("cancelled
+ by the user"). Switch: `[voice].realtime_unblock_pending_tool_calls`
+ (default on). Native asynchronous calls stay a follow-up: no supported
+ default model offers them today, so no plumbing is shipped for them.
+2. **Rest is per delivery.** `_session_is_at_rest()` and
+ `deliver_announcement` no longer refuse because a delegate task exists;
+ they refuse while a trusted reply is between injection and its readback
+ (`_delegate_delivery_in_flight`, the BUG-143 window). A background order
+ that is still computing does not hold a ready result hostage. Running
+ orders stay reachable after their turn closed
+ (`_delegate_states_by_turn`).
+3. **No expiry.** The late-result flush polls until spoken or until the
+ session ends (`end()` re-routes what is left to the detached completion
+ channel, named in the log); the 30 s bound is gone. A parked result
+ carries `request_text`, and the `late=True` rendering order names it so
+ the live model ties the answer back to the request.
+4. **Wait queries are owned by the orchestrator, before the planner.**
+ `classify_wait_query` runs on every final while an order runs or a result
+ is parked: a PROGRESS question, or a RESULT request while the order
+ still runs, speaks ONE line grounded in the running tool
+ (`ActionProposed` → SEARCH / READ / SCREEN pools; unknown tool → the
+ BUG-070 status pool); a RESULT request while a result is parked delivers
+ that result now — as this turn's response on transports that generate
+ only on request, or the moment the probe turn closes on transports that
+ answer on their own VAD (their freestyle answer is dropped). Neither
+ becomes a brain dispatch. The grounded line still rides the surface-TTS
+ status channel BUG-070 established; moving it into the live voice needs
+ the bridge to own a non-delegate turn and is a follow-up for a live
+ session.
+5. Everything else stays: the withhold of the model's own words for the
+ turn that owns a running delegate (BUG-054), the repeat-order refusal,
+ `_DELEGATE_PENDING_DIRECTIVE`, the endpoint protection.
+
+### 3. Classic pipeline
+
+1. **Heavy turns park.** When `plan_turn(text).requires_orchestrator` and
+ `[voice].background_heavy_turns` (default `True`), the pipeline arms the
+ instant ack as before, then runs the brain call in a background task
+ (`_run_parked_turn`) and returns to `LISTENING` at once. The continuation
+ window is marked idle at that moment (2.5 s grace as for a finished
+ turn), so a fast follow still recombines and a later utterance is a new
+ turn.
+2. **New utterances are ordinary turns**, answered by the same brain
+ concurrently (`generate_stream` holds no lock; history is appended per
+ completed generation). Three utterance classes are answered by the
+ orchestrator instead: stop/redirect (`interrupt_intent`) cancels the
+ parked turn and speaks the cancel line; presence probe / progress
+ question speaks ONE grounded progress line; a re-issue of the same order
+ is refused with the progress line (no double execution).
+3. **The result is delivered as a completion readback**:
+ `AnnouncementRequested(kind="completion", source_layer="speech.parked_turn")`
+ — deferred while the user holds the floor or a turn is being processed
+ or spoken, flushed at the next `LISTENING`/`IDLE` boundary, scrubbed by
+ the same output filter, recorded on the transcript as the answer to the
+ parked turn. The instant ack, the progress line and `_on_action_proposed`
+ read the parked turn's own "still running" state, no longer the global
+ `PROCESSING` state.
+4. Failure and silence are spoken: a timeout, an exception, or an empty
+ answer speaks the existing honest lines (`_speak_brain_timeout`,
+ `_handle_silent_brain_turn`) — never a silent drop (AD-OE6).
+5. `[voice].background_heavy_turns = false` restores the inline await.
+
+### 4. Text surfaces
+
+Chat (desktop, headless launcher, browser) and channel adapters keep their
+per-message concurrency; the pre-ack bubble stays. A heavy reply that lands
+after a later exchange is prefixed with the same re-anchoring line
+(`ParkedResult` prefix pool) so the message names the request it answers.
+
+### 5. Observability
+
+Two frozen bus events: `ParkedResultQueued(request_text, delivery_id,
+surface)` and `ParkedResultDelivered(delivery_id, surface, waited_s,
+delivered_on_demand)`; a named WARNING for every parked result a session
+end abandons. Nothing about a parked result is silent (AP-30).
+
+## Consequences
+
+- The user can keep talking through any heavy turn on every engine and
+ transport; the answer arrives when there is room for it, introduced by
+ what it answers, and never disappears after 30 s.
+- Two chargeable calls stay the ceiling per heavy turn (ack + interim);
+ the parked delivery reuses the result already computed.
+- Ordering of brain history changes: a parked pair is appended when it
+ completes, possibly after a later fast turn. The fast turn sees "still
+ running" via the pending directive / the parked-turn state; this is the
+ honest picture.
+- The interim answer to an open function call is the universal baseline
+ and needs no provider cooperation; a native asynchronous path (Gemini
+ `NON_BLOCKING` + `WHEN_IDLE`, OpenAI's asynchronous wire) is a declared
+ capability for later, once a supported model offers it and a live session
+ can verify it.
+- Amends BUG-051's rule "a bridge must be a bystander" — unchanged — and
+ BUG-070's presence probe: same trigger, now a grounded line in the live
+ voice.
+
+## Alternatives considered
+
+- **Keep the 30 s expiry, just longer** — any bound drops a result the user
+ asked for; a result may only leave the ledger for a named reason.
+- **Native asynchronous calls as the design** — unavailable on Vertex,
+ not offered by the 3.1 Live model on the Developer API, meaningless on
+ the pipeline; it is one cell of the matrix, so it stays a capability.
+- **Answer every provider call with the interim payload immediately** —
+ would change the wire for the common fast case (result inside the same
+ turn) that 25 live-forensic tests pin, for no user-visible gain; the
+ block only hurts once the user actually moves on, so that is when the
+ call is answered.
+- **Deliver the parked result through the surface TTS in realtime** — a
+ second voice mid-call (BUG-090); rejected, the live model renders it.
+- **Keep the pipeline inline and only shorten the wait** — the ack (ADR-0033)
+ already does that; it does not give the floor back.
+- **Interrupt the user for a ready result** (`scheduling=INTERRUPT`) — the
+ maintainer's explicit fear; a result waits for rest or for being asked.
+
+## References
+
+- `jarvis/voice/parked_results.py`, `tests/unit/voice/test_parked_results.py`
+- `jarvis/realtime/session.py` (`_session_is_at_rest`, `_flush_late_delegate_results`,
+ `_handle_tool_call`, `_speak_pending_action_status`), `jarvis/realtime/protocol.py`
+- `jarvis/plugins/realtime/gemini_live.py`, `jarvis/plugins/realtime/openai_realtime.py`
+- `jarvis/speech/pipeline.py` (`_run_parked_turn`, `_deliver_parked_result`)
+- ADR-0033, BUG-051, BUG-054, BUG-070, BUG-090, BUG-143, BUG-144
+- CLAUDE.md §3 "Provider & mode parity", CLOUD.md Rule #2
diff --git a/jarvis/core/config.py b/jarvis/core/config.py
index 8bd4eb147a..aa1d5cfffd 100644
--- a/jarvis/core/config.py
+++ b/jarvis/core/config.py
@@ -2657,6 +2657,24 @@ class VoiceConfig(BaseModel):
# opt-in. Read once per session; unknown values fail closed to the compact
# delegate mode.
realtime_tool_mode: str = "delegate"
+ # ADR-0034 (2026-08-18). When the user opens a NEW turn while an earlier
+ # order's provider function call (jarvis_action) is still unanswered on
+ # the wire, the session answers that call at once with a closed "still
+ # executing" payload so a transport with blocking function calls (Gemini
+ # Live; NON_BLOCKING is unsupported on Vertex and on the 3.1 Live model)
+ # can answer the new turn; the order keeps running and its result is
+ # spoken later as a parked follow-up. Fast orders that finish inside their
+ # own turn are untouched. False = pre-2026-08-18 behaviour (the transport
+ # waits for the result). Read per event by the realtime session.
+ realtime_unblock_pending_tool_calls: bool = True
+ # ADR-0034 (2026-08-18). Classic pipeline: a heavy turn (one the
+ # deterministic planner routes to the orchestrator / tool model) runs in
+ # the background and the microphone stays open — the user keeps talking,
+ # ordinary utterances are answered meanwhile, and the heavy answer is
+ # spoken at the next quiet turn boundary, tied back to the request. False
+ # = the pre-2026-08-18 inline wait (speech during the wait cancels and
+ # recombines). Read per turn by the pipeline.
+ background_heavy_turns: bool = True
# Per-gap budget after which a stale pending fragment is silently
# discarded (user-mandated 2026-05-26 — was: flushed/spoken). NOT a total
# budget — every continuation resets the timer. Bumped from 8 s to 15 s
diff --git a/jarvis/realtime/session.py b/jarvis/realtime/session.py
index 31213b15af..5b59d7b11e 100644
--- a/jarvis/realtime/session.py
+++ b/jarvis/realtime/session.py
@@ -94,6 +94,13 @@ from jarvis.voice.instant_ack import (
pick_progress_text,
plan_instant_ack,
)
+from jarvis.voice.parked_results import (
+ WAIT_QUERY_PROGRESS,
+ WAIT_QUERY_RESULT,
+ ParkedResult,
+ classify_wait_query,
+ requested_result,
+)
log = logging.getLogger(__name__)
@@ -746,9 +753,31 @@ _DELEGATE_DISCOURAGED_DIRECTIVE_HANDOFF = _handoff_variant(
)
# Delivering a result whose turn already closed must never race the live turn:
# the session waits until it is at rest, then speaks the result as an explicit
-# follow-up. The bound only decides how long a result may wait for that silence.
-_LATE_DELEGATE_DELIVERY_TIMEOUT_S = 30.0
+# follow-up. There is NO time bound on that wait (ADR-0034): a result the user
+# asked for leaves the queue only delivered, cancelled, superseded, or — at
+# session end — re-routed to the detached completion channel. The 30 s bound
+# that stood here until 2026-08-18 dropped the answer whenever the user simply
+# kept talking. The poll interval only sets how quickly a rest is noticed;
+# the "still parked" log line keeps a long wait visible (AP-30).
_LATE_DELEGATE_POLL_S = 0.15
+_LATE_DELEGATE_STILL_PARKED_LOG_S = 60.0
+# A provider-requested ``jarvis_action`` that is still pending when the user
+# opens a NEW turn is answered on the wire with this interim payload, so a
+# transport that waits for the function response (Gemini Live: blocking calls
+# are the default and NON_BLOCKING is unsupported on Vertex and on the 3.1
+# Live model) can answer the new turn. The real result travels the late-result
+# path (a developer text turn at rest), never a late tool result.
+_PENDING_TOOL_CALL_INTERIM_RESULT: dict[str, Any] = {
+ "success": False,
+ "status": "in_progress",
+ "error": (
+ "Still executing. The Jarvis orchestrator is running this request in "
+ "the background and has no result yet. Do not announce, promise, or "
+ "invent an outcome and do not call the function again for it; answer "
+ "the user's NEW request now. The trusted result will be injected as a "
+ "separate message as soon as it is ready."
+ ),
+}
# Let tasks that are only unwinding a readback verifier observe ``_ended``
# before process-scope retention. Real action work remains untouched after
# this tiny teardown-only grace and is transferred below.
@@ -977,6 +1006,7 @@ def _delegate_result_prompt(
success: bool,
late: bool = False,
already_said: str = "",
+ request_text: str = "",
) -> str:
"""Wrap one trusted Brain result for tool-free native voice rendering.
@@ -1006,15 +1036,23 @@ def _delegate_result_prompt(
if dot > _DELEGATE_RESULT_MAX_CHARS // 2:
cut = cut[: dot + 1]
text = cut + " [result shortened]"
- framing = (
- (
+ framing = ""
+ if late:
+ framing = (
"This is the outcome of the user's earlier request, which finished "
"only now. Open with one short phrase that ties it back to that "
"earlier request, then state the result. "
)
- if late
- else ""
- )
+ request = " ".join(str(request_text or "").split())
+ if request:
+ # ADR-0034: the conversation may have moved on several turns; the
+ # tie-back must name WHAT is being answered, in the user's words.
+ if len(request) > 160:
+ request = request[:157].rstrip() + "..."
+ framing += (
+ f'The earlier request was: "{request}" — refer to its topic '
+ "in that opening phrase. "
+ )
spoken = str(already_said or "").strip()
if spoken:
# Instant acknowledgment (2026-08-17): the user already heard an
@@ -1367,6 +1405,11 @@ class _DelegateTurnState:
user_text: str = ""
result_payload: dict[str, Any] = field(default_factory=dict)
pending_tool_calls: list[tuple[str, str]] = field(default_factory=list)
+ # ADR-0034: the provider's function call(s) for this order were answered
+ # on the wire with the interim "still executing" payload because the user
+ # opened a new turn while the order ran. The real result then travels the
+ # late-result path (a developer text turn at rest), never a tool result.
+ interim_tool_reply_sent: bool = False
seen_tool_call_ids: set[str] = field(default_factory=set)
dispatch_started: bool = False
bridge_delivery_started: bool = False
@@ -1421,14 +1464,11 @@ class _ExternalUpdateState:
detail: str | None = None
-@dataclass(slots=True)
-class _LateDelegateResult:
- """One executed action whose trusted result outlived its realtime turn."""
-
- text: str
- success: bool
- language: str
- delivery_id: str
+#: One executed action whose trusted result outlived its realtime turn. Since
+#: ADR-0034 this is the engine-neutral ``ParkedResult`` (``request_text`` names
+#: the order it answers, ``queued_at`` how long it has waited); the alias keeps
+#: the session's own vocabulary and its tests readable.
+_LateDelegateResult = ParkedResult
_TOOL_ROLE_DIRECTIVE = (
@@ -1927,6 +1967,7 @@ class RealtimeVoiceSession:
self._tool_bridge = tool_bridge
self._delegate_tasks: set[asyncio.Task[None]] = set()
self._delegate_tasks_by_turn: dict[str, set[asyncio.Task[None]]] = {}
+ self._delegate_states_by_turn: dict[str, _DelegateTurnState] = {}
# BUG-051: the dead-air bridge is deliberately NOT a tracked delegate
# task — it must never hold a turn open, defer a VAD edge, or refuse
# an announcement on behalf of work that is merely a sleeping timer.
@@ -3851,6 +3892,29 @@ class RealtimeVoiceSession:
# by the ordinary path below, now that the order
# it replaces is gone.
turn_plan = self._plan_turn(self._last_user_text)
+ # ADR-0034: the user opened a NEW turn while an
+ # earlier order's provider function call is still
+ # unanswered on the wire. A transport that waits for
+ # that answer would answer nothing new — free it now;
+ # the order keeps running and its result is parked.
+ await self._unblock_pending_provider_calls()
+ # ADR-0034: "how far are you?" / "what came out of
+ # it?" spoken into a wait or after a parked result is
+ # owned by the orchestrator BEFORE the planner can
+ # read it as an order of its own ("what did you find"
+ # is search-shaped): a grounded progress line, or the
+ # parked result itself. Anything else stays native.
+ wait_query_claimed = bool(
+ (
+ self._has_pending_delegate_from_earlier_turn()
+ or self._late_delegate_results
+ )
+ and not self._response_requested_for_turn
+ and not self._user_is_speaking()
+ and not self._answers_open_delegate_question()
+ and not self._brain_awaits_voice_confirm()
+ and await self._answer_wait_query(self._last_user_text)
+ )
if (
turn_plan.requires_orchestrator
and not self._active_provider_supports_direct_tools()
@@ -3907,6 +3971,7 @@ class RealtimeVoiceSession:
if (
self._last_user_text
and deterministic_delegate_available
+ and not wait_query_claimed
and (
self._delegate_enabled
or screen_context_turn
@@ -3920,6 +3985,10 @@ class RealtimeVoiceSession:
or self._brain_awaits_voice_confirm()
or self._answers_open_delegate_question()
)
+ if wait_query_claimed:
+ # The orchestrator answered this turn; no
+ # dispatch, no grounding, no provider response.
+ self._delegate_required_for_turn = False
refresh_tools = getattr(
self._tool_bridge, "refresh_from_source", None
)
@@ -7858,23 +7927,41 @@ class RealtimeVoiceSession:
self._output_drop_count = 0
def _track_delegate_task(
- self, turn_id: str, task: asyncio.Task[None]
+ self,
+ turn_id: str,
+ task: asyncio.Task[None],
+ state: _DelegateTurnState | None = None,
) -> None:
self._delegate_tasks.add(task)
turn_tasks = self._delegate_tasks_by_turn.setdefault(turn_id, set())
turn_tasks.add(task)
+ if state is not None:
+ # ``_delegate_turns`` is popped when the TURN completes, while the
+ # order keeps running; everything that must still reach a running
+ # order's state after that (ADR-0034: the interim wire answer, the
+ # per-delivery rest gate, the executing-order texts) reads it here.
+ self._delegate_states_by_turn[turn_id] = state
def _discard(done: asyncio.Task[None]) -> None:
self._delegate_tasks.discard(done)
tracked = self._delegate_tasks_by_turn.get(turn_id)
if tracked is None:
+ self._delegate_states_by_turn.pop(turn_id, None)
return
tracked.discard(done)
if not tracked:
self._delegate_tasks_by_turn.pop(turn_id, None)
+ self._delegate_states_by_turn.pop(turn_id, None)
task.add_done_callback(_discard)
+ def _running_delegate_state(self, turn_id: str) -> _DelegateTurnState | None:
+ """The delegate state of ``turn_id`` while its order runs — open turn or not."""
+ state = self._delegate_turns.get(turn_id)
+ if state is None:
+ state = self._delegate_states_by_turn.get(turn_id)
+ return state
+
def _retain_detached_delegate_task(
self,
turn_id: str,
@@ -8065,6 +8152,70 @@ class RealtimeVoiceSession:
for turn_id, tasks in self._delegate_tasks_by_turn.items()
)
+ def _unblock_pending_tool_calls_enabled(self) -> bool:
+ voice_cfg = getattr(self._config, "voice", None)
+ return bool(
+ getattr(voice_cfg, "realtime_unblock_pending_tool_calls", True)
+ )
+
+ async def _unblock_pending_provider_calls(self) -> int:
+ """Answer an EARLIER turn's still-pending function calls with the interim payload.
+
+ ADR-0034 §2.1. Function calls are blocking by default on the Live API
+ (and ``NON_BLOCKING`` is unsupported on Vertex and on the 3.1 Live
+ model), so a ``jarvis_action`` the provider requested for a slow order
+ held the whole transport: the user spoke a new turn into that wait and
+ the model could not answer it. When the user opens a new turn, every
+ earlier order whose calls are still unanswered gets the closed
+ interim payload — "still executing, do not invent an outcome, answer
+ the new request" — on the wire, and its eventual result is delivered
+ as a parked follow-up (``_queue_late_delegate_result``), never as a
+ late tool result. Fast orders that finish inside their own turn are
+ untouched: their result still answers the call directly.
+
+ Returns the number of calls answered. Never raises: a torn wire is
+ logged and the order keeps running.
+ """
+ if not self._unblock_pending_tool_calls_enabled():
+ return 0
+ if self._session is None or not self._session_takes_tool_results():
+ return 0
+ answered = 0
+ for turn_id in tuple(self._delegate_tasks_by_turn):
+ if turn_id == self._turn_id or not self._turn_has_pending_delegate(turn_id):
+ continue
+ state = self._running_delegate_state(turn_id)
+ if state is None or not state.pending_tool_calls or state.result_complete:
+ continue
+ calls = tuple(state.pending_tool_calls)
+ for call_id, wire_name in calls:
+ try:
+ await self._session.send_tool_result(
+ call_id, wire_name, dict(_PENDING_TOOL_CALL_INTERIM_RESULT)
+ )
+ except Exception: # noqa: BLE001 — the order keeps running either way
+ log.warning(
+ "realtime[%s] could not answer pending function call %s "
+ "with the interim payload",
+ self.session_id,
+ call_id,
+ exc_info=True,
+ )
+ continue
+ answered += 1
+ state.pending_tool_calls.clear()
+ state.interim_tool_reply_sent = True
+ log.info(
+ "realtime[%s] user opened a new turn while an order still runs "
+ "— answered %d pending function call(s) with the interim "
+ "payload; the result will follow as a parked follow-up "
+ "(request: %s)",
+ self.session_id,
+ len(calls),
+ safe_preview(state.user_text, max_chars=80),
+ )
+ return answered
+
async def _check_readback_fidelity(
self,
rendering: str,
@@ -8203,7 +8354,7 @@ class RealtimeVoiceSession:
for turn_id, tasks in self._delegate_tasks_by_turn.items():
if turn_id == self._turn_id or all(task.done() for task in tasks):
continue
- state = self._delegate_turns.get(turn_id)
+ state = self._running_delegate_state(turn_id)
if state is None or state.result_complete:
continue
order = str(state.user_text or "").strip()
@@ -8307,11 +8458,15 @@ class RealtimeVoiceSession:
success=turn_state.result_success,
language=str(turn_state.language or self._language),
delivery_id=turn_state.delivery_id,
+ request_text=str(turn_state.user_text or ""),
+ queued_at=time.monotonic(),
)
)
log.info(
- "realtime[%s] action result outlived its turn — queued as a follow-up",
+ "realtime[%s] action result outlived its turn — parked as a "
+ "follow-up (request: %s)",
self.session_id,
+ safe_preview(str(turn_state.user_text or ""), max_chars=80),
)
self._schedule_late_delegate_flush()
@@ -8347,32 +8502,76 @@ class RealtimeVoiceSession:
# A discarded generation is still streaming from the provider;
# a follow-up injected now would land inside it (BUG-143).
or self._stale_generation_drop_active()
- or self._delegate_tasks
+ # ADR-0034: rest is judged per DELIVERY, not per session. A
+ # background order that is still computing (another turn's
+ # delegate, no result yet) is no reason to hold a ready result;
+ # only a delivery in flight — a trusted reply handed to the
+ # provider whose readback has not landed — must not be raced.
+ or self._delegate_delivery_in_flight()
or self._pending_tool_events
or self._response_requested_for_turn
)
+ def _delegate_delivery_in_flight(self) -> bool:
+ """A trusted delegate reply is between injection and its readback.
+
+ Distinct from "a delegate task exists": the task keeps running while
+ the tool model still computes (harmless to a follow-up) and lingers in
+ the readback verifier after delivery (that phase already stands down
+ turn holds). What a follow-up must never race is the window from
+ ``delivery_started`` to ``delivery_completed`` — the provider is about
+ to render, or rendering, a result (BUG-143 double delivery class).
+ """
+ for turn_id, tasks in self._delegate_tasks_by_turn.items():
+ if not any(not task.done() for task in tasks):
+ continue
+ state = self._running_delegate_state(turn_id)
+ if state is None:
+ # An untracked delegate task: nothing proves its result is not
+ # mid-delivery, so keep the conservative pre-ADR-0034 answer.
+ return True
+ if state.delivery_started and not state.delivery_completed:
+ return True
+ return False
+
async def _flush_late_delegate_results(self) -> None:
- loop = asyncio.get_running_loop()
- deadline = loop.time() + _LATE_DELEGATE_DELIVERY_TIMEOUT_S
+ """Speak every parked result as soon as the session is at rest.
+
+ No deadline (ADR-0034): the loop polls until the queue is empty or the
+ session ends; ``end()`` re-routes whatever is still parked to the
+ detached completion channel, so nothing is lost silently. A result
+ that waits long is logged (bounded, once per minute per item) so a
+ call that never rests is visible in the log rather than mysterious.
+ """
+ logged_wait: dict[str, float] = {}
while self._late_delegate_results and not self._ended:
if self._session_is_at_rest():
pending = self._late_delegate_results[0]
if not await self._speak_late_delegate_result(pending):
+ # A torn-down wire; the session is ending or rebuilding.
+ # Leave the debt in the queue: the next flush (or ``end()``)
+ # owns it.
break
- self._late_delegate_results.pop(0)
+ if pending in self._late_delegate_results:
+ self._late_delegate_results.remove(pending)
continue
- if loop.time() >= deadline:
- break
+ oldest = self._late_delegate_results[0]
+ waited = time.monotonic() - float(oldest.queued_at or time.monotonic())
+ marker = oldest.delivery_id
+ if (
+ waited >= _LATE_DELEGATE_STILL_PARKED_LOG_S
+ and waited - logged_wait.get(marker, 0.0)
+ >= _LATE_DELEGATE_STILL_PARKED_LOG_S
+ ):
+ logged_wait[marker] = waited
+ log.info(
+ "realtime[%s] parked action result still waiting for a "
+ "quiet moment after %.0f s (request: %s)",
+ self.session_id,
+ waited,
+ safe_preview(oldest.request_text, max_chars=80),
+ )
await asyncio.sleep(_LATE_DELEGATE_POLL_S)
- for lost in self._late_delegate_results:
- # The action itself ran; only its spoken confirmation was lost.
- log.warning(
- "realtime[%s] executed action result could not be spoken: %s",
- self.session_id,
- safe_preview(lost.text, max_chars=200),
- )
- self._late_delegate_results.clear()
async def _speak_late_delegate_result(
self, pending: _LateDelegateResult
@@ -8400,6 +8599,7 @@ class RealtimeVoiceSession:
language=pending.language,
success=pending.success,
late=True,
+ request_text=pending.request_text,
)
)
except Exception: # noqa: BLE001 — a torn-down wire must not lose the log
@@ -8429,8 +8629,15 @@ class RealtimeVoiceSession:
TTS and drops the provider's freestyle response for this turn. The
late-result flush still delivers the real answer once the session is
at rest — both drop flags are cleared by that injection path.
+
+ ADR-0034: the line is grounded in the tool the running order is
+ actually executing when one is known (``ActionProposed`` →
+ ``progress_pool``: "still searching", "still on the screen"); the
+ closed bridge pool is the fallback, never a stock line for a known
+ activity. A handover (a background agent took over) says nothing —
+ the spawn reply already stated it — so the generic pool speaks then.
"""
- status_text = _pick_delegate_bridge_text(self._language)
+ status_text = self._grounded_pending_status_text()
self._response_requested_for_turn = True
self._drop_provider_output_until_user_turn = True
# Recording the line as this turn's output keeps the exported
@@ -8444,6 +8651,79 @@ class RealtimeVoiceSession:
)
await self._send_json(self._surface_speech_message(status_text))
+ def _grounded_pending_status_text(self) -> str:
+ """One progress line for a running order: grounded when a tool is known.
+
+ SEARCH / READ / SCREEN speak their own honest line; an unknown tool
+ (OTHER) and a handover fall back to the session's closed status pool
+ — the same lines BUG-070 pinned, so a probe answered before any tool
+ was proposed sounds exactly as before.
+ """
+ activity = classify_tool_activity(self._running_tool_name)
+ if activity not in (ToolActivity.OTHER, ToolActivity.HANDOVER):
+ grounded = pick_progress_text(activity, self._language)
+ if grounded:
+ return grounded
+ return _pick_delegate_bridge_text(self._language)
+
+ async def _answer_wait_query(self, text: str) -> bool:
+ """Answer "how far are you?" / "what came out of it?" deterministically.
+
+ ADR-0034. Two closed vocabularies (``classify_wait_query``), one rule:
+ the orchestrator owns the turn whenever the only correct answer is
+ known in advance (BUG-070 lesson).
+
+ * A RESULT request while a parked result is ready: the provider's
+ freestyle answer for this turn is dropped and the late-result flush
+ is nudged — it speaks the parked result the moment this probe turn
+ closes, in the live voice, tied back to the request. No canned line
+ in front of it: the user asked for the result, and a "still working"
+ line before a ready result would be a lie.
+ * A PROGRESS question, or a RESULT request while the order still
+ runs: one grounded progress line (``_speak_pending_action_status``).
+
+ Returns ``True`` when the turn was claimed. Anything the vocabulary
+ does not recognise stays with the provider.
+ """
+ kind = classify_wait_query(text)
+ if not kind:
+ return False
+ running = self._has_pending_delegate_from_earlier_turn()
+ if kind == WAIT_QUERY_RESULT and self._late_delegate_results:
+ wanted = requested_result(self._late_delegate_results, text)
+ if wanted is not None:
+ self._late_delegate_results.remove(wanted)
+ self._late_delegate_results.insert(0, wanted)
+ log.info(
+ "realtime[%s] user asked for a parked result (waited "
+ "%.0f s) — delivering it now",
+ self.session_id,
+ wanted.waited_s(),
+ )
+ if bool(
+ getattr(self._session, "creates_responses_automatically", False)
+ ):
+ # This transport is already answering the probe on its
+ # own VAD; that freestyle answer is dropped and the flush
+ # speaks the parked result the moment the probe turn
+ # closes — injecting now would race the generation.
+ self._response_requested_for_turn = True
+ self._drop_provider_output_until_user_turn = True
+ self._schedule_late_delegate_flush()
+ return True
+ # No response was requested for this turn and none will be:
+ # the parked result IS this turn's response.
+ self._late_delegate_results.remove(wanted)
+ if await self._speak_late_delegate_result(wanted):
+ return True
+ self._late_delegate_results.insert(0, wanted)
+ self._schedule_late_delegate_flush()
+ return True
+ if kind in (WAIT_QUERY_PROGRESS, WAIT_QUERY_RESULT) and running:
+ await self._speak_pending_action_status()
+ return True
+ return False
+
async def _cancel_running_delegates(self, *, reason: str) -> int:
"""Abandon every still-running delegated action. Returns the count.
@@ -8480,7 +8760,7 @@ class RealtimeVoiceSession:
if not pending and not dropped:
return 0
for turn_id in turn_ids:
- state = self._delegate_turns.get(turn_id)
+ state = self._running_delegate_state(turn_id)
if state is None:
continue
# Latch the delivery so _queue_late_delegate_result refuses this
@@ -8490,6 +8770,34 @@ class RealtimeVoiceSession:
self._delegate_delivery_status[state.delivery_id] = (
"cancelled_by_user"
)
+ # ADR-0034: a provider function call still open on the wire would
+ # keep a blocking transport waiting for an answer that now never
+ # comes. Close it honestly so the model can take the next turn.
+ if state.pending_tool_calls and self._session is not None and (
+ self._session_takes_tool_results()
+ ):
+ for call_id, wire_name in tuple(state.pending_tool_calls):
+ try:
+ await self._session.send_tool_result(
+ call_id,
+ wire_name,
+ {
+ "success": False,
+ "error": (
+ "Cancelled by the user. Do not retry it "
+ "and do not describe an outcome."
+ ),
+ },
+ )
+ except Exception: # noqa: BLE001 — the cancel itself stands
+ log.debug(
+ "realtime[%s] cancelled function call %s could not "
+ "be answered on the wire",
+ self.session_id,
+ call_id,
+ exc_info=True,
+ )
+ state.pending_tool_calls.clear()
for task in pending:
await self._cancel_and_reap(task)
log.info(
@@ -8806,7 +9114,7 @@ class RealtimeVoiceSession:
self._run_deterministic_delegate(turn_id, turn_state),
name=f"rt-deterministic-delegate-{self.session_id}",
)
- self._track_delegate_task(turn_id, task)
+ self._track_delegate_task(turn_id, task, turn_state)
previous_bridge = self._delegate_bridge_task
if previous_bridge is not None and not previous_bridge.done():
previous_bridge.cancel()
@@ -9877,7 +10185,7 @@ class RealtimeVoiceSession:
self._run_delegate(turn_id, turn_state),
name=f"rt-delegate-{self.session_id}",
)
- self._track_delegate_task(turn_id, task)
+ self._track_delegate_task(turn_id, task, turn_state)
async def _run_delegate(
self,
diff --git a/jarvis/voice/parked_results.py b/jarvis/voice/parked_results.py
index 65202ab2c3..91bc0cedb9 100644
--- a/jarvis/voice/parked_results.py
+++ b/jarvis/voice/parked_results.py
@@ -32,7 +32,7 @@ from __future__ import annotations
import re
import time
from collections import deque
-from collections.abc import Callable, Iterable
+from collections.abc import Callable, Iterable, Sequence
from dataclasses import dataclass
from typing import Final
from uuid import uuid4
@@ -77,7 +77,10 @@ class ParkedResult:
text: str
language: str
- request_text: str
+ #: The user's own words for the order this answers. Empty only for a
+ #: legacy caller that never knew the request; the re-anchoring prefix then
+ #: falls back to its topic-less form.
+ request_text: str = ""
success: bool = True
request_turn_id: str = ""
delivery_id: str = ""
@@ -173,22 +176,28 @@ class ParkedResultLedger:
return self.cancel_all()
def retrieve_for(self, query: str) -> ParkedResult | None:
- """The parked result the user is asking for, if the query names one.
-
- With exactly one parked result any result request means it. With
- several, the query has to share a content word with the request text
- of one of them; otherwise the oldest is returned — the user asked for
- "the result", and the oldest debt is the most likely one.
- """
- if not self._items:
- return None
- if len(self._items) == 1:
- return self._items[0]
- query_words = _content_words(query)
- for item in self._items:
- if query_words & _content_words(item.request_text):
- return item
- return self._items[0]
+ """The parked result the user is asking for (see :func:`requested_result`)."""
+ return requested_result(self._items, query)
+
+
+def requested_result(items: Sequence[ParkedResult], query: str) -> ParkedResult | None:
+ """The parked result a user's result request refers to, if any.
+
+ With exactly one parked result any result request means it. With several,
+ the query has to share a content word with the request text of one of
+ them; otherwise the oldest is returned — the user asked for "the result",
+ and the oldest debt is the most likely one. Shared by the ledger and by
+ engines that keep their own list.
+ """
+ if not items:
+ return None
+ if len(items) == 1:
+ return items[0]
+ query_words = _content_words(query)
+ for item in items:
+ if query_words & _content_words(item.request_text):
+ return item
+ return items[0]
# ---------------------------------------------------------------------------
@@ -428,5 +437,6 @@ __all__ = [
"anchor_pool",
"classify_wait_query",
"reanchor",
+ "requested_result",
"topic_of",
]
diff --git a/tests/unit/realtime/test_non_blocking_heavy_turns.py b/tests/unit/realtime/test_non_blocking_heavy_turns.py
new file mode 100644
index 0000000000..d12f1c6c34
--- /dev/null
+++ b/tests/unit/realtime/test_non_blocking_heavy_turns.py
@@ -0,0 +1,405 @@
+"""ADR-0034 in the realtime engine: the user keeps talking while the tool model works.
+
+Four guarantees, each pinned by one test shape:
+
+* a parked (late) result never expires — it is spoken at the next rest however
+ long the conversation went on;
+* rest is judged per delivery, not per session — a background order that is
+ still computing does not hold a ready result hostage;
+* a provider function call still open on the wire is answered with the interim
+ payload the moment the user opens a new turn (blocking transports can then
+ answer that turn), and cancelled orders close their calls too;
+* "how far are you?" / "what came out of it?" are owned by the orchestrator:
+ a grounded progress line, or the parked result itself.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import time
+
+import pytest
+
+from jarvis.realtime.protocol import RealtimeEvent
+from jarvis.realtime.session import (
+ _DELEGATE_BRIDGE_TEXTS,
+ _PENDING_TOOL_CALL_INTERIM_RESULT,
+ _delegate_result_prompt,
+ _DelegateTurnState,
+)
+from jarvis.voice.instant_ack import ToolActivity, progress_pool
+from jarvis.voice.parked_results import ParkedResult
+from tests.unit.realtime.test_session import (
+ FakeBrain,
+ FakeProvider,
+ FakeSession,
+ _session,
+)
+
+
+def _surface_lines(jsons):
+ return [m["text"] for m in jsons if m.get("type") == "error_spoken"]
+
+
+def _all_status_lines() -> set[str]:
+ pool = {text for texts in _DELEGATE_BRIDGE_TEXTS.values() for text in texts}
+ for activity in ToolActivity:
+ for language in ("de", "en", "es"):
+ pool.update(progress_pool(activity, language))
+ return pool
+
+
+# --- no expiry -----------------------------------------------------------------
+
+
+async def test_parked_result_is_spoken_however_long_it_waited():
+ """A result parked an hour ago is still owed; the flush waits for rest, not a clock."""
+ provider = FakeProvider([])
+ sess = _session(provider, brain=FakeBrain())
+ fake = FakeSession([])
+ sess._session = fake
+ sess._late_delegate_results.append(
+ ParkedResult(
+ text="Berlin: 21 degrees and sunny.",
+ language="en",
+ success=True,
+ delivery_id="late-1",
+ request_text="what's the weather in Berlin",
+ queued_at=time.monotonic() - 3600.0,
+ )
+ )
+ # Not at rest yet: the user is talking.
+ sess._user_speech_active = True
+ flush = asyncio.create_task(sess._flush_late_delegate_results())
+ await asyncio.sleep(0.4)
+ assert fake.text_inputs == []
+ assert len(sess._late_delegate_results) == 1
+ # The floor clears → the parked result is injected, tied to its request.
+ sess._user_speech_active = False
+ await asyncio.wait_for(flush, timeout=3.0)
+ assert sess._late_delegate_results == []
+ assert len(fake.text_inputs) == 1
+ injected = fake.text_inputs[0]
+ assert "<trusted_action_result>\nBerlin: 21 degrees and sunny.\n" in injected
+ assert "what's the weather in Berlin" in injected
+ await sess.end(reason="test")
+
+
+def test_late_result_prompt_ties_back_to_the_request_text():
+ prompt = _delegate_result_prompt(
+ "Done.",
+ language="en",
+ success=True,
+ late=True,
+ request_text="send the travel plan to Anna",
+ )
+ assert "earlier request" in prompt
+ assert '"send the travel plan to Anna"' in prompt
+ plain = _delegate_result_prompt("Done.", language="en", success=True)
+ assert "earlier request" not in plain
+
+
+# --- rest is per delivery --------------------------------------------------------
+
+
+async def test_rest_ignores_a_running_order_but_not_a_delivery_in_flight():
+ provider = FakeProvider([])
+ sess = _session(provider, brain=FakeBrain())
+ sess._session = FakeSession([])
+ assert sess._session_is_at_rest()
+
+ gate = asyncio.Event()
+
+ async def _never_done():
+ await gate.wait()
+
+ task = asyncio.create_task(_never_done())
+ try:
+ state = _DelegateTurnState(user_text="write the plan to my wiki")
+ sess._delegate_turns["turn-a"] = state
+ sess._track_delegate_task("turn-a", task)
+ # Still computing, nothing handed to the provider: a follow-up may go.
+ assert sess._session_is_at_rest()
+ # A trusted reply is with the provider and its readback has not landed:
+ # a follow-up now would race it (BUG-143 class) — hold.
+ state.delivery_started = True
+ assert not sess._session_is_at_rest()
+ state.delivery_completed = True
+ assert sess._session_is_at_rest()
+ finally:
+ gate.set()
+ await task
+ await sess.end(reason="test")
+
+
+# --- the wire is freed for the new turn -----------------------------------------
+
+
+async def test_new_turn_answers_the_pending_provider_call_with_the_interim_payload():
+ """The user talks into a slow provider-requested action: the open call is closed."""
+ gate = asyncio.Event()
+ brain = FakeBrain(replies=("The travel plan is in your wiki.",), gate=gate)
+ jsons = []
+ provider = FakeProvider(
+ [
+ RealtimeEvent(
+ type="input_transcript",
+ text="Write the travel plan to my wiki.",
+ is_final=True,
+ ),
+ RealtimeEvent(
+ type="tool_call",
+ call_id="call-1",
+ tool_name="jarvis_action",
+ tool_args={"request": "Write the travel plan to my wiki."},
+ ),
+ RealtimeEvent(type="interrupted"),
+ RealtimeEvent(
+ type="input_transcript",
+ text="By the way, what is the capital of Portugal?",
+ is_final=True,
+ ),
+ ]
+ )
+ sess = _session(provider, brain=brain, jsons=jsons)
+
+ await sess.handle_control({"type": "audio_start", "sample_rate": 16_000})
+ await sess.wait_finished()
+ await asyncio.sleep(0.05)
+
+ interim = [r for r in provider.session.tool_results if r[0] == "call-1"]
+ assert len(interim) == 1
+ assert interim[0][2]["status"] == "in_progress"
+ assert interim[0][2] == _PENDING_TOOL_CALL_INTERIM_RESULT
+ # The new turn itself was NOT dispatched to the tool model and NOT answered
+ # with a canned line: it stays native.
+ assert all("Portugal" not in call[0] for call in brain.calls)
+ assert _surface_lines(jsons) == []
+ state = next(iter(sess._delegate_states_by_turn.values()))
+ assert state.interim_tool_reply_sent
+ assert state.pending_tool_calls == []
+
+ # The order finishes later: its result is parked, never sent as a late tool
+ # result against the answered call id.
+ gate.set()
+ await asyncio.sleep(0.1)
+ assert [r for r in provider.session.tool_results if r[0] == "call-1"] == interim
+ parked = sess._late_delegate_results
+ injected = "".join(provider.session.text_inputs)
+ assert (
+ parked and parked[0].request_text == "Write the travel plan to my wiki."
+ ) or "The travel plan is in your wiki." in injected
+ await sess.end(reason="test")
+
+
+async def test_new_turn_leaves_the_call_open_when_the_switch_is_off():
+ gate = asyncio.Event()
+ brain = FakeBrain(gate=gate)
+ provider = FakeProvider(
+ [
+ RealtimeEvent(
+ type="input_transcript", text="Write the travel plan to my wiki.", is_final=True
+ ),
+ RealtimeEvent(
+ type="tool_call",
+ call_id="call-1",
+ tool_name="jarvis_action",
+ tool_args={"request": "Write the travel plan to my wiki."},
+ ),
+ RealtimeEvent(type="interrupted"),
+ RealtimeEvent(
+ type="input_transcript",
+ text="By the way, what is the capital of Portugal?",
+ is_final=True,
+ ),
+ ]
+ )
+ sess = _session(provider, brain=brain)
+ sess._config.voice.realtime_unblock_pending_tool_calls = False
+
+ await sess.handle_control({"type": "audio_start", "sample_rate": 16_000})
+ await sess.wait_finished()
+ await asyncio.sleep(0.05)
+
+ assert provider.session.tool_results == []
+ gate.set()
+ await sess.end(reason="test")
+
+
+async def test_stop_word_closes_the_pending_provider_call_on_the_wire():
+ gate = asyncio.Event()
+ brain = FakeBrain(gate=gate)
+ jsons = []
+ provider = FakeProvider(
+ [
+ RealtimeEvent(
+ type="input_transcript", text="Write the travel plan to my wiki.", is_final=True
+ ),
+ RealtimeEvent(
+ type="tool_call",
+ call_id="call-1",
+ tool_name="jarvis_action",
+ tool_args={"request": "Write the travel plan to my wiki."},
+ ),
+ RealtimeEvent(type="interrupted"),
+ RealtimeEvent(type="input_transcript", text="Stop.", is_final=True),
+ ]
+ )
+ sess = _session(provider, brain=brain, jsons=jsons)
+
+ await sess.handle_control({"type": "audio_start", "sample_rate": 16_000})
+ await sess.wait_finished()
+ await asyncio.sleep(0.05)
+
+ answers = [r for r in provider.session.tool_results if r[0] == "call-1"]
+ assert len(answers) == 1
+ assert answers[0][2]["success"] is False
+ assert "Cancelled" in answers[0][2]["error"]
+ assert brain.cancelled
+ await sess.end(reason="test")
+
+
+# --- what the user says into the wait -----------------------------------------
+
+
+async def test_progress_question_gets_one_grounded_line_from_the_running_tool():
+ gate = asyncio.Event() # the order outlives the test
+ brain = FakeBrain(gate=gate)
+ jsons = []
+ provider = FakeProvider(
+ [
+ RealtimeEvent(
+ type="input_transcript", text="Write the travel plan to my wiki.", is_final=True
+ ),
+ RealtimeEvent(
+ type="tool_call",
+ call_id="call-1",
+ tool_name="jarvis_action",
+ tool_args={"request": "Write the travel plan to my wiki."},
+ ),
+ RealtimeEvent(type="interrupted"),
+ RealtimeEvent(type="input_transcript", text="How far are you?", is_final=True),
+ ]
+ )
+ sess = _session(provider, brain=brain, jsons=jsons)
+ sess._running_tool_name = "web_search"
+
+ await sess.handle_control({"type": "audio_start", "sample_rate": 16_000})
+ await sess.wait_finished()
+ await asyncio.sleep(0.05)
+
+ spoken = _surface_lines(jsons)
+ assert len(spoken) == 1
+ assert spoken[0] in progress_pool(ToolActivity.SEARCH, "en")
+ # The probe never became a brain turn.
+ assert all("far" not in call[0] for call in brain.calls)
+ gate.set()
+ await sess.end(reason="test")
+
+
+async def test_progress_question_falls_back_to_the_status_pool_without_a_known_tool():
+ gate = asyncio.Event()
+ brain = FakeBrain(gate=gate)
+ jsons = []
+ provider = FakeProvider(
+ [
+ RealtimeEvent(
+ type="input_transcript", text="Write the travel plan to my wiki.", is_final=True
+ ),
+ RealtimeEvent(
+ type="tool_call",
+ call_id="call-1",
+ tool_name="jarvis_action",
+ tool_args={"request": "Write the travel plan to my wiki."},
+ ),
+ RealtimeEvent(type="interrupted"),
+ RealtimeEvent(type="input_transcript", text="Are you done yet?", is_final=True),
+ ]
+ )
+ sess = _session(provider, brain=brain, jsons=jsons)
+
+ await sess.handle_control({"type": "audio_start", "sample_rate": 16_000})
+ await sess.wait_finished()
+ await asyncio.sleep(0.05)
+
+ spoken = _surface_lines(jsons)
+ assert len(spoken) == 1
+ assert spoken[0] in _all_status_lines()
+ gate.set()
+ await sess.end(reason="test")
+
+
+async def test_result_request_delivers_the_parked_result_as_this_turns_answer():
+ """No order running, one result parked: "what did you find" gets it now."""
+ jsons = []
+ provider = FakeProvider(
+ [
+ RealtimeEvent(type="input_transcript", text="What did you find?", is_final=True),
+ ]
+ )
+ sess = _session(provider, brain=FakeBrain(), jsons=jsons)
+ sess._late_delegate_results.append(
+ ParkedResult(
+ text="Rome: 21 degrees, clear sky.",
+ language="en",
+ success=True,
+ delivery_id="late-rome",
+ request_text="look up the weather in Rome",
+ queued_at=time.monotonic() - 90.0,
+ )
+ )
+
+ await sess.handle_control({"type": "audio_start", "sample_rate": 16_000})
+ await sess.wait_finished()
+ await asyncio.sleep(0.05)
+
+ injected = "".join(provider.session.text_inputs)
+ assert "<trusted_action_result>\nRome: 21 degrees, clear sky.\n" in injected
+ assert "look up the weather in Rome" in injected
+ assert sess._late_delegate_results == []
+ # No canned "still working" line in front of a ready result.
+ assert _surface_lines(jsons) == []
+ await sess.end(reason="test")
+
+
+async def test_unrelated_new_turn_stays_native_while_a_result_is_parked():
+ jsons = []
+ provider = FakeProvider(
+ [
+ RealtimeEvent(
+ type="input_transcript",
+ text="Tell me a fun fact about octopuses.",
+ is_final=True,
+ ),
+ ]
+ )
+ sess = _session(provider, brain=FakeBrain(), jsons=jsons)
+ sess._late_delegate_results.append(
+ ParkedResult(
+ text="Rome: 21 degrees.",
+ language="en",
+ success=True,
+ delivery_id="late-rome",
+ request_text="look up the weather in Rome",
+ )
+ )
+
+ await sess.handle_control({"type": "audio_start", "sample_rate": 16_000})
+ await sess.wait_finished()
+ await asyncio.sleep(0.05)
+
+ # The turn belongs to the provider: one ordinary response request, no
+ # surface line, and the parked result was not forced into this turn.
+ assert provider.session.response_requests == 1
+ assert _surface_lines(jsons) == []
+ assert not any("<trusted_action_result>" in t for t in provider.session.text_inputs)
+ await sess.end(reason="test")