Dodaios is an open-source agent OS for your desktop: it runs your coding agents, browser and apps by voice or chat, on your own machine. Formerly Personal Jarvis.
Python agent-orchestrationai-agentai-coding-assistantclaude-codecoding-agentcomputer-usedesktop-applocal-firstmcppersonal-jarvisvoice-assistantvoice-coding
fix(realtime): a split utterance stays ONE order — refuse the tail's second executor
rubenluetke10-beep9d24762ffc98
diff --git a/docs/BUGS.md b/docs/BUGS.mdindex 76c0ca19fd..47289c4c92 100644--- a/docs/BUGS.md+++ b/docs/BUGS.md@@ -9015,3 +9015,51 @@ parser that executes speech has to model the repair marker itself, or it willexecute the draft. And when a parser has only two buckets ("fleet" / "task forthe fleet"), everything that fits neither lands in the more dangerous one:never hand a fresh agent a task the workspace itself should have executed.++## BUG-131: one spoken order briefs the same pane twice — the provider's VAD splits the utterance and the tail becomes a second executor (HIGH, FIXED 2026-08-12)++**Symptoms (2026-08-12 16:09, gemini-live).** The maintainer asked, in one+breath, for a skills deep-dive: "Can you please prompt Gemini Pro that he+looks at our skills and makes a deep dive … It never uses the skills I've+built. It doesn't really — you know, recognize the skills." Pane T4 then+received TWO complete deep-dive briefs three seconds apart (16:10:40 and+16:10:43, `prompts_sent: 2` in the workspace snapshot) — two differently+worded expansions of the same request, each opening its own agent turn —+while the spoken announcement claimed a single handover.++**Root.** Gemini-live's VAD read the thinking pause after "It doesn't+really" as end-of-turn, so ONE utterance arrived as two final turns. Each+final is planned independently, and the 5-word tail carried planner evidence+of its own (the word "skills" matches `_SKILL_RE`), so it became a second+orchestrator turn with a second deterministic delegate. Both Brain turns saw+the same conversation history, both correctly reconstructed the same request,+and each briefed T4 — two blind executors for one order. The existing+repeat-order guard (`_order_already_executing`, built for the 2026-07-27+provider-echo shape) is deliberately scoped to a turn that asked for NOTHING+of its own; a tail fragment WITH planner evidence walked straight past it.++**Fix.** `jarvis/realtime/session.py::_continues_executing_order` — a+refusal that needs four independent probes to agree the fragment cannot+stand alone as a new order: an earlier turn's delegate is still executing+without a result; the fragment carries no self-standing order evidence (no+ACTION/MISSION/WORKSPACE reason, no addressed pane); every planner reason it+does carry is already covered by the running order's own reasons; and it is+fragment-short. A refused turn gets the deterministic progress line and the+trusted result via the late flush. Enforced at BOTH doors a second executor+can open through: the deterministic dispatch site and the provider's+`jarvis_action` call path.++**Guards:** `tests/unit/realtime/test_split_utterance_single_order.py` — the+live transcripts verbatim against the real planner, plus the counter-cases+that must keep their dispatch: a command-verb follow-up, a question carrying+new evidence, an addressed pane, a long same-topic follow-up, a completed+order (clarify/confirm flows), and the provider tool-call door.++**Lesson.** A provider's VAD is a turn-splitter, not a request-splitter:+"final" marks where the model stopped listening, not where the user stopped+meaning. When two segments of one utterance each carry planner evidence,+text-level deduplication cannot see the repeat — the comparison has to+happen at the EVIDENCE level, where a segment whose reasons are a subset of+the running order's is a continuation, never a new order. And the asymmetry+decides the ties: a wrongly refused turn degrades honestly into "still+working on it", while a wrongly allowed turn executes a user order twice.diff --git a/jarvis/realtime/session.py b/jarvis/realtime/session.pyindex 740b2baef4..cc15283f79 100644--- a/jarvis/realtime/session.py+++ b/jarvis/realtime/session.py@@ -614,6 +614,23 @@ _PUBLIC_FACT_UNCERTAINTY: dict[str, str] = {# are pulled back to the orchestrator; a longer utterance is a new topic._DELEGATE_ANSWER_MAX_TOKENS = 6+# A trailing speech fragment can only CONTINUE an order already executing when+# it stays within this length. Live 2026-08-12 16:09: the provider's VAD read+# a thinking pause as end-of-turn and chopped ONE spoken request in two; the+# 5-word tail "You know, recognize the skills." became its own turn and its+# own second executor. A follow-up LONGER than this carries enough words to be+# a request of its own even when every other continuation probe agrees, so it+# keeps its dispatch.+_CONTINUATION_FRAGMENT_MAX_TOKENS = 12++# Plan reasons that make a turn a self-standing ORDER: a command verb, a+# background mission, or an addressed workspace pane. A turn carrying any of+# these asked for something new in its own words and must never be folded+# into an earlier order as a continuation.+_SELF_STANDING_ORDER_REASONS = frozenset(+ {TurnReason.ACTION, TurnReason.MISSION, TurnReason.WORKSPACE}+)+# While a delegated action still runs, the wait is silent (or worse: a scrub# hold has just cut a running answer mid-sentence). A user speaking a bare# "hello? are you there?" into that silence is probing whether the assistant@@ -3586,13 +3603,34 @@ class RealtimeVoiceSession:and input_observedand self._delegate_required_for_turn):- # This branch runs on a FINAL input transcript, so the- # utterance is provably over — no boundary wait needed.- self._start_deterministic_delegate(- self._last_user_text,- input_final=True,- turn_plan=turn_plan,- )+ if self._continues_executing_order(turn_plan):+ # A provider VAD that reads a thinking pause as+ # end-of-turn finalizes ONE spoken request as two+ # turns; a second executor for the tail briefs+ # the same pane twice (live 2026-08-12 16:09).+ # The running order keeps this turn: the user+ # hears the deterministic progress line now and+ # the trusted result via the late flush. A later+ # final that grows this turn into a real new+ # order re-plans and dispatches normally.+ self._delegate_required_for_turn = False+ log.info(+ "realtime[%s] refused a deterministic "+ "dispatch that can only continue the order "+ "already executing",+ self.session_id,+ )+ if not self._response_requested_for_turn:+ await self._speak_pending_action_status()+ else:+ # This branch runs on a FINAL input transcript,+ # so the utterance is provably over — no+ # boundary wait needed.+ self._start_deterministic_delegate(+ self._last_user_text,+ input_final=True,+ turn_plan=turn_plan,+ )if (event.is_finaland input_observed@@ -6939,6 +6977,78 @@ class RealtimeVoiceSession:or self._late_delegate_results)+ def _executing_order_texts(self) -> tuple[str, ...]:+ """User texts of earlier-turn delegates still executing, no result yet.++ Deliberately excludes the CURRENT turn (its own delegate is what a+ provider function call coalesces with) and every turn whose result is+ already complete — a finished order that ended in a clarify question+ must keep owning the user's short answer+ (``_answers_open_delegate_question``), and a finished confirmation+ must keep owning the "yes" (``_brain_awaits_voice_confirm``).+ """+ texts: list[str] = []+ for turn_id, tasks in self._delegate_tasks_by_turn.items():+ if turn_id == self._turn_id or all(task.done() for task in tasks):+ continue+ state = self._delegate_turns.get(turn_id)+ if state is None or state.result_complete:+ continue+ order = str(state.user_text or "").strip()+ if order:+ texts.append(order)+ return tuple(texts)++ def _continues_executing_order(self, turn_plan: TurnPlan) -> bool:+ """True when this final can only CONTINUE the order already executing.++ The live 2026-08-12 16:09 failure in one line: ONE spoken request+ briefed the same coding pane twice. The provider's VAD read a+ thinking pause as end-of-turn, so "…the skill system doesn't work+ properly. It doesn't really — you know, recognize the skills" became+ TWO turns. The first dispatched its deterministic delegate; the+ 5-word tail then planned as an orchestrator turn of its own (the+ word "skills" is planner evidence), opened a SECOND delegate, and+ both executors briefed pane T4 with the same deep-dive three seconds+ apart. ``_order_already_executing`` never saw it: that guard is for+ a turn that asked for NOTHING of its own, and the tail carried a+ planner reason.++ Refusal therefore needs FOUR independent probes to agree that the+ fragment cannot stand alone as a new order:++ 1. an earlier turn's order is still executing without a result+ (``_executing_order_texts``) — a completed order, including one+ awaiting a clarify answer or a confirmation, never captures the+ next turn;+ 2. the fragment carries no self-standing order evidence: no command+ verb, no mission, no addressed pane+ (``_SELF_STANDING_ORDER_REASONS``, plus the workspace's own+ ``owns_turn`` sweep) — "and turn on the lights" stays a real+ second order;+ 3. every planner reason the fragment DOES carry is already covered+ by the running order's own reasons — "what's on my calendar?"+ spoken while an email check runs brings CURRENT/PRIVATE evidence+ of its own and keeps its dispatch;+ 4. the fragment is short (``_CONTINUATION_FRAGMENT_MAX_TOKENS``) — a+ long same-topic follow-up carries new content by sheer length.++ A wrongly refused turn degrades honestly (the deterministic progress+ line now, the trusted result via the late flush); a wrongly allowed+ turn executes a user order TWICE. The asymmetry decides the ties.+ """+ text = str(self._last_user_text or "").strip()+ if not text or len(text.split()) > _CONTINUATION_FRAGMENT_MAX_TOKENS:+ return False+ if turn_plan.reasons & _SELF_STANDING_ORDER_REASONS:+ return False+ if self._workspace_owns_turn(text):+ return False+ return any(+ turn_plan.reasons <= self._plan_turn(order_text).reasons+ for order_text in self._executing_order_texts()+ )+def _queue_late_delegate_result(self, turn_state: _DelegateTurnState) -> None:"""Keep a trusted result whose turn closed before the action finished.@@ -7154,7 +7264,15 @@ class RealtimeVoiceSession:},)return- if self._order_already_executing(local_plan):+ # Two disjoint repeat shapes share one refusal: a turn that asked+ # for nothing of its own (the provider re-answering an old order,+ # ``_order_already_executing``), and a turn that IS a fragment of+ # the executing order itself (a provider VAD chopped one request+ # in two and the tail carries planner evidence,+ # ``_continues_executing_order``).+ if self._order_already_executing(local_plan) or (+ self._continues_executing_order(local_plan)+ ):log.info("realtime[%s] refused a delegate call that repeats an ""order already executing",diff --git a/tests/unit/realtime/test_split_utterance_single_order.py b/tests/unit/realtime/test_split_utterance_single_order.pynew file mode 100644index 0000000000..525f2c9993--- /dev/null+++ b/tests/unit/realtime/test_split_utterance_single_order.py@@ -0,0 +1,253 @@+"""One spoken request chopped in two by the provider stays ONE order.++Live 2026-08-12 16:09 (gemini-live): the provider's VAD read a thinking pause+as end-of-turn, so "…the skill system doesn't work properly. It doesn't+really — you know, recognize the skills" arrived as TWO final turns. The+first dispatched its deterministic delegate; the 5-word tail then planned as+an orchestrator turn of its own (the word "skills" is planner evidence),+opened a SECOND executor, and pane T4 was briefed with the same deep-dive+twice, three seconds apart. ``_order_already_executing`` never fired: that+guard covers a turn that asked for NOTHING of its own, and the tail carried+a planner reason.++Pinned here: ``_continues_executing_order`` refuses the second executor for+a short fragment whose evidence the running order already covers, while every+shape that IS a new order — a command verb, new evidence, an addressed pane,+sheer length, or no running order at all — keeps its dispatch. The live+transcripts are used verbatim against the real planner.+"""++from __future__ import annotations++import asyncio+from types import SimpleNamespace+from typing import Any++from jarvis.realtime.session import (+ _DELEGATE_DECLARATION,+ RealtimeVoiceSession,+ _DelegateTurnState,+)++RATE = 24_000++# The live 2026-08-12 16:09 transcripts, verbatim (session+# 5f6eb2eb-2966-49ce-bc64-b85273c25266, turns 0 and 1).+LIVE_ORDER = (+ "Can you please prompt Gemini Pro that he looks at our skills and makes "+ "a deep dive and how we can improve the skills drastically? I have the "+ "feeling that the skill system doesn't work properly. It never uses the "+ "skills I've built. It doesn't really"+)+LIVE_FRAGMENT = "You know, recognize the skills."+++class _IdleWire:+ """Wire shape for tests that never start the receive pump."""++ session_id = "split-utterance-wire"+ supports_tool_updates = True+ creates_responses_automatically = True+ isolates_response_generations = True++ def __init__(self) -> None:+ self.tool_results: list[tuple[str, str, dict[str, Any]]] = []++ async def send_tool_result(self, call_id: str, name: str, payload: dict[str, Any]) -> None:+ self.tool_results.append((call_id, name, payload))++ async def send_json(self, *_args: Any) -> None:+ return None+++class _IdleProvider:+ name = "split-utterance"+ supports_realtime = True+ supports_direct_tools = True+ input_sample_rate = RATE+ output_sample_rate = RATE++ async def can_open_duplex_session(self) -> bool:+ return True++ async def open_session(self, _config: Any) -> _IdleWire:+ return _IdleWire()+++def _config() -> SimpleNamespace:+ return SimpleNamespace(+ brain=SimpleNamespace(reply_language="en", providers={}),+ stt=SimpleNamespace(language="auto"),+ voice=SimpleNamespace(mode="realtime", realtime_tool_mode="delegate"),+ latency=SimpleNamespace(enabled=False),+ )+++def _session(messages: list[dict[str, Any]] | None = None) -> RealtimeVoiceSession:+ sink = messages if messages is not None else []+ session = RealtimeVoiceSession(+ session_id="split-utterance-single-order",+ send_binary=lambda _data: asyncio.sleep(0),+ send_json=lambda message: sink.append(message) or asyncio.sleep(0),+ providers=[_IdleProvider()],+ config=_config(),+ bus=None,+ surface="desktop",+ half_duplex=True,+ browser_sample_rate=RATE,+ # No ``plan_turn`` attribute: ``_plan_turn`` falls back to the REAL+ # shared planner, so these tests exercise the production vocabulary+ # on the live transcripts instead of a scripted plan.+ brain=SimpleNamespace(conversation_language="en"),+ )+ session._language = "en"+ return session+++def _pending_order(+ session: RealtimeVoiceSession, text: str, turn_id: str = "turn-a"+) -> asyncio.Task[None]:+ """Model an earlier turn's delegate that is still executing."""+ state = _DelegateTurnState(deterministic=True, user_text=text)+ session._delegate_turns[turn_id] = state+ task = asyncio.get_running_loop().create_task(asyncio.sleep(60))+ session._delegate_tasks_by_turn[turn_id] = {task}+ return task+++def _guard(session: RealtimeVoiceSession, fragment: str) -> bool:+ session._turn_id = "turn-b"+ session._last_user_text = fragment+ return session._continues_executing_order(session._plan_turn(fragment))+++async def test_the_live_split_utterance_opens_no_second_executor() -> None:+ session = _session()+ task = _pending_order(session, LIVE_ORDER)+ try:+ assert _guard(session, LIVE_FRAGMENT), (+ "the live tail fragment must be refused as a continuation — "+ "dispatching it briefed pane T4 twice on 2026-08-12"+ )+ finally:+ task.cancel()+++async def test_a_command_verb_follow_up_keeps_its_dispatch() -> None:+ # The running order ALSO plans as a plain action, so only the+ # self-standing-order probe (a command verb of its own) protects the+ # follow-up here — the reason-subset probe alone would refuse it.+ session = _session()+ task = _pending_order(session, "Please open the garage and switch the heating to eco mode")+ try:+ assert not _guard(session, "And turn on the lights."), (+ "a short follow-up carrying its own command verb is a second "+ "order, never a continuation"+ )+ finally:+ task.cancel()+++async def test_new_evidence_keeps_its_dispatch() -> None:+ session = _session()+ task = _pending_order(session, LIVE_ORDER)+ try:+ assert not _guard(session, "What is on my calendar today?"), (+ "a question bringing evidence the running order never carried "+ "(calendar/current) is a new request and keeps its dispatch"+ )+ finally:+ task.cancel()+++async def test_an_addressed_pane_keeps_its_dispatch(monkeypatch: Any) -> None:+ session = _session()+ monkeypatch.setattr(+ RealtimeVoiceSession,+ "_workspace_call_signs",+ staticmethod(lambda: ("Dana",)),+ )+ task = _pending_order(session, LIVE_ORDER)+ try:+ assert not _guard(session, "Dana should look at the skills too."), (+ "naming an open pane is as strong as order evidence gets — the "+ "workspace turn must dispatch even while another order runs"+ )+ finally:+ task.cancel()+++async def test_a_long_same_topic_follow_up_keeps_its_dispatch() -> None:+ session = _session()+ task = _pending_order(session, LIVE_ORDER)+ long_follow_up = (+ "You know I want you to really recognize every single skill I have built for this system"+ )+ try:+ assert not _guard(session, long_follow_up), (+ "a follow-up past the fragment length bound carries new content "+ "by sheer length and keeps its dispatch"+ )+ finally:+ task.cancel()+++async def test_a_completed_order_never_captures_the_next_turn() -> None:+ # The clarify-answer and voice-confirm flows depend on this: once the+ # earlier delegate has its result, a short reply belongs to THOSE paths,+ # not to the continuation refusal.+ session = _session()+ task = _pending_order(session, LIVE_ORDER)+ session._delegate_turns["turn-a"].result_complete = True+ try:+ assert not _guard(session, LIVE_FRAGMENT)+ finally:+ task.cancel()++ done: asyncio.Task[None] = asyncio.get_running_loop().create_task(asyncio.sleep(0))+ await done+ session._delegate_turns["turn-a"].result_complete = False+ session._delegate_tasks_by_turn["turn-a"] = {done}+ assert not _guard(session, LIVE_FRAGMENT)+++async def test_no_running_order_means_no_refusal() -> None:+ session = _session()+ assert not _guard(session, LIVE_FRAGMENT)+++async def test_the_provider_tool_call_path_refuses_the_fragment_too() -> None:+ """The same fragment arriving as a provider ``jarvis_action`` call.++ It plans as requiring the orchestrator, so ``_order_already_executing``+ (built for a turn that asked nothing of its own) lets it through — the+ continuation guard must cover this door as well.+ """+ messages: list[dict[str, Any]] = []+ session = _session(messages)+ wire = _IdleWire()+ session._session = wire+ session._delegate_enabled = True+ task = _pending_order(session, LIVE_ORDER)+ session._turn_id = "turn-b"+ session._last_user_text = LIVE_FRAGMENT+ event = SimpleNamespace(+ call_id="call-1",+ tool_name=str(_DELEGATE_DECLARATION["name"]),+ tool_args={"request": LIVE_FRAGMENT},+ )+ try:+ await session._handle_tool_call(event)+ finally:+ task.cancel()++ assert len(wire.tool_results) == 1+ _call_id, _name, payload = wire.tool_results[0]+ assert payload["success"] is False+ assert "Do not start it again" in payload["error"]+ assert "turn-b" not in session._delegate_tasks_by_turn, (+ "the refused call must not have started a second executor"+ )+ assert any(m.get("type") for m in messages), (+ "the refusal must not leave a silent turn — the deterministic progress line answers it"+ )