Skip to content

Dodaios is an open-source agent OS for your desktop: it runs your coding agents, browser and apps by voice or chat, on your own machine. Formerly Personal Jarvis.

fix(realtime): a split utterance stays ONE order — refuse the tail's second executor

rubenluetke10-beep9d24762ffc98
diff --git a/docs/BUGS.md b/docs/BUGS.md
index 76c0ca19fd..47289c4c92 100644
--- a/docs/BUGS.md
+++ b/docs/BUGS.md
@@ -9015,3 +9015,51 @@ parser that executes speech has to model the repair marker itself, or it will
execute the draft. And when a parser has only two buckets ("fleet" / "task for
the fleet"), everything that fits neither lands in the more dangerous one:
never hand a fresh agent a task the workspace itself should have executed.
+
+## BUG-131: one spoken order briefs the same pane twice — the provider's VAD splits the utterance and the tail becomes a second executor (HIGH, FIXED 2026-08-12)
+
+**Symptoms (2026-08-12 16:09, gemini-live).** The maintainer asked, in one
+breath, for a skills deep-dive: "Can you please prompt Gemini Pro that he
+looks at our skills and makes a deep dive … It never uses the skills I've
+built. It doesn't really — you know, recognize the skills." Pane T4 then
+received TWO complete deep-dive briefs three seconds apart (16:10:40 and
+16:10:43, `prompts_sent: 2` in the workspace snapshot) — two differently
+worded expansions of the same request, each opening its own agent turn —
+while the spoken announcement claimed a single handover.
+
+**Root.** Gemini-live's VAD read the thinking pause after "It doesn't
+really" as end-of-turn, so ONE utterance arrived as two final turns. Each
+final is planned independently, and the 5-word tail carried planner evidence
+of its own (the word "skills" matches `_SKILL_RE`), so it became a second
+orchestrator turn with a second deterministic delegate. Both Brain turns saw
+the same conversation history, both correctly reconstructed the same request,
+and each briefed T4 — two blind executors for one order. The existing
+repeat-order guard (`_order_already_executing`, built for the 2026-07-27
+provider-echo shape) is deliberately scoped to a turn that asked for NOTHING
+of its own; a tail fragment WITH planner evidence walked straight past it.
+
+**Fix.** `jarvis/realtime/session.py::_continues_executing_order` — a
+refusal that needs four independent probes to agree the fragment cannot
+stand alone as a new order: an earlier turn's delegate is still executing
+without a result; the fragment carries no self-standing order evidence (no
+ACTION/MISSION/WORKSPACE reason, no addressed pane); every planner reason it
+does carry is already covered by the running order's own reasons; and it is
+fragment-short. A refused turn gets the deterministic progress line and the
+trusted result via the late flush. Enforced at BOTH doors a second executor
+can open through: the deterministic dispatch site and the provider's
+`jarvis_action` call path.
+
+**Guards:** `tests/unit/realtime/test_split_utterance_single_order.py` — the
+live transcripts verbatim against the real planner, plus the counter-cases
+that must keep their dispatch: a command-verb follow-up, a question carrying
+new evidence, an addressed pane, a long same-topic follow-up, a completed
+order (clarify/confirm flows), and the provider tool-call door.
+
+**Lesson.** A provider's VAD is a turn-splitter, not a request-splitter:
+"final" marks where the model stopped listening, not where the user stopped
+meaning. When two segments of one utterance each carry planner evidence,
+text-level deduplication cannot see the repeat — the comparison has to
+happen at the EVIDENCE level, where a segment whose reasons are a subset of
+the running order's is a continuation, never a new order. And the asymmetry
+decides the ties: a wrongly refused turn degrades honestly into "still
+working on it", while a wrongly allowed turn executes a user order twice.
diff --git a/jarvis/realtime/session.py b/jarvis/realtime/session.py
index 740b2baef4..cc15283f79 100644
--- a/jarvis/realtime/session.py
+++ b/jarvis/realtime/session.py
@@ -614,6 +614,23 @@ _PUBLIC_FACT_UNCERTAINTY: dict[str, str] = {
# are pulled back to the orchestrator; a longer utterance is a new topic.
_DELEGATE_ANSWER_MAX_TOKENS = 6
+# A trailing speech fragment can only CONTINUE an order already executing when
+# it stays within this length. Live 2026-08-12 16:09: the provider's VAD read
+# a thinking pause as end-of-turn and chopped ONE spoken request in two; the
+# 5-word tail "You know, recognize the skills." became its own turn and its
+# own second executor. A follow-up LONGER than this carries enough words to be
+# a request of its own even when every other continuation probe agrees, so it
+# keeps its dispatch.
+_CONTINUATION_FRAGMENT_MAX_TOKENS = 12
+
+# Plan reasons that make a turn a self-standing ORDER: a command verb, a
+# background mission, or an addressed workspace pane. A turn carrying any of
+# these asked for something new in its own words and must never be folded
+# into an earlier order as a continuation.
+_SELF_STANDING_ORDER_REASONS = frozenset(
+ {TurnReason.ACTION, TurnReason.MISSION, TurnReason.WORKSPACE}
+)
+
# While a delegated action still runs, the wait is silent (or worse: a scrub
# hold has just cut a running answer mid-sentence). A user speaking a bare
# "hello? are you there?" into that silence is probing whether the assistant
@@ -3586,13 +3603,34 @@ class RealtimeVoiceSession:
and input_observed
and self._delegate_required_for_turn
):
- # This branch runs on a FINAL input transcript, so the
- # utterance is provably over — no boundary wait needed.
- self._start_deterministic_delegate(
- self._last_user_text,
- input_final=True,
- turn_plan=turn_plan,
- )
+ if self._continues_executing_order(turn_plan):
+ # A provider VAD that reads a thinking pause as
+ # end-of-turn finalizes ONE spoken request as two
+ # turns; a second executor for the tail briefs
+ # the same pane twice (live 2026-08-12 16:09).
+ # The running order keeps this turn: the user
+ # hears the deterministic progress line now and
+ # the trusted result via the late flush. A later
+ # final that grows this turn into a real new
+ # order re-plans and dispatches normally.
+ self._delegate_required_for_turn = False
+ log.info(
+ "realtime[%s] refused a deterministic "
+ "dispatch that can only continue the order "
+ "already executing",
+ self.session_id,
+ )
+ if not self._response_requested_for_turn:
+ await self._speak_pending_action_status()
+ else:
+ # This branch runs on a FINAL input transcript,
+ # so the utterance is provably over — no
+ # boundary wait needed.
+ self._start_deterministic_delegate(
+ self._last_user_text,
+ input_final=True,
+ turn_plan=turn_plan,
+ )
if (
event.is_final
and input_observed
@@ -6939,6 +6977,78 @@ class RealtimeVoiceSession:
or self._late_delegate_results
)
+ def _executing_order_texts(self) -> tuple[str, ...]:
+ """User texts of earlier-turn delegates still executing, no result yet.
+
+ Deliberately excludes the CURRENT turn (its own delegate is what a
+ provider function call coalesces with) and every turn whose result is
+ already complete — a finished order that ended in a clarify question
+ must keep owning the user's short answer
+ (``_answers_open_delegate_question``), and a finished confirmation
+ must keep owning the "yes" (``_brain_awaits_voice_confirm``).
+ """
+ texts: list[str] = []
+ for turn_id, tasks in self._delegate_tasks_by_turn.items():
+ if turn_id == self._turn_id or all(task.done() for task in tasks):
+ continue
+ state = self._delegate_turns.get(turn_id)
+ if state is None or state.result_complete:
+ continue
+ order = str(state.user_text or "").strip()
+ if order:
+ texts.append(order)
+ return tuple(texts)
+
+ def _continues_executing_order(self, turn_plan: TurnPlan) -> bool:
+ """True when this final can only CONTINUE the order already executing.
+
+ The live 2026-08-12 16:09 failure in one line: ONE spoken request
+ briefed the same coding pane twice. The provider's VAD read a
+ thinking pause as end-of-turn, so "…the skill system doesn't work
+ properly. It doesn't really — you know, recognize the skills" became
+ TWO turns. The first dispatched its deterministic delegate; the
+ 5-word tail then planned as an orchestrator turn of its own (the
+ word "skills" is planner evidence), opened a SECOND delegate, and
+ both executors briefed pane T4 with the same deep-dive three seconds
+ apart. ``_order_already_executing`` never saw it: that guard is for
+ a turn that asked for NOTHING of its own, and the tail carried a
+ planner reason.
+
+ Refusal therefore needs FOUR independent probes to agree that the
+ fragment cannot stand alone as a new order:
+
+ 1. an earlier turn's order is still executing without a result
+ (``_executing_order_texts``) — a completed order, including one
+ awaiting a clarify answer or a confirmation, never captures the
+ next turn;
+ 2. the fragment carries no self-standing order evidence: no command
+ verb, no mission, no addressed pane
+ (``_SELF_STANDING_ORDER_REASONS``, plus the workspace's own
+ ``owns_turn`` sweep) — "and turn on the lights" stays a real
+ second order;
+ 3. every planner reason the fragment DOES carry is already covered
+ by the running order's own reasons — "what's on my calendar?"
+ spoken while an email check runs brings CURRENT/PRIVATE evidence
+ of its own and keeps its dispatch;
+ 4. the fragment is short (``_CONTINUATION_FRAGMENT_MAX_TOKENS``) — a
+ long same-topic follow-up carries new content by sheer length.
+
+ A wrongly refused turn degrades honestly (the deterministic progress
+ line now, the trusted result via the late flush); a wrongly allowed
+ turn executes a user order TWICE. The asymmetry decides the ties.
+ """
+ text = str(self._last_user_text or "").strip()
+ if not text or len(text.split()) > _CONTINUATION_FRAGMENT_MAX_TOKENS:
+ return False
+ if turn_plan.reasons & _SELF_STANDING_ORDER_REASONS:
+ return False
+ if self._workspace_owns_turn(text):
+ return False
+ return any(
+ turn_plan.reasons <= self._plan_turn(order_text).reasons
+ for order_text in self._executing_order_texts()
+ )
+
def _queue_late_delegate_result(self, turn_state: _DelegateTurnState) -> None:
"""Keep a trusted result whose turn closed before the action finished.
@@ -7154,7 +7264,15 @@ class RealtimeVoiceSession:
},
)
return
- if self._order_already_executing(local_plan):
+ # Two disjoint repeat shapes share one refusal: a turn that asked
+ # for nothing of its own (the provider re-answering an old order,
+ # ``_order_already_executing``), and a turn that IS a fragment of
+ # the executing order itself (a provider VAD chopped one request
+ # in two and the tail carries planner evidence,
+ # ``_continues_executing_order``).
+ if self._order_already_executing(local_plan) or (
+ self._continues_executing_order(local_plan)
+ ):
log.info(
"realtime[%s] refused a delegate call that repeats an "
"order already executing",
diff --git a/tests/unit/realtime/test_split_utterance_single_order.py b/tests/unit/realtime/test_split_utterance_single_order.py
new file mode 100644
index 0000000000..525f2c9993
--- /dev/null
+++ b/tests/unit/realtime/test_split_utterance_single_order.py
@@ -0,0 +1,253 @@
+"""One spoken request chopped in two by the provider stays ONE order.
+
+Live 2026-08-12 16:09 (gemini-live): the provider's VAD read a thinking pause
+as end-of-turn, so "…the skill system doesn't work properly. It doesn't
+really — you know, recognize the skills" arrived as TWO final turns. The
+first dispatched its deterministic delegate; the 5-word tail then planned as
+an orchestrator turn of its own (the word "skills" is planner evidence),
+opened a SECOND executor, and pane T4 was briefed with the same deep-dive
+twice, three seconds apart. ``_order_already_executing`` never fired: that
+guard covers a turn that asked for NOTHING of its own, and the tail carried
+a planner reason.
+
+Pinned here: ``_continues_executing_order`` refuses the second executor for
+a short fragment whose evidence the running order already covers, while every
+shape that IS a new order — a command verb, new evidence, an addressed pane,
+sheer length, or no running order at all — keeps its dispatch. The live
+transcripts are used verbatim against the real planner.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from types import SimpleNamespace
+from typing import Any
+
+from jarvis.realtime.session import (
+ _DELEGATE_DECLARATION,
+ RealtimeVoiceSession,
+ _DelegateTurnState,
+)
+
+RATE = 24_000
+
+# The live 2026-08-12 16:09 transcripts, verbatim (session
+# 5f6eb2eb-2966-49ce-bc64-b85273c25266, turns 0 and 1).
+LIVE_ORDER = (
+ "Can you please prompt Gemini Pro that he looks at our skills and makes "
+ "a deep dive and how we can improve the skills drastically? I have the "
+ "feeling that the skill system doesn't work properly. It never uses the "
+ "skills I've built. It doesn't really"
+)
+LIVE_FRAGMENT = "You know, recognize the skills."
+
+
+class _IdleWire:
+ """Wire shape for tests that never start the receive pump."""
+
+ session_id = "split-utterance-wire"
+ supports_tool_updates = True
+ creates_responses_automatically = True
+ isolates_response_generations = True
+
+ def __init__(self) -> None:
+ self.tool_results: list[tuple[str, str, dict[str, Any]]] = []
+
+ async def send_tool_result(self, call_id: str, name: str, payload: dict[str, Any]) -> None:
+ self.tool_results.append((call_id, name, payload))
+
+ async def send_json(self, *_args: Any) -> None:
+ return None
+
+
+class _IdleProvider:
+ name = "split-utterance"
+ supports_realtime = True
+ supports_direct_tools = True
+ input_sample_rate = RATE
+ output_sample_rate = RATE
+
+ async def can_open_duplex_session(self) -> bool:
+ return True
+
+ async def open_session(self, _config: Any) -> _IdleWire:
+ return _IdleWire()
+
+
+def _config() -> SimpleNamespace:
+ return SimpleNamespace(
+ brain=SimpleNamespace(reply_language="en", providers={}),
+ stt=SimpleNamespace(language="auto"),
+ voice=SimpleNamespace(mode="realtime", realtime_tool_mode="delegate"),
+ latency=SimpleNamespace(enabled=False),
+ )
+
+
+def _session(messages: list[dict[str, Any]] | None = None) -> RealtimeVoiceSession:
+ sink = messages if messages is not None else []
+ session = RealtimeVoiceSession(
+ session_id="split-utterance-single-order",
+ send_binary=lambda _data: asyncio.sleep(0),
+ send_json=lambda message: sink.append(message) or asyncio.sleep(0),
+ providers=[_IdleProvider()],
+ config=_config(),
+ bus=None,
+ surface="desktop",
+ half_duplex=True,
+ browser_sample_rate=RATE,
+ # No ``plan_turn`` attribute: ``_plan_turn`` falls back to the REAL
+ # shared planner, so these tests exercise the production vocabulary
+ # on the live transcripts instead of a scripted plan.
+ brain=SimpleNamespace(conversation_language="en"),
+ )
+ session._language = "en"
+ return session
+
+
+def _pending_order(
+ session: RealtimeVoiceSession, text: str, turn_id: str = "turn-a"
+) -> asyncio.Task[None]:
+ """Model an earlier turn's delegate that is still executing."""
+ state = _DelegateTurnState(deterministic=True, user_text=text)
+ session._delegate_turns[turn_id] = state
+ task = asyncio.get_running_loop().create_task(asyncio.sleep(60))
+ session._delegate_tasks_by_turn[turn_id] = {task}
+ return task
+
+
+def _guard(session: RealtimeVoiceSession, fragment: str) -> bool:
+ session._turn_id = "turn-b"
+ session._last_user_text = fragment
+ return session._continues_executing_order(session._plan_turn(fragment))
+
+
+async def test_the_live_split_utterance_opens_no_second_executor() -> None:
+ session = _session()
+ task = _pending_order(session, LIVE_ORDER)
+ try:
+ assert _guard(session, LIVE_FRAGMENT), (
+ "the live tail fragment must be refused as a continuation — "
+ "dispatching it briefed pane T4 twice on 2026-08-12"
+ )
+ finally:
+ task.cancel()
+
+
+async def test_a_command_verb_follow_up_keeps_its_dispatch() -> None:
+ # The running order ALSO plans as a plain action, so only the
+ # self-standing-order probe (a command verb of its own) protects the
+ # follow-up here — the reason-subset probe alone would refuse it.
+ session = _session()
+ task = _pending_order(session, "Please open the garage and switch the heating to eco mode")
+ try:
+ assert not _guard(session, "And turn on the lights."), (
+ "a short follow-up carrying its own command verb is a second "
+ "order, never a continuation"
+ )
+ finally:
+ task.cancel()
+
+
+async def test_new_evidence_keeps_its_dispatch() -> None:
+ session = _session()
+ task = _pending_order(session, LIVE_ORDER)
+ try:
+ assert not _guard(session, "What is on my calendar today?"), (
+ "a question bringing evidence the running order never carried "
+ "(calendar/current) is a new request and keeps its dispatch"
+ )
+ finally:
+ task.cancel()
+
+
+async def test_an_addressed_pane_keeps_its_dispatch(monkeypatch: Any) -> None:
+ session = _session()
+ monkeypatch.setattr(
+ RealtimeVoiceSession,
+ "_workspace_call_signs",
+ staticmethod(lambda: ("Dana",)),
+ )
+ task = _pending_order(session, LIVE_ORDER)
+ try:
+ assert not _guard(session, "Dana should look at the skills too."), (
+ "naming an open pane is as strong as order evidence gets — the "
+ "workspace turn must dispatch even while another order runs"
+ )
+ finally:
+ task.cancel()
+
+
+async def test_a_long_same_topic_follow_up_keeps_its_dispatch() -> None:
+ session = _session()
+ task = _pending_order(session, LIVE_ORDER)
+ long_follow_up = (
+ "You know I want you to really recognize every single skill I have built for this system"
+ )
+ try:
+ assert not _guard(session, long_follow_up), (
+ "a follow-up past the fragment length bound carries new content "
+ "by sheer length and keeps its dispatch"
+ )
+ finally:
+ task.cancel()
+
+
+async def test_a_completed_order_never_captures_the_next_turn() -> None:
+ # The clarify-answer and voice-confirm flows depend on this: once the
+ # earlier delegate has its result, a short reply belongs to THOSE paths,
+ # not to the continuation refusal.
+ session = _session()
+ task = _pending_order(session, LIVE_ORDER)
+ session._delegate_turns["turn-a"].result_complete = True
+ try:
+ assert not _guard(session, LIVE_FRAGMENT)
+ finally:
+ task.cancel()
+
+ done: asyncio.Task[None] = asyncio.get_running_loop().create_task(asyncio.sleep(0))
+ await done
+ session._delegate_turns["turn-a"].result_complete = False
+ session._delegate_tasks_by_turn["turn-a"] = {done}
+ assert not _guard(session, LIVE_FRAGMENT)
+
+
+async def test_no_running_order_means_no_refusal() -> None:
+ session = _session()
+ assert not _guard(session, LIVE_FRAGMENT)
+
+
+async def test_the_provider_tool_call_path_refuses_the_fragment_too() -> None:
+ """The same fragment arriving as a provider ``jarvis_action`` call.
+
+ It plans as requiring the orchestrator, so ``_order_already_executing``
+ (built for a turn that asked nothing of its own) lets it through — the
+ continuation guard must cover this door as well.
+ """
+ messages: list[dict[str, Any]] = []
+ session = _session(messages)
+ wire = _IdleWire()
+ session._session = wire
+ session._delegate_enabled = True
+ task = _pending_order(session, LIVE_ORDER)
+ session._turn_id = "turn-b"
+ session._last_user_text = LIVE_FRAGMENT
+ event = SimpleNamespace(
+ call_id="call-1",
+ tool_name=str(_DELEGATE_DECLARATION["name"]),
+ tool_args={"request": LIVE_FRAGMENT},
+ )
+ try:
+ await session._handle_tool_call(event)
+ finally:
+ task.cancel()
+
+ assert len(wire.tool_results) == 1
+ _call_id, _name, payload = wire.tool_results[0]
+ assert payload["success"] is False
+ assert "Do not start it again" in payload["error"]
+ assert "turn-b" not in session._delegate_tasks_by_turn, (
+ "the refused call must not have started a second executor"
+ )
+ assert any(m.get("type") for m in messages), (
+ "the refusal must not leave a silent turn — the deterministic progress line answers it"
+ )