Dodaios is an open-source agent OS for your desktop: it runs your coding agents, browser and apps by voice or chat, on your own machine. Formerly Personal Jarvis.
Python agent-orchestrationai-agentai-coding-assistantclaude-codecoding-agentcomputer-usedesktop-applocal-firstmcppersonal-jarvisvoice-assistantvoice-coding
fix(music): a blind media key is an attempt, not a result — the voice no longer announces music that never started
rubenluetke10-beep7c94f7b9d225
diff --git a/docs/BUGS.md b/docs/BUGS.mdindex 58a7591028..4cf2c19522 100644--- a/docs/BUGS.md+++ b/docs/BUGS.md@@ -13045,3 +13045,82 @@ and five sentences ending in dictionary words that must survive),whatever goes into a Whisper prompt needs a guard on the way out. And "oneletter apart" is a property of spelling — the person who registers a word isdescribing how it sounds.++## BUG-186: "There you go, I've just started some cool music on YouTube Music" — over a silent machine. The tool pressed a media key blindly, called that a resume, and the voice sold it as done (HIGH, FIXED 2026-08-26)++**Symptom.** Gemini Live, 2026-08-26 19:50, "just use like the Google Music+plugin and play some cool music." The trace shows `youtube_music · play` in+0.0 s, then the voice: "One moment, let's get that music playing for you.+There you go, I've just started some cool music on YouTube Music. Just so+you know, that just toggles play or pause …". No browser opened, nothing+played. The maintainer's reading: the assistant confirms things it did not+do.++**Root cause.** Three layers agreed on a lie, and the model was the most+honest of them.++1. The model called `play` without a title (`{"action": "play", "type":+ "song"}`) — the schema says a bare `play` resumes what is paused, which is+ a fair reading of "play some music" when nothing names a song.+2. `youtube_music_rest._control("play")` on this box: the Windows media+ session cannot READ (`winrt-*` was added to the `[desktop]` extras after+ the app's interpreter was last installed — `pywebview`, `pystray`, `pycaw`+ present, `winrt` absent), so `cap.can_read` was false, no "is anything+ paused" check ran, the play/pause TOGGLE key went to whatever the OS+ considered current, and the result was+ `{'ok': True, 'action': 'play', 'note': 'Sent as a media key: this toggles+ play/pause blindly.', 'resumed': True}` in 17 ms.+3. The system prompt forbids claiming an outcome "unless a successful+ function result explicitly supports that claim" — and this one did. The+ realtime honesty guard (`action_honesty.py`, `_has_execution_evidence`)+ only fires when NO tool ran; a tool that ran and said `ok` is evidence by+ construction. The model read the note, added its "just toggles" caveat,+ and still opened with "There you go, I've just started".++**Fix.** Truth comes from the tool; the bridge makes it audible.++* `jarvis/plugins/tool/youtube_music_rest.py` — `_control` splits at+ `cap.can_read`. A readable session (background player, WinRT, MPRIS,+ nowplaying-cli) answers `verified: true` as before. A blind machine goes+ through `_control_blind`: `play` is REFUSED unless this tool itself sent a+ blind pause just before (the key is a toggle; with nothing known paused it+ starts a random tab or stops a running one), and `play()` then answers its+ existing "say what to play" error without pressing anything. Pause, next+ and previous still go out as best effort but come back+ `verified: false` with a note that nothing on this machine can confirm the+ effect. `_open_and_confirm` and the background-player path carry+ `verified` = `playback_confirmed`; a browser opened on a machine that+ cannot read the session is `verified: false` too.+* `jarvis/realtime/tools.py` — `_bounded_result` sees+ `output["verified"] is False` on a successful call and adds+ `unverified: true` plus an `instruction` ("This result is an ATTEMPT, not a+ confirmed outcome … never say it is done, playing, running, or started.+ Offer the next step in the same breath") — the same field the model+ already obeys for confirmation questions. It survives truncation.+* `jarvis/realtime/session.py` — the tool-use system prompt names the+ contract: a result marked unverified is an attempt, not a confirmation.+* This box: the four `winrt-*` packages installed into the app's Python+ (needs an app restart to take effect); with them the plugin reads what+ plays and resumes only a real pause.++**Why not gate it in the honesty guard.** Withholding the tool from+`_executed_tool_names` would make `_recover_unbacked_action_claim` replace+the whole answer with "I did not start an action for that" — false the other+way (a key WAS sent) and a dead stop in the conversation. An unverified+result is data the model can speak honestly at full speed; the fix keeps the+turn's timing untouched (the tool still answers in milliseconds) and changes+only what is true in it.++**Regression tests.**+`tests/unit/plugins/tool/test_youtube_music_rest.py::test_blind_play_with_nothing_known_paused_presses_nothing_and_says_so`,+`::test_blind_pause_is_an_attempt_not_a_result`,+`::test_blind_play_after_a_blind_pause_is_a_resume_attempt`,+`::test_readable_session_control_is_verified`,+`::test_blind_play_of_a_song_opens_the_link_but_never_claims_playback`;+`tests/unit/realtime/test_tools.py::test_unverified_outcome_carries_the_attempt_not_outcome_instruction`,+`::test_verified_outcome_carries_no_instruction`.++**Lesson.** `ok: true` is a claim about the ACTION, and the voice speaks+about the OUTCOME. Every tool that acts without being able to look owes the+model the difference in a field it cannot miss — a note in prose is a caveat+the model will politely paraphrase after the good news.diff --git a/jarvis/plugins/tool/youtube_music_rest.py b/jarvis/plugins/tool/youtube_music_rest.pyindex da8b60ccb8..575f42692d 100644--- a/jarvis/plugins/tool/youtube_music_rest.py+++ b/jarvis/plugins/tool/youtube_music_rest.py@@ -404,6 +404,10 @@ class YouTubeMusicRestTool:self._player = playerself._playback_mode = playback_mode or _default_playback_modeself._player_confirm_timeout_s = player_confirm_timeout_s+ # Whether the last blind media key this tool sent was a pause — the+ # only state in which a blind "play" is a resume rather than a guess+ # (``_control_blind``).+ self._blind_pause_sent = Falseself._pool = HttpClientPool(transport=transport)# search.list is the rationed call (100/day). Same words → same answer# within a session, so a repeat costs nothing.@@ -558,7 +562,12 @@ class YouTubeMusicRestTool:ok, why = False, str(exc)if ok:try:- return await self._play_in_player(url, started)+ out = await self._play_in_player(url, started)+ # Only a player that reported "playing" counts as done;+ # a cookie wall, a paused page or a page still loading+ # is an attempt the voice must not sell as a result.+ out["verified"] = bool(out.get("playback_confirmed"))+ return outexcept Exception as exc: # noqa: BLE001 — player unusable → browser, and say solog.info("background player unusable, opening the browser: %s", exc)why = f"The background player could not start ({exc})."@@ -823,7 +832,12 @@ class YouTubeMusicRestTool:return {"error": f"The background player did not accept '{verb}': {exc}"}if not ok:return {"error": f"The background player did not accept '{verb}'."}- via_player: dict[str, Any] = {"ok": True, "action": verb, "app": _PLAYER_APP}+ via_player: dict[str, Any] = {+ "ok": True,+ "action": verb,+ "app": _PLAYER_APP,+ "verified": True,+ }if verb in ("next", "previous"):await asyncio.sleep(1.2)after = await self._player_state()@@ -837,27 +851,62 @@ class YouTubeMusicRestTool:cap = await media.capability()if not cap.can_control:return {"error": "This machine cannot control playback. " + (cap.note or "")}- before = await media.now_playing() if cap.can_read else None- if cap.can_read and before is None:+ if not cap.can_read:+ return await self._control_blind(media, verb, cap.note)+ before = await media.now_playing()+ if before is None:return {"error": _NO_SESSION_READ}ok = bool(await getattr(media, verb)())if not ok:return {"error": f"The player did not accept '{verb}'."}- out: dict[str, Any] = {"ok": True, "action": verb}- if before is not None:- out["app"] = before.app- if verb in ("next", "previous"):- await asyncio.sleep(0.8)- after = await media.now_playing()- if after is not None:- out["now"] = after.as_dict()- else:- out["track"] = before.title- out["artist"] = before.artist- elif cap.backend == "media-keys":- out["note"] = "Sent as a media key: this toggles play/pause blindly."+ out: dict[str, Any] = {"ok": True, "action": verb, "verified": True, "app": before.app}+ if verb in ("next", "previous"):+ await asyncio.sleep(0.8)+ after = await media.now_playing()+ if after is not None:+ out["now"] = after.as_dict()+ else:+ out["track"] = before.title+ out["artist"] = before.artistreturn out+ async def _control_blind(self, media: Any, verb: str, note: str | None) -> dict[str, Any]:+ """Steer through media keys on a machine that cannot read its media+ session (Windows without the ``winrt`` extras).++ A media key reaches whatever player the OS considers current and+ reports nothing back, so the result is an ATTEMPT and says so+ (``verified: false``): the realtime bridge turns that into a spoken+ caveat, never a completion. ``play`` is the one verb refused outright+ unless this tool itself paused something the same way — the key is+ the play/pause TOGGLE, so with nothing known to be paused it starts a+ random tab or stops a running one. Live 2026-08-26 19:50: "play some+ cool music" arrived without a title, this path pressed the toggle,+ answered ``ok: true, resumed: true``, and the voice said "I've just+ started some cool music on YouTube Music" over a silent machine.+ """+ if verb == "play" and not self._blind_pause_sent:+ return {+ "error": (+ "Nothing is known to be paused, and this machine cannot see "+ "what is playing, so play was not pressed."+ )+ }+ ok = bool(await getattr(media, verb)())+ if not ok:+ return {"error": f"The player did not accept '{verb}'."}+ self._blind_pause_sent = verb == "pause"+ return {+ "ok": True,+ "action": verb,+ "verified": False,+ "note": (+ "Sent as a media key to whichever player is current; this machine "+ "cannot confirm that anything changed. "+ + (note or "")+ ).strip(),+ }+async def pause(self) -> dict[str, Any]:return await self._control("pause")@@ -927,12 +976,23 @@ class YouTubeMusicRestTool:confirmed = Truebreakout["playback_confirmed"] = confirmed+ out["verified"] = confirmedif not confirmed:out["note"] = ("YouTube Music opened, but the browser has not reported playback ""yet — if it stays silent, press play once (browsers block ""autoplay on a site until it has been used there).")+ else:+ # The page opened; whether it plays is beyond what this machine+ # can see, so the result is an attempt (``verified: false``),+ # never "it is playing".+ out["playback_confirmed"] = False+ out["verified"] = False+ out["note"] = (+ "YouTube Music opened in the browser; this machine cannot confirm "+ "that playback started. " + (cap.note or "")+ ).strip()return outasync def play(diff --git a/jarvis/realtime/session.py b/jarvis/realtime/session.pyindex 334e11dfed..b4a6834d94 100644--- a/jarvis/realtime/session.py+++ b/jarvis/realtime/session.py@@ -881,7 +881,9 @@ _HYBRID_ROLE_DIRECTIVE = ("however heavy the topic sounds. Never claim that an action or mission ""was started, completed, saved, opened, or changed unless a successful ""function result explicitly supports that claim; a promise or an "- "intention is not a result. When a request names SEVERAL targets, report "+ "intention is not a result, and a result marked unverified is an attempt, "+ "not a confirmation — say what was tried and that you cannot confirm it. "+ "When a request names SEVERAL targets, report ""ONLY the ones a result actually names as done. If a function returns a ""spoken confirmation question, ask the user and call the same function ""again only after a clear affirmative answer. jarvis_action returns "diff --git a/jarvis/realtime/tools.py b/jarvis/realtime/tools.pyindex 07a105da83..199d8e0b6c 100644--- a/jarvis/realtime/tools.py+++ b/jarvis/realtime/tools.py@@ -131,16 +131,41 @@ def _json_safe(value: Any) -> Any:return json.loads(json.dumps(value, ensure_ascii=False, default=str))+#: Spoken-outcome contract for a tool that acted but could not check the+#: effect (``output["verified"] is False``). The system prompt already forbids+#: claiming an outcome no successful result supports; this makes the result+#: itself say "attempt, not outcome", in the field the model reads for+#: confirmation questions too. Live 2026-08-26 19:50: a blind media-key+#: "play" came back ``ok: true`` and the voice said "I've just started some+#: cool music on YouTube Music" while nothing played.+_UNVERIFIED_OUTCOME_INSTRUCTION = (+ "This result is an ATTEMPT, not a confirmed outcome: the tool acted but "+ "could not verify that anything changed. Tell the user in one short "+ "sentence what was tried and that you cannot confirm it from here; never "+ "say it is done, playing, running, or started. Offer the next step in the "+ "same breath instead of asking them to wait."+)+++def _outcome_unverified(output: Any) -> bool:+ """Whether a tool explicitly marked its own outcome as unchecked."""+ return isinstance(output, dict) and output.get("verified") is False++def _bounded_result(success: bool, output: Any, error: str | None) -> dict[str, Any]:- payload = {+ payload: dict[str, Any] = {"success": bool(success),"output": _json_safe(output),"error": str(error) if error else None,}+ unverified = success and _outcome_unverified(output)+ if unverified:+ payload["unverified"] = True+ payload["instruction"] = _UNVERIFIED_OUTCOME_INSTRUCTIONserialized = json.dumps(payload, ensure_ascii=False, default=str)if len(serialized) <= _MAX_RESULT_CHARS:return payload- return {+ truncated: dict[str, Any] = {"success": bool(success),"output": serialized[:_MAX_RESULT_CHARS],"error": (@@ -149,6 +174,10 @@ def _bounded_result(success: bool, output: Any, error: str | None) -> dict[str,),"truncated": True,}+ if unverified:+ truncated["unverified"] = True+ truncated["instruction"] = _UNVERIFIED_OUTCOME_INSTRUCTION+ return truncateddef _compact_text(text: str, limit: int) -> str:diff --git a/tests/unit/plugins/tool/test_youtube_music_rest.py b/tests/unit/plugins/tool/test_youtube_music_rest.pyindex 951c9208b1..55d8b697c8 100644--- a/tests/unit/plugins/tool/test_youtube_music_rest.py+++ b/tests/unit/plugins/tool/test_youtube_music_rest.py@@ -447,6 +447,61 @@ async def test_control_with_nothing_registered_is_an_error():assert "Nothing is registered" in out["error"]+# -- blind media keys (a machine that cannot read its media session) ---------+#+# Live 2026-08-26 19:50: "play some cool music" arrived as play without a+# title, the machine had no winrt, the tool pressed the play/pause toggle+# blindly and answered ``ok: true, resumed: true`` — the voice said "I've+# just started some cool music on YouTube Music" over a silent machine.+++def _blind() -> FakeMedia:+ return FakeMedia(now=None, can_read=False, backend="media-keys", note="install X")+++async def test_blind_play_with_nothing_known_paused_presses_nothing_and_says_so():+ media = _blind()+ out = await _tool(_search_handler, media=media).play(query="")+ assert media.calls == []+ assert "say what to play" in out["error"]+++async def test_blind_pause_is_an_attempt_not_a_result():+ media = _blind()+ out = await _tool(_search_handler, media=media).pause()+ assert media.calls == ["pause"]+ assert out["ok"] is True and out["verified"] is False+ assert "cannot confirm" in out["note"] and "install X" in out["note"]+ assert "track" not in out+++async def test_blind_play_after_a_blind_pause_is_a_resume_attempt():+ media = _blind()+ tool = _tool(_search_handler, media=media)+ await tool.pause()+ out = await tool.play(query="")+ assert media.calls == ["pause", "play"]+ assert out["ok"] is True and out["resumed"] is True and out["verified"] is False+ # The resume spent the pause: a second bare play is a guess again.+ out = await tool.play(query="")+ assert media.calls == ["pause", "play"] and "say what to play" in out["error"]+++async def test_readable_session_control_is_verified():+ media = FakeMedia(now=_playing())+ out = await _tool(_search_handler, media=media).pause()+ assert out["ok"] is True and out["verified"] is True and out["track"] == "Karma Police"+++async def test_blind_play_of_a_song_opens_the_link_but_never_claims_playback():+ media = _blind()+ opener = Opener(media=media)+ out = await _tool(_search_handler, media=media, opener=opener).play(query="Karma Police")+ assert opener.urls == [song_url("vid123")]+ assert out["ok"] is True and out["playback_confirmed"] is False+ assert out["verified"] is False and "cannot confirm" in out["note"]++# -- library writes -----------------------------------------------------------diff --git a/tests/unit/realtime/test_tools.py b/tests/unit/realtime/test_tools.pyindex b1dd705747..1c016e74ec 100644--- a/tests/unit/realtime/test_tools.py+++ b/tests/unit/realtime/test_tools.py@@ -28,8 +28,9 @@ class FakeTool:class FakeExecutor:- def __init__(self, *, confirmation_required: bool = False) -> None:+ def __init__(self, *, confirmation_required: bool = False, output="opened") -> None:self.confirmation_required = confirmation_required+ self.output = outputself.execute_calls = []self.confirmed_calls = []self.cancelled = []@@ -43,7 +44,7 @@ class FakeExecutor:output={"tool_name": tool.name},error=VOICE_CONFIRM_SENTINEL,)- return SimpleNamespace(success=True, output="opened", error=None)+ return SimpleNamespace(success=True, output=self.output, error=None)async def execute_confirmed(self, trace_id, **kwargs):self.confirmed_calls.append((trace_id, kwargs))@@ -121,6 +122,41 @@ async def test_available_tool_runs_only_through_tool_executor():assert executor.execute_calls[0][2]["config_snapshot"]["voice_confirm"] is True+async def test_unverified_outcome_carries_the_attempt_not_outcome_instruction():+ """A tool that acted blind (``verified: false``) must reach the model as+ an attempt: live 2026-08-26 19:50 a media-key "play" came back ok and+ the voice announced music that never started."""+ tool = FakeTool("youtube_music")+ executor = FakeExecutor(output={"ok": True, "action": "play", "verified": False})+ bridge = RealtimeToolBridge(tools={"youtube_music": tool}, executor=executor, language="en")++ _name, result = await bridge.execute(+ wire_name="youtube_music", arguments={"app_name": "x"}+ )++ assert result["success"] is True and result["unverified"] is True+ assert "not a confirmed outcome" in result["instruction"]+ assert "never say it is done" in result["instruction"]+++async def test_verified_outcome_carries_no_instruction():+ tool = FakeTool("youtube_music")+ executor = FakeExecutor(output={"ok": True, "action": "play", "verified": True})+ bridge = RealtimeToolBridge(tools={"youtube_music": tool}, executor=executor, language="en")++ _name, result = await bridge.execute(+ wire_name="youtube_music", arguments={"app_name": "x"}+ )++ assert result == {+ "success": True,+ "output": {"ok": True, "action": "play", "verified": True},+ "error": None,+ }++@pytest.mark.asyncioasync def test_vertex_toolset_prefix_resolves_to_the_declared_tool():"""Vertex Live prefixes user-declared functions with the tool-set name.