Skip to content

Dodaios is an open-source agent OS for your desktop: it runs your coding agents, browser and apps by voice or chat, on your own machine. Formerly Personal Jarvis.

fix(music): a blind media key is an attempt, not a result — the voice no longer announces music that never started

rubenluetke10-beep7c94f7b9d225
diff --git a/docs/BUGS.md b/docs/BUGS.md
index 58a7591028..4cf2c19522 100644
--- a/docs/BUGS.md
+++ b/docs/BUGS.md
@@ -13045,3 +13045,82 @@ and five sentences ending in dictionary words that must survive),
whatever goes into a Whisper prompt needs a guard on the way out. And "one
letter apart" is a property of spelling — the person who registers a word is
describing how it sounds.
+
+## BUG-186: "There you go, I've just started some cool music on YouTube Music" — over a silent machine. The tool pressed a media key blindly, called that a resume, and the voice sold it as done (HIGH, FIXED 2026-08-26)
+
+**Symptom.** Gemini Live, 2026-08-26 19:50, "just use like the Google Music
+plugin and play some cool music." The trace shows `youtube_music · play` in
+0.0 s, then the voice: "One moment, let's get that music playing for you.
+There you go, I've just started some cool music on YouTube Music. Just so
+you know, that just toggles play or pause …". No browser opened, nothing
+played. The maintainer's reading: the assistant confirms things it did not
+do.
+
+**Root cause.** Three layers agreed on a lie, and the model was the most
+honest of them.
+
+1. The model called `play` without a title (`{"action": "play", "type":
+ "song"}`) — the schema says a bare `play` resumes what is paused, which is
+ a fair reading of "play some music" when nothing names a song.
+2. `youtube_music_rest._control("play")` on this box: the Windows media
+ session cannot READ (`winrt-*` was added to the `[desktop]` extras after
+ the app's interpreter was last installed — `pywebview`, `pystray`, `pycaw`
+ present, `winrt` absent), so `cap.can_read` was false, no "is anything
+ paused" check ran, the play/pause TOGGLE key went to whatever the OS
+ considered current, and the result was
+ `{'ok': True, 'action': 'play', 'note': 'Sent as a media key: this toggles
+ play/pause blindly.', 'resumed': True}` in 17 ms.
+3. The system prompt forbids claiming an outcome "unless a successful
+ function result explicitly supports that claim" — and this one did. The
+ realtime honesty guard (`action_honesty.py`, `_has_execution_evidence`)
+ only fires when NO tool ran; a tool that ran and said `ok` is evidence by
+ construction. The model read the note, added its "just toggles" caveat,
+ and still opened with "There you go, I've just started".
+
+**Fix.** Truth comes from the tool; the bridge makes it audible.
+
+* `jarvis/plugins/tool/youtube_music_rest.py` — `_control` splits at
+ `cap.can_read`. A readable session (background player, WinRT, MPRIS,
+ nowplaying-cli) answers `verified: true` as before. A blind machine goes
+ through `_control_blind`: `play` is REFUSED unless this tool itself sent a
+ blind pause just before (the key is a toggle; with nothing known paused it
+ starts a random tab or stops a running one), and `play()` then answers its
+ existing "say what to play" error without pressing anything. Pause, next
+ and previous still go out as best effort but come back
+ `verified: false` with a note that nothing on this machine can confirm the
+ effect. `_open_and_confirm` and the background-player path carry
+ `verified` = `playback_confirmed`; a browser opened on a machine that
+ cannot read the session is `verified: false` too.
+* `jarvis/realtime/tools.py` — `_bounded_result` sees
+ `output["verified"] is False` on a successful call and adds
+ `unverified: true` plus an `instruction` ("This result is an ATTEMPT, not a
+ confirmed outcome … never say it is done, playing, running, or started.
+ Offer the next step in the same breath") — the same field the model
+ already obeys for confirmation questions. It survives truncation.
+* `jarvis/realtime/session.py` — the tool-use system prompt names the
+ contract: a result marked unverified is an attempt, not a confirmation.
+* This box: the four `winrt-*` packages installed into the app's Python
+ (needs an app restart to take effect); with them the plugin reads what
+ plays and resumes only a real pause.
+
+**Why not gate it in the honesty guard.** Withholding the tool from
+`_executed_tool_names` would make `_recover_unbacked_action_claim` replace
+the whole answer with "I did not start an action for that" — false the other
+way (a key WAS sent) and a dead stop in the conversation. An unverified
+result is data the model can speak honestly at full speed; the fix keeps the
+turn's timing untouched (the tool still answers in milliseconds) and changes
+only what is true in it.
+
+**Regression tests.**
+`tests/unit/plugins/tool/test_youtube_music_rest.py::test_blind_play_with_nothing_known_paused_presses_nothing_and_says_so`,
+`::test_blind_pause_is_an_attempt_not_a_result`,
+`::test_blind_play_after_a_blind_pause_is_a_resume_attempt`,
+`::test_readable_session_control_is_verified`,
+`::test_blind_play_of_a_song_opens_the_link_but_never_claims_playback`;
+`tests/unit/realtime/test_tools.py::test_unverified_outcome_carries_the_attempt_not_outcome_instruction`,
+`::test_verified_outcome_carries_no_instruction`.
+
+**Lesson.** `ok: true` is a claim about the ACTION, and the voice speaks
+about the OUTCOME. Every tool that acts without being able to look owes the
+model the difference in a field it cannot miss — a note in prose is a caveat
+the model will politely paraphrase after the good news.
diff --git a/jarvis/plugins/tool/youtube_music_rest.py b/jarvis/plugins/tool/youtube_music_rest.py
index da8b60ccb8..575f42692d 100644
--- a/jarvis/plugins/tool/youtube_music_rest.py
+++ b/jarvis/plugins/tool/youtube_music_rest.py
@@ -404,6 +404,10 @@ class YouTubeMusicRestTool:
self._player = player
self._playback_mode = playback_mode or _default_playback_mode
self._player_confirm_timeout_s = player_confirm_timeout_s
+ # Whether the last blind media key this tool sent was a pause — the
+ # only state in which a blind "play" is a resume rather than a guess
+ # (``_control_blind``).
+ self._blind_pause_sent = False
self._pool = HttpClientPool(transport=transport)
# search.list is the rationed call (100/day). Same words → same answer
# within a session, so a repeat costs nothing.
@@ -558,7 +562,12 @@ class YouTubeMusicRestTool:
ok, why = False, str(exc)
if ok:
try:
- return await self._play_in_player(url, started)
+ out = await self._play_in_player(url, started)
+ # Only a player that reported "playing" counts as done;
+ # a cookie wall, a paused page or a page still loading
+ # is an attempt the voice must not sell as a result.
+ out["verified"] = bool(out.get("playback_confirmed"))
+ return out
except Exception as exc: # noqa: BLE001 — player unusable → browser, and say so
log.info("background player unusable, opening the browser: %s", exc)
why = f"The background player could not start ({exc})."
@@ -823,7 +832,12 @@ class YouTubeMusicRestTool:
return {"error": f"The background player did not accept '{verb}': {exc}"}
if not ok:
return {"error": f"The background player did not accept '{verb}'."}
- via_player: dict[str, Any] = {"ok": True, "action": verb, "app": _PLAYER_APP}
+ via_player: dict[str, Any] = {
+ "ok": True,
+ "action": verb,
+ "app": _PLAYER_APP,
+ "verified": True,
+ }
if verb in ("next", "previous"):
await asyncio.sleep(1.2)
after = await self._player_state()
@@ -837,27 +851,62 @@ class YouTubeMusicRestTool:
cap = await media.capability()
if not cap.can_control:
return {"error": "This machine cannot control playback. " + (cap.note or "")}
- before = await media.now_playing() if cap.can_read else None
- if cap.can_read and before is None:
+ if not cap.can_read:
+ return await self._control_blind(media, verb, cap.note)
+ before = await media.now_playing()
+ if before is None:
return {"error": _NO_SESSION_READ}
ok = bool(await getattr(media, verb)())
if not ok:
return {"error": f"The player did not accept '{verb}'."}
- out: dict[str, Any] = {"ok": True, "action": verb}
- if before is not None:
- out["app"] = before.app
- if verb in ("next", "previous"):
- await asyncio.sleep(0.8)
- after = await media.now_playing()
- if after is not None:
- out["now"] = after.as_dict()
- else:
- out["track"] = before.title
- out["artist"] = before.artist
- elif cap.backend == "media-keys":
- out["note"] = "Sent as a media key: this toggles play/pause blindly."
+ out: dict[str, Any] = {"ok": True, "action": verb, "verified": True, "app": before.app}
+ if verb in ("next", "previous"):
+ await asyncio.sleep(0.8)
+ after = await media.now_playing()
+ if after is not None:
+ out["now"] = after.as_dict()
+ else:
+ out["track"] = before.title
+ out["artist"] = before.artist
return out
+ async def _control_blind(self, media: Any, verb: str, note: str | None) -> dict[str, Any]:
+ """Steer through media keys on a machine that cannot read its media
+ session (Windows without the ``winrt`` extras).
+
+ A media key reaches whatever player the OS considers current and
+ reports nothing back, so the result is an ATTEMPT and says so
+ (``verified: false``): the realtime bridge turns that into a spoken
+ caveat, never a completion. ``play`` is the one verb refused outright
+ unless this tool itself paused something the same way — the key is
+ the play/pause TOGGLE, so with nothing known to be paused it starts a
+ random tab or stops a running one. Live 2026-08-26 19:50: "play some
+ cool music" arrived without a title, this path pressed the toggle,
+ answered ``ok: true, resumed: true``, and the voice said "I've just
+ started some cool music on YouTube Music" over a silent machine.
+ """
+ if verb == "play" and not self._blind_pause_sent:
+ return {
+ "error": (
+ "Nothing is known to be paused, and this machine cannot see "
+ "what is playing, so play was not pressed."
+ )
+ }
+ ok = bool(await getattr(media, verb)())
+ if not ok:
+ return {"error": f"The player did not accept '{verb}'."}
+ self._blind_pause_sent = verb == "pause"
+ return {
+ "ok": True,
+ "action": verb,
+ "verified": False,
+ "note": (
+ "Sent as a media key to whichever player is current; this machine "
+ "cannot confirm that anything changed. "
+ + (note or "")
+ ).strip(),
+ }
+
async def pause(self) -> dict[str, Any]:
return await self._control("pause")
@@ -927,12 +976,23 @@ class YouTubeMusicRestTool:
confirmed = True
break
out["playback_confirmed"] = confirmed
+ out["verified"] = confirmed
if not confirmed:
out["note"] = (
"YouTube Music opened, but the browser has not reported playback "
"yet — if it stays silent, press play once (browsers block "
"autoplay on a site until it has been used there)."
)
+ else:
+ # The page opened; whether it plays is beyond what this machine
+ # can see, so the result is an attempt (``verified: false``),
+ # never "it is playing".
+ out["playback_confirmed"] = False
+ out["verified"] = False
+ out["note"] = (
+ "YouTube Music opened in the browser; this machine cannot confirm "
+ "that playback started. " + (cap.note or "")
+ ).strip()
return out
async def play(
diff --git a/jarvis/realtime/session.py b/jarvis/realtime/session.py
index 334e11dfed..b4a6834d94 100644
--- a/jarvis/realtime/session.py
+++ b/jarvis/realtime/session.py
@@ -881,7 +881,9 @@ _HYBRID_ROLE_DIRECTIVE = (
"however heavy the topic sounds. Never claim that an action or mission "
"was started, completed, saved, opened, or changed unless a successful "
"function result explicitly supports that claim; a promise or an "
- "intention is not a result. When a request names SEVERAL targets, report "
+ "intention is not a result, and a result marked unverified is an attempt, "
+ "not a confirmation — say what was tried and that you cannot confirm it. "
+ "When a request names SEVERAL targets, report "
"ONLY the ones a result actually names as done. If a function returns a "
"spoken confirmation question, ask the user and call the same function "
"again only after a clear affirmative answer. jarvis_action returns "
diff --git a/jarvis/realtime/tools.py b/jarvis/realtime/tools.py
index 07a105da83..199d8e0b6c 100644
--- a/jarvis/realtime/tools.py
+++ b/jarvis/realtime/tools.py
@@ -131,16 +131,41 @@ def _json_safe(value: Any) -> Any:
return json.loads(json.dumps(value, ensure_ascii=False, default=str))
+#: Spoken-outcome contract for a tool that acted but could not check the
+#: effect (``output["verified"] is False``). The system prompt already forbids
+#: claiming an outcome no successful result supports; this makes the result
+#: itself say "attempt, not outcome", in the field the model reads for
+#: confirmation questions too. Live 2026-08-26 19:50: a blind media-key
+#: "play" came back ``ok: true`` and the voice said "I've just started some
+#: cool music on YouTube Music" while nothing played.
+_UNVERIFIED_OUTCOME_INSTRUCTION = (
+ "This result is an ATTEMPT, not a confirmed outcome: the tool acted but "
+ "could not verify that anything changed. Tell the user in one short "
+ "sentence what was tried and that you cannot confirm it from here; never "
+ "say it is done, playing, running, or started. Offer the next step in the "
+ "same breath instead of asking them to wait."
+)
+
+
+def _outcome_unverified(output: Any) -> bool:
+ """Whether a tool explicitly marked its own outcome as unchecked."""
+ return isinstance(output, dict) and output.get("verified") is False
+
+
def _bounded_result(success: bool, output: Any, error: str | None) -> dict[str, Any]:
- payload = {
+ payload: dict[str, Any] = {
"success": bool(success),
"output": _json_safe(output),
"error": str(error) if error else None,
}
+ unverified = success and _outcome_unverified(output)
+ if unverified:
+ payload["unverified"] = True
+ payload["instruction"] = _UNVERIFIED_OUTCOME_INSTRUCTION
serialized = json.dumps(payload, ensure_ascii=False, default=str)
if len(serialized) <= _MAX_RESULT_CHARS:
return payload
- return {
+ truncated: dict[str, Any] = {
"success": bool(success),
"output": serialized[:_MAX_RESULT_CHARS],
"error": (
@@ -149,6 +174,10 @@ def _bounded_result(success: bool, output: Any, error: str | None) -> dict[str,
),
"truncated": True,
}
+ if unverified:
+ truncated["unverified"] = True
+ truncated["instruction"] = _UNVERIFIED_OUTCOME_INSTRUCTION
+ return truncated
def _compact_text(text: str, limit: int) -> str:
diff --git a/tests/unit/plugins/tool/test_youtube_music_rest.py b/tests/unit/plugins/tool/test_youtube_music_rest.py
index 951c9208b1..55d8b697c8 100644
--- a/tests/unit/plugins/tool/test_youtube_music_rest.py
+++ b/tests/unit/plugins/tool/test_youtube_music_rest.py
@@ -447,6 +447,61 @@ async def test_control_with_nothing_registered_is_an_error():
assert "Nothing is registered" in out["error"]
+# -- blind media keys (a machine that cannot read its media session) ---------
+#
+# Live 2026-08-26 19:50: "play some cool music" arrived as play without a
+# title, the machine had no winrt, the tool pressed the play/pause toggle
+# blindly and answered ``ok: true, resumed: true`` — the voice said "I've
+# just started some cool music on YouTube Music" over a silent machine.
+
+
+def _blind() -> FakeMedia:
+ return FakeMedia(now=None, can_read=False, backend="media-keys", note="install X")
+
+
+async def test_blind_play_with_nothing_known_paused_presses_nothing_and_says_so():
+ media = _blind()
+ out = await _tool(_search_handler, media=media).play(query="")
+ assert media.calls == []
+ assert "say what to play" in out["error"]
+
+
+async def test_blind_pause_is_an_attempt_not_a_result():
+ media = _blind()
+ out = await _tool(_search_handler, media=media).pause()
+ assert media.calls == ["pause"]
+ assert out["ok"] is True and out["verified"] is False
+ assert "cannot confirm" in out["note"] and "install X" in out["note"]
+ assert "track" not in out
+
+
+async def test_blind_play_after_a_blind_pause_is_a_resume_attempt():
+ media = _blind()
+ tool = _tool(_search_handler, media=media)
+ await tool.pause()
+ out = await tool.play(query="")
+ assert media.calls == ["pause", "play"]
+ assert out["ok"] is True and out["resumed"] is True and out["verified"] is False
+ # The resume spent the pause: a second bare play is a guess again.
+ out = await tool.play(query="")
+ assert media.calls == ["pause", "play"] and "say what to play" in out["error"]
+
+
+async def test_readable_session_control_is_verified():
+ media = FakeMedia(now=_playing())
+ out = await _tool(_search_handler, media=media).pause()
+ assert out["ok"] is True and out["verified"] is True and out["track"] == "Karma Police"
+
+
+async def test_blind_play_of_a_song_opens_the_link_but_never_claims_playback():
+ media = _blind()
+ opener = Opener(media=media)
+ out = await _tool(_search_handler, media=media, opener=opener).play(query="Karma Police")
+ assert opener.urls == [song_url("vid123")]
+ assert out["ok"] is True and out["playback_confirmed"] is False
+ assert out["verified"] is False and "cannot confirm" in out["note"]
+
+
# -- library writes -----------------------------------------------------------
diff --git a/tests/unit/realtime/test_tools.py b/tests/unit/realtime/test_tools.py
index b1dd705747..1c016e74ec 100644
--- a/tests/unit/realtime/test_tools.py
+++ b/tests/unit/realtime/test_tools.py
@@ -28,8 +28,9 @@ class FakeTool:
class FakeExecutor:
- def __init__(self, *, confirmation_required: bool = False) -> None:
+ def __init__(self, *, confirmation_required: bool = False, output="opened") -> None:
self.confirmation_required = confirmation_required
+ self.output = output
self.execute_calls = []
self.confirmed_calls = []
self.cancelled = []
@@ -43,7 +44,7 @@ class FakeExecutor:
output={"tool_name": tool.name},
error=VOICE_CONFIRM_SENTINEL,
)
- return SimpleNamespace(success=True, output="opened", error=None)
+ return SimpleNamespace(success=True, output=self.output, error=None)
async def execute_confirmed(self, trace_id, **kwargs):
self.confirmed_calls.append((trace_id, kwargs))
@@ -121,6 +122,41 @@ async def test_available_tool_runs_only_through_tool_executor():
assert executor.execute_calls[0][2]["config_snapshot"]["voice_confirm"] is True
+async def test_unverified_outcome_carries_the_attempt_not_outcome_instruction():
+ """A tool that acted blind (``verified: false``) must reach the model as
+ an attempt: live 2026-08-26 19:50 a media-key "play" came back ok and
+ the voice announced music that never started."""
+ tool = FakeTool("youtube_music")
+ executor = FakeExecutor(output={"ok": True, "action": "play", "verified": False})
+ bridge = RealtimeToolBridge(tools={"youtube_music": tool}, executor=executor, language="en")
+
+ _name, result = await bridge.execute(
+ wire_name="youtube_music", arguments={"app_name": "x"}
+ )
+
+ assert result["success"] is True and result["unverified"] is True
+ assert "not a confirmed outcome" in result["instruction"]
+ assert "never say it is done" in result["instruction"]
+
+
+async def test_verified_outcome_carries_no_instruction():
+ tool = FakeTool("youtube_music")
+ executor = FakeExecutor(output={"ok": True, "action": "play", "verified": True})
+ bridge = RealtimeToolBridge(tools={"youtube_music": tool}, executor=executor, language="en")
+
+ _name, result = await bridge.execute(
+ wire_name="youtube_music", arguments={"app_name": "x"}
+ )
+
+ assert result == {
+ "success": True,
+ "output": {"ok": True, "action": "play", "verified": True},
+ "error": None,
+ }
+
+
@pytest.mark.asyncio
async def test_vertex_toolset_prefix_resolves_to_the_declared_tool():
"""Vertex Live prefixes user-declared functions with the tool-set name.