Skip to content

Build and run agents you can see, understand and trust.

feat(realtime): realtime voice agent with DashScope backends and local transport (#2547)

DavdGaof8b40f60f534
diff --git a/README.md b/README.md
index d60bcd5c4..7390bcb40 100644
--- a/README.md
+++ b/README.md
@@ -73,6 +73,7 @@ rather than constraining them with strict prompts and opinionated orchestrations
## News
<!-- BEGIN NEWS -->
+- **[2026-09] `FEAT` `Experimental`:** Realtime voice agent supported — full-duplex speech-to-speech via `RealtimeAgent` on the DashScope Realtime API (Qwen-Omni, Qwen-Audio-3.0), with barge-in and tool calls. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/realtime)
- **[2026-09] `FEAT`:** A2A protocol supported — chat with any remote A2A agent via `A2AAgent`. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/a2a) | [Docs](https://docs.agentscope.io/latest/en/building-blocks/a2a)
- **[2026-08] `FEAT`:** Pipeline supported — run multiple agents by a fixed logic behind one event stream. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/pipeline) | [Docs](https://docs.agentscope.io/latest/en/building-blocks/pipeline/overview)
- **[2026-08] `INTE`:** DingTalk channel supported. [Docs](https://docs.agentscope.io/latest/en/deploy/channel/dingtalk)
@@ -82,7 +83,6 @@ rather than constraining them with strict prompts and opinionated orchestrations
- **[2026-08] `INTE`:** GitHub MCP Registry and ClawHub supported as built-in hubs. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent_service) | [Docs](https://docs.agentscope.io/latest/en/deploy/hub)
- **[2026-08] `FEAT`:** MCP & Skill Hub supported — browse a hub, install into your library, add to a workspace. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent_service) | [Docs](https://docs.agentscope.io/latest/en/deploy/hub)
- **[2026-07] `INTE`:** Daytona-based workspace/sandbox supported. [Docs](https://docs.agentscope.io/latest/en/building-blocks/workspace)
-- **[2026-07] `INTE`:** K8s, OpenSandbox-based workspace/sandbox supported. [Docs](https://docs.agentscope.io/latest/en/building-blocks/workspace)
<!-- END NEWS -->
[More news →](./docs/NEWS.md)
diff --git a/README_zh.md b/README_zh.md
index 510638c05..24871d43d 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -72,6 +72,7 @@ AgentScope 的目标是充分发挥大模型的推理与工具调用能力,
## 新闻
<!-- BEGIN NEWS -->
+- **[2026-09] `功能` `实验性`:** 支持实时语音智能体 —— 通过 `RealtimeAgent` 接入 DashScope Realtime API(Qwen-Omni、Qwen-Audio-3.0),全双工语音对话、打断与工具调用。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/realtime)
- **[2026-09] `功能`:** 支持 A2A 协议 —— 通过 `A2AAgent` 与任意远端 A2A 智能体对话。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/a2a) | [文档](https://docs.agentscope.io/latest/zh/building-blocks/a2a)
- **[2026-08] `功能`:** 支持流水线 —— 按照固化逻辑运行多智能体,并向外提供统一接口。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/pipeline) | [文档](https://docs.agentscope.io/latest/zh/building-blocks/pipeline/overview)
- **[2026-08] `集成`:** 支持钉钉(DingTalk)频道。[文档](https://docs.agentscope.io/latest/zh/deploy/channel/dingtalk)
@@ -81,7 +82,6 @@ AgentScope 的目标是充分发挥大模型的推理与工具调用能力,
- **[2026-08] `集成`:** 内置集成 GitHub MCP Registry 与 ClawHub。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent_service) | [文档](https://docs.agentscope.io/latest/zh/deploy/hub)
- **[2026-08] `功能`:** 支持 MCP & Skill Hub —— 浏览 hub、安装到个人库、再添加到工作区。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent_service) | [文档](https://docs.agentscope.io/latest/zh/deploy/hub)
- **[2026-07] `集成`:** 集成 Daytona 工作区/沙箱实现。 [文档](https://docs.agentscope.io/latest/en/building-blocks/workspace)
-- **[2026-07] `集成`:** 集成 K8s,OpenSandbox 工作区/沙箱实现。 [文档](https://docs.agentscope.io/latest/en/building-blocks/workspace)
<!-- END NEWS -->
[更多新闻 →](./docs/NEWS_zh.md)
diff --git a/docs/NEWS.md b/docs/NEWS.md
index 6db505a9a..d1f6f9ea5 100644
--- a/docs/NEWS.md
+++ b/docs/NEWS.md
@@ -2,6 +2,7 @@
<!-- The first 10 items are automatically synced to README.md and README_zh.md via GitHub Actions. -->
<!-- To update news in READMEs, modify this file and push to trigger the workflow. -->
+- **[2026-09] `FEAT` `Experimental`:** Realtime voice agent supported — full-duplex speech-to-speech via `RealtimeAgent` on the DashScope Realtime API (Qwen-Omni, Qwen-Audio-3.0), with barge-in and tool calls. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/realtime)
- **[2026-09] `FEAT`:** A2A protocol supported — chat with any remote A2A agent via `A2AAgent`. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/a2a) | [Docs](https://docs.agentscope.io/latest/en/building-blocks/a2a)
- **[2026-08] `FEAT`:** Pipeline supported — run multiple agents by a fixed logic behind one event stream. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/pipeline) | [Docs](https://docs.agentscope.io/latest/en/building-blocks/pipeline/overview)
- **[2026-08] `INTE`:** DingTalk channel supported. [Docs](https://docs.agentscope.io/latest/en/deploy/channel/dingtalk)
diff --git a/docs/NEWS_zh.md b/docs/NEWS_zh.md
index 1727d1308..304323df0 100644
--- a/docs/NEWS_zh.md
+++ b/docs/NEWS_zh.md
@@ -2,6 +2,7 @@
<!-- The first 10 items are automatically synced to README.md and README_zh.md via GitHub Actions. -->
<!-- To update news in READMEs, modify this file and push to trigger the workflow. -->
+- **[2026-09] `功能` `实验性`:** 支持实时语音智能体 —— 通过 `RealtimeAgent` 接入 DashScope Realtime API(Qwen-Omni、Qwen-Audio-3.0),全双工语音对话、打断与工具调用。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/realtime)
- **[2026-09] `功能`:** 支持 A2A 协议 —— 通过 `A2AAgent` 与任意远端 A2A 智能体对话。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/a2a) | [文档](https://docs.agentscope.io/latest/zh/building-blocks/a2a)
- **[2026-08] `功能`:** 支持流水线 —— 按照固化逻辑运行多智能体,并向外提供统一接口。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/pipeline) | [文档](https://docs.agentscope.io/latest/zh/building-blocks/pipeline/overview)
- **[2026-08] `集成`:** 支持钉钉(DingTalk)频道。[文档](https://docs.agentscope.io/latest/zh/deploy/channel/dingtalk)
diff --git a/examples/realtime/README.md b/examples/realtime/README.md
new file mode 100644
index 000000000..33cf864ce
--- /dev/null
+++ b/examples/realtime/README.md
@@ -0,0 +1,93 @@
+# Realtime Voice Agent
+
+Talk to a DashScope speech-to-speech model through your microphone.
+`local_mic.py` wires up the three pieces of `agentscope.realtime`:
+
+- `DashScopeAudioRealtimeModel` / `DashScopeRealtimeModel` — the provider
+ session, resolved from the model card so any listed model works.
+- `LocalAudioTransport` — the sound card: capture, playback, playout
+ accounting and a short fade when the reply is cut off.
+- `RealtimeAgent` — the turn-taking state machine in between: barge-in,
+ tool calls with permission prompts, per-turn latency metrics.
+
+The agent owns the model session, you own the transport, and one
+`agent.run(transport)` borrows both until the transport ends.
+
+## Prerequisites
+
+- Python 3.11 or newer
+- AgentScope with the realtime extra:
+
+ ```bash
+ pip install "agentscope[realtime]"
+ ```
+
+ `sounddevice` needs PortAudio: it is bundled on macOS and Windows; on
+ Debian/Ubuntu run `apt install libportaudio2`.
+- A DashScope API key with access to a realtime model.
+
+## Run
+
+```bash
+export DASHSCOPE_API_KEY=sk-...
+python examples/realtime/local_mic.py
+```
+
+Speak, hear the reply, and speak over it to interrupt. `Ctrl-C` quits.
+
+The default model is `qwen-audio-3.0-realtime-plus`. Any card listed for the
+credential works; the class is looked up from the card, the same way the
+service layer does it:
+
+```bash
+REALTIME_MODEL=qwen3.5-omni-flash-realtime python examples/realtime/local_mic.py
+```
+
+An unknown name prints the available ones.
+
+Every reply ends with a diagnostic line: how it finished, the model's time to
+first audio byte (`ttfb`), the user's end-to-end wait (`e2e`), and the
+assistant message as it now stands in the context — after a barge-in that is
+only the part you actually heard.
+
+### Tools
+
+The example registers `Bash`, `Edit`, `Write` and `Read`. When the model
+calls one that needs permission, the terminal asks:
+
+```
+ [permission] Bash({"command": "ls"})
+ allow? [y/N]
+```
+
+The prompt runs in a thread, so audio keeps flowing while you decide. The
+agent gives up on a prompt after five minutes.
+
+## Tips
+
+- **Use headphones.** With speakers, the microphone picks up the agent's own
+ voice; the provider's VAD then hears "the user" start speaking and cuts the
+ reply off — the agent interrupts itself. `LocalAudioTransport` does no echo
+ cancellation (a browser would), so headphones are the fix.
+- **Do not route both directions through one Bluetooth headset.** macOS
+ switches AirPods and similar devices to the hands-free profile when they
+ are used for input and output at once, and PortAudio's separate
+ input/output streams on that device often produce no sound. Pick devices
+ by index from `python -m sounddevice`, e.g. the headset's microphone with
+ the built-in speakers:
+
+ ```bash
+ REALTIME_INPUT_DEVICE=3 REALTIME_OUTPUT_DEVICE=2 python examples/realtime/local_mic.py
+ ```
+
+- **Silence is fine.** DashScope closes a session after about three minutes
+ without a response. The example logs that it happened and keeps the
+ microphone open; the next thing you say reconnects the model. The
+ conversation history stays in `agent.state`, but the provider starts fresh —
+ it cannot be seeded with prior turns.
+- **Model limits are per turn, not per token.** `qwen3-omni-flash-realtime`
+ remembers eight turns, `qwen-audio-3.0-realtime-*` fifty; older turns are
+ dropped silently on the provider side. The cards under
+ `agentscope/realtime/_dashscope/` list each model's limits.
+- **Typed input** only works on models that accept text
+ (`qwen-audio-3.0-realtime-*`); the Omni models are audio-in only.
diff --git a/examples/realtime/local_mic.py b/examples/realtime/local_mic.py
new file mode 100644
index 000000000..d4d91f7d3
--- /dev/null
+++ b/examples/realtime/local_mic.py
@@ -0,0 +1,153 @@
+# -*- coding: utf-8 -*-
+"""Talk to a DashScope realtime model through the local microphone, with
+tools the model may call and a terminal prompt for the ones that need
+your permission.
+
+ export DASHSCOPE_API_KEY=sk-...
+ python examples/realtime/local_mic.py
+
+Any of the DashScope realtime models works; the class is looked up from
+the model card, the same way the service layer does it:
+
+ REALTIME_MODEL=qwen3.5-omni-flash-realtime python ...
+
+Pick devices by index from ``python -m sounddevice`` when the defaults
+are wrong, e.g. a Bluetooth headset used for both directions:
+
+ REALTIME_INPUT_DEVICE=3 REALTIME_OUTPUT_DEVICE=2 python ...
+
+Speak, hear the reply, and speak over it to interrupt. Ctrl-C to quit.
+"""
+import asyncio
+import os
+
+from agentscope.agent import RealtimeAgent
+from agentscope.credential import DashScopeCredential
+from agentscope.event import (
+ ConfirmResult,
+ ReplyEndEvent,
+ ReplyStartEvent,
+ RequireUserConfirmEvent,
+ TextBlockDeltaEvent,
+ ToolResultEndEvent,
+ ToolResultStartEvent,
+ UserConfirmResultEvent,
+ UserInputAudioStartEvent,
+ UserInputTranscriptionEvent,
+)
+from agentscope.realtime import LocalAudioTransport
+from agentscope.tool import Bash, Edit, Read, Toolkit, Write
+
+
+async def main() -> None:
+ """Run one voice session until interrupted."""
+ api_key = os.environ.get("DASHSCOPE_API_KEY")
+ if not api_key:
+ raise SystemExit("Set DASHSCOPE_API_KEY first.")
+ credential = DashScopeCredential(api_key=api_key)
+
+ # Resolve the model class from its card rather than hard-coding it:
+ # a credential can serve several realtime APIs, and each card carries
+ # the tag of the class that produced it.
+ name = os.environ.get("REALTIME_MODEL", "qwen-audio-3.0-realtime-plus")
+ cards = {c.name: c for c in credential.list_realtime_models()}
+ classes = {c.type: c for c in credential.get_realtime_model_classes()}
+ if name not in cards:
+ raise SystemExit(f"Unknown model {name!r}; try: {sorted(cards)}")
+ card = cards[name]
+ model = classes[card.model_type](name, credential, model_card=card)
+
+ agent = RealtimeAgent(
+ name="Friday",
+ sys_prompt="你是一个中文语音助手,回答尽量简短。",
+ model=model,
+ toolkit=Toolkit(tools=[Bash(), Edit(), Write(), Read()]),
+ )
+ transport = LocalAudioTransport(
+ input_sample_rate=model.input_sample_rate,
+ output_sample_rate=model.output_sample_rate,
+ input_device=_device("REALTIME_INPUT_DEVICE"),
+ output_device=_device("REALTIME_OUTPUT_DEVICE"),
+ )
+
+ print(f"[{name}] listening... (Ctrl-C to quit)")
+ # The agent owns the model session, we own the transport, and one
+ # run() borrows both until the transport ends.
+ async with agent, transport:
+ async for event in agent.run(transport):
+ match event:
+ case UserInputAudioStartEvent():
+ print("\n[you] ...", end="", flush=True)
+ case UserInputTranscriptionEvent():
+ print(f"\r[you] {event.transcript}")
+ case ReplyStartEvent():
+ print(f"[{agent.name}] ", end="", flush=True)
+ case TextBlockDeltaEvent():
+ print(event.delta, end="", flush=True)
+ case RequireUserConfirmEvent():
+ await confirm(agent, event)
+ case ToolResultStartEvent():
+ print(f"\n [tool] {event.tool_call_name} running...")
+ case ToolResultEndEvent():
+ print(f" [tool] {event.state}")
+ case ReplyEndEvent():
+ m = agent.last_turn_metrics
+ print(
+ f"\n ({event.finished_reason}"
+ f" | ttfb={_ms(m.backend_ttfb)}"
+ f" | e2e={_ms(m.e2e_latency)})",
+ )
+ if agent.state.context:
+ tail = agent.state.context[-1]
+ print(
+ f" context[-1] = {tail.role}: "
+ f"{tail.get_text_content()!r}",
+ )
+
+
+async def confirm(
+ agent: RealtimeAgent,
+ event: RequireUserConfirmEvent,
+) -> None:
+ """Ask on the terminal whether each pending tool call may run.
+
+ The prompt runs in a thread so the audio pumps keep going while we
+ wait; the agent itself gives up after five minutes.
+ """
+ loop = asyncio.get_running_loop()
+ results = []
+ for call in event.tool_calls:
+ print(f"\n [permission] {call.name}({call.input})")
+ answer = await loop.run_in_executor(None, input, " allow? [y/N] ")
+ results.append(
+ ConfirmResult(
+ tool_call=call,
+ confirmed=answer.strip().lower() in ("y", "yes"),
+ ),
+ )
+ await agent.send(
+ UserConfirmResultEvent(
+ reply_id=event.reply_id,
+ confirm_results=results,
+ ),
+ )
+
+
+def _device(env: str) -> int | str | None:
+ """A sounddevice index or name from the environment, if given."""
+ value = os.environ.get(env)
+ if value is None:
+ return None
+ return int(value) if value.isdigit() else value
+
+
+def _ms(seconds: float | None) -> str:
+ """Format a latency for the console."""
+ return "n/a" if seconds is None else f"{seconds * 1000:.0f}ms"
+
+
+if __name__ == "__main__":
+ try:
+ asyncio.run(main())
+ except KeyboardInterrupt:
+ print("\nbye")
diff --git a/pyproject.toml b/pyproject.toml
index a490ba158..8ba7f9f98 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -106,6 +106,11 @@ workspace = [
tools = [
"ripgrep>=14.1,<15",
]
+# ------------ Realtime voice ------------
+realtime = [
+ "websockets>=13",
+ "sounddevice",
+]
# ------------ A2A ------------
a2a = [
"a2a-sdk>=1.1.1",
diff --git a/src/agentscope/agent/__init__.py b/src/agentscope/agent/__init__.py
index 54b852973..b1b1f3052 100644
--- a/src/agentscope/agent/__init__.py
+++ b/src/agentscope/agent/__init__.py
@@ -3,10 +3,14 @@
from ._agent import Agent
from ._a2a_agent import A2AAgent
from ._config import ContextConfig, InjectionConfig, ModelConfig, ReActConfig
+from ._realtime import RealtimeAgent, TurnAggregator, TurnMetrics
__all__ = [
"Agent",
"A2AAgent",
+ "RealtimeAgent",
+ "TurnAggregator",
+ "TurnMetrics",
"ContextConfig",
"InjectionConfig",
"ModelConfig",
diff --git a/src/agentscope/agent/_realtime/__init__.py b/src/agentscope/agent/_realtime/__init__.py
new file mode 100644
index 000000000..2891b12a0
--- /dev/null
+++ b/src/agentscope/agent/_realtime/__init__.py
@@ -0,0 +1,7 @@
+# -*- coding: utf-8 -*-
+"""The realtime voice agent."""
+from ._agent import RealtimeAgent
+from ._aggregator import TurnAggregator
+from ._metrics import TurnMetrics
+
+__all__ = ["RealtimeAgent", "TurnAggregator", "TurnMetrics"]
diff --git a/src/agentscope/agent/_realtime/_agent.py b/src/agentscope/agent/_realtime/_agent.py
new file mode 100644
index 000000000..ba3cb228b
--- /dev/null
+++ b/src/agentscope/agent/_realtime/_agent.py
@@ -0,0 +1,1034 @@
+# -*- coding: utf-8 -*-
+"""The realtime voice agent."""
+import asyncio
+import base64
+import time
+import uuid
+from dataclasses import dataclass, field
+from typing import Any, AsyncIterator
+
+from ...realtime import _events as me
+from ._aggregator import TurnAggregator
+from ...realtime._base import ModelDisconnectedError, RealtimeModelBase
+from ._metrics import TurnMetrics
+from ...realtime._transport._base import (
+ AudioFrame,
+ ControlFrame,
+ ControlFrameType,
+ TransportBase,
+)
+from ...realtime._vad import SpeechTransition, VADBase
+from ..._logging import logger
+from ..._utils._common import _json_loads_with_repair
+from ...event import (
+ AgentEvent,
+ ConfirmResult,
+ DataBlockDeltaEvent,
+ DataBlockEndEvent,
+ DataBlockStartEvent,
+ ExternalExecutionResultEvent,
+ ModelCallEndEvent,
+ ModelCallStartEvent,
+ ReplyEndEvent,
+ ReplyStartEvent,
+ RequireUserConfirmEvent,
+ TextBlockDeltaEvent,
+ TextBlockEndEvent,
+ TextBlockStartEvent,
+ ToolCallEndEvent,
+ ToolCallStartEvent,
+ ToolResultEndEvent,
+ ToolResultStartEvent,
+ ToolResultTextDeltaEvent,
+ UserConfirmResultEvent,
+ UserInputAudioStartEvent,
+ UserInputTranscriptionEvent,
+ UserInterruptEvent,
+)
+from ...message import (
+ Msg,
+ TextBlock,
+ ToolCallBlock,
+ ToolResultBlock,
+ ToolResultState,
+ UserMsg,
+)
+from ...permission import PermissionBehavior, PermissionEngine
+from ...state import AgentState
+from ...tool import ToolChunk, ToolResponse, Toolkit
+from ...types import ReplyFinishedReason
+
+# Audio buffered while the model is being reconnected: 10 s at 100 ms chunks.
+_BACKLOG_FRAMES = 100
+
+
+@dataclass
+class _Reply:
+ """One in-flight assistant turn, plus the text/audio alignment needed
+ to work out what the user actually heard."""
+
+ item_id: str
+ text_block_id: str = field(default_factory=lambda: uuid.uuid4().hex)
+ audio_block_id: str = field(default_factory=lambda: uuid.uuid4().hex)
+ text: str = ""
+ audio_ms: float = 0.0
+ text_started: bool = False
+ audio_started: bool = False
+ marks: list[tuple[float, int]] = field(default_factory=list)
+
+ def on_audio(self, pcm: bytes, sample_rate: int) -> None:
+ """Account for one chunk of generated audio."""
+ self.audio_ms += len(pcm) / (sample_rate * 2) * 1000
+
+ def on_text(self, delta: str) -> None:
+ """Record a transcript delta against the audio generated so far."""
+ self.text += delta
+ self.marks.append((self.audio_ms, len(self.text)))
+
+ def spoken_prefix(self, played_ms: int) -> str:
+ """The transcript prefix matching *played_ms* of playback.
+
+ Transcript deltas usually run slightly ahead of the audio they
+ describe, so this errs towards keeping one word too many.
+ """
+ length = 0
+ for at_ms, text_len in self.marks:
+ if at_ms > played_ms:
+ break
+ length = text_len
+ return self.text[:length]
+
+
+class RealtimeAgent:
+ """A voice agent: a realtime model on one side, a transport on the
+ other, and the turn-taking state machine in between.
+
+ Unlike :class:`~agentscope.agent.Agent` it is bidirectional and has no
+ request/reply boundary — audio flows in continuously while events flow
+ out of :meth:`run`.
+
+ Three lifetimes are kept apart on purpose. The agent owns the model
+ session and the state; a transport is owned by whoever created it; a
+ :meth:`run` borrows both for as long as both are alive. So a client
+ can drop and reconnect without losing the model session, and the model
+ session can time out during a long silence and be re-established on
+ the next word without touching the transport.
+
+ The public methods are the only entry point for discrete input. A
+ transport carrying a browser's ``ControlFrame`` calls exactly those
+ methods rather than reaching into the agent, so one code path handles
+ the semantics whether the caller is Python or a browser. Continuous
+ audio is separate and always arrives through the transport.
+
+ Example:
+ .. code-block:: python
+
+ agent = RealtimeAgent("Friday", "Be brief.", model)
+ async with agent: # model session
+ async with LocalAudioTransport() as t: # sound card
+ async for event in agent.run(t): # this call
+ print(event)
+ """
+
+ def __init__(
+ self,
+ name: str,
+ sys_prompt: str,
+ model: RealtimeModelBase,
+ toolkit: Toolkit | None = None,
+ state: AgentState | None = None,
+ vad: VADBase | None = None,
+ aggregator: TurnAggregator | None = None,
+ ) -> None:
+ """Initialize the realtime agent.
+
+ Args:
+ name (`str`):
+ Display name stamped on assistant messages and events.
+ sys_prompt (`str`):
+ System instructions sent to the model on connect, with the
+ toolkit's skill instructions appended.
+ model (`RealtimeModelBase`):
+ The realtime model. Its session is opened by
+ :meth:`connect` and lives as long as this agent — not as
+ long as any one transport — and is re-established on the
+ next user audio if the provider closes it.
+ toolkit (`Toolkit | None`, optional):
+ Tools the model may call. Executed here, with permission
+ checks against ``state.permission_context``.
+ state (`AgentState | None`, optional):
+ Conversation history, permission rules and tool context.
+ A new one is created if omitted.
+ vad (`VADBase | None`, optional):
+ Server-side voice activity detection over the audio
+ arriving from the transport. ``STARTED`` cuts off a reply
+ the user speaks over; ``ENDED`` closes the user's turn and
+ hands it to the model. When given, the provider's own turn
+ detection is switched off so that exactly one source
+ decides turn boundaries; when ``None``, the provider
+ decides and this agent only reacts to what it reports.
+ aggregator (`TurnAggregator | None`, optional):
+ Collapses the provider's transcripts into clean user turns:
+ merging one that endpointing split, dropping bare
+ acknowledgements. Subclass it to change what counts as a
+ turn. A default with no backchannel list is used if
+ omitted.
+ """
+ self.name = name
+ self.sys_prompt = sys_prompt
+ self.model = model
+ self.toolkit = toolkit
+ self.state = state or AgentState()
+ self.vad = vad
+ self.aggregator = aggregator or TurnAggregator()
+
+ self._engine = PermissionEngine(self.state.permission_context)
+ self._transport: TransportBase | None = None
+ self._out: asyncio.Queue = asyncio.Queue()
+ self._reply: _Reply | None = None
+ self._finished_item = ""
+ self._metrics = TurnMetrics()
+ self._pending_tools: dict[str, ToolCallBlock] = {}
+ self._confirmations: dict[str, asyncio.Future[ConfirmResult]] = {}
+ self._tasks: set[asyncio.Task] = set()
+ self._barge_lock = asyncio.Lock()
+
+ # Model session: connected flag, the downlink pump that outlives
+ # any transport, and reconnect bookkeeping.
+ self._connected = False
+ self._connected_event = asyncio.Event()
+ self._downlink: asyncio.Task | None = None
+ self._backlog: list[bytes] = []
+ self._retry_at = 0.0
+ self._backoff = 1.0
+
+ # ------------------------------------------------------------------
+ # Lifecycle: the model session
+ # ------------------------------------------------------------------
+
+ async def __aenter__(self) -> "RealtimeAgent":
+ """Connect on entry."""
+ await self.connect()
+ return self
+
+ async def __aexit__(self, *exc: Any) -> None:
+ """Disconnect on exit."""
+ await self.close()
+
+ async def connect(self) -> None:
+ """Open the model session. Safe to call again after the provider
+ closed it; a no-op while connected."""
+ if self._connected:
+ return
+
+ instructions = self.sys_prompt
+ tools = None
+ if self.toolkit is not None:
+ groups = self.state.tool_context.activated_groups
+ skills = await self.toolkit.get_skill_instructions(groups)
+ if skills:
+ instructions = f"{instructions}\n\n{skills}"
+ if self.model.card.supports_tools:
+ tools = await self.toolkit.get_tool_schemas(groups)
+
+ # TODO(realtime): tools and instructions are sent once, here.
+ # Activating a tool group or installing a skill mid-session —
+ # ResetTools, the meta tool — therefore has no effect until the
+ # next connect, even though the model is told it can do it.
+ #
+ # Fix: a `RealtimeModelBase.update_session(instructions, tools)`
+ # re-sent whenever `state.tool_context.activated_groups` changes.
+ # OpenAI, DashScope and xAI accept `session.update` on the open
+ # connection; Gemini forbids it and must reconnect with a
+ # `sessionResumption` handle plus a new `setup`, which keeps the
+ # context. No provider documents whether the update applies
+ # retroactively, so treat it as affecting future turns only. Do
+ # not let it change `voice`: OpenAI locks it after first audio.
+ await self.model.connect(
+ context=self.state.context,
+ instructions=instructions,
+ tools=tools,
+ turn_detection_disabled=self.vad is not None,
+ )
+ if self.vad is not None:
+ self.vad.reset()
+ self.aggregator.reset()
+ self._connected = True
+ self._connected_event.set()
+ self._backoff = 1.0
+ if self._downlink is None:
+ self._downlink = asyncio.create_task(
+ self._pump_downlink(),
+ name="rt-downlink",
+ )
+ self._downlink.add_done_callback(self._on_downlink_done)
+
+ async def close(self) -> None:
+ """Cancel everything in flight and close the model session."""
+ for future in self._confirmations.values():
+ future.cancel()
+ self._confirmations.clear()
+ for task in list(self._tasks):
+ task.cancel()
+ if self._tasks:
+ await asyncio.gather(*self._tasks, return_exceptions=True)
+ if self._downlink is not None:
+ self._downlink.cancel()
+ await asyncio.gather(self._downlink, return_exceptions=True)
+ self._downlink = None
+ self._connected = False
+ self._connected_event.clear()
+ await self.model.close()
+
+ def _on_downlink_done(self, task: asyncio.Task) -> None:
+ """Surface a crashed downlink instead of a silently dead session."""
+ if task.cancelled() or task.exception() is None:
+ return
+ self._connected = False
+ self._connected_event.clear()
+ logger.error(
+ "RealtimeAgent: downlink pump crashed",
+ exc_info=task.exception(),
+ )
+
+ async def _try_connect(self) -> bool:
+ """Reconnect the model with backoff; ``False`` while still down."""
+ now = time.monotonic()
+ if now < self._retry_at:
+ return False
+ try:
+ await self.connect()
+ except Exception as exc: # noqa: BLE001
+ self._retry_at = now + self._backoff
+ self._backoff = min(self._backoff * 2, 30.0)
+ logger.warning(
+ "RealtimeAgent: reconnect failed (%s); retrying in %.0fs",
+ exc,
+ self._backoff,
+ )
+ return False
+ return True
+
+ # ------------------------------------------------------------------
+ # Lifecycle: one transport
+ # ------------------------------------------------------------------
+
+ async def run(self, transport: TransportBase) -> AsyncIterator[AgentEvent]:
+ """Pump *transport* and yield agent events until it ends.
+
+ The transport is borrowed, not owned: it must already be started
+ and is not closed here, so the caller can keep it, reuse it or hand
+ it on. The stream ends when the transport's input ends — a client
+ disconnecting — not when a reply ends; there may be many replies.
+ Call again with a new transport to resume the same session.
+
+ Breaking out of the loop early does not run cleanup immediately;
+ wrap the generator in :func:`contextlib.aclosing` when that
+ matters.
+ """
+ if self._transport is not None:
+ raise RuntimeError("RealtimeAgent.run is already active.")
+ # Audio is forwarded as-is in both directions, so the rates must
+ # already agree; resampling belongs to the transport, not here.
+ pairs = {
+ "input": (
+ transport.input_sample_rate,
+ self.model.input_sample_rate,
+ ),
+ "output": (
+ transport.output_sample_rate,
+ self.model.output_sample_rate,
+ ),
+ }
+ if self.vad is not None:
+ pairs["vad"] = (self.vad.sample_rate, transport.input_sample_rate)
+ for what, (got, want) in pairs.items():
+ if got != want:
+ raise ValueError(
+ f"{what} sample rate mismatch: {got} Hz delivered, "
+ f"{want} Hz expected.",
+ )
+
+ self._transport = transport
+ uplink = asyncio.create_task(
+ self._pump_uplink(transport),
+ name="rt-up",
+ )
+ try:
+ while not uplink.done():
+ getter = asyncio.ensure_future(self._out.get())
+ done, _ = await asyncio.wait(
+ {getter, uplink},
+ return_when=asyncio.FIRST_COMPLETED,
+ )
+ if getter in done:
+ yield getter.result()
+ else:
+ getter.cancel()
+ # The transport is gone: cut off any reply still in flight so
+ # the model stops talking to nobody, then hand the caller the
+ # events that produced (the interrupted ReplyEnd) before the
+ # stream ends rather than leaking them into the next run.
+ await self._barge_in()
+ while not self._out.empty():
+ yield self._out.get_nowait()
+ # A transport failure must not look like a clean disconnect.
+ if not uplink.cancelled() and uplink.exception() is not None:
+ raise uplink.exception() # type: ignore[misc]
+ finally:
+ if not uplink.done():
+ uplink.cancel()
+ await asyncio.gather(uplink, return_exceptions=True)
+ self._transport = None
+
+ # ------------------------------------------------------------------
+ # Discrete input
+ # ------------------------------------------------------------------
+
+ async def send(
+ self,
+ inputs: (
+ str
+ | Msg
+ | UserConfirmResultEvent
+ | UserInterruptEvent
+ | ExternalExecutionResultEvent
+ ),
+ ) -> None:
+ """Feed the agent anything that is not audio.
+
+ Mirrors :meth:`Agent.reply`'s ``inputs``: a text turn (which cuts
+ off any reply in progress first), the outcome of a permission
+ prompt, an interrupt, or the result of a tool run elsewhere.
+
+ Raises:
+ `NotImplementedError`: For a text turn when the provider takes
+ no text input, or for external execution results, which
+ are not supported yet.
+ """
+ match inputs:
+ case UserInterruptEvent():
+ await self._barge_in()
+
+ case UserConfirmResultEvent():
+ for result in inputs.confirm_results:
+ future = self._confirmations.get(result.tool_call.id)
+ if future and not future.done():
+ future.set_result(result)
+
+ case ExternalExecutionResultEvent():
+ raise NotImplementedError(
+ "External tool execution is not supported yet.",
+ )
+
+ case str() | Msg():
+ if not self.model.supports_text_input:
+ raise NotImplementedError(
+ f"{type(self.model).__name__} accepts no text input.",
+ )
+ msg = (
+ UserMsg(name="user", content=inputs)
+ if isinstance(inputs, str)
+ else inputs
+ )
+ text = msg.get_text_content() or ""
+ await self._barge_in()
+ self.state.context.append(msg)
+ await self.model.push_text(text)
+
+ async def interrupt(self) -> None:
+ """Stop the active reply, as when the user presses stop."""
+ await self.send(UserInterruptEvent())
+
+ @property
+ def last_turn_metrics(self) -> TurnMetrics:
+ """Latency breakdown of the most recent turn."""
+ return self._metrics
+
+ # ------------------------------------------------------------------
+ # Uplink: transport -> model
+ # ------------------------------------------------------------------
+
+ async def _pump_uplink(self, transport: TransportBase) -> None:
+ """Forward user audio and control frames until *transport* ends."""
+ async for frame in transport.incoming():
+ if isinstance(frame, AudioFrame):
+ await self._on_audio(frame)
+ else:
+ await self._on_control(frame)
+
+ async def _on_audio(self, frame: AudioFrame) -> None:
+ """Run our VAD if we own it, reconnect if needed, forward audio."""
+ speech = self.vad.push(frame.pcm) if self.vad is not None else None
+ try:
+ await self._forward_audio(frame.pcm, speech)
+ except ModelDisconnectedError as exc:
+ # The provider closed on us between two frames; the downlink
+ # may not have noticed yet. Keep the frame and reconnect on
+ # the next one — this is the idle-timeout path, not an error.
+ self._mark_disconnected()
+ self._backlog.append(frame.pcm)
+ logger.info(
+ "RealtimeAgent: model session closed (%s); keep talking "
+ "and it reconnects on the next audio.",
+ exc,
+ )
+
+ def _mark_disconnected(self) -> None:
+ """Forget the model session so the next audio reconnects."""
+ self._connected = False
+ self._connected_event.clear()
+
+ async def _forward_audio(
+ self,
+ pcm: bytes,
+ speech: SpeechTransition | None,
+ ) -> None:
+ """Body of :meth:`_on_audio`; raises on a closed model session."""
+ pushed = False
+ if not self._connected:
+ self._backlog.append(pcm)
+ del self._backlog[:-_BACKLOG_FRAMES]
+ if not await self._try_connect():
+ return
+ for buffered in self._backlog:
+ await self.model.push_audio(buffered)
+ self._backlog.clear()
+ pushed = True
+
+ if speech is SpeechTransition.STARTED:
+ await self._barge_in()
+ elif speech is SpeechTransition.ENDED:
+ now = time.monotonic()
+ self._metrics.user_speech_end_at = now
+ await self.model.commit_turn()
+ self._metrics.turn_committed_at = time.monotonic()
+
+ if not pushed:
+ await self.model.push_audio(pcm)
+
+ async def _on_control(self, frame: ControlFrame) -> None:
+ """Translate one upstream control frame into :meth:`send`."""
+ match frame.type:
+ case ControlFrameType.TEXT:
+ await self.send(frame.data.get("text", ""))
+ case ControlFrameType.USER_CONFIRM:
+ await self.send(UserConfirmResultEvent(**frame.data))
+ case ControlFrameType.INTERRUPT:
+ await self.send(UserInterruptEvent())
+ case ControlFrameType.EXTERNAL_EXECUTION_RESULT:
+ await self.send(ExternalExecutionResultEvent(**frame.data))
+ case _:
+ logger.debug("RealtimeAgent: ignoring %s frame", frame.type)
+
+ # ------------------------------------------------------------------
+ # Barge-in
+ # ------------------------------------------------------------------
+
+ async def _barge_in(self) -> None:
+ """Cut the reply short and correct both contexts to what was heard.
+
+ Whether a given overlap counts as an interruption is decided
+ before this is called — by the provider when it owns turn
+ detection, by the VAD's own debounce otherwise.
+
+ Reached from the uplink pump, the downlink pump, :meth:`send` and
+ :meth:`run`'s exit, so it is serialised; the losers find the reply
+ already closed and return.
+ """
+ async with self._barge_lock:
+ await self._barge_in_locked()
+
+ async def _barge_in_locked(self) -> None:
+ """Body of :meth:`_barge_in`, run under the lock."""
+ reply = self._reply
+ if reply is None:
+ return
+
+ spoken, played_ms = (
+ "",
+ 0,
+ ) # nothing reaches the ear without a transport
+ if self._transport is not None:
+ position = await self._transport.clear_audio()
+ if position.item_id and position.item_id != reply.item_id:
+ logger.warning(
+ "RealtimeAgent: playout reports %s but %s is open; "
+ "not truncating.",
+ position.item_id,
+ reply.item_id,
+ )
+ return
+ played_ms = position.played_ms
+ spoken = reply.spoken_prefix(played_ms)
+
+ self._truncate_reply(spoken)
+ if self._connected:
+ await self.model.truncate(reply.item_id, played_ms, spoken)
+ await self.model.cancel_response()
+ self._finish_reply(ReplyFinishedReason.INTERRUPTED)
+
+ def _truncate_reply(self, spoken: str) -> None:
+ """Rewrite the current reply in context to the part heard.
+
+ Non-text blocks stay: a tool call that already ran belongs in the
+ record even though the sentence around it was never heard.
+ """
+ if not self.state.context:
+ return
+ tail = self.state.context[-1]
+ if tail.role != "assistant" or tail.name != self.name:
+ return
+
+ others = (
+ []
+ if isinstance(tail.content, str)
+ else [_ for _ in tail.content if not isinstance(_, TextBlock)]
+ )
+ if spoken.strip():
+ tail.content = [TextBlock(text=spoken), *others]
+ elif others:
+ tail.content = others
+ else:
+ self.state.context.pop()
+
+ # ------------------------------------------------------------------
+ # Downlink: model -> transport + events (lives with the agent)
+ # ------------------------------------------------------------------
+
+ async def _pump_downlink(self) -> None:
+ """Translate model events for as long as the agent is open.
+
+ When the provider closes the session the pump does not exit: it
+ marks the model disconnected and waits for :meth:`connect` to be
+ called again, which the uplink does on the next user audio.
+ """
+ while True:
+ await self._connected_event.wait()
+ async for event in self.model.events():
+ await self._on_model_event(event)
+ self._mark_disconnected()
+ self._finish_reply(ReplyFinishedReason.ERROR)
+ logger.info(
+ "RealtimeAgent: model session ended; keep talking and it "
+ "reconnects on the next audio.",
+ )
+
+ async def _on_model_event(self, event: me.ModelEvent) -> None:
+ """Handle one model event."""
+ rate = self.model.output_sample_rate
+ match event:
+ case me.SpeechStartedEvent():
+ self._emit(
+ UserInputAudioStartEvent(
+ session_id=self.state.session_id,
+ item_id=event.item_id,
+ ),
+ )
+ await self._barge_in()
+
+ case me.SpeechEndedEvent():
+ # With provider turn detection this is also its commit.
+ now = time.monotonic()
+ self._metrics.user_speech_end_at = now
+ if self.vad is None:
+ self._metrics.turn_committed_at = now
+
+ case me.InputTranscriptionEvent():
+ self._on_transcription(event)
+
+ case me.ResponseCreatedEvent():
+ self._start_reply(event.item_id)
+
+ case me.AudioDeltaEvent():
+ if self._transport is None:
+ await self._barge_in() # nobody listening
+ return
+ reply = self._start_reply(event.item_id)
+ if reply is None:
+ return
+ reply.on_audio(event.pcm, rate)
+ await self._transport.send_audio(event.pcm, reply.item_id)
+ self._emit_audio(reply, event.pcm, rate)
+ self._metrics.backend_first_audio_at = (
+ self._metrics.backend_first_audio_at or time.monotonic()
+ )
+
+ case me.TranscriptDeltaEvent():
+ reply = self._start_reply(event.item_id)
+ if reply is None:
+ return
+ reply.on_text(event.delta)
+ self._emit_text(reply, event.delta)
+
+ case me.ToolCallEvent():
+ if self._start_reply(event.item_id) is not None:
+ self._pending_tools[event.tool_call.id] = event.tool_call
+
+ case me.ResponseDoneEvent():
+ self._metrics.input_tokens = event.input_tokens
+ self._metrics.output_tokens = event.output_tokens
+ self._finish_reply(ReplyFinishedReason.COMPLETED)
+ self._schedule_tools()
+
+ case me.ModelErrorEvent():
+ logger.error(
+ "RealtimeAgent: model error %s: %s",
+ event.code,
+ event.message,
+ )
+ self._finish_reply(ReplyFinishedReason.ERROR)
+
+ case me.SessionEndedEvent():
+ pass # the events() iterator ends right after this
+
+ def _on_transcription(self, event: me.InputTranscriptionEvent) -> None:
+ """Record a settled user turn, merging a split one back together."""
+ turn = self.aggregator.take(event.text)
+ if turn is None:
+ logger.debug("RealtimeAgent: dropping %r", event.text)
+ return
+
+ if self.aggregator.merges_with_previous() and self._merge_user(turn):
+ transcript = self.state.context[-1].get_text_content() or turn
+ else:
+ self.state.context.append(UserMsg(name="user", content=turn))
+ transcript = turn
+
+ self._emit(
+ UserInputTranscriptionEvent(
+ session_id=self.state.session_id,
+ item_id=event.item_id,
+ transcript=transcript,
+ ),
+ )
+
+ def _merge_user(self, text: str) -> bool:
+ """Append *text* to the previous user turn that endpointing split.
+
+ The stub assistant message between the two halves is dropped: it
+ is whatever the model managed to say before being cut off, which
+ answers half a question nobody finished asking.
+ """
+ context = self.state.context
+ if context and context[-1].role == "assistant":
+ if not (context[-1].get_text_content() or "").strip():
+ context.pop()
+ if not context or context[-1].role != "user":
+ return False
+ previous = context[-1].get_text_content() or ""
+ context[-1].content = [TextBlock(text=f"{previous}{text}")]
+ return True
+
+ def _start_reply(self, item_id: str) -> _Reply | None:
+ """Open a reply for *item_id*, emitting its start events once.
+
+ Returns ``None`` for an item already closed — deltas still in
+ flight after a barge-in must not reopen it.
+ """
+ if self._reply is not None and self._reply.item_id == item_id:
+ return self._reply
+ if item_id == self._finished_item:
+ return None
+
+ self._reply = _Reply(item_id=item_id)
+ self.state.reply_id = item_id
+ self._metrics = TurnMetrics(
+ user_speech_end_at=self._metrics.user_speech_end_at,
+ turn_committed_at=self._metrics.turn_committed_at,
+ )
+ self._emit(
+ ReplyStartEvent(
+ session_id=self.state.session_id,
+ reply_id=item_id,
+ name=self.name,
+ ),
+ )
+ self._emit(
+ ModelCallStartEvent(
+ reply_id=item_id,
+ model_name=self.model.model_name,
+ ),
+ )
+ return self._reply
+
+ def _finish_reply(self, reason: ReplyFinishedReason) -> None:
+ """Close the open reply, if any."""
+ reply = self._reply
+ if reply is None:
+ return
+ if self._transport is not None:
+ position = self._transport.playout()
+ if position.item_id == reply.item_id:
+ self._metrics.first_audio_played_at = position.first_played_at
+ if reply.text_started:
+ self._emit(
+ TextBlockEndEvent(
+ reply_id=reply.item_id,
+ block_id=reply.text_block_id,
+ ),
+ )
+ if reply.audio_started:
+ self._emit(
+ DataBlockEndEvent(
+ reply_id=reply.item_id,
+ block_id=reply.audio_block_id,
+ ),
+ )
+ self._emit(
+ ModelCallEndEvent(
+ reply_id=reply.item_id,
+ input_tokens=self._metrics.input_tokens,
+ output_tokens=self._metrics.output_tokens,
+ ),
+ )
+ self._emit(
+ ReplyEndEvent(
+ session_id=self.state.session_id,
+ reply_id=reply.item_id,
+ finished_reason=reason,
+ ),
+ )
+ self._finished_item = reply.item_id
+ self._reply = None
+
+ def _emit_text(self, reply: _Reply, delta: str) -> None:
+ """Emit a transcript delta, opening the block on first use."""
+ if not reply.text_started:
+ reply.text_started = True
+ self._emit(
+ TextBlockStartEvent(
+ reply_id=reply.item_id,
+ block_id=reply.text_block_id,
+ ),
+ )
+ # Grow the tail text block rather than appending one per delta,
+ # or the context ends up as dozens of one-word blocks.
+ tail = self.state.context[-1] if self.state.context else None
+ blocks = (
+ tail.content
+ if tail is not None and tail.id == reply.item_id
+ else None
+ )
+ if (
+ isinstance(blocks, list)
+ and blocks
+ and isinstance(blocks[-1], TextBlock)
+ ):
+ blocks[-1].text += delta
+ else:
+ self.state.append_context(self.name, [TextBlock(text=delta)])
+ self._emit(
+ TextBlockDeltaEvent(
+ reply_id=reply.item_id,
+ block_id=reply.text_block_id,
+ delta=delta,
+ ),
+ )
+
+ def _emit_audio(self, reply: _Reply, pcm: bytes, rate: int) -> None:
+ """Emit an audio delta, opening the block on first use."""
+ media_type = f"audio/pcm;rate={rate}"
+ if not reply.audio_started:
+ reply.audio_started = True
+ self._emit(
+ DataBlockStartEvent(
+ reply_id=reply.item_id,
+ block_id=reply.audio_block_id,
+ media_type=media_type,
+ ),
+ )
+ self._emit(
+ DataBlockDeltaEvent(
+ reply_id=reply.item_id,
+ block_id=reply.audio_block_id,
+ data=base64.b64encode(pcm).decode("ascii"),
+ media_type=media_type,
+ ),
+ )
+
+ def _emit(self, event: AgentEvent) -> None:
+ """Queue one event for :meth:`run`."""
+ self._out.put_nowait(event)
+
+ # ------------------------------------------------------------------
+ # Tools
+ # ------------------------------------------------------------------
+
+ def _schedule_tools(self) -> None:
+ """Run the tool calls of the finished reply, then ask for more."""
+ if not self._pending_tools or self.toolkit is None:
+ return
+ calls = list(self._pending_tools.values())
+ self._pending_tools.clear()
+ task = asyncio.create_task(self._run_tools(calls), name="rt-tools")
+ self._tasks.add(task)
+ task.add_done_callback(self._tasks.discard)
+
+ async def _run_tools(self, calls: list[ToolCallBlock]) -> None:
+ """Execute *calls* in order, then trigger the follow-up response."""
+ reply_id = self.state.reply_id
+ try:
+ for call in calls:
+ await self._run_tool(reply_id, call)
+ await self.model.request_response()
+ except Exception: # noqa: BLE001
+ logger.exception("RealtimeAgent: tool execution failed")
+
+ async def _run_tool(self, reply_id: str, call: ToolCallBlock) -> None:
+ """Check permission for one call, run it, and report the result."""
+ assert self.toolkit is not None
+ self._emit(
+ ToolCallStartEvent(
+ reply_id=reply_id,
+ tool_call_id=call.id,
+ tool_call_name=call.name,
+ ),
+ )
+ self._emit(ToolCallEndEvent(reply_id=reply_id, tool_call_id=call.id))
+
+ try:
+ tool = await self.toolkit.check_tool_available(
+ call.name,
+ self.state.tool_context.activated_groups,
+ )
+ tool_input = _json_loads_with_repair(call.input, tool.input_schema)
+ except Exception as exc: # noqa: BLE001
+ await self._report_tool(reply_id, call, str(exc))
+ return
+
+ decision = await self._engine.check_permission(tool, tool_input)
+ if decision.behavior in (
+ PermissionBehavior.ASK,
+ PermissionBehavior.PASSTHROUGH,
+ ):
+ confirmed = await self._ask_user(reply_id, call, decision)
+ if not confirmed:
+ await self._report_tool(
+ reply_id,
+ call,
+ f'Tool "{call.name}" denied by user.',
+ state=ToolResultState.DENIED,
+ )
+ return
+ elif decision.behavior is PermissionBehavior.DENY:
+ await self._report_tool(
+ reply_id,
+ call,
+ decision.message or f'Tool "{call.name}" denied by policy.',
+ state=ToolResultState.DENIED,
+ )
+ return
+
+ self._emit(
+ ToolResultStartEvent(
+ reply_id=reply_id,
+ tool_call_id=call.id,
+ tool_call_name=call.name,
+ ),
+ )
+ parts: list[str] = []
+ streamed = False
+ result_state = ToolResultState.SUCCESS
+ try:
+ async for chunk in self.toolkit.call_tool(call, self.state):
+ texts = [
+ b.text for b in chunk.content if isinstance(b, TextBlock)
+ ]
+ if isinstance(chunk, ToolResponse):
+ # The completed result. Chunks before it were display
+ # only, so this alone is what the provider receives.
+ parts = texts
+ result_state = chunk.state
+ if not streamed:
+ for text in texts:
+ self._emit_tool_delta(reply_id, call.id, text)
+ break
+ if not isinstance(chunk, ToolChunk):
+ break
+ streamed = True
+ for text in texts:
+ self._emit_tool_delta(reply_id, call.id, text)
+ except Exception: # noqa: BLE001
+ logger.exception("RealtimeAgent: tool %s failed", call.name)
+ parts = [f"Error executing tool {call.name}."]
+ result_state = ToolResultState.ERROR
+
+ await self._report_tool(
+ reply_id,
+ call,
+ "".join(parts) or "Tool executed successfully.",
+ state=result_state,
+ started=True,
+ )
+
+ def _emit_tool_delta(self, reply_id: str, call_id: str, text: str) -> None:
+ """Emit one fragment of tool output."""
+ self._emit(
+ ToolResultTextDeltaEvent(
+ reply_id=reply_id,
+ tool_call_id=call_id,
+ delta=text,
+ ),
+ )
+
+ async def _ask_user(
+ self,
+ reply_id: str,
+ call: ToolCallBlock,
+ decision: Any,
+ ) -> bool:
+ """Ask the user to confirm *call* and wait for the answer."""
+ call.suggested_rules = decision.suggested_rules or []
+ self._emit(
+ RequireUserConfirmEvent(reply_id=reply_id, tool_calls=[call]),
+ )
+ future: asyncio.Future[
+ ConfirmResult
+ ] = asyncio.get_running_loop().create_future()
+ self._confirmations[call.id] = future
+ try:
+ result = await asyncio.wait_for(future, timeout=300)
+ except (asyncio.TimeoutError, asyncio.CancelledError):
+ return False
+ finally:
+ self._confirmations.pop(call.id, None)
+
+ for rule in result.rules or []:
+ self._engine.add_rule(rule)
+ return result.confirmed
+
+ async def _report_tool(
+ self,
+ reply_id: str,
+ call: ToolCallBlock,
+ output: str,
+ state: ToolResultState = ToolResultState.ERROR,
+ started: bool = False,
+ ) -> None:
+ """Close the tool result lifecycle and send the output back."""
+ if not started:
+ self._emit(
+ ToolResultStartEvent(
+ reply_id=reply_id,
+ tool_call_id=call.id,
+ tool_call_name=call.name,
+ ),
+ )
+ self._emit(
+ ToolResultTextDeltaEvent(
+ reply_id=reply_id,
+ tool_call_id=call.id,
+ delta=output,
+ ),
+ )
+ self._emit(
+ ToolResultEndEvent(
+ reply_id=reply_id,
+ tool_call_id=call.id,
+ state=state,
+ ),
+ )
+ await self.model.push_tool_result(
+ ToolResultBlock(id=call.id, name=call.name, output=output),
+ )
diff --git a/src/agentscope/agent/_realtime/_aggregator.py b/src/agentscope/agent/_realtime/_aggregator.py
new file mode 100644
index 000000000..c94ae1d6a
--- /dev/null
+++ b/src/agentscope/agent/_realtime/_aggregator.py
@@ -0,0 +1,75 @@
+# -*- coding: utf-8 -*-
+"""Turn aggregation."""
+import time
+
+_TRAILING = " 。,,.!?!?、"
+
+
+class TurnAggregator:
+ """Collapses the transcripts a provider emits into clean user turns.
+
+ Turn boundaries in speech are inferred, and inferred wrongly often
+ enough to matter: one sentence gets split across two turns when the
+ speaker pauses, and a stray "mm-hmm" gets promoted to a turn of its
+ own. Neither is visible to the provider once it has answered, but both
+ end up in our own context, where they outlive the call.
+ """
+
+ def __init__(
+ self,
+ merge_window_ms: int = 800,
+ backchannels: frozenset[str] = frozenset(),
+ min_chars: int = 1,
+ ) -> None:
+ """Initialize the aggregator.
+
+ Args:
+ merge_window_ms (`int`, defaults to `800`):
+ A transcript arriving this soon after the previous turn
+ continues it rather than starting a new one.
+ backchannels (`frozenset[str]`, optional):
+ Acknowledgements that never constitute a turn. Language
+ specific, so empty by default.
+ min_chars (`int`, defaults to `1`):
+ Transcripts shorter than this are dropped.
+ """
+ self.merge_window_ms = merge_window_ms
+ self.backchannels = frozenset(backchannels)
+ self.min_chars = min_chars
+ self._last_at: float | None = None
+ self._merges = False
+
+ def take(self, transcript: str) -> str | None:
+ """Accept a settled transcript and decide whether it is a turn.
+
+ Args:
+ transcript (`str`):
+ The transcript the provider settled on.
+
+ Returns:
+ `str | None`:
+ The turn text, or ``None`` for an empty transcript or a
+ bare acknowledgement.
+ """
+ text = transcript.strip()
+ if len(text) < self.min_chars:
+ return None
+ if text.strip(_TRAILING) in self.backchannels:
+ return None
+
+ now = time.monotonic()
+ self._merges = (
+ self._last_at is not None
+ and (now - self._last_at) * 1000 <= self.merge_window_ms
+ )
+ self._last_at = now
+ return text
+
+ def merges_with_previous(self) -> bool:
+ """Whether the last :meth:`take` continued the previous turn."""
+ return self._merges
+
+ def reset(self) -> None:
+ """Forget the previous turn, e.g. on reconnect."""
+ self._last_at = None
+ self._merges = False
diff --git a/src/agentscope/agent/_realtime/_metrics.py b/src/agentscope/agent/_realtime/_metrics.py
new file mode 100644
index 000000000..43fa62356
--- /dev/null
+++ b/src/agentscope/agent/_realtime/_metrics.py
@@ -0,0 +1,62 @@
+# -*- coding: utf-8 -*-
+"""Per-turn latency metrics."""
+from pydantic import BaseModel, Field
+
+
+class TurnMetrics(BaseModel):
+ """The four timestamps of one turn, and the delays derived from them.
+
+ All timestamps are :func:`time.monotonic` readings, in seconds.
+ """
+
+ user_speech_end_at: float | None = None
+ """The user stopped speaking."""
+
+ turn_committed_at: float | None = None
+ """Endpointing settled and the turn was handed to the backend."""
+
+ backend_first_audio_at: float | None = None
+ """The backend produced its first audio byte."""
+
+ first_audio_played_at: float | None = None
+ """That audio actually reached the speaker. Reported by the transport,
+ and the only timestamp that reflects what the user perceives."""
+
+ input_tokens: int = Field(default=0, ge=0)
+ output_tokens: int = Field(default=0, ge=0)
+
+ @property
+ def endpointing_delay(self) -> float | None:
+ """Time spent deciding the user had finished."""
+ return self._delta(self.user_speech_end_at, self.turn_committed_at)
+
+ @property
+ def backend_ttfb(self) -> float | None:
+ """Time the backend took to start answering."""
+ return self._delta(
+ self.turn_committed_at,
+ self.backend_first_audio_at,
+ )
+
+ @property
+ def transport_delay(self) -> float | None:
+ """Time between audio leaving the server and reaching the ear."""
+ return self._delta(
+ self.backend_first_audio_at,
+ self.first_audio_played_at,
+ )
+
+ @property
+ def e2e_latency(self) -> float | None:
+ """The user stopped speaking until they heard an answer."""
+ return self._delta(
+ self.user_speech_end_at,
+ self.first_audio_played_at,
+ )
+
+ @staticmethod
+ def _delta(start: float | None, end: float | None) -> float | None:
+ """The gap between two optional timestamps."""
+ if start is None or end is None:
+ return None
+ return end - start
diff --git a/src/agentscope/credential/_base.py b/src/agentscope/credential/_base.py
index 906beab76..262141706 100644
--- a/src/agentscope/credential/_base.py
+++ b/src/agentscope/credential/_base.py
@@ -9,6 +9,7 @@ from .._utils._common import _generate_id
if TYPE_CHECKING:
from ..embedding import EmbeddingModelBase
from ..model import ChatModelBase, ModelCard
+ from ..realtime import RealtimeModelBase, RealtimeModelCard
from ..tts import TTSModelBase
from ..tts._tts_model_card import TTSModelCard
@@ -67,6 +68,23 @@ class CredentialBase(BaseModel):
cards.extend(tts_cls.list_models())
return cards
+ @classmethod
+ def get_realtime_model_classes(
+ cls,
+ ) -> list[Type["RealtimeModelBase"]]:
+ """Return the realtime model classes supported by this credential.
+ The default returns an empty list (provider does not support
+ realtime)."""
+ return []
+
+ @classmethod
+ def list_realtime_models(cls) -> list["RealtimeModelCard"]:
+ """List the candidate realtime models under this credential."""
+ cards: list["RealtimeModelCard"] = []
+ for rt_cls in cls.get_realtime_model_classes():
+ cards.extend(rt_cls.list_models())
+ return cards
+
@classmethod
def list_models(cls) -> list["ModelCard"]:
"""List the candidate chat models that are available under this
diff --git a/src/agentscope/credential/_dashscope.py b/src/agentscope/credential/_dashscope.py
index 11e6ab59f..9799b5203 100644
--- a/src/agentscope/credential/_dashscope.py
+++ b/src/agentscope/credential/_dashscope.py
@@ -9,6 +9,7 @@ from ._base import CredentialBase
if TYPE_CHECKING:
from ..embedding import EmbeddingModelBase
from ..model import ChatModelBase
+ from ..realtime import RealtimeModelBase
from ..tts import TTSModelBase
_DASHSCOPE_BASE_URL = "https://dashscope.aliyuncs.com/compatible-mode/v1"
@@ -59,6 +60,18 @@ class DashScopeCredential(CredentialBase):
DashScopeCosyVoiceTTSModel,
]
+ @classmethod
+ def get_realtime_model_classes(
+ cls,
+ ) -> list[Type["RealtimeModelBase"]]:
+ """Return the DashScope realtime model classes."""
+ from ..realtime import (
+ DashScopeAudioRealtimeModel,
+ DashScopeRealtimeModel,
+ )
+
+ return [DashScopeRealtimeModel, DashScopeAudioRealtimeModel]
+
@classmethod
def get_embedding_model_class(cls) -> Type["EmbeddingModelBase"]:
"""Return the DashScopeEmbeddingModel class."""
diff --git a/src/agentscope/event/__init__.py b/src/agentscope/event/__init__.py
index 51632a731..dd9e40ac8 100644
--- a/src/agentscope/event/__init__.py
+++ b/src/agentscope/event/__init__.py
@@ -35,6 +35,8 @@ from ._event import (
ExternalExecutionResultEvent,
CustomEvent,
AgentEvent,
+ UserInputAudioStartEvent,
+ UserInputTranscriptionEvent,
ConfirmResult,
)
@@ -73,5 +75,7 @@ __all__ = [
"ExternalExecutionResultEvent",
"CustomEvent",
"AgentEvent",
+ "UserInputAudioStartEvent",
+ "UserInputTranscriptionEvent",
"ConfirmResult",
]
diff --git a/src/agentscope/event/_event.py b/src/agentscope/event/_event.py
index 59d5f8d4d..8bce51cbd 100644
--- a/src/agentscope/event/_event.py
+++ b/src/agentscope/event/_event.py
@@ -64,6 +64,9 @@ class EventType(StrEnum):
USER_INTERRUPT = "USER_INTERRUPT"
EXTERNAL_EXECUTION_RESULT = "EXTERNAL_EXECUTION_RESULT"
+ USER_INPUT_AUDIO_START = "USER_INPUT_AUDIO_START"
+ USER_INPUT_TRANSCRIPTION = "USER_INPUT_TRANSCRIPTION"
+
CUSTOM = "CUSTOM"
@@ -562,6 +565,34 @@ class CustomEvent(EventBase):
"""Arbitrary payload."""
+class UserInputAudioStartEvent(EventBase):
+ """The user started speaking, as detected by VAD."""
+
+ type: Literal[
+ EventType.USER_INPUT_AUDIO_START
+ ] = EventType.USER_INPUT_AUDIO_START
+ """Event type."""
+ session_id: str
+ """ID of the session this input belongs to."""
+ item_id: str
+ """ID of the user input item the provider assigned."""
+
+
+class UserInputTranscriptionEvent(EventBase):
+ """The settled transcript of one spoken user turn."""
+
+ type: Literal[
+ EventType.USER_INPUT_TRANSCRIPTION
+ ] = EventType.USER_INPUT_TRANSCRIPTION
+ """Event type."""
+ session_id: str
+ """ID of the session this input belongs to."""
+ item_id: str
+ """ID of the user input item the provider assigned."""
+ transcript: str
+ """What the user said."""
+
+
AgentEvent: TypeAlias = (
ReplyStartEvent
| ReplyEndEvent
@@ -590,5 +621,7 @@ AgentEvent: TypeAlias = (
| UserConfirmResultEvent
| UserInterruptEvent
| ExternalExecutionResultEvent
+ | UserInputAudioStartEvent
+ | UserInputTranscriptionEvent
| CustomEvent
)
diff --git a/src/agentscope/realtime/DECISIONS.md b/src/agentscope/realtime/DECISIONS.md
new file mode 100644
index 000000000..d8b734a2b
--- /dev/null
+++ b/src/agentscope/realtime/DECISIONS.md
@@ -0,0 +1,57 @@
+# Realtime 设计决策记录
+
+已定的事项不因单个提问重开;改变需要说明新事实。
+
+## 已定
+
+| 决策 | 依据 |
+|---|---|
+| 三层事实划分:类属性 = adapter 实现;card = 模型是什么;property = 各家归属不同的(采样率、turn detection) | 四家 API 调研:xAI 采样率可调、DashScope 固定 |
+| `TruncationSupport` 三态 NONE / SERVER / EXPLICIT | OpenAI(WS)=EXPLICIT、Gemini=SERVER、DashScope/xAI=NONE |
+| card 四个上下文限制字段并存,全部可选 | DashScope 按轮数+秒数、OpenAI/Gemini 按 token、同时生效 |
+| 公开方法是离散输入的唯一入口,`ControlFrame` 是其线上形态 | livekit / openai SDK / pipecat / vocode 四家一致 |
+| `_barge_in` 串行化,三个调用点共用一把锁 | pipecat CHANGELOG:4712 记录的双队列插队 bug |
+| 复用 `AgentState`,不另建 `VoiceContext` | `append_context` 已覆盖进行中回合的追加语义 |
+| 打断截断留在 `RealtimeAgent._truncate_reply`,不进 `AgentState` | 用户决定,不扩散到 `state/` |
+| `SessionConfig` 二分:`chat`(Agent 驱动)/ `voice`(RealtimeAgent 驱动),内层再按 backend 判别 | 分类依据是驱动的运行时类,不是字段形状 |
+| `TurnAggregator` 在 M1 做上下文合并那一半 | SERVER 模式无提交权,但脏历史会进持久化/重连/压缩 |
+| M1 只做 DashScope,不做 OpenAI | 打断闭环除 provider RPC 外整条链 DashScope 都会跑到 |
+| 不复用 `Agent` 实现级联 | 用户决定 |
+| VAD 归 agent(`RealtimeAgent(vad=)`),不归 transport;无客户端上报的 speech 字段 | livekit 放 session 层、pipecat 主动从 transport 搬出、无一家采信浏览器 speech |
+| 给了 `vad=` 就关 provider 的 turn detection,一个来源判回合;`TurnMode` 枚举删除,由 `vad is None` 推导 | livekit `_resolve_rt_turn_detection_enabled`;pipecat 警告双源发重复帧 |
+| transport 由创建者拥有,`run(transport)` 只借用,不 start/close | livekit `start(room=)` 不关 room;三者平级:agent / transport / run |
+| 下行泵归 agent(`connect()` 起),上行泵归 `run()`;两次 run 之间模型队列照常消费 | 否则空档里工具结果、模型帧无人处理 |
+| 模型会话被 provider 关掉后**不主动重连**,下一帧用户音频触发 `connect()`(可重入,带退避,攒帧) | 静默时挂着 API 连接没必要 |
+| 离散输入统一为 `send(inputs)`,镜像 `Agent.reply(inputs=)`;`interrupt()` 为快捷方式 | `send_text` / `send_confirm` 合并;`ExternalExecutionResultEvent` 入口留着,M1 抛 NotImplementedError |
+| 流式接口叫 `run`,async generator,起动即迭代 | 与 `reply_stream` 结束条件不同(连接断才结束),不复用其名 |
+| `TurnAggregator` 作为实体注入(`aggregator=`),不经 config;pydantic `TurnConfig` 只存在于 app 层 | 它不依赖 agent 内部件,可子类化(pipecat 的 turn strategy 家族) |
+| `fade_ms` 归 transport 构造参数 | 淡出由 transport 执行 |
+| provider 连接异常由 adapter 翻译成 `ModelDisconnectedError`,agent 只认它、不 import websockets;发送失败 = 断开,攒帧等下一帧重连,上行泵不死 | 真机复现:DashScope 180s 空闲关闭后 `push_audio` 撞上已关 socket,比下行泵察觉更早 |
+| `run()` 入口校验 transport / VAD 与 model 的采样率,不一致直接拒绝;agent 不做重采样 | Copilot review;重采样归 transport |
+| 工具结果只取 `ToolResponse.content`,`ToolChunk` 仅用于展示 | `ToolResponse` 是完整结果(与 `Agent` 一致),否则流式工具的文本会重复 |
+| `RealtimeAgent` / `TurnAggregator` / `TurnMetrics` 放 `agent/_realtime/`;`realtime/` 只放模型侧(model、card、事件、transport、VAD) | 与 `Agent` 同目录;`agent/` 单向依赖 `realtime/` |
+| 模型事件统一 `Event` 后缀(`AudioDeltaEvent` 等),基类 `ModelEvent` | 对齐 `agentscope.event` 的命名 |
+| 命名保留 `realtime`,不改 `live`;agent 层将来挂级联后端时再评估 `VoiceAgent` | 六家里五家叫 Realtime,M1 的 provider 产品名就是 Qwen-Omni-Realtime |
+| DashScope 的 Omni 和 Audio-3.0 两个 S2S API 用两个类,不在一个类里按模型名路由 | `supports_text_input` 等是 per-API 事实;`tts/_dashscope/` 一个 provider 三个类的先例 |
+| 反向查找靠 card 上的 `model_type`(产出它的 class 的 `type` tag),服务层建 `{cls.type: cls}` dict,不扫名字 | 同名模型可出现在两个 class 的 card 里(OpenAI chat/responses)|
+| `TurnMetrics` 是 agent 的**产出**(四个时刻),不是配置,不进构造函数;`last_turn_metrics` 读取 | 每回合的指标事件(livekit `metrics_collected`)待前端需要时再加 |
+| `PlayoutPosition.first_played_at` 由 transport 在音频线程记录,agent 在回合结束时读取 | e2e_latency 的唯一真实来源;随 `LocalAudioTransport` 落地 |
+| 用户打字先打断当前回复 | livekit / pipecat 默认;不打断需要额外的待处理状态 |
+
+## 已推迟(保留概念,届时形态如下)
+
+| 事项 | 阶段 | 预期形态 | 为什么现在不做 |
+|---|---|---|---|
+| filler(工具执行期间的填充语) | M2 | 本地 TTS 播放,不走 provider | DashScope 不收文本输入 |
+| 独立的打断判定(叠在 VAD 之上) | M2 | 词数门槛(pipecat)或独立模型(livekit) | SERVER 模式下 provider 已决定;M1 无本地 VAD |
+| `update_session(instructions, tools)` | M2 | OpenAI/DashScope/xAI 发 `session.update`;Gemini 重连+resumption handle;不改 `voice` | 见 `_agent.py` connect() 处 TODO |
+| `ModelEvent.SessionResumption` | Gemini 接入时 | 随 Gemini adapter 一起回来 | Gemini 唯一的改 instructions 路径 |
+| 语义端点检测 | M3 之后 | 先接 pipecat smart-turn ONNX | OpenAI/DashScope 已有 `semantic_vad` |
+| 上下文压缩 | 重连功能时 | `summarize(messages) -> str` 灌进新 session | S2S 下压缩本地历史不影响 provider |
+| WebSocket transport 兜底 | Safari 支持面确认后 | 同 9 字节头 + 二进制 Opus | 优先 WebTransport |
+
+## 待定
+
+- `SessionConfig` 二分改造单独 PR 的时机
+- 模型重连后 UI 提示事件(M1 只记日志)
+- `AudioDelta` 事件的 base64 惰性编码(local 场景全部白做)
diff --git a/src/agentscope/realtime/__init__.py b/src/agentscope/realtime/__init__.py
new file mode 100644
index 000000000..97e1d3def
--- /dev/null
+++ b/src/agentscope/realtime/__init__.py
@@ -0,0 +1,69 @@
+# -*- coding: utf-8 -*-
+"""Realtime voice sessions in AgentScope.
+
+The model side of a voice session: realtime models, their cards and
+events, transports and VAD. The agent that ties them together is
+:class:`agentscope.agent.RealtimeAgent`.
+"""
+from ._base import ModelDisconnectedError, RealtimeModelBase, TruncationSupport
+from ._dashscope import (
+ DashScopeAudioRealtimeModel,
+ DashScopeRealtimeModel,
+)
+from ._events import (
+ AudioDeltaEvent,
+ InputTranscriptionEvent,
+ ModelErrorEvent,
+ ModelEvent,
+ ResponseCreatedEvent,
+ ResponseDoneEvent,
+ SessionEndedEvent,
+ SpeechEndedEvent,
+ SpeechStartedEvent,
+ ToolCallEvent,
+ TranscriptDeltaEvent,
+)
+from ._model_card import RealtimeModelCard
+from ._playout import PlayoutPosition
+from ._transport import (
+ AudioFrame,
+ ControlFrame,
+ ControlFrameType,
+ LocalAudioTransport,
+ TransportBase,
+ TransportFrame,
+)
+from ._vad import SpeechTransition, VADBase
+
+__all__ = [
+ # Model
+ "RealtimeModelBase",
+ "DashScopeRealtimeModel",
+ "DashScopeAudioRealtimeModel",
+ "RealtimeModelCard",
+ "TruncationSupport",
+ "ModelDisconnectedError",
+ # Transport
+ "TransportBase",
+ "LocalAudioTransport",
+ "TransportFrame",
+ "AudioFrame",
+ "ControlFrame",
+ "ControlFrameType",
+ "PlayoutPosition",
+ # Turn taking
+ "VADBase",
+ "SpeechTransition",
+ # Model events
+ "ModelEvent",
+ "SessionEndedEvent",
+ "SpeechStartedEvent",
+ "SpeechEndedEvent",
+ "InputTranscriptionEvent",
+ "ResponseCreatedEvent",
+ "AudioDeltaEvent",
+ "TranscriptDeltaEvent",
+ "ToolCallEvent",
+ "ResponseDoneEvent",
+ "ModelErrorEvent",
+]
diff --git a/src/agentscope/realtime/_base.py b/src/agentscope/realtime/_base.py
new file mode 100644
index 000000000..f3d82b2db
--- /dev/null
+++ b/src/agentscope/realtime/_base.py
@@ -0,0 +1,242 @@
+# -*- coding: utf-8 -*-
+"""The realtime model base class."""
+import inspect
+from abc import ABC, abstractmethod
+from enum import StrEnum
+from pathlib import Path
+from typing import Any, AsyncIterator
+
+from pydantic import BaseModel
+
+from ._events import ModelEvent
+from ._model_card import RealtimeModelCard
+from ..credential import CredentialBase
+from ..message import Msg, ToolResultBlock
+
+
+class ModelDisconnectedError(ConnectionError):
+ """The provider closed the session; the next user audio reconnects."""
+
+
+class TruncationSupport(StrEnum):
+ """How the provider lets an interrupted turn be corrected. A protocol
+ property, constant across the models of one API."""
+
+ NONE = "none"
+ SERVER = "server"
+ """The provider corrects itself; ``truncate`` is a no-op."""
+
+ EXPLICIT = "explicit"
+ """The provider accepts an explicit truncate frame."""
+
+
+class RealtimeModelBase(ABC):
+ """A bidirectional realtime model: audio in, audio and tool calls out.
+
+ Facts live in three places: class attributes for what the adapter
+ implements, the model card for what the model is, and properties for
+ what is a constant on one provider and a parameter on another.
+ """
+
+ class Parameters(BaseModel):
+ """Provider-specific tuneables, surfaced to the UI through the
+ card's ``parameter_schema``."""
+
+ # ------------------------------------------------------------------
+ # Adapter facts — constant across every model of this API
+ # ------------------------------------------------------------------
+
+ type: str = ""
+ """Identifies the adapter; stamped onto every card it lists so a stored
+ config can be mapped back to this class."""
+
+ truncation: TruncationSupport = TruncationSupport.NONE
+ """Whether and how an interrupted turn can be corrected."""
+
+ supports_text_input: bool = False
+ """Whether a text turn can be injected mid-session."""
+
+ def __init__(
+ self,
+ model_name: str,
+ credential: CredentialBase,
+ parameters: "RealtimeModelBase.Parameters | None" = None,
+ model_card: RealtimeModelCard | None = None,
+ ) -> None:
+ """Initialize the realtime model.
+
+ Args:
+ model_name (`str`):
+ The model name, e.g. ``"qwen3-omni-flash-realtime"``.
+ credential (`CredentialBase`):
+ The credential used to authenticate against the provider.
+ parameters (`RealtimeModelBase.Parameters | None`, optional):
+ Provider-specific parameters; defaults are used if omitted.
+ model_card (`RealtimeModelCard | None`, optional):
+ The model card. When ``None`` it is looked up by
+ *model_name* among the cards shipped with this class, so
+ that constructing a model by name alone still yields the
+ right sample rates and limits.
+
+ Raises:
+ `ValueError`: If no card is given and none matches *model_name*.
+ """
+ self.model_name = model_name
+ self.credential = credential
+ self.parameters = parameters or self.Parameters()
+ self.card = model_card or self._find_card(model_name)
+
+ @classmethod
+ def _find_card(cls, model_name: str) -> RealtimeModelCard:
+ """Look up the shipped card for *model_name*."""
+ for card in cls.list_models():
+ if card.name == model_name:
+ return card
+ raise ValueError(
+ f"No realtime model card found for {model_name!r} in "
+ f"{cls.__name__}. Pass `model_card` explicitly for a model "
+ f"that ships no card.",
+ )
+
+ @classmethod
+ def list_models(
+ cls,
+ custom_yaml_dir: str | None = None,
+ ) -> list[RealtimeModelCard]:
+ """List the candidate models of this adapter.
+
+ Args:
+ custom_yaml_dir (`str | None`, optional):
+ Directory to scan. Defaults to the ``_models`` directory
+ next to the concrete subclass's source file.
+
+ Returns:
+ `list[RealtimeModelCard]`: The cards found, oldest error
+ skipped with a warning.
+ """
+ yaml_dir = (
+ Path(custom_yaml_dir)
+ if custom_yaml_dir
+ else Path(inspect.getfile(cls)).parent / "_models"
+ )
+ cards = RealtimeModelCard.list_from_directory(yaml_dir, cls.Parameters)
+ for card in cards:
+ card.model_type = cls.type
+ return cards
+
+ # ------------------------------------------------------------------
+ # Facts that are constant on some providers and tuneable on others
+ # ------------------------------------------------------------------
+
+ @property
+ def input_sample_rate(self) -> int:
+ """The PCM rate :meth:`push_audio` expects, in Hz."""
+ return self.card.input_sample_rate
+
+ @property
+ def output_sample_rate(self) -> int:
+ """The PCM rate of the audio in :class:`AudioDelta`, in Hz."""
+ return self.card.output_sample_rate
+
+ @property
+ @abstractmethod
+ def turn_detection_enabled(self) -> bool:
+ """Whether the provider decides turn boundaries. When ``False``
+ the caller owns endpointing and must call :meth:`commit_turn`."""
+
+ # ------------------------------------------------------------------
+ # Lifecycle
+ # ------------------------------------------------------------------
+
+ @abstractmethod
+ async def connect(
+ self,
+ context: list[Msg],
+ instructions: str,
+ tools: list[dict] | None = None,
+ **kwargs: Any,
+ ) -> None:
+ """Open the session, seeding it with prior *context*.
+
+ Args:
+ context (`list[Msg]`):
+ Prior conversation, e.g. from ``AgentState.context``.
+ instructions (`str`):
+ System instructions.
+ tools (`list[dict] | None`, optional):
+ Tool JSON schemas. Ignored unless the card declares
+ ``supports_tools``.
+ **kwargs (`Any`):
+ Extra provider fields. ``turn_detection_disabled=True``
+ asks the provider not to detect turns because the caller
+ runs its own VAD; a resumption handle may also be passed.
+ """
+
+ @abstractmethod
+ async def close(self) -> None:
+ """Close the session and release the connection."""
+
+ @abstractmethod
+ async def events(self) -> AsyncIterator[ModelEvent]:
+ """Yield model events until the session ends."""
+ raise NotImplementedError(f"{type(self).__name__}.events")
+ yield # pylint: disable=unreachable
+
+ # ------------------------------------------------------------------
+ # Input
+ # ------------------------------------------------------------------
+
+ @abstractmethod
+ async def push_audio(self, pcm: bytes) -> None:
+ """Feed user audio: PCM16 mono at :attr:`input_sample_rate`."""
+
+ @abstractmethod
+ async def push_text(self, text: str) -> None:
+ """Feed a text user turn.
+
+ Raises:
+ `NotImplementedError`: If :attr:`supports_text_input` is
+ ``False``.
+ """
+
+ @abstractmethod
+ async def push_tool_result(self, block: ToolResultBlock) -> None:
+ """Feed the result of a tool the caller executed."""
+
+ @abstractmethod
+ async def commit_turn(self) -> None:
+ """Declare the user turn finished.
+
+ Only called when :attr:`turn_detection_enabled` is ``False``.
+ """
+
+ # ------------------------------------------------------------------
+ # Response control
+ # ------------------------------------------------------------------
+
+ @abstractmethod
+ async def request_response(self) -> None:
+ """Ask the model to start generating a reply."""
+
+ @abstractmethod
+ async def cancel_response(self) -> None:
+ """Stop the active reply without closing the session."""
+
+ @abstractmethod
+ async def truncate(
+ self,
+ item_id: str,
+ played_ms: int,
+ played_text: str,
+ ) -> None:
+ """Rewrite an assistant turn to what the user actually heard. A
+ no-op unless :attr:`truncation` is ``EXPLICIT``.
+
+ Args:
+ item_id (`str`):
+ The interrupted assistant item.
+ played_ms (`int`):
+ Milliseconds of its audio that reached the speaker.
+ played_text (`str`):
+ The corresponding prefix of its spoken text.
+ """
diff --git a/src/agentscope/realtime/_dashscope/__init__.py b/src/agentscope/realtime/_dashscope/__init__.py
new file mode 100644
index 000000000..09dbfdfe3
--- /dev/null
+++ b/src/agentscope/realtime/_dashscope/__init__.py
@@ -0,0 +1,6 @@
+# -*- coding: utf-8 -*-
+"""The DashScope realtime models."""
+from ._audio_model import DashScopeAudioRealtimeModel
+from ._model import DashScopeRealtimeModel
+
+__all__ = ["DashScopeRealtimeModel", "DashScopeAudioRealtimeModel"]
diff --git a/src/agentscope/realtime/_dashscope/_audio_model.py b/src/agentscope/realtime/_dashscope/_audio_model.py
new file mode 100644
index 000000000..bf868908b
--- /dev/null
+++ b/src/agentscope/realtime/_dashscope/_audio_model.py
@@ -0,0 +1,126 @@
+# -*- coding: utf-8 -*-
+"""The DashScope Qwen-Audio realtime model."""
+from pathlib import Path
+from typing import Any, Literal
+
+from pydantic import Field
+
+from ._model import DashScopeRealtimeModel
+from .. import _events as me
+from .._base import RealtimeModelBase
+from .._model_card import RealtimeModelCard
+from ..._logging import logger
+
+
+class DashScopeAudioRealtimeModel(DashScopeRealtimeModel):
+ """The Qwen-Audio-3.0-Realtime API over WebSocket.
+
+ Same endpoint and framing as Qwen-Omni, but it accepts text turns,
+ spells both PCM formats as ``"pcm"``, and offers ``smart_turn``
+ endpointing in place of ``semantic_vad``.
+ """
+
+ class Parameters(RealtimeModelBase.Parameters):
+ """Tuneables surfaced to the UI."""
+
+ voice: str = Field(default="longanqian", title="Voice")
+ turn_detection: Literal["server_vad", "smart_turn", "none"] = Field(
+ default="server_vad",
+ title="Turn Detection",
+ description="``none`` hands endpointing to the caller.",
+ )
+ vad_threshold: float = Field(default=0.5, ge=-1.0, le=1.0)
+ vad_silence_duration_ms: int = Field(default=800, ge=200, le=6000)
+ voiceprint_audio_urls: list[str] = Field(
+ default_factory=list,
+ title="Voiceprint Audio URLs",
+ description="Speaker references, used by ``smart_turn`` only.",
+ )
+ max_history_turns: int = Field(default=20, ge=1, le=50)
+
+ type = "dashscope_audio_realtime"
+ supports_text_input = True
+
+ parameters: "DashScopeAudioRealtimeModel.Parameters"
+
+ @classmethod
+ def list_models(
+ cls,
+ custom_yaml_dir: str | None = None,
+ ) -> list[RealtimeModelCard]:
+ """List the Qwen-Audio cards, which live beside the Omni ones."""
+ return super().list_models(
+ custom_yaml_dir or str(Path(__file__).parent / "_audio_models"),
+ )
+
+ async def push_text(self, text: str) -> None:
+ """Send a text user turn and ask for the reply."""
+ await self._send(
+ {
+ "type": "conversation.item.create",
+ "item": {
+ "type": "message",
+ "role": "user",
+ "content": [{"type": "input_text", "text": text}],
+ },
+ },
+ )
+ await self.request_response()
+
+ def _session_update(
+ self,
+ instructions: str,
+ tools: list[dict] | None,
+ ) -> dict:
+ """Build the ``session.update`` message. ``voice`` only takes
+ effect here, on the first update of the session."""
+ p = self.parameters
+ turn_detection: dict[str, Any] | None = None
+ if p.turn_detection == "server_vad":
+ turn_detection = {
+ "type": "server_vad",
+ "threshold": p.vad_threshold,
+ "silence_duration_ms": p.vad_silence_duration_ms,
+ }
+ elif p.turn_detection == "smart_turn":
+ urls = p.voiceprint_audio_urls
+ turn_detection = {"type": "smart_turn"}
+ if urls:
+ turn_detection["voiceprint_audio_urls"] = urls
+
+ session: dict[str, Any] = {
+ "instructions": instructions,
+ "modalities": ["audio", "text"],
+ "voice": p.voice,
+ "input_audio_format": "pcm",
+ "output_audio_format": "pcm",
+ "max_history_turns": p.max_history_turns,
+ "turn_detection": turn_detection,
+ }
+ if tools and self.card.supports_tools:
+ session["tools"] = tools
+ return {"type": "session.update", "session": session}
+
+ def _parse(self, data: dict) -> me.ModelEvent | None:
+ """Translate one server frame, dropping the two families that are
+ specific to this API before falling back to the Omni parser."""
+ match data.get("type", ""):
+ # Speech that ``smart_turn`` judged not to be addressed at the
+ # model, and the progress of voiceprint registration.
+ case "conversation.item.ambient_audio_transcription.delta":
+ logger.debug(
+ "DashScopeAudioRealtimeModel: ambient speech %r",
+ data.get("delta", ""),
+ )
+ return None
+
+ case (
+ "voiceprint_audio_list.in_progress"
+ | "voiceprint_audio_list.completed"
+ | "voiceprint_audio_list.failed"
+ ) as kind:
+ logger.debug("DashScopeAudioRealtimeModel: %s", kind)
+ return None
+
+ case _:
+ return super()._parse(data)
diff --git a/src/agentscope/realtime/_dashscope/_audio_models/qwen-audio-3.0-realtime-flash.yaml b/src/agentscope/realtime/_dashscope/_audio_models/qwen-audio-3.0-realtime-flash.yaml
new file mode 100644
index 000000000..5582971fd
--- /dev/null
+++ b/src/agentscope/realtime/_dashscope/_audio_models/qwen-audio-3.0-realtime-flash.yaml
@@ -0,0 +1,32 @@
+type: realtime_model
+name: qwen-audio-3.0-realtime-flash
+label: Qwen-Audio-3.0-Realtime-Flash
+status: active
+
+input_types:
+ - audio/pcm
+ - text/plain
+output_types:
+ - audio/pcm
+ - text/plain
+
+input_sample_rate: 16000
+output_sample_rate: 24000
+
+supports_tools: true
+supports_session_resumption: false
+
+# Turns and cumulative audio seconds both apply; the provider discards
+# the oldest turn once either is exceeded.
+max_audio_turns: 50
+max_audio_duration_s: 300
+
+parameter_overrides:
+ voice:
+ default: "longanqian"
+ enum:
+ - "longanqian"
+ - "longanlingxin"
+ - "longanlingxi"
+ - "longanxiaoxin"
+ - "longanlufeng"
diff --git a/src/agentscope/realtime/_dashscope/_audio_models/qwen-audio-3.0-realtime-plus.yaml b/src/agentscope/realtime/_dashscope/_audio_models/qwen-audio-3.0-realtime-plus.yaml
new file mode 100644
index 000000000..c02378a69
--- /dev/null
+++ b/src/agentscope/realtime/_dashscope/_audio_models/qwen-audio-3.0-realtime-plus.yaml
@@ -0,0 +1,32 @@
+type: realtime_model
+name: qwen-audio-3.0-realtime-plus
+label: Qwen-Audio-3.0-Realtime-Plus
+status: active
+
+input_types:
+ - audio/pcm
+ - text/plain
+output_types:
+ - audio/pcm
+ - text/plain
+
+input_sample_rate: 16000
+output_sample_rate: 24000
+
+supports_tools: true
+supports_session_resumption: false
+
+# Turns and cumulative audio seconds both apply; the provider discards
+# the oldest turn once either is exceeded.
+max_audio_turns: 50
+max_audio_duration_s: 300
+
+parameter_overrides:
+ voice:
+ default: "longanqian"
+ enum:
+ - "longanqian"
+ - "longanlingxin"
+ - "longanlingxi"
+ - "longanxiaoxin"
+ - "longanlufeng"
diff --git a/src/agentscope/realtime/_dashscope/_model.py b/src/agentscope/realtime/_dashscope/_model.py
new file mode 100644
index 000000000..9afd6461f
--- /dev/null
+++ b/src/agentscope/realtime/_dashscope/_model.py
@@ -0,0 +1,387 @@
+# -*- coding: utf-8 -*-
+"""The DashScope realtime model."""
+import asyncio
+import base64
+import json
+from typing import Any, AsyncIterator, Literal
+
+from pydantic import Field
+
+from .. import _events as me
+from .._base import (
+ ModelDisconnectedError,
+ RealtimeModelBase,
+ TruncationSupport,
+)
+from .._model_card import RealtimeModelCard
+from ..._logging import logger
+from ...credential import DashScopeCredential
+from ...message import Msg, TextBlock, ToolCallBlock, ToolResultBlock
+
+_REALTIME_URL = "wss://dashscope.aliyuncs.com/api-ws/v1/realtime"
+
+
+class DashScopeRealtimeModel(RealtimeModelBase):
+ """The Qwen-Omni realtime API over WebSocket.
+
+ The protocol is OpenAI-shaped (``session.update``,
+ ``input_audio_buffer.append``, ``response.create``) but narrower: no
+ text turns, no ``conversation.item.truncate``, and history that cannot
+ be seeded on connect.
+ """
+
+ class Parameters(RealtimeModelBase.Parameters):
+ """Tuneables surfaced to the UI."""
+
+ voice: str = Field(default="Cherry", title="Voice")
+ turn_detection: Literal["server_vad", "semantic_vad", "none"] = Field(
+ default="server_vad",
+ title="Turn Detection",
+ description="``none`` hands endpointing to the caller.",
+ )
+ vad_threshold: float = Field(default=0.5, ge=0.0, le=1.0)
+ vad_silence_duration_ms: int = Field(default=800, ge=0)
+ input_audio_transcription: bool = Field(
+ default=True,
+ title="Transcribe Input",
+ description="Whether to transcribe the user's speech.",
+ )
+
+ type = "dashscope_omni_realtime"
+ truncation = TruncationSupport.NONE
+ supports_text_input = False
+
+ def __init__(
+ self,
+ model_name: str,
+ credential: DashScopeCredential,
+ parameters: "DashScopeRealtimeModel.Parameters | None" = None,
+ model_card: RealtimeModelCard | None = None,
+ ) -> None:
+ """Initialize the DashScope realtime model.
+
+ Args:
+ model_name (`str`):
+ The model name, e.g. ``"qwen3-omni-flash-realtime"``.
+ credential (`DashScopeCredential`):
+ The DashScope credential.
+ parameters (`DashScopeRealtimeModel.Parameters | None`, optional):
+ Provider parameters; defaults are used if omitted.
+ model_card (`RealtimeModelCard | None`, optional):
+ The model card, looked up by name if omitted.
+ """
+ super().__init__(model_name, credential, parameters, model_card)
+ self.parameters: DashScopeRealtimeModel.Parameters
+ self._ws: Any = None
+ self._reader: asyncio.Task | None = None
+ self._queue: asyncio.Queue[me.ModelEvent | None] = asyncio.Queue()
+ self._item_id = ""
+ self._tool_args: dict[str, str] = {}
+ self._tool_names: dict[str, str] = {}
+
+ @property
+ def turn_detection_enabled(self) -> bool:
+ """Whether the provider decides turn boundaries."""
+ return self.parameters.turn_detection != "none"
+
+ # ------------------------------------------------------------------
+ # Lifecycle
+ # ------------------------------------------------------------------
+
+ async def connect(
+ self,
+ context: list[Msg],
+ instructions: str,
+ tools: list[dict] | None = None,
+ **kwargs: Any,
+ ) -> None:
+ """Open the WebSocket and send the session config.
+
+ DashScope offers no way to replay *context* into a new session, so
+ a non-empty one is logged and dropped rather than silently lost.
+ """
+ import websockets
+
+ if context:
+ logger.warning(
+ "DashScopeRealtimeModel: %d prior messages cannot be "
+ "seeded into the session; the model starts fresh.",
+ len(context),
+ )
+
+ if kwargs.get("turn_detection_disabled"):
+ self.parameters = self.parameters.model_copy(
+ update={"turn_detection": "none"},
+ )
+
+ credential: DashScopeCredential = self.credential # type: ignore
+ self._ws = await websockets.connect(
+ f"{_REALTIME_URL}?model={self.model_name}",
+ additional_headers={
+ "Authorization": f"Bearer "
+ f"{credential.api_key.get_secret_value()}",
+ "X-DashScope-DataInspection": "disable",
+ },
+ )
+ self._reader = asyncio.create_task(self._read(), name="dashscope-rt")
+ await self._send(self._session_update(instructions, tools))
+
+ async def close(self) -> None:
+ """Stop reading and close the WebSocket."""
+ if self._reader is not None:
+ self._reader.cancel()
+ await asyncio.gather(self._reader, return_exceptions=True)
+ self._reader = None
+ if self._ws is not None:
+ await self._ws.close()
+ self._ws = None
+
+ async def events(self) -> AsyncIterator[me.ModelEvent]:
+ """Yield events until the session ends."""
+ while True:
+ event = await self._queue.get()
+ if event is None:
+ return
+ yield event
+
+ # ------------------------------------------------------------------
+ # Input
+ # ------------------------------------------------------------------
+
+ async def push_audio(self, pcm: bytes) -> None:
+ """Append PCM16 mono audio at 16 kHz to the input buffer."""
+ await self._send(
+ {
+ "type": "input_audio_buffer.append",
+ "audio": base64.b64encode(pcm).decode("ascii"),
+ },
+ )
+
+ async def push_text(self, text: str) -> None:
+ """Unsupported: the API accepts audio and images only."""
+ raise NotImplementedError(
+ "DashScope realtime accepts no text input.",
+ )
+
+ async def push_tool_result(self, block: ToolResultBlock) -> None:
+ """Send a ``function_call_output`` item."""
+ output = (
+ block.output
+ if isinstance(block.output, str)
+ else "".join(
+ _.text for _ in block.output if isinstance(_, TextBlock)
+ )
+ )
+ await self._send(
+ {
+ "type": "conversation.item.create",
+ "item": {
+ "type": "function_call_output",
+ "call_id": block.id,
+ "output": output,
+ },
+ },
+ )
+
+ async def commit_turn(self) -> None:
+ """Commit the input buffer as one user turn."""
+ await self._send({"type": "input_audio_buffer.commit"})
+
+ # ------------------------------------------------------------------
+ # Response control
+ # ------------------------------------------------------------------
+
+ async def request_response(self) -> None:
+ """Ask for a reply, cancelling one already in flight first."""
+ if self._item_id:
+ await self.cancel_response()
+ await self._send(
+ {
+ "type": "response.create",
+ "response": {"modalities": ["text", "audio"]},
+ },
+ )
+
+ async def cancel_response(self) -> None:
+ """Cancel the reply in flight, if any."""
+ if self._item_id:
+ await self._send({"type": "response.cancel"})
+
+ async def truncate(
+ self,
+ item_id: str,
+ played_ms: int,
+ played_text: str,
+ ) -> None:
+ """No-op: the protocol has no truncate frame."""
+
+ # ------------------------------------------------------------------
+ # Wire
+ # ------------------------------------------------------------------
+
+ def _session_update(
+ self,
+ instructions: str,
+ tools: list[dict] | None,
+ ) -> dict:
+ """Build the ``session.update`` message."""
+ p = self.parameters
+ session: dict[str, Any] = {
+ "instructions": instructions,
+ "modalities": ["audio", "text"],
+ "voice": p.voice,
+ "input_audio_format": f"pcm{self.input_sample_rate // 1000}",
+ "output_audio_format": f"pcm{self.output_sample_rate // 1000}",
+ "turn_detection": None
+ if p.turn_detection == "none"
+ else {
+ "type": p.turn_detection,
+ "threshold": p.vad_threshold,
+ "silence_duration_ms": p.vad_silence_duration_ms,
+ },
+ }
+ if p.input_audio_transcription:
+ session["input_audio_transcription"] = {
+ "model": "gummy-realtime-v1",
+ }
+ if tools and self.card.supports_tools:
+ session["tools"] = tools
+ return {"type": "session.update", "session": session}
+
+ async def _send(self, payload: dict) -> None:
+ """Send one JSON frame.
+
+ Raises:
+ `ModelDisconnectedError`: If the provider has closed the
+ session, e.g. after its idle timeout.
+ """
+ from websockets.exceptions import ConnectionClosed
+
+ if self._ws is None:
+ raise ModelDisconnectedError("Not connected.")
+ try:
+ await self._ws.send(json.dumps(payload, ensure_ascii=False))
+ except ConnectionClosed as exc:
+ self._ws = None
+ raise ModelDisconnectedError(str(exc.rcvd or exc)) from exc
+
+ async def _read(self) -> None:
+ """Drain the WebSocket into the event queue until it closes."""
+ try:
+ async for raw in self._ws:
+ if isinstance(raw, bytes):
+ raw = raw.decode("utf-8")
+ try:
+ event = self._parse(json.loads(raw))
+ except Exception: # noqa: BLE001
+ logger.exception("DashScopeRealtimeModel: bad frame")
+ continue
+ if event is not None:
+ self._queue.put_nowait(event)
+ except Exception as exc: # noqa: BLE001
+ logger.error("DashScopeRealtimeModel: connection lost: %s", exc)
+ finally:
+ self._queue.put_nowait(me.SessionEndedEvent(reason="closed"))
+ self._queue.put_nowait(None)
+
+ # pylint: disable=too-many-return-statements
+ def _parse(self, data: dict) -> me.ModelEvent | None:
+ """Translate one server frame into a model event."""
+ kind = data.get("type", "")
+ item_id = data.get("item_id", "")
+
+ match kind:
+ case "response.created":
+ self._item_id = data.get("response", {}).get("id", "")
+ return me.ResponseCreatedEvent(item_id=self._item_id)
+
+ case "response.done":
+ response = data.get("response", {})
+ usage = response.get("usage") or {}
+ event = me.ResponseDoneEvent(
+ item_id=response.get("id") or self._item_id,
+ input_tokens=usage.get("input_tokens", 0),
+ output_tokens=usage.get("output_tokens", 0),
+ )
+ self._item_id = ""
+ return event
+
+ case "response.audio.delta":
+ delta = data.get("delta")
+ if not delta:
+ return None
+ return me.AudioDeltaEvent(
+ item_id=self._item_id,
+ pcm=base64.b64decode(delta),
+ sample_rate=self.output_sample_rate,
+ )
+
+ case "response.audio_transcript.delta":
+ delta = data.get("delta")
+ if not delta:
+ return None
+ return me.TranscriptDeltaEvent(
+ item_id=self._item_id,
+ delta=delta,
+ )
+
+ case "conversation.item.input_audio_transcription.completed":
+ return me.InputTranscriptionEvent(
+ item_id=item_id,
+ text=data.get("transcript", ""),
+ )
+
+ case "input_audio_buffer.speech_started":
+ return me.SpeechStartedEvent(
+ item_id=item_id,
+ at_ms=data.get("audio_start_ms", 0),
+ )
+
+ case "input_audio_buffer.speech_stopped":
+ return me.SpeechEndedEvent(
+ item_id=item_id,
+ at_ms=data.get("audio_end_ms", 0),
+ )
+
+ case "response.output_item.added" | "response.output_item.done":
+ item = data.get("item", {})
+ if item.get("type") == "function_call" and item.get("name"):
+ self._tool_names[item["call_id"]] = item["name"]
+ return None
+
+ case "response.function_call_arguments.delta":
+ call_id = data.get("call_id", "")
+ self._tool_args[call_id] = self._tool_args.get(
+ call_id,
+ "",
+ ) + data.get("delta", "")
+ if data.get("name"):
+ self._tool_names[call_id] = data["name"]
+ return None
+
+ case "response.function_call_arguments.done":
+ # The done frame is authoritative; accumulated deltas are
+ # only the fallback for a frame that omits ``arguments``.
+ call_id = data.get("call_id", "")
+ accumulated = self._tool_args.pop(call_id, "")
+ arguments = data.get("arguments") or accumulated
+ name = data.get("name") or self._tool_names.pop(call_id, "")
+ return me.ToolCallEvent(
+ item_id=self._item_id,
+ tool_call=ToolCallBlock(
+ id=call_id,
+ name=name,
+ input=arguments,
+ ),
+ )
+
+ case "error":
+ err = data.get("error", {})
+ return me.ModelErrorEvent(
+ code=err.get("code", ""),
+ message=err.get("message", ""),
+ )
+
+ case _:
+ logger.debug("DashScopeRealtimeModel: ignoring %s", kind)
+ return None
diff --git a/src/agentscope/realtime/_dashscope/_models/qwen-omni-turbo-realtime.yaml b/src/agentscope/realtime/_dashscope/_models/qwen-omni-turbo-realtime.yaml
new file mode 100644
index 000000000..3aad9e615
--- /dev/null
+++ b/src/agentscope/realtime/_dashscope/_models/qwen-omni-turbo-realtime.yaml
@@ -0,0 +1,27 @@
+type: realtime_model
+name: qwen-omni-turbo-realtime
+label: Qwen-Omni-Turbo Realtime
+status: active
+
+input_types:
+ - audio/pcm
+ - image/jpeg
+output_types:
+ - audio/pcm
+ - text/plain
+
+input_sample_rate: 16000
+output_sample_rate: 24000
+
+# The model page lists "Function Calling: 不支持".
+supports_tools: false
+supports_session_resumption: false
+
+# The model page states the context in tokens; the realtime guide's
+# turn/duration table does not cover this model.
+max_context_tokens: 32768
+max_session_duration_s: 7200
+
+parameter_overrides:
+ voice:
+ enum: ["Cherry", "Serena", "Ethan", "Chelsie"]
diff --git a/src/agentscope/realtime/_dashscope/_models/qwen3-omni-flash-realtime.yaml b/src/agentscope/realtime/_dashscope/_models/qwen3-omni-flash-realtime.yaml
new file mode 100644
index 000000000..8e3296584
--- /dev/null
+++ b/src/agentscope/realtime/_dashscope/_models/qwen3-omni-flash-realtime.yaml
@@ -0,0 +1,27 @@
+type: realtime_model
+name: qwen3-omni-flash-realtime
+label: Qwen3-Omni-Flash Realtime
+status: active
+
+input_types:
+ - audio/pcm
+ - image/jpeg
+output_types:
+ - audio/pcm
+ - text/plain
+
+input_sample_rate: 16000
+output_sample_rate: 24000
+
+# Function calling is only offered on the qwen3.5 series.
+supports_tools: false
+supports_session_resumption: false
+
+# History is bounded by dialog turns, not tokens; the provider discards
+# the oldest turn once exceeded.
+max_audio_turns: 8
+max_session_duration_s: 7200
+
+parameter_overrides:
+ voice:
+ enum: ["Cherry", "Serena", "Ethan", "Chelsie"]
diff --git a/src/agentscope/realtime/_dashscope/_models/qwen3.5-omni-flash-realtime.yaml b/src/agentscope/realtime/_dashscope/_models/qwen3.5-omni-flash-realtime.yaml
new file mode 100644
index 000000000..617b97472
--- /dev/null
+++ b/src/agentscope/realtime/_dashscope/_models/qwen3.5-omni-flash-realtime.yaml
@@ -0,0 +1,28 @@
+type: realtime_model
+name: qwen3.5-omni-flash-realtime
+label: Qwen3.5-Omni-Flash Realtime
+status: active
+
+input_types:
+ - audio/pcm
+ - image/jpeg
+output_types:
+ - audio/pcm
+ - text/plain
+
+input_sample_rate: 16000
+output_sample_rate: 24000
+
+supports_tools: true
+supports_session_resumption: false
+
+# Turns and cumulative audio seconds both apply; the provider discards
+# the oldest turn once either is exceeded.
+max_audio_turns: 80
+max_audio_duration_s: 480
+max_session_duration_s: 7200
+
+parameter_overrides:
+ voice:
+ default: "Tina"
+ enum: ["Cherry", "Serena", "Ethan", "Chelsie", "Tina"]
diff --git a/src/agentscope/realtime/_dashscope/_models/qwen3.5-omni-plus-realtime.yaml b/src/agentscope/realtime/_dashscope/_models/qwen3.5-omni-plus-realtime.yaml
new file mode 100644
index 000000000..8235b8999
--- /dev/null
+++ b/src/agentscope/realtime/_dashscope/_models/qwen3.5-omni-plus-realtime.yaml
@@ -0,0 +1,28 @@
+type: realtime_model
+name: qwen3.5-omni-plus-realtime
+label: Qwen3.5-Omni-Plus Realtime
+status: active
+
+input_types:
+ - audio/pcm
+ - image/jpeg
+output_types:
+ - audio/pcm
+ - text/plain
+
+input_sample_rate: 16000
+output_sample_rate: 24000
+
+supports_tools: true
+supports_session_resumption: false
+
+# Turns and cumulative audio seconds both apply; the provider discards
+# the oldest turn once either is exceeded.
+max_audio_turns: 100
+max_audio_duration_s: 600
+max_session_duration_s: 7200
+
+parameter_overrides:
+ voice:
+ default: "Tina"
+ enum: ["Cherry", "Serena", "Ethan", "Chelsie", "Tina"]
diff --git a/src/agentscope/realtime/_events.py b/src/agentscope/realtime/_events.py
new file mode 100644
index 000000000..dd22e2862
--- /dev/null
+++ b/src/agentscope/realtime/_events.py
@@ -0,0 +1,84 @@
+# -*- coding: utf-8 -*-
+"""Events emitted by a realtime model.
+
+These are internal to :mod:`agentscope.realtime`: adapters normalise
+vendor frames into them, and :class:`RealtimeAgent` translates them into
+the public :class:`~agentscope.event.AgentEvent` stream.
+"""
+from pydantic import BaseModel, Field
+
+from ..message import ToolCallBlock
+
+
+class ModelEvent(BaseModel):
+ """Common base class, for type hinting only."""
+
+
+class SessionEndedEvent(ModelEvent):
+ """The provider session is gone; no further events will arrive."""
+
+ reason: str = ""
+
+
+class SpeechStartedEvent(ModelEvent):
+ """The provider's own VAD detected the user starting to speak."""
+
+ item_id: str = ""
+ at_ms: int = 0
+
+
+class SpeechEndedEvent(ModelEvent):
+ """The provider's own VAD detected the user stopping."""
+
+ item_id: str = ""
+ at_ms: int = 0
+
+
+class InputTranscriptionEvent(ModelEvent):
+ """The settled transcript of one user turn."""
+
+ item_id: str = ""
+ text: str
+
+
+class ResponseCreatedEvent(ModelEvent):
+ """The provider started producing a reply."""
+
+ item_id: str
+
+
+class AudioDeltaEvent(ModelEvent):
+ """A chunk of assistant speech."""
+
+ item_id: str
+ pcm: bytes
+ sample_rate: int
+
+
+class TranscriptDeltaEvent(ModelEvent):
+ """A chunk of the assistant's spoken text."""
+
+ item_id: str
+ delta: str
+
+
+class ToolCallEvent(ModelEvent):
+ """A complete tool call requested by the provider."""
+
+ item_id: str
+ tool_call: ToolCallBlock
+
+
+class ResponseDoneEvent(ModelEvent):
+ """The provider finished producing a reply."""
+
+ item_id: str
+ input_tokens: int = Field(default=0, ge=0)
+ output_tokens: int = Field(default=0, ge=0)
+
+
+class ModelErrorEvent(ModelEvent):
+ """An error reported by the provider."""
+
+ code: str = ""
+ message: str = ""
diff --git a/src/agentscope/realtime/_model_card.py b/src/agentscope/realtime/_model_card.py
new file mode 100644
index 000000000..77eeed6fa
--- /dev/null
+++ b/src/agentscope/realtime/_model_card.py
@@ -0,0 +1,136 @@
+# -*- coding: utf-8 -*-
+"""The realtime model card."""
+import copy
+from datetime import datetime
+from pathlib import Path
+from typing import Literal, Self, Type
+
+import yaml
+from pydantic import BaseModel, Field
+
+from .._logging import logger
+
+
+class RealtimeModelCard(BaseModel):
+ """Describes one realtime model. Everything here varies with the model
+ name; what the adapter implements lives on
+ :class:`RealtimeModelBase`."""
+
+ type: Literal["realtime_model"] = "realtime_model"
+
+ model_type: str = Field(
+ default="",
+ description="The adapter class this model belongs to, e.g. "
+ "``dashscope_omni_realtime``.",
+ )
+
+ name: str = Field(description="The model identifier.")
+ label: str = Field(description="Human-readable label for the UI.")
+
+ status: Literal["active", "deprecated", "sunset"] = Field(
+ default="active",
+ description="Lifecycle status of the model.",
+ )
+ deprecated_at: datetime | None = Field(
+ default=None,
+ description="When the model was deprecated, if applicable.",
+ )
+
+ input_types: list[str] = Field(
+ default_factory=lambda: ["audio/pcm"],
+ description="Accepted input media types.",
+ )
+ output_types: list[str] = Field(
+ default_factory=lambda: ["audio/pcm", "text/plain"],
+ description="Media types the model produces.",
+ )
+
+ input_sample_rate: int = Field(
+ default=16000,
+ gt=0,
+ description="Advertised input PCM sample rate in Hz.",
+ )
+ output_sample_rate: int = Field(
+ default=24000,
+ gt=0,
+ description="Advertised output PCM sample rate in Hz.",
+ )
+
+ supports_tools: bool = Field(
+ default=False,
+ description="Whether this model accepts tool schemas.",
+ )
+ supports_session_resumption: bool = Field(
+ default=False,
+ description="Whether a previous session can be resumed by handle.",
+ )
+
+ # Context limits. Providers use different units and several can apply
+ # at once, so all are optional; ``None`` means unlimited or unknown.
+ max_context_tokens: int | None = Field(default=None, gt=0)
+ max_audio_turns: int | None = Field(default=None, gt=0)
+ max_audio_duration_s: int | None = Field(default=None, gt=0)
+ max_session_duration_s: int | None = Field(default=None, gt=0)
+
+ parameter_schema: dict = Field(default_factory=dict)
+ """JSON Schema of the tuneable parameters, surfaced to the UI."""
+
+ parameter_overrides: dict[str, dict] = Field(default_factory=dict)
+ """Per-parameter narrowing merged into :attr:`parameter_schema`."""
+
+ @classmethod
+ def from_yaml(
+ cls,
+ yaml_path: str | Path,
+ parameter_class: Type[BaseModel],
+ ) -> Self:
+ """Load a card from YAML, seeding the parameter schema from
+ *parameter_class*.
+
+ Args:
+ yaml_path (`str | Path`):
+ Path to the YAML file.
+ parameter_class (`Type[BaseModel]`):
+ The adapter's ``Parameters`` class, whose JSON schema seeds
+ :attr:`parameter_schema`.
+
+ Returns:
+ `RealtimeModelCard`: The loaded card.
+ """
+ with open(yaml_path, "r", encoding="utf-8") as fh:
+ config = yaml.safe_load(fh)
+
+ properties = copy.deepcopy(
+ parameter_class.model_json_schema().get("properties", {}),
+ )
+ overrides = config.pop("parameter_overrides", {}) or {}
+ for param, override in overrides.items():
+ if override is None or override.get("hidden"):
+ properties.pop(param, None)
+ elif param in properties:
+ properties[param] = {**properties[param], **override}
+
+ return cls(
+ **config,
+ parameter_schema={"type": "object", "properties": properties},
+ parameter_overrides=overrides,
+ )
+
+ @classmethod
+ def list_from_directory(
+ cls,
+ yaml_dir: str | Path,
+ parameter_class: Type[BaseModel],
+ ) -> list["RealtimeModelCard"]:
+ """Load every ``*.yaml`` card in *yaml_dir*, skipping broken ones."""
+ cards: list["RealtimeModelCard"] = []
+ for yaml_file in sorted(Path(yaml_dir).glob("*.yaml")):
+ try:
+ cards.append(cls.from_yaml(yaml_file, parameter_class))
+ except Exception as exc: # noqa: BLE001
+ logger.warning(
+ "Failed to load realtime model card %s: %s",
+ yaml_file,
+ exc,
+ )
+ return cards
diff --git a/src/agentscope/realtime/_playout.py b/src/agentscope/realtime/_playout.py
new file mode 100644
index 000000000..f7753b124
--- /dev/null
+++ b/src/agentscope/realtime/_playout.py
@@ -0,0 +1,18 @@
+# -*- coding: utf-8 -*-
+"""Playback position of assistant audio."""
+from pydantic import BaseModel, Field
+
+
+class PlayoutPosition(BaseModel):
+ """How much assistant audio actually left the speaker — the only sound
+ basis for truncating a barge-in."""
+
+ item_id: str
+ """The assistant item this position refers to."""
+
+ played_ms: int = Field(default=0, ge=0)
+ """Milliseconds of audio actually played out."""
+
+ first_played_at: float | None = None
+ """:func:`time.monotonic` when the first sample of this item reached
+ the speaker — the only timestamp that reflects what the user hears."""
diff --git a/src/agentscope/realtime/_transport/__init__.py b/src/agentscope/realtime/_transport/__init__.py
new file mode 100644
index 000000000..634f7befb
--- /dev/null
+++ b/src/agentscope/realtime/_transport/__init__.py
@@ -0,0 +1,19 @@
+# -*- coding: utf-8 -*-
+"""Transports connecting a realtime agent to audio endpoints."""
+from ._local import LocalAudioTransport
+from ._base import (
+ AudioFrame,
+ ControlFrame,
+ ControlFrameType,
+ TransportBase,
+ TransportFrame,
+)
+
+__all__ = [
+ "LocalAudioTransport",
+ "AudioFrame",
+ "ControlFrame",
+ "ControlFrameType",
+ "TransportBase",
+ "TransportFrame",
+]
diff --git a/src/agentscope/realtime/_transport/_base.py b/src/agentscope/realtime/_transport/_base.py
new file mode 100644
index 000000000..a430daa69
--- /dev/null
+++ b/src/agentscope/realtime/_transport/_base.py
@@ -0,0 +1,124 @@
+# -*- coding: utf-8 -*-
+"""The transport base class."""
+from abc import ABC, abstractmethod
+from enum import StrEnum
+from typing import AsyncIterator
+
+from pydantic import BaseModel
+
+from .._playout import PlayoutPosition
+
+
+class AudioFrame(BaseModel):
+ """A chunk of upstream user audio."""
+
+ pcm: bytes
+ """PCM16 mono at the transport's :attr:`input_sample_rate`."""
+
+
+class ControlFrameType(StrEnum):
+ """The non-audio things a remote client can say.
+
+ Each one is the wire form of a public :class:`RealtimeAgent` method,
+ so that a browser can do what a Python caller does directly. Traffic
+ a transport handles for itself — handshakes, playback reports — never
+ reaches the agent and has no member here.
+ """
+
+ TEXT = "text"
+ """A typed user turn."""
+
+ USER_CONFIRM = "user_confirm"
+ """The outcome of a tool permission prompt."""
+
+ INTERRUPT = "interrupt"
+ """The user pressed stop, as opposed to speaking over the reply."""
+
+ EXTERNAL_EXECUTION_RESULT = "external_execution_result"
+ """The result of a tool the client ran itself."""
+
+
+class ControlFrame(BaseModel):
+ """One upstream control event."""
+
+ type: ControlFrameType
+ data: dict = {}
+
+
+TransportFrame = AudioFrame | ControlFrame
+
+
+class TransportBase(ABC):
+ """Connects a :class:`RealtimeAgent` to where audio comes from and
+ goes to.
+
+ A transport is a browser connection, a local sound card, or a queue
+ the caller drives — the agent cannot tell the difference. It is owned
+ by whoever created it, not by the agent; :meth:`RealtimeAgent.run`
+ only borrows it.
+ """
+
+ input_sample_rate: int
+ """The rate of the PCM delivered by :meth:`incoming`."""
+
+ output_sample_rate: int
+ """The rate :meth:`send_audio` expects."""
+
+ async def __aenter__(self) -> "TransportBase":
+ """Start on entry; the transport's owner is whoever entered."""
+ await self.start()
+ return self
+
+ async def __aexit__(self, *exc: object) -> None:
+ """Close on exit."""
+ await self.close()
+
+ @abstractmethod
+ async def start(self) -> None:
+ """Open the transport and begin capturing."""
+
+ @abstractmethod
+ async def close(self) -> None:
+ """Release every resource held by the transport."""
+
+ @abstractmethod
+ async def incoming(self) -> AsyncIterator[TransportFrame]:
+ """Yield upstream frames until the transport closes."""
+ raise NotImplementedError(f"{type(self).__name__}.incoming")
+ yield # pylint: disable=unreachable
+
+ @abstractmethod
+ async def send_audio(self, pcm: bytes, item_id: str) -> None:
+ """Queue assistant audio for playback.
+
+ Args:
+ pcm (`bytes`):
+ PCM16 mono at :attr:`output_sample_rate`.
+ item_id (`str`):
+ The assistant item this audio belongs to, so that playback
+ position can be attributed to it.
+ """
+
+ @abstractmethod
+ async def send_event(self, event: dict) -> None:
+ """Send one control event downstream."""
+
+ @abstractmethod
+ async def clear_audio(self) -> PlayoutPosition:
+ """Cut playback short and report how much was actually heard.
+
+ Fades out briefly to avoid an audible click, then discards queued
+ and in-flight audio.
+
+ Clearing and reporting are one call on purpose: the returned
+ position is what the caller must feed to
+ :meth:`VoiceBackendBase.truncate`, so the two cannot drift apart
+ and the truncation cannot be forgotten.
+
+ Returns:
+ `PlayoutPosition`: What the user heard before the cut.
+ """
+
+ @abstractmethod
+ def playout(self) -> PlayoutPosition:
+ """The current playback position, without interrupting."""
diff --git a/src/agentscope/realtime/_transport/_local.py b/src/agentscope/realtime/_transport/_local.py
new file mode 100644
index 000000000..52339232a
--- /dev/null
+++ b/src/agentscope/realtime/_transport/_local.py
@@ -0,0 +1,202 @@
+# -*- coding: utf-8 -*-
+"""The local sound-card transport."""
+import asyncio
+import threading
+import time
+from typing import Any, AsyncIterator
+
+import numpy as np
+
+from ._base import AudioFrame, TransportBase, TransportFrame
+from .._playout import PlayoutPosition
+from ..._logging import logger
+
+
+class LocalAudioTransport(TransportBase):
+ """Microphone in, speaker out, via ``sounddevice``.
+
+ Audio callbacks run on PortAudio's thread, not the event loop: captured
+ chunks are handed to the loop with ``call_soon_threadsafe`` and
+ playback pulls from a lock-guarded buffer. Playout accounting lives in
+ that callback, the same place a browser's AudioWorklet does it.
+ """
+
+ def __init__(
+ self,
+ input_sample_rate: int = 16000,
+ output_sample_rate: int = 24000,
+ input_device: int | str | None = None,
+ output_device: int | str | None = None,
+ chunk_ms: int = 100,
+ fade_ms: int = 30,
+ ) -> None:
+ """Initialize the transport.
+
+ Args:
+ input_sample_rate (`int`, defaults to `16000`):
+ Capture rate; should match the model's input rate.
+ output_sample_rate (`int`, defaults to `24000`):
+ Playback rate; should match the model's output rate.
+ input_device (`int | str | None`, optional):
+ ``sounddevice`` input device, default if ``None``.
+ output_device (`int | str | None`, optional):
+ ``sounddevice`` output device, default if ``None``.
+ chunk_ms (`int`, defaults to `100`):
+ Capture chunk length handed to :meth:`incoming`.
+ fade_ms (`int`, defaults to `30`):
+ Fade-out applied by :meth:`clear_audio` to avoid a click.
+ """
+ self.input_sample_rate = input_sample_rate
+ self.output_sample_rate = output_sample_rate
+ self._input_device = input_device
+ self._output_device = output_device
+ self._chunk_frames = input_sample_rate * chunk_ms // 1000
+ self._fade_ms = fade_ms
+ # Ten seconds of capture; beyond that the oldest chunk is dropped.
+ self._max_queued = max(1, 10_000 // chunk_ms)
+
+ self._loop: asyncio.AbstractEventLoop | None = None
+ self._in_queue: asyncio.Queue[AudioFrame | None] = asyncio.Queue()
+ self._in_stream: Any = None
+ self._out_stream: Any = None
+
+ # Playback state, guarded by ``_lock`` (shared with the audio thread).
+ self._lock = threading.Lock()
+ self._pending = bytearray()
+ self._item_id = ""
+ self._played_samples = 0
+ self._first_played_at: float | None = None
+
+ # ------------------------------------------------------------------
+ # Lifecycle
+ # ------------------------------------------------------------------
+
+ async def start(self) -> None:
+ """Open both streams."""
+ import sounddevice as sd
+
+ self._loop = asyncio.get_running_loop()
+ self._in_stream = sd.InputStream(
+ samplerate=self.input_sample_rate,
+ channels=1,
+ dtype="int16",
+ blocksize=self._chunk_frames,
+ device=self._input_device,
+ callback=self._on_input,
+ )
+ self._out_stream = sd.OutputStream(
+ samplerate=self.output_sample_rate,
+ channels=1,
+ dtype="int16",
+ device=self._output_device,
+ callback=self._on_output,
+ )
+ self._in_stream.start()
+ self._out_stream.start()
+
+ async def close(self) -> None:
+ """Stop both streams and end :meth:`incoming`."""
+ for stream in (self._in_stream, self._out_stream):
+ if stream is not None:
+ stream.stop()
+ stream.close()
+ self._in_stream = self._out_stream = None
+ self._in_queue.put_nowait(None)
+
+ # ------------------------------------------------------------------
+ # Upstream
+ # ------------------------------------------------------------------
+
+ async def incoming(self) -> AsyncIterator[TransportFrame]:
+ """Yield captured chunks until :meth:`close`."""
+ while True:
+ frame = await self._in_queue.get()
+ if frame is None:
+ return
+ yield frame
+
+ def _on_input(
+ self,
+ indata: Any,
+ _frames: int,
+ _t: Any,
+ status: Any,
+ ) -> None:
+ """Audio thread: hand one captured chunk to the loop."""
+ if status:
+ logger.debug("LocalAudioTransport: input %s", status)
+ if self._loop is None:
+ return
+ frame = AudioFrame(pcm=bytes(indata))
+ self._loop.call_soon_threadsafe(self._enqueue, frame)
+
+ def _enqueue(self, frame: AudioFrame) -> None:
+ """Loop thread: queue a chunk, dropping the oldest when nobody drains
+ fast enough, so a stalled consumer never replays stale audio."""
+ if self._in_queue.qsize() >= self._max_queued:
+ self._in_queue.get_nowait()
+ self._in_queue.put_nowait(frame)
+
+ # ------------------------------------------------------------------
+ # Downstream
+ # ------------------------------------------------------------------
+
+ async def send_audio(self, pcm: bytes, item_id: str) -> None:
+ """Queue assistant audio. A new item resets the playout counters."""
+ with self._lock:
+ if item_id != self._item_id:
+ self._item_id = item_id
+ self._pending.clear()
+ self._played_samples = 0
+ self._first_played_at = None
+ self._pending.extend(pcm)
+
+ async def send_event(self, event: dict) -> None:
+ """No peer to send to; events are consumed via the agent."""
+
+ def _on_output(
+ self,
+ outdata: Any,
+ frames: int,
+ _t: Any,
+ status: Any,
+ ) -> None:
+ """Audio thread: fill one block from the buffer, count what played."""
+ if status:
+ logger.debug("LocalAudioTransport: output %s", status)
+ wanted = frames * 2
+ with self._lock:
+ chunk = bytes(self._pending[:wanted])
+ del self._pending[:wanted]
+ if chunk:
+ self._played_samples += len(chunk) // 2
+ if self._first_played_at is None:
+ self._first_played_at = time.monotonic()
+ n = len(chunk) // 2 # bytes -> int16 samples
+ outdata[:n] = np.frombuffer(chunk, dtype=np.int16).reshape(-1, 1)
+ outdata[n:] = 0
+
+ async def clear_audio(self) -> PlayoutPosition:
+ """Fade the head of the queue out, drop the rest, report the cut."""
+ with self._lock:
+ position = self._position()
+ fade_bytes = self.output_sample_rate * self._fade_ms // 1000 * 2
+ head = np.frombuffer(bytes(self._pending[:fade_bytes]), np.int16)
+ self._pending.clear()
+ if len(head):
+ ramp = np.linspace(1.0, 0.0, len(head), dtype=np.float32)
+ self._pending.extend((head * ramp).astype(np.int16).tobytes())
+ return position
+
+ def playout(self) -> PlayoutPosition:
+ """The current playback position."""
+ with self._lock:
+ return self._position()
+
+ def _position(self) -> PlayoutPosition:
+ """Build a position; caller holds the lock."""
+ return PlayoutPosition(
+ item_id=self._item_id,
+ played_ms=self._played_samples * 1000 // self.output_sample_rate,
+ first_played_at=self._first_played_at,
+ )
diff --git a/src/agentscope/realtime/_vad.py b/src/agentscope/realtime/_vad.py
new file mode 100644
index 000000000..bbbeba615
--- /dev/null
+++ b/src/agentscope/realtime/_vad.py
@@ -0,0 +1,48 @@
+# -*- coding: utf-8 -*-
+"""Voice activity detection."""
+from abc import ABC, abstractmethod
+from enum import StrEnum
+
+
+class SpeechTransition(StrEnum):
+ """A transition in voice activity, as detected by a VAD. Not an event
+ class; the model-side ``SpeechStartedEvent`` is what a provider reports."""
+
+ STARTED = "started"
+ ENDED = "ended"
+
+
+class VADBase(ABC):
+ """Voice activity detection over a stream of PCM16 mono audio.
+
+ Implementations are stateful: ``push`` is called with consecutive
+ chunks and returns a transition only on the frame where it happens.
+ """
+
+ sample_rate: int
+ """The PCM sample rate the implementation expects."""
+
+ @abstractmethod
+ def push(self, pcm: bytes) -> SpeechTransition | None:
+ """Feed one chunk of audio and report a transition, if any.
+
+ This is an edge detector, not a level query: it returns a value
+ only on the chunk where speech starts or stops. Chunk sizes are
+ whatever the transport produces, while models want a fixed frame
+ (Silero v5 wants 512 samples), so implementations buffer and slice
+ internally and report at most one transition per call.
+
+ Args:
+ pcm (`bytes`):
+ PCM16 mono audio at :attr:`sample_rate`.
+
+ Returns:
+ `SpeechTransition | None`:
+ The transition detected on this chunk, or ``None``.
+ """
+
+ @abstractmethod
+ def reset(self) -> None:
+ """Drop internal state. Called whenever the audio stream has a
+ gap — a reconnect, or the microphone being unmuted — since the
+ silence counters carried across it would be stale."""
diff --git a/tests/realtime_agent_test.py b/tests/realtime_agent_test.py
new file mode 100644
index 000000000..b01b6806f
--- /dev/null
+++ b/tests/realtime_agent_test.py
@@ -0,0 +1,987 @@
+# -*- coding: utf-8 -*-
+"""Unit tests for RealtimeAgent, driven by a scripted model and a fake
+transport — no network, no sound card."""
+# pylint: disable=protected-access, unused-argument
+import asyncio
+from typing import Any, AsyncIterator
+from unittest.async_case import IsolatedAsyncioTestCase
+from utils import AnyString
+
+from agentscope.agent import RealtimeAgent, TurnAggregator
+from agentscope.credential import DashScopeCredential
+from agentscope.event import (
+ ReplyEndEvent,
+ UserInputTranscriptionEvent,
+)
+from agentscope.realtime import (
+ AudioFrame,
+ ModelDisconnectedError,
+ PlayoutPosition,
+ RealtimeModelBase,
+ RealtimeModelCard,
+ SpeechTransition,
+ TransportBase,
+ TruncationSupport,
+ VADBase,
+)
+from agentscope.event import (
+ ConfirmResult,
+ RequireUserConfirmEvent,
+ ToolCallEndEvent,
+ ToolCallStartEvent,
+ ToolResultEndEvent,
+ ToolResultStartEvent,
+ ToolResultTextDeltaEvent,
+ UserConfirmResultEvent,
+)
+from agentscope.message import Msg, TextBlock, ToolCallBlock, ToolResultBlock
+from agentscope.permission import (
+ PermissionBehavior,
+ PermissionContext,
+ PermissionDecision,
+)
+from agentscope.tool import ToolBase, ToolChunk, ToolResponse, Toolkit
+from agentscope.realtime import _events as me
+
+PCM_100MS = b"\x01\x00" * 2400
+
+REPLY_R1 = [
+ me.SpeechEndedEvent(item_id="u1"),
+ me.InputTranscriptionEvent(item_id="u1", text="讲个故事"),
+ me.ResponseCreatedEvent(item_id="r1"),
+]
+for _word in ["从前", "有座山", "山里", "有座庙", "庙里", "有个", "老和尚"]:
+ REPLY_R1 += [
+ me.TranscriptDeltaEvent(item_id="r1", delta=_word),
+ me.AudioDeltaEvent(item_id="r1", pcm=PCM_100MS, sample_rate=24000),
+ ]
+
+REPLY_R2 = [
+ me.InputTranscriptionEvent(item_id="u3", text="你好"),
+ me.ResponseCreatedEvent(item_id="r2"),
+ me.TranscriptDeltaEvent(item_id="r2", delta="你好呀"),
+ me.AudioDeltaEvent(item_id="r2", pcm=PCM_100MS, sample_rate=24000),
+ me.ResponseDoneEvent(item_id="r2", input_tokens=10, output_tokens=3),
+]
+
+
+class ScriptedModel(RealtimeModelBase):
+ """Plays one event script per session and records every call."""
+
+ truncation = TruncationSupport.NONE
+ type = "scripted"
+
+ def __init__(self, scripts: list[list[Any]]) -> None:
+ card = RealtimeModelCard(
+ name="scripted",
+ label="scripted",
+ input_sample_rate=16000,
+ output_sample_rate=24000,
+ )
+ super().__init__(
+ "scripted",
+ DashScopeCredential(api_key="sk-x"),
+ model_card=card,
+ )
+ self.scripts = scripts
+ self.calls: list[str] = []
+ self.sessions = 0
+ self._open = asyncio.Event()
+ self._requested = asyncio.Event()
+
+ @property
+ def turn_detection_enabled(self) -> bool:
+ """The provider owns turn boundaries unless a VAD is given."""
+ return True
+
+ async def connect(
+ self,
+ context: list[Msg],
+ instructions: str,
+ tools: list[dict] | None = None,
+ **kwargs: Any,
+ ) -> None:
+ """Record a session open and the turn-detection request."""
+ self.sessions += 1
+ self._open.clear()
+ self.calls.append(
+ f"connect(session={self.sessions},ctx={len(context)},"
+ f"td_off={kwargs.get('turn_detection_disabled')})",
+ )
+
+ async def close(self) -> None:
+ """Record the close and release a live session."""
+ self.calls.append("close")
+ self._open.set()
+
+ async def events(self) -> AsyncIterator[me.ModelEvent]:
+ """Play the script for the current session."""
+ script = self.scripts[self.sessions - 1]
+ for event in script:
+ if event == "WAIT": # the follow-up reply after a tool call
+ await self._requested.wait()
+ continue
+ yield event
+ await asyncio.sleep(0)
+ if not script or not isinstance(script[-1], me.SessionEndedEvent):
+ await self._open.wait() # a live session stays open
+
+ async def push_audio(self, pcm: bytes) -> None:
+ """Count audio pushes."""
+ self.calls.append("push_audio")
+
+ async def push_text(self, text: str) -> None:
+ """Record a text turn; the agent gates on ``supports_text_input``
+ before ever calling this."""
+ self.calls.append(f"push_text({text!r})")
+
+ async def push_tool_result(self, block: ToolResultBlock) -> None:
+ """Record exactly what the provider would receive."""
+ self.calls.append(f"tool_result({block.id},{block.output!r})")
+
+ async def commit_turn(self) -> None:
+ """Record the commit."""
+ self.calls.append("commit_turn")
+
+ async def request_response(self) -> None:
+ """Record the request and let a script waiting on it continue."""
+ self.calls.append("request_response")
+ self._requested.set()
+
+ async def cancel_response(self) -> None:
+ """Record the cancel."""
+ self.calls.append("cancel")
+
+ async def truncate(
+ self,
+ item_id: str,
+ played_ms: int,
+ played_text: str,
+ ) -> None:
+ """Record what the agent thinks the user heard."""
+ self.calls.append(f"truncate({item_id},{played_ms}ms,{played_text!r})")
+
+
+class FakeTransport(TransportBase):
+ """Emits ``frames`` chunks of silence, reports 320 ms played."""
+
+ input_sample_rate = 16000
+ output_sample_rate = 24000
+
+ def __init__(self, frames: int) -> None:
+ self.frames = frames
+ self.item = ""
+ self.cleared = 0
+ self.started = 0
+ self.closed = 0
+
+ async def start(self) -> None:
+ """Count starts; the owner is the test."""
+ self.started += 1
+
+ async def close(self) -> None:
+ """Count closes."""
+ self.closed += 1
+
+ async def incoming(self) -> AsyncIterator[AudioFrame]:
+ """Emit silence on a 20 ms clock."""
+ for _ in range(self.frames):
+ await asyncio.sleep(0.02)
+ yield AudioFrame(pcm=b"\x00" * 3200)
+
+ async def send_audio(self, pcm: bytes, item_id: str) -> None:
+ """Remember which item is playing."""
+ self.item = item_id
+
+ async def send_event(self, event: dict) -> None:
+ """No peer to send to."""
+
+ async def clear_audio(self) -> PlayoutPosition:
+ """Count cuts and report the fixed playout position."""
+ self.cleared += 1
+ return self.playout()
+
+ def playout(self) -> PlayoutPosition:
+ """Always 320 ms into the current item."""
+ return PlayoutPosition(
+ item_id=self.item,
+ played_ms=320,
+ first_played_at=1.0,
+ )
+
+
+class EndOnSecondFrameVAD(VADBase):
+ """Reports the user stopping on the second chunk."""
+
+ sample_rate = 16000
+
+ def __init__(self) -> None:
+ self.seen = 0
+
+ def push(self, pcm: bytes) -> SpeechTransition | None:
+ """ENDED on the second chunk, otherwise nothing."""
+ self.seen += 1
+ return SpeechTransition.ENDED if self.seen == 2 else None
+
+ def reset(self) -> None:
+ """Start counting again."""
+ self.seen = 0
+
+
+class RealtimeAgentTest(IsolatedAsyncioTestCase):
+ """Behaviour of the turn-taking state machine."""
+
+ async def _collect(
+ self,
+ agent: RealtimeAgent,
+ transport: FakeTransport,
+ ) -> list[tuple[str, Any]]:
+ """Run the agent over *transport* and summarise the events."""
+ summary: list[tuple[str, Any]] = []
+ async with transport:
+ async for event in agent.run(transport):
+ if isinstance(event, ReplyEndEvent):
+ summary.append(("reply_end", event.finished_reason))
+ elif isinstance(event, UserInputTranscriptionEvent):
+ summary.append(("user", event.transcript))
+ return summary
+
+ async def test_barge_in_truncates_to_what_was_heard(self) -> None:
+ """A barge-in mid-reply cuts both contexts to the played prefix
+ and the duplicate speech_started is swallowed by the lock."""
+ script = REPLY_R1 + [
+ me.SpeechStartedEvent(item_id="u2"),
+ me.SpeechStartedEvent(item_id="u2"),
+ ]
+ model = ScriptedModel([script])
+ agent = RealtimeAgent("Friday", "be brief", model)
+ transport = FakeTransport(frames=3)
+
+ async with agent:
+ summary = await self._collect(agent, transport)
+
+ self.assertListEqual(
+ summary,
+ [("user", "讲个故事"), ("reply_end", "interrupted")],
+ )
+ self.assertEqual(transport.cleared, 1)
+ # Audio frames interleave with model events on the transport's
+ # clock, so they are counted rather than positioned.
+ self.assertListEqual(
+ [c for c in model.calls if c != "push_audio"],
+ [
+ "connect(session=1,ctx=0,td_off=False)",
+ "truncate(r1,320ms,'从前有座山山里有座庙')",
+ "cancel",
+ "close",
+ ],
+ )
+ self.assertEqual(model.calls.count("push_audio"), 3)
+ self.assertListEqual(
+ [(m.role, m.get_text_content()) for m in agent.state.context],
+ [("user", "讲个故事"), ("assistant", "从前有座山山里有座庙")],
+ )
+ self.assertEqual(agent.last_turn_metrics.first_audio_played_at, 1.0)
+
+ async def test_provider_timeout_reconnects_on_next_audio(self) -> None:
+ """When the provider closes the session, nothing reconnects until
+ the next user audio, which reconnects with the current context."""
+ model = ScriptedModel(
+ [
+ REPLY_R1 + [me.SessionEndedEvent(reason="idle")],
+ REPLY_R2,
+ ],
+ )
+ agent = RealtimeAgent(
+ "Friday",
+ "be brief",
+ model,
+ aggregator=TurnAggregator(merge_window_ms=0),
+ )
+
+ async with agent:
+ # No transport, no audio: the provider times the session out
+ # and nothing reconnects.
+ await asyncio.sleep(0.1)
+ self.assertFalse(agent._connected) # pylint: disable=W0212
+ self.assertEqual(model.sessions, 1)
+
+ summary = await self._collect(agent, FakeTransport(frames=2))
+
+ self.assertEqual(model.sessions, 2)
+ # The orphaned reply was dropped, so one message carried over.
+ self.assertIn("connect(session=2,ctx=1,td_off=False)", model.calls)
+ # Events produced while no run was active are delivered first;
+ # a reply streamed with nobody listening is cut off exactly once.
+ self.assertListEqual(
+ summary,
+ [
+ ("user", "讲个故事"),
+ ("reply_end", "interrupted"),
+ ("user", "你好"),
+ ("reply_end", "completed"),
+ ],
+ )
+ self.assertListEqual(
+ [(m.role, m.get_text_content()) for m in agent.state.context],
+ [("user", "讲个故事"), ("user", "你好"), ("assistant", "你好呀")],
+ )
+
+ async def test_local_vad_owns_turns(self) -> None:
+ """Passing a VAD disables provider turn detection and commits the
+ turn when the VAD reports the user stopped."""
+ model = ScriptedModel([[]])
+ agent = RealtimeAgent(
+ "Friday",
+ "be brief",
+ model,
+ vad=EndOnSecondFrameVAD(),
+ )
+ async with agent:
+ await self._collect(agent, FakeTransport(frames=3))
+
+ self.assertListEqual(
+ model.calls,
+ [
+ "connect(session=1,ctx=0,td_off=True)",
+ "push_audio",
+ "commit_turn",
+ "push_audio",
+ "push_audio",
+ "close",
+ ],
+ )
+
+ async def test_run_exit_cancels_reply_in_flight(self) -> None:
+ """The transport ending mid-reply cancels the reply so the model
+ does not keep talking to nobody."""
+ model = ScriptedModel([REPLY_R1]) # never sends response.done
+ agent = RealtimeAgent("Friday", "be brief", model)
+ async with agent:
+ summary = await self._collect(agent, FakeTransport(frames=1))
+
+ self.assertListEqual(
+ summary,
+ [("user", "讲个故事"), ("reply_end", "interrupted")],
+ )
+ self.assertIn("cancel", model.calls)
+
+ async def test_text_rejected_by_audio_only_model(self) -> None:
+ """A provider without text input refuses typed turns."""
+ agent = RealtimeAgent("Friday", "be brief", ScriptedModel([[]]))
+ with self.assertRaises(NotImplementedError):
+ await agent.send("hi")
+
+ async def test_backchannel_is_dropped(self) -> None:
+ """A bare acknowledgement never becomes a turn."""
+ model = ScriptedModel(
+ [[me.InputTranscriptionEvent(item_id="u1", text="嗯。")]],
+ )
+ agent = RealtimeAgent(
+ "Friday",
+ "be brief",
+ model,
+ aggregator=TurnAggregator(backchannels=frozenset({"嗯"})),
+ )
+ async with agent:
+ summary = await self._collect(agent, FakeTransport(frames=1))
+
+ self.assertListEqual(summary, [])
+ self.assertListEqual(agent.state.context, [])
+
+
+class StreamTool(ToolBase):
+ """Streams two chunks, then the completed result."""
+
+ name: str = "stream_tool"
+ description: str = "streams"
+ input_schema: dict[str, Any] = {
+ "type": "object",
+ "properties": {"q": {"type": "string"}},
+ "required": ["q"],
+ }
+ is_concurrency_safe: bool = True
+ is_read_only: bool = True
+ is_external_tool: bool = False
+ is_mcp: bool = False
+
+ async def check_permissions(
+ self,
+ tool_input: dict[str, Any],
+ context: PermissionContext,
+ ) -> PermissionDecision:
+ """Run freely."""
+ return PermissionDecision(
+ behavior=PermissionBehavior.ALLOW,
+ decision_reason="test",
+ message="test",
+ )
+
+ async def __call__(self, q: str, **kwargs: Any) -> Any:
+ """Yield chunks then the final response."""
+ yield ToolChunk(content=[TextBlock(text=f"{q}-a")])
+ yield ToolChunk(content=[TextBlock(text=f"{q}-b")])
+ yield ToolResponse(content=[TextBlock(text=f"{q}-final")])
+
+
+class AskTool(StreamTool):
+ """Requires user confirmation before running. Not read-only, or the
+ permission engine's read-only fast path would allow it unasked."""
+
+ name: str = "ask_tool"
+ is_read_only: bool = False
+
+ async def check_permissions(
+ self,
+ tool_input: dict[str, Any],
+ context: PermissionContext,
+ ) -> PermissionDecision:
+ """Always ask."""
+ return PermissionDecision(
+ behavior=PermissionBehavior.ASK,
+ decision_reason="test",
+ message="test",
+ )
+
+
+class BrokenTool(StreamTool):
+ """Raises while running."""
+
+ name: str = "broken_tool"
+
+ async def __call__(self, q: str, **kwargs: Any) -> Any:
+ """Fail."""
+ raise RuntimeError("boom")
+ yield # pylint: disable=unreachable
+
+
+def _tool_script(name: str) -> list[me.ModelEvent]:
+ """A reply that only calls *name* and completes."""
+ return [
+ me.ResponseCreatedEvent(item_id="r1"),
+ me.ToolCallEvent(
+ item_id="r1",
+ tool_call=ToolCallBlock(id="c1", name=name, input='{"q": "x"}'),
+ ),
+ me.ResponseDoneEvent(item_id="r1"),
+ ]
+
+
+class RealtimeAgentToolTest(IsolatedAsyncioTestCase):
+ """Tool calls: permission, execution, result delivery."""
+
+ async def _run_tool_scenario(
+ self,
+ tool: ToolBase,
+ confirm: bool | None = None,
+ ) -> tuple[list[tuple[str, Any]], ScriptedModel]:
+ """Run one tool-calling reply; answer a permission prompt with
+ *confirm* if one appears. Returns the tool-related events."""
+ model = ScriptedModel([_tool_script(tool.name)])
+ agent = RealtimeAgent(
+ "Friday",
+ "be brief",
+ model,
+ toolkit=Toolkit(tools=[tool]),
+ )
+ events: list[tuple[str, Any]] = []
+ async with agent:
+ transport = FakeTransport(frames=4)
+ async with transport:
+ async for event in agent.run(transport):
+ match event:
+ case ToolCallStartEvent():
+ events.append(("call_start", event.tool_call_name))
+ case ToolCallEndEvent():
+ events.append(("call_end", event.tool_call_id))
+ case RequireUserConfirmEvent():
+ events.append(
+ ("ask", [c.name for c in event.tool_calls]),
+ )
+ await agent.send(
+ UserConfirmResultEvent(
+ reply_id=event.reply_id,
+ confirm_results=[
+ ConfirmResult(
+ tool_call=event.tool_calls[0],
+ confirmed=bool(confirm),
+ ),
+ ],
+ ),
+ )
+ case ToolResultStartEvent():
+ events.append(
+ ("result_start", event.tool_call_name),
+ )
+ case ToolResultTextDeltaEvent():
+ events.append(("delta", event.delta))
+ case ToolResultEndEvent():
+ events.append(("result_end", event.state))
+ return events, model
+
+ async def test_streamed_tool_result_is_not_duplicated(self) -> None:
+ """Chunks are shown as they come; the provider gets only the
+ completed result, once."""
+ events, model = await self._run_tool_scenario(StreamTool())
+
+ self.assertListEqual(
+ events,
+ [
+ ("call_start", "stream_tool"),
+ ("call_end", "c1"),
+ ("result_start", "stream_tool"),
+ ("delta", "x-a"),
+ ("delta", "x-b"),
+ ("result_end", "success"),
+ ],
+ )
+ self.assertListEqual(
+ [c for c in model.calls if not c.startswith("push_audio")],
+ [
+ "connect(session=1,ctx=0,td_off=False)",
+ "tool_result(c1,'x-final')",
+ "request_response",
+ "close",
+ ],
+ )
+
+ async def test_confirmed_tool_runs(self) -> None:
+ """A confirmed permission prompt lets the tool run."""
+ events, model = await self._run_tool_scenario(AskTool(), confirm=True)
+
+ self.assertListEqual(
+ events,
+ [
+ ("call_start", "ask_tool"),
+ ("call_end", "c1"),
+ ("ask", ["ask_tool"]),
+ ("result_start", "ask_tool"),
+ ("delta", "x-a"),
+ ("delta", "x-b"),
+ ("result_end", "success"),
+ ],
+ )
+ self.assertIn("tool_result(c1,'x-final')", model.calls)
+
+ async def test_denied_tool_reports_denial(self) -> None:
+ """A refused prompt sends a denial to the provider, runs nothing."""
+ events, model = await self._run_tool_scenario(AskTool(), confirm=False)
+
+ self.assertListEqual(
+ events,
+ [
+ ("call_start", "ask_tool"),
+ ("call_end", "c1"),
+ ("ask", ["ask_tool"]),
+ ("result_start", "ask_tool"),
+ ("delta", 'Tool "ask_tool" denied by user.'),
+ ("result_end", "denied"),
+ ],
+ )
+ self.assertIn(
+ "tool_result(c1,'Tool \"ask_tool\" denied by user.')",
+ model.calls,
+ )
+
+ async def test_failing_tool_reports_error(self) -> None:
+ """The toolkit turns an exception into an error response, which
+ is forwarded as-is."""
+ events, model = await self._run_tool_scenario(BrokenTool())
+
+ self.assertListEqual(
+ events,
+ [
+ ("call_start", "broken_tool"),
+ ("call_end", "c1"),
+ ("result_start", "broken_tool"),
+ ("delta", "boom"),
+ ("result_end", "error"),
+ ],
+ )
+ self.assertIn("tool_result(c1,'boom')", model.calls)
+
+
+class RealtimeAgentFullStreamTest(IsolatedAsyncioTestCase):
+ """The complete event stream of a turn that calls a tool and then
+ answers with speech, asserted as one structure."""
+
+ async def test_tool_call_then_spoken_reply(self) -> None:
+ """User asks → model speaks, calls a tool → tool runs → model
+ speaks the answer. Every event, in order."""
+ model = ScriptedModel(
+ [
+ [
+ me.SpeechEndedEvent(item_id="u1"),
+ me.InputTranscriptionEvent(item_id="u1", text="查天气"),
+ me.ResponseCreatedEvent(item_id="r1"),
+ me.TranscriptDeltaEvent(item_id="r1", delta="我查一下"),
+ me.AudioDeltaEvent(
+ item_id="r1",
+ pcm=b"\x01\x00",
+ sample_rate=24000,
+ ),
+ me.ToolCallEvent(
+ item_id="r1",
+ tool_call=ToolCallBlock(
+ id="c1",
+ name="stream_tool",
+ input='{"q": "x"}',
+ ),
+ ),
+ me.ResponseDoneEvent(
+ item_id="r1",
+ input_tokens=5,
+ output_tokens=2,
+ ),
+ "WAIT",
+ me.ResponseCreatedEvent(item_id="r2"),
+ me.TranscriptDeltaEvent(item_id="r2", delta="今天晴"),
+ me.AudioDeltaEvent(
+ item_id="r2",
+ pcm=b"\x01\x00",
+ sample_rate=24000,
+ ),
+ me.ResponseDoneEvent(
+ item_id="r2",
+ input_tokens=9,
+ output_tokens=3,
+ ),
+ ],
+ ],
+ )
+ agent = RealtimeAgent(
+ "Friday",
+ "be brief",
+ model,
+ toolkit=Toolkit(tools=[StreamTool()]),
+ )
+ events = []
+ async with agent:
+ transport = FakeTransport(frames=6)
+ async with transport:
+ async for event in agent.run(transport):
+ events.append(event.model_dump(mode="json"))
+
+ self.assertListEqual(
+ events,
+ [
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "USER_INPUT_TRANSCRIPTION",
+ "session_id": AnyString(),
+ "item_id": "u1",
+ "transcript": "查天气",
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "REPLY_START",
+ "session_id": AnyString(),
+ "reply_id": "r1",
+ "name": "Friday",
+ "role": "assistant",
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "MODEL_CALL_START",
+ "reply_id": "r1",
+ "model_name": "scripted",
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "TEXT_BLOCK_START",
+ "reply_id": "r1",
+ "block_id": AnyString(),
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "TEXT_BLOCK_DELTA",
+ "reply_id": "r1",
+ "block_id": AnyString(),
+ "delta": "我查一下",
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "DATA_BLOCK_START",
+ "reply_id": "r1",
+ "block_id": AnyString(),
+ "media_type": "audio/pcm;rate=24000",
+ "name": None,
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "DATA_BLOCK_DELTA",
+ "reply_id": "r1",
+ "block_id": AnyString(),
+ "media_type": "audio/pcm;rate=24000",
+ "data": "AQA=",
+ "url": None,
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "TEXT_BLOCK_END",
+ "reply_id": "r1",
+ "block_id": AnyString(),
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "DATA_BLOCK_END",
+ "reply_id": "r1",
+ "block_id": AnyString(),
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "MODEL_CALL_END",
+ "reply_id": "r1",
+ "input_tokens": 5,
+ "output_tokens": 2,
+ "cache_input_tokens": 0,
+ "cache_creation_input_tokens": 0,
+ "finished_reason": "completed",
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "REPLY_END",
+ "session_id": AnyString(),
+ "reply_id": "r1",
+ "finished_reason": "completed",
+ "error": None,
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "TOOL_CALL_START",
+ "reply_id": "r1",
+ "tool_call_id": "c1",
+ "tool_call_name": "stream_tool",
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "TOOL_CALL_END",
+ "reply_id": "r1",
+ "tool_call_id": "c1",
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "TOOL_RESULT_START",
+ "reply_id": "r1",
+ "tool_call_id": "c1",
+ "tool_call_name": "stream_tool",
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "TOOL_RESULT_TEXT_DELTA",
+ "reply_id": "r1",
+ "tool_call_id": "c1",
+ "delta": "x-a",
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "TOOL_RESULT_TEXT_DELTA",
+ "reply_id": "r1",
+ "tool_call_id": "c1",
+ "delta": "x-b",
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "TOOL_RESULT_END",
+ "reply_id": "r1",
+ "tool_call_id": "c1",
+ "state": "success",
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "REPLY_START",
+ "session_id": AnyString(),
+ "reply_id": "r2",
+ "name": "Friday",
+ "role": "assistant",
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "MODEL_CALL_START",
+ "reply_id": "r2",
+ "model_name": "scripted",
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "TEXT_BLOCK_START",
+ "reply_id": "r2",
+ "block_id": AnyString(),
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "TEXT_BLOCK_DELTA",
+ "reply_id": "r2",
+ "block_id": AnyString(),
+ "delta": "今天晴",
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "DATA_BLOCK_START",
+ "reply_id": "r2",
+ "block_id": AnyString(),
+ "media_type": "audio/pcm;rate=24000",
+ "name": None,
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "DATA_BLOCK_DELTA",
+ "reply_id": "r2",
+ "block_id": AnyString(),
+ "media_type": "audio/pcm;rate=24000",
+ "data": "AQA=",
+ "url": None,
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "TEXT_BLOCK_END",
+ "reply_id": "r2",
+ "block_id": AnyString(),
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "DATA_BLOCK_END",
+ "reply_id": "r2",
+ "block_id": AnyString(),
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "MODEL_CALL_END",
+ "reply_id": "r2",
+ "input_tokens": 9,
+ "output_tokens": 3,
+ "cache_input_tokens": 0,
+ "cache_creation_input_tokens": 0,
+ "finished_reason": "completed",
+ },
+ {
+ "id": AnyString(),
+ "created_at": AnyString(),
+ "metadata": {},
+ "type": "REPLY_END",
+ "session_id": AnyString(),
+ "reply_id": "r2",
+ "finished_reason": "completed",
+ "error": None,
+ },
+ ],
+ )
+ self.assertListEqual(
+ [(m.role, m.get_text_content()) for m in agent.state.context],
+ [
+ ("user", "查天气"),
+ ("assistant", "我查一下"),
+ ("assistant", "今天晴"),
+ ],
+ )
+ self.assertListEqual(
+ [c for c in model.calls if c != "push_audio"],
+ [
+ "connect(session=1,ctx=0,td_off=False)",
+ "tool_result(c1,'x-final')",
+ "request_response",
+ "close",
+ ],
+ )
+
+
+class DropsSocketModel(ScriptedModel):
+ """Raises on the first push after the provider closed the socket,
+ the way a real WebSocket does before the reader notices."""
+
+ def __init__(self) -> None:
+ super().__init__([[], []])
+ self.dropped = False
+
+ async def push_audio(self, pcm: bytes) -> None:
+ """Fail exactly once, then behave."""
+ if self.sessions == 1 and not self.dropped:
+ self.dropped = True
+ raise ModelDisconnectedError("idle for 180 seconds")
+ await super().push_audio(pcm)
+
+
+class RealtimeAgentDisconnectTest(IsolatedAsyncioTestCase):
+ """A send that hits a closed provider socket must not kill the run."""
+
+ async def test_send_on_closed_socket_reconnects_on_next_audio(
+ self,
+ ) -> None:
+ """The failing frame is kept, the run survives, and the next frame
+ reconnects and flushes it."""
+ model = DropsSocketModel()
+ agent = RealtimeAgent("Friday", "be brief", model)
+ async with agent:
+ transport = FakeTransport(frames=3)
+ with self.assertLogs("as", level="INFO") as logs:
+ async with transport:
+ async for _ in agent.run(transport):
+ pass
+
+ self.assertEqual(model.sessions, 2)
+ self.assertListEqual(
+ [c for c in model.calls if c != "push_audio"],
+ [
+ "connect(session=1,ctx=0,td_off=False)",
+ "connect(session=2,ctx=0,td_off=False)",
+ "close",
+ ],
+ )
+ # Three frames captured; the failed one was replayed, so all
+ # three reach the provider in the end.
+ self.assertEqual(model.calls.count("push_audio"), 3)
+ self.assertTrue(
+ any("keep talking" in line for line in logs.output),
+ logs.output,
+ )
diff --git a/tests/realtime_aggregator_test.py b/tests/realtime_aggregator_test.py
new file mode 100644
index 000000000..83a1f4cda
--- /dev/null
+++ b/tests/realtime_aggregator_test.py
@@ -0,0 +1,56 @@
+# -*- coding: utf-8 -*-
+"""Unit tests for TurnAggregator."""
+import time
+import unittest
+
+from agentscope.agent import TurnAggregator
+
+
+class TurnAggregatorTest(unittest.TestCase):
+ """Collapsing transcripts into turns."""
+
+ def test_continuation_within_window_merges(self) -> None:
+ """A transcript arriving inside the merge window continues the
+ previous turn; one arriving later starts a new one."""
+ aggregator = TurnAggregator(merge_window_ms=300)
+
+ self.assertListEqual(
+ [
+ aggregator.take("帮我订张机票"),
+ aggregator.merges_with_previous(),
+ aggregator.take("去上海的"),
+ aggregator.merges_with_previous(),
+ ],
+ ["帮我订张机票", False, "去上海的", True],
+ )
+
+ time.sleep(0.4)
+ self.assertListEqual(
+ [aggregator.take("几点的"), aggregator.merges_with_previous()],
+ ["几点的", False],
+ )
+
+ def test_backchannels_and_empty_transcripts_are_not_turns(self) -> None:
+ """Acknowledgements are matched ignoring trailing punctuation."""
+ aggregator = TurnAggregator(backchannels=frozenset({"嗯", "对"}))
+
+ self.assertListEqual(
+ [
+ aggregator.take("嗯。"),
+ aggregator.take("对,"),
+ aggregator.take(" "),
+ aggregator.take("对的"),
+ ],
+ [None, None, None, "对的"],
+ )
+
+ def test_reset_forgets_previous_turn(self) -> None:
+ """After a reset the next transcript never merges."""
+ aggregator = TurnAggregator(merge_window_ms=10_000)
+ aggregator.take("第一句")
+ aggregator.reset()
+
+ self.assertListEqual(
+ [aggregator.take("第二句"), aggregator.merges_with_previous()],
+ ["第二句", False],
+ )
diff --git a/tests/realtime_dashscope_test.py b/tests/realtime_dashscope_test.py
new file mode 100644
index 000000000..33c1eb012
--- /dev/null
+++ b/tests/realtime_dashscope_test.py
@@ -0,0 +1,413 @@
+# -*- coding: utf-8 -*-
+"""Unit tests for the DashScope realtime adapters: cards, session config
+and frame parsing. Nothing here opens a connection."""
+# pylint: disable=protected-access
+import base64
+import unittest
+from unittest.async_case import IsolatedAsyncioTestCase
+
+from agentscope.credential import DashScopeCredential
+from agentscope.realtime import (
+ DashScopeAudioRealtimeModel,
+ DashScopeRealtimeModel,
+ ModelDisconnectedError,
+ TruncationSupport,
+)
+from agentscope.realtime import _events as me
+
+CRED = DashScopeCredential(api_key="sk-x")
+TRANSCRIPTION_DONE = "conversation.item.input_audio_transcription.completed"
+AMBIENT_DELTA = "conversation.item.ambient_audio_transcription.delta"
+
+
+class DashScopeCardsTest(unittest.TestCase):
+ """Each adapter lists only its own cards, tagged with its type."""
+
+ def test_omni_cards(self) -> None:
+ """The Omni adapter lists the four Omni cards."""
+ self.assertListEqual(
+ [
+ (c.name, c.model_type, c.supports_tools, c.max_audio_turns)
+ for c in DashScopeRealtimeModel.list_models()
+ ],
+ [
+ (
+ "qwen-omni-turbo-realtime",
+ "dashscope_omni_realtime",
+ False,
+ None,
+ ),
+ (
+ "qwen3-omni-flash-realtime",
+ "dashscope_omni_realtime",
+ False,
+ 8,
+ ),
+ (
+ "qwen3.5-omni-flash-realtime",
+ "dashscope_omni_realtime",
+ True,
+ 80,
+ ),
+ (
+ "qwen3.5-omni-plus-realtime",
+ "dashscope_omni_realtime",
+ True,
+ 100,
+ ),
+ ],
+ )
+
+ def test_audio_cards(self) -> None:
+ """The Audio adapter lists the two Audio cards."""
+ self.assertListEqual(
+ [
+ (
+ c.name,
+ c.model_type,
+ c.max_audio_turns,
+ c.max_audio_duration_s,
+ )
+ for c in DashScopeAudioRealtimeModel.list_models()
+ ],
+ [
+ (
+ "qwen-audio-3.0-realtime-flash",
+ "dashscope_audio_realtime",
+ 50,
+ 300,
+ ),
+ (
+ "qwen-audio-3.0-realtime-plus",
+ "dashscope_audio_realtime",
+ 50,
+ 300,
+ ),
+ ],
+ )
+
+ def test_credential_maps_card_back_to_class(self) -> None:
+ """The service-layer lookup: card.model_type -> class, no scan."""
+ classes = {c.type: c for c in CRED.get_realtime_model_classes()}
+ self.assertDictEqual(
+ {
+ card.name: classes[card.model_type].__name__
+ for card in CRED.list_realtime_models()
+ },
+ {
+ "qwen-audio-3.0-realtime-flash": "DashScopeAudioRealtimeModel",
+ "qwen-audio-3.0-realtime-plus": "DashScopeAudioRealtimeModel",
+ "qwen-omni-turbo-realtime": "DashScopeRealtimeModel",
+ "qwen3-omni-flash-realtime": "DashScopeRealtimeModel",
+ "qwen3.5-omni-flash-realtime": "DashScopeRealtimeModel",
+ "qwen3.5-omni-plus-realtime": "DashScopeRealtimeModel",
+ },
+ )
+
+ def test_unknown_model_name_is_rejected(self) -> None:
+ """A name with no card fails at construction."""
+ with self.assertRaises(ValueError):
+ DashScopeRealtimeModel("no-such-model", CRED)
+
+
+class DashScopeSessionUpdateTest(unittest.TestCase):
+ """The session.update payload each adapter sends on connect."""
+
+ def test_omni_payload(self) -> None:
+ """Omni session.update with tools and transcription."""
+ model = DashScopeRealtimeModel("qwen3.5-omni-flash-realtime", CRED)
+ self.assertDictEqual(
+ model._session_update("be nice", [{"type": "function"}]),
+ {
+ "type": "session.update",
+ "session": {
+ "instructions": "be nice",
+ "modalities": ["audio", "text"],
+ "voice": "Cherry",
+ "input_audio_format": "pcm16",
+ "output_audio_format": "pcm24",
+ "turn_detection": {
+ "type": "server_vad",
+ "threshold": 0.5,
+ "silence_duration_ms": 800,
+ },
+ "input_audio_transcription": {
+ "model": "gummy-realtime-v1",
+ },
+ "tools": [{"type": "function"}],
+ },
+ },
+ )
+
+ def test_omni_withholds_tools_the_card_does_not_support(self) -> None:
+ """Tools are not sent for a model whose card declines them."""
+ model = DashScopeRealtimeModel("qwen3-omni-flash-realtime", CRED)
+ session = model._session_update("x", [{"type": "function"}])["session"]
+ self.assertNotIn("tools", session)
+
+ def test_audio_payload_with_smart_turn(self) -> None:
+ """Audio session.update with smart_turn and a voiceprint."""
+ model = DashScopeAudioRealtimeModel(
+ "qwen-audio-3.0-realtime-plus",
+ CRED,
+ parameters=DashScopeAudioRealtimeModel.Parameters(
+ turn_detection="smart_turn",
+ voiceprint_audio_urls=["https://x/a.wav"],
+ max_history_turns=30,
+ ),
+ )
+ self.assertDictEqual(
+ model._session_update("be nice", None),
+ {
+ "type": "session.update",
+ "session": {
+ "instructions": "be nice",
+ "modalities": ["audio", "text"],
+ "voice": "longanqian",
+ "input_audio_format": "pcm",
+ "output_audio_format": "pcm",
+ "max_history_turns": 30,
+ "turn_detection": {
+ "type": "smart_turn",
+ "voiceprint_audio_urls": ["https://x/a.wav"],
+ },
+ },
+ },
+ )
+
+ def test_turn_detection_none_hands_endpointing_to_caller(self) -> None:
+ """``none`` sends null and marks endpointing as ours."""
+ model = DashScopeAudioRealtimeModel(
+ "qwen-audio-3.0-realtime-flash",
+ CRED,
+ parameters=DashScopeAudioRealtimeModel.Parameters(
+ turn_detection="none",
+ ),
+ )
+ session = model._session_update("x", None)["session"]
+ self.assertIsNone(session["turn_detection"])
+ self.assertFalse(model.turn_detection_enabled)
+
+ def test_adapter_facts(self) -> None:
+ """Protocol facts differ per adapter, not per model."""
+ omni = DashScopeRealtimeModel("qwen3.5-omni-flash-realtime", CRED)
+ audio = DashScopeAudioRealtimeModel(
+ "qwen-audio-3.0-realtime-plus",
+ CRED,
+ )
+ self.assertListEqual(
+ [
+ (omni.truncation, omni.supports_text_input),
+ (audio.truncation, audio.supports_text_input),
+ ],
+ [
+ (TruncationSupport.NONE, False),
+ (TruncationSupport.NONE, True),
+ ],
+ )
+
+
+class DashScopeParseTest(unittest.TestCase):
+ """Server frames -> model events."""
+
+ def setUp(self) -> None:
+ """Open a response so deltas have an item to attach to."""
+ self.model = DashScopeRealtimeModel(
+ "qwen3.5-omni-flash-realtime",
+ CRED,
+ )
+ self.model._parse(
+ {"type": "response.created", "response": {"id": "r1"}},
+ )
+
+ def test_reply_frames(self) -> None:
+ """Every reply-side frame maps to one model event."""
+ frames = [
+ {"type": "response.audio_transcript.delta", "delta": "你好"},
+ {
+ "type": "response.audio.delta",
+ "delta": base64.b64encode(b"\x01\x00").decode(),
+ },
+ {
+ "type": "input_audio_buffer.speech_started",
+ "item_id": "u1",
+ "audio_start_ms": 120,
+ },
+ {
+ "type": TRANSCRIPTION_DONE,
+ "item_id": "u1",
+ "transcript": "天气",
+ },
+ {
+ "type": "response.done",
+ "response": {
+ "id": "r1",
+ "usage": {"input_tokens": 10, "output_tokens": 5},
+ },
+ },
+ {"type": "session.updated"},
+ ]
+ self.assertListEqual(
+ [self.model._parse(f) for f in frames],
+ [
+ me.TranscriptDeltaEvent(item_id="r1", delta="你好"),
+ me.AudioDeltaEvent(
+ item_id="r1",
+ pcm=b"\x01\x00",
+ sample_rate=24000,
+ ),
+ me.SpeechStartedEvent(item_id="u1", at_ms=120),
+ me.InputTranscriptionEvent(item_id="u1", text="天气"),
+ me.ResponseDoneEvent(
+ item_id="r1",
+ input_tokens=10,
+ output_tokens=5,
+ ),
+ None,
+ ],
+ )
+
+ def test_tool_call_done_frame_is_authoritative(self) -> None:
+ """Accumulated deltas are only a fallback for a done frame that
+ omits ``arguments``."""
+ self.model._parse(
+ {
+ "type": "response.output_item.added",
+ "item": {
+ "type": "function_call",
+ "call_id": "c1",
+ "name": "get_weather",
+ },
+ },
+ )
+ self.model._parse(
+ {
+ "type": "response.function_call_arguments.delta",
+ "call_id": "c1",
+ "delta": '{"city":',
+ },
+ )
+ with_args = self.model._parse(
+ {
+ "type": "response.function_call_arguments.done",
+ "call_id": "c1",
+ "arguments": '{"city":"sh"}',
+ },
+ )
+ self.model._parse(
+ {
+ "type": "response.function_call_arguments.delta",
+ "call_id": "c2",
+ "name": "f",
+ "delta": '{"a":1}',
+ },
+ )
+ without_args = self.model._parse(
+ {"type": "response.function_call_arguments.done", "call_id": "c2"},
+ )
+
+ self.assertListEqual(
+ [
+ (with_args.tool_call.name, with_args.tool_call.input),
+ (without_args.tool_call.name, without_args.tool_call.input),
+ ],
+ [("get_weather", '{"city":"sh"}'), ("f", '{"a":1}')],
+ )
+
+ def test_audio_specific_frames_are_known_and_dropped(self) -> None:
+ """Audio-only frames are recognised and dropped, errors kept."""
+ model = DashScopeAudioRealtimeModel(
+ "qwen-audio-3.0-realtime-plus",
+ CRED,
+ )
+ self.assertListEqual(
+ [
+ model._parse(
+ {
+ "type": AMBIENT_DELTA,
+ "delta": "x",
+ },
+ ),
+ model._parse({"type": "voiceprint_audio_list.completed"}),
+ model._parse(
+ {
+ "type": "error",
+ "error": {"code": "E1", "message": "boom"},
+ },
+ ),
+ ],
+ [None, None, me.ModelErrorEvent(code="E1", message="boom")],
+ )
+
+
+class DashScopeAudioTextInputTest(IsolatedAsyncioTestCase):
+ """push_text sends a text item and asks for the reply."""
+
+ async def test_push_text_wire_shape(self) -> None:
+ """A text turn is one item plus a response request."""
+ model = DashScopeAudioRealtimeModel(
+ "qwen-audio-3.0-realtime-plus",
+ CRED,
+ )
+ sent: list[dict] = []
+
+ async def capture(payload: dict) -> None:
+ sent.append(payload)
+
+ model._send = capture # type: ignore[method-assign]
+ await model.push_text("你好")
+
+ self.assertListEqual(
+ sent,
+ [
+ {
+ "type": "conversation.item.create",
+ "item": {
+ "type": "message",
+ "role": "user",
+ "content": [{"type": "input_text", "text": "你好"}],
+ },
+ },
+ {
+ "type": "response.create",
+ "response": {"modalities": ["text", "audio"]},
+ },
+ ],
+ )
+
+
+class DashScopeDisconnectTest(IsolatedAsyncioTestCase):
+ """A closed WebSocket surfaces as ModelDisconnectedError."""
+
+ async def test_send_on_closed_socket(self) -> None:
+ """websockets' ConnectionClosed becomes the realtime-level error
+ and the socket reference is dropped."""
+ from websockets.exceptions import ConnectionClosedError
+ from websockets.frames import Close
+
+ class ClosedSocket:
+ """Raises like a socket the provider already closed."""
+
+ async def send(self, _payload: str) -> None:
+ """Fail with the provider's close frame."""
+ close = Close(1007, "idle 180s")
+ raise ConnectionClosedError(close, close, True)
+
+ model = DashScopeRealtimeModel("qwen3.5-omni-flash-realtime", CRED)
+ model._ws = ClosedSocket()
+
+ with self.assertRaises(ModelDisconnectedError) as ctx:
+ await model.push_audio(b"\x00\x00")
+ self.assertEqual(
+ (str(ctx.exception), model._ws),
+ (
+ "1007 (invalid frame payload data) idle 180s",
+ None,
+ ),
+ )
+
+ async def test_send_before_connect(self) -> None:
+ """No socket at all is the same condition."""
+ model = DashScopeRealtimeModel("qwen3.5-omni-flash-realtime", CRED)
+ with self.assertRaises(ModelDisconnectedError):
+ await model.commit_turn()
diff --git a/tests/realtime_local_transport_test.py b/tests/realtime_local_transport_test.py
new file mode 100644
index 000000000..590d15f46
--- /dev/null
+++ b/tests/realtime_local_transport_test.py
@@ -0,0 +1,145 @@
+# -*- coding: utf-8 -*-
+"""Unit tests for LocalAudioTransport, driving the PortAudio callbacks
+directly — no sound card is opened."""
+# pylint: disable=protected-access
+import asyncio
+from unittest.async_case import IsolatedAsyncioTestCase
+
+import numpy as np
+
+from agentscope.realtime import (
+ AudioFrame,
+ LocalAudioTransport,
+ PlayoutPosition,
+)
+
+BLOCK = 2400 # 100 ms at 24 kHz
+
+
+def _play(transport: LocalAudioTransport, blocks: int) -> np.ndarray:
+ """Run the output callback *blocks* times and return the last block."""
+ out = np.zeros((BLOCK, 1), np.int16)
+ for _ in range(blocks):
+ transport._on_output(out, BLOCK, None, None)
+ return out
+
+
+class LocalAudioTransportTest(IsolatedAsyncioTestCase):
+ """Playback buffering, playout accounting and the interruption fade."""
+
+ async def test_playout_accounting_and_first_played_at(self) -> None:
+ """Position advances by what the callback consumed, and the first
+ played timestamp is set once."""
+ transport = LocalAudioTransport(output_sample_rate=24000)
+ await transport.send_audio(
+ np.full(24000, 1000, np.int16).tobytes(),
+ "r1",
+ )
+
+ self.assertEqual(
+ transport.playout().model_dump(),
+ {"item_id": "r1", "played_ms": 0, "first_played_at": None},
+ )
+ _play(transport, 5)
+ position = transport.playout()
+ self.assertEqual(
+ (
+ position.item_id,
+ position.played_ms,
+ position.first_played_at is not None,
+ ),
+ ("r1", 500, True),
+ )
+
+ async def test_clear_audio_reports_cut_then_fades(self) -> None:
+ """The reported position excludes the fade tail; the tail ramps to
+ silence and the remainder of the block is zero-filled."""
+ transport = LocalAudioTransport(output_sample_rate=24000, fade_ms=30)
+ await transport.send_audio(
+ np.full(24000, 1000, np.int16).tobytes(),
+ "r1",
+ )
+ _play(transport, 5)
+
+ cut = await transport.clear_audio()
+ tail = np.frombuffer(bytes(transport._pending), np.int16)
+ self.assertEqual(
+ (
+ cut.played_ms,
+ len(tail),
+ int(tail[0]),
+ int(tail[-1]),
+ bool(np.all(np.diff(tail) <= 0)),
+ ),
+ (500, 720, 1000, 0, True),
+ )
+
+ out = _play(transport, 1)
+ self.assertEqual(
+ (
+ len(transport._pending),
+ int(out[0, 0]),
+ int(out[719, 0]),
+ int(out[720, 0]),
+ int(out[-1, 0]),
+ ),
+ (0, 1000, 0, 0, 0),
+ )
+
+ async def test_new_item_resets_counters(self) -> None:
+ """Audio for a new item starts a fresh position."""
+ transport = LocalAudioTransport(output_sample_rate=24000)
+ await transport.send_audio(
+ np.full(24000, 1000, np.int16).tobytes(),
+ "r1",
+ )
+ _play(transport, 3)
+ await transport.send_audio(b"\x00\x00" * 10, "r2")
+
+ self.assertEqual(
+ transport.playout().model_dump(),
+ {"item_id": "r2", "played_ms": 0, "first_played_at": None},
+ )
+
+ async def test_partial_block_is_zero_filled(self) -> None:
+ """A block larger than what is queued plays the remainder as silence
+ and counts only what was queued."""
+ transport = LocalAudioTransport(output_sample_rate=24000)
+ await transport.send_audio(np.full(600, 7, np.int16).tobytes(), "r1")
+
+ out = _play(transport, 1)
+ self.assertEqual(
+ (
+ int(out[0, 0]),
+ int(out[599, 0]),
+ int(out[600, 0]),
+ transport.playout().played_ms,
+ ),
+ (7, 7, 0, 25),
+ )
+
+ async def test_input_queue_drops_oldest_when_full(self) -> None:
+ """Capture never grows past ten seconds; the oldest chunk goes."""
+ transport = LocalAudioTransport(input_sample_rate=16000, chunk_ms=100)
+ transport._loop = asyncio.get_running_loop()
+ for i in range(transport._max_queued + 3):
+ transport._enqueue(AudioFrame(pcm=bytes([i % 256]) * 2))
+
+ queued = []
+ while not transport._in_queue.empty():
+ queued.append(transport._in_queue.get_nowait().pcm[0])
+ self.assertEqual(
+ (len(queued), queued[0], queued[-1]),
+ (transport._max_queued, 3, (transport._max_queued + 2) % 256),
+ )
+
+ async def test_playout_position_shape(self) -> None:
+ """The position is a plain value object."""
+ self.assertEqual(
+ PlayoutPosition(
+ item_id="x",
+ played_ms=1,
+ first_played_at=2.0,
+ ).model_dump(),
+ {"item_id": "x", "played_ms": 1, "first_played_at": 2.0},
+ )