Build and run agents you can see, understand and trust.
Python agentchatbotlarge-language-modelsllmllm-agentmcpmulti-agentmulti-modalreact-agentrealtime-agentvoice-agentvoice-assistant
feat(realtime): realtime voice agent with DashScope backends and local transport (#2547)
DavdGaof8b40f60f534
diff --git a/README.md b/README.mdindex d60bcd5c4..7390bcb40 100644--- a/README.md+++ b/README.md@@ -73,6 +73,7 @@ rather than constraining them with strict prompts and opinionated orchestrations## News<!-- BEGIN NEWS -->+- **[2026-09] `FEAT` `Experimental`:** Realtime voice agent supported — full-duplex speech-to-speech via `RealtimeAgent` on the DashScope Realtime API (Qwen-Omni, Qwen-Audio-3.0), with barge-in and tool calls. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/realtime)- **[2026-09] `FEAT`:** A2A protocol supported — chat with any remote A2A agent via `A2AAgent`. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/a2a) | [Docs](https://docs.agentscope.io/latest/en/building-blocks/a2a)- **[2026-08] `FEAT`:** Pipeline supported — run multiple agents by a fixed logic behind one event stream. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/pipeline) | [Docs](https://docs.agentscope.io/latest/en/building-blocks/pipeline/overview)- **[2026-08] `INTE`:** DingTalk channel supported. [Docs](https://docs.agentscope.io/latest/en/deploy/channel/dingtalk)@@ -82,7 +83,6 @@ rather than constraining them with strict prompts and opinionated orchestrations- **[2026-08] `INTE`:** GitHub MCP Registry and ClawHub supported as built-in hubs. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent_service) | [Docs](https://docs.agentscope.io/latest/en/deploy/hub)- **[2026-08] `FEAT`:** MCP & Skill Hub supported — browse a hub, install into your library, add to a workspace. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent_service) | [Docs](https://docs.agentscope.io/latest/en/deploy/hub)- **[2026-07] `INTE`:** Daytona-based workspace/sandbox supported. [Docs](https://docs.agentscope.io/latest/en/building-blocks/workspace)-- **[2026-07] `INTE`:** K8s, OpenSandbox-based workspace/sandbox supported. [Docs](https://docs.agentscope.io/latest/en/building-blocks/workspace)<!-- END NEWS -->[More news →](./docs/NEWS.md)diff --git a/README_zh.md b/README_zh.mdindex 510638c05..24871d43d 100644--- a/README_zh.md+++ b/README_zh.md@@ -72,6 +72,7 @@ AgentScope 的目标是充分发挥大模型的推理与工具调用能力,## 新闻<!-- BEGIN NEWS -->+- **[2026-09] `功能` `实验性`:** 支持实时语音智能体 —— 通过 `RealtimeAgent` 接入 DashScope Realtime API(Qwen-Omni、Qwen-Audio-3.0),全双工语音对话、打断与工具调用。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/realtime)- **[2026-09] `功能`:** 支持 A2A 协议 —— 通过 `A2AAgent` 与任意远端 A2A 智能体对话。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/a2a) | [文档](https://docs.agentscope.io/latest/zh/building-blocks/a2a)- **[2026-08] `功能`:** 支持流水线 —— 按照固化逻辑运行多智能体,并向外提供统一接口。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/pipeline) | [文档](https://docs.agentscope.io/latest/zh/building-blocks/pipeline/overview)- **[2026-08] `集成`:** 支持钉钉(DingTalk)频道。[文档](https://docs.agentscope.io/latest/zh/deploy/channel/dingtalk)@@ -81,7 +82,6 @@ AgentScope 的目标是充分发挥大模型的推理与工具调用能力,- **[2026-08] `集成`:** 内置集成 GitHub MCP Registry 与 ClawHub。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent_service) | [文档](https://docs.agentscope.io/latest/zh/deploy/hub)- **[2026-08] `功能`:** 支持 MCP & Skill Hub —— 浏览 hub、安装到个人库、再添加到工作区。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent_service) | [文档](https://docs.agentscope.io/latest/zh/deploy/hub)- **[2026-07] `集成`:** 集成 Daytona 工作区/沙箱实现。 [文档](https://docs.agentscope.io/latest/en/building-blocks/workspace)-- **[2026-07] `集成`:** 集成 K8s,OpenSandbox 工作区/沙箱实现。 [文档](https://docs.agentscope.io/latest/en/building-blocks/workspace)<!-- END NEWS -->[更多新闻 →](./docs/NEWS_zh.md)diff --git a/docs/NEWS.md b/docs/NEWS.mdindex 6db505a9a..d1f6f9ea5 100644--- a/docs/NEWS.md+++ b/docs/NEWS.md@@ -2,6 +2,7 @@<!-- The first 10 items are automatically synced to README.md and README_zh.md via GitHub Actions. --><!-- To update news in READMEs, modify this file and push to trigger the workflow. -->+- **[2026-09] `FEAT` `Experimental`:** Realtime voice agent supported — full-duplex speech-to-speech via `RealtimeAgent` on the DashScope Realtime API (Qwen-Omni, Qwen-Audio-3.0), with barge-in and tool calls. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/realtime)- **[2026-09] `FEAT`:** A2A protocol supported — chat with any remote A2A agent via `A2AAgent`. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/a2a) | [Docs](https://docs.agentscope.io/latest/en/building-blocks/a2a)- **[2026-08] `FEAT`:** Pipeline supported — run multiple agents by a fixed logic behind one event stream. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/pipeline) | [Docs](https://docs.agentscope.io/latest/en/building-blocks/pipeline/overview)- **[2026-08] `INTE`:** DingTalk channel supported. [Docs](https://docs.agentscope.io/latest/en/deploy/channel/dingtalk)diff --git a/docs/NEWS_zh.md b/docs/NEWS_zh.mdindex 1727d1308..304323df0 100644--- a/docs/NEWS_zh.md+++ b/docs/NEWS_zh.md@@ -2,6 +2,7 @@<!-- The first 10 items are automatically synced to README.md and README_zh.md via GitHub Actions. --><!-- To update news in READMEs, modify this file and push to trigger the workflow. -->+- **[2026-09] `功能` `实验性`:** 支持实时语音智能体 —— 通过 `RealtimeAgent` 接入 DashScope Realtime API(Qwen-Omni、Qwen-Audio-3.0),全双工语音对话、打断与工具调用。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/realtime)- **[2026-09] `功能`:** 支持 A2A 协议 —— 通过 `A2AAgent` 与任意远端 A2A 智能体对话。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/a2a) | [文档](https://docs.agentscope.io/latest/zh/building-blocks/a2a)- **[2026-08] `功能`:** 支持流水线 —— 按照固化逻辑运行多智能体,并向外提供统一接口。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/pipeline) | [文档](https://docs.agentscope.io/latest/zh/building-blocks/pipeline/overview)- **[2026-08] `集成`:** 支持钉钉(DingTalk)频道。[文档](https://docs.agentscope.io/latest/zh/deploy/channel/dingtalk)diff --git a/examples/realtime/README.md b/examples/realtime/README.mdnew file mode 100644index 000000000..33cf864ce--- /dev/null+++ b/examples/realtime/README.md@@ -0,0 +1,93 @@+# Realtime Voice Agent++Talk to a DashScope speech-to-speech model through your microphone.+`local_mic.py` wires up the three pieces of `agentscope.realtime`:++- `DashScopeAudioRealtimeModel` / `DashScopeRealtimeModel` — the provider+ session, resolved from the model card so any listed model works.+- `LocalAudioTransport` — the sound card: capture, playback, playout+ accounting and a short fade when the reply is cut off.+- `RealtimeAgent` — the turn-taking state machine in between: barge-in,+ tool calls with permission prompts, per-turn latency metrics.++The agent owns the model session, you own the transport, and one+`agent.run(transport)` borrows both until the transport ends.++## Prerequisites++- Python 3.11 or newer+- AgentScope with the realtime extra:++ ```bash+ pip install "agentscope[realtime]"+ ```++ `sounddevice` needs PortAudio: it is bundled on macOS and Windows; on+ Debian/Ubuntu run `apt install libportaudio2`.+- A DashScope API key with access to a realtime model.++## Run++```bash+export DASHSCOPE_API_KEY=sk-...+python examples/realtime/local_mic.py+```++Speak, hear the reply, and speak over it to interrupt. `Ctrl-C` quits.++The default model is `qwen-audio-3.0-realtime-plus`. Any card listed for the+credential works; the class is looked up from the card, the same way the+service layer does it:++```bash+REALTIME_MODEL=qwen3.5-omni-flash-realtime python examples/realtime/local_mic.py+```++An unknown name prints the available ones.++Every reply ends with a diagnostic line: how it finished, the model's time to+first audio byte (`ttfb`), the user's end-to-end wait (`e2e`), and the+assistant message as it now stands in the context — after a barge-in that is+only the part you actually heard.++### Tools++The example registers `Bash`, `Edit`, `Write` and `Read`. When the model+calls one that needs permission, the terminal asks:++```+ [permission] Bash({"command": "ls"})+ allow? [y/N]+```++The prompt runs in a thread, so audio keeps flowing while you decide. The+agent gives up on a prompt after five minutes.++## Tips++- **Use headphones.** With speakers, the microphone picks up the agent's own+ voice; the provider's VAD then hears "the user" start speaking and cuts the+ reply off — the agent interrupts itself. `LocalAudioTransport` does no echo+ cancellation (a browser would), so headphones are the fix.+- **Do not route both directions through one Bluetooth headset.** macOS+ switches AirPods and similar devices to the hands-free profile when they+ are used for input and output at once, and PortAudio's separate+ input/output streams on that device often produce no sound. Pick devices+ by index from `python -m sounddevice`, e.g. the headset's microphone with+ the built-in speakers:++ ```bash+ REALTIME_INPUT_DEVICE=3 REALTIME_OUTPUT_DEVICE=2 python examples/realtime/local_mic.py+ ```++- **Silence is fine.** DashScope closes a session after about three minutes+ without a response. The example logs that it happened and keeps the+ microphone open; the next thing you say reconnects the model. The+ conversation history stays in `agent.state`, but the provider starts fresh —+ it cannot be seeded with prior turns.+- **Model limits are per turn, not per token.** `qwen3-omni-flash-realtime`+ remembers eight turns, `qwen-audio-3.0-realtime-*` fifty; older turns are+ dropped silently on the provider side. The cards under+ `agentscope/realtime/_dashscope/` list each model's limits.+- **Typed input** only works on models that accept text+ (`qwen-audio-3.0-realtime-*`); the Omni models are audio-in only.diff --git a/examples/realtime/local_mic.py b/examples/realtime/local_mic.pynew file mode 100644index 000000000..d4d91f7d3--- /dev/null+++ b/examples/realtime/local_mic.py@@ -0,0 +1,153 @@+# -*- coding: utf-8 -*-+"""Talk to a DashScope realtime model through the local microphone, with+tools the model may call and a terminal prompt for the ones that need+your permission.++ export DASHSCOPE_API_KEY=sk-...+ python examples/realtime/local_mic.py++Any of the DashScope realtime models works; the class is looked up from+the model card, the same way the service layer does it:++ REALTIME_MODEL=qwen3.5-omni-flash-realtime python ...++Pick devices by index from ``python -m sounddevice`` when the defaults+are wrong, e.g. a Bluetooth headset used for both directions:++ REALTIME_INPUT_DEVICE=3 REALTIME_OUTPUT_DEVICE=2 python ...++Speak, hear the reply, and speak over it to interrupt. Ctrl-C to quit.+"""+import asyncio+import os++from agentscope.agent import RealtimeAgent+from agentscope.credential import DashScopeCredential+from agentscope.event import (+ ConfirmResult,+ ReplyEndEvent,+ ReplyStartEvent,+ RequireUserConfirmEvent,+ TextBlockDeltaEvent,+ ToolResultEndEvent,+ ToolResultStartEvent,+ UserConfirmResultEvent,+ UserInputAudioStartEvent,+ UserInputTranscriptionEvent,+)+from agentscope.realtime import LocalAudioTransport+from agentscope.tool import Bash, Edit, Read, Toolkit, Write+++async def main() -> None:+ """Run one voice session until interrupted."""+ api_key = os.environ.get("DASHSCOPE_API_KEY")+ if not api_key:+ raise SystemExit("Set DASHSCOPE_API_KEY first.")+ credential = DashScopeCredential(api_key=api_key)++ # Resolve the model class from its card rather than hard-coding it:+ # a credential can serve several realtime APIs, and each card carries+ # the tag of the class that produced it.+ name = os.environ.get("REALTIME_MODEL", "qwen-audio-3.0-realtime-plus")+ cards = {c.name: c for c in credential.list_realtime_models()}+ classes = {c.type: c for c in credential.get_realtime_model_classes()}+ if name not in cards:+ raise SystemExit(f"Unknown model {name!r}; try: {sorted(cards)}")+ card = cards[name]+ model = classes[card.model_type](name, credential, model_card=card)++ agent = RealtimeAgent(+ name="Friday",+ sys_prompt="你是一个中文语音助手,回答尽量简短。",+ model=model,+ toolkit=Toolkit(tools=[Bash(), Edit(), Write(), Read()]),+ )+ transport = LocalAudioTransport(+ input_sample_rate=model.input_sample_rate,+ output_sample_rate=model.output_sample_rate,+ input_device=_device("REALTIME_INPUT_DEVICE"),+ output_device=_device("REALTIME_OUTPUT_DEVICE"),+ )++ print(f"[{name}] listening... (Ctrl-C to quit)")+ # The agent owns the model session, we own the transport, and one+ # run() borrows both until the transport ends.+ async with agent, transport:+ async for event in agent.run(transport):+ match event:+ case UserInputAudioStartEvent():+ print("\n[you] ...", end="", flush=True)+ case UserInputTranscriptionEvent():+ print(f"\r[you] {event.transcript}")+ case ReplyStartEvent():+ print(f"[{agent.name}] ", end="", flush=True)+ case TextBlockDeltaEvent():+ print(event.delta, end="", flush=True)+ case RequireUserConfirmEvent():+ await confirm(agent, event)+ case ToolResultStartEvent():+ print(f"\n [tool] {event.tool_call_name} running...")+ case ToolResultEndEvent():+ print(f" [tool] {event.state}")+ case ReplyEndEvent():+ m = agent.last_turn_metrics+ print(+ f"\n ({event.finished_reason}"+ f" | ttfb={_ms(m.backend_ttfb)}"+ f" | e2e={_ms(m.e2e_latency)})",+ )+ if agent.state.context:+ tail = agent.state.context[-1]+ print(+ f" context[-1] = {tail.role}: "+ f"{tail.get_text_content()!r}",+ )+++async def confirm(+ agent: RealtimeAgent,+ event: RequireUserConfirmEvent,+) -> None:+ """Ask on the terminal whether each pending tool call may run.++ The prompt runs in a thread so the audio pumps keep going while we+ wait; the agent itself gives up after five minutes.+ """+ loop = asyncio.get_running_loop()+ results = []+ for call in event.tool_calls:+ print(f"\n [permission] {call.name}({call.input})")+ answer = await loop.run_in_executor(None, input, " allow? [y/N] ")+ results.append(+ ConfirmResult(+ tool_call=call,+ confirmed=answer.strip().lower() in ("y", "yes"),+ ),+ )+ await agent.send(+ UserConfirmResultEvent(+ reply_id=event.reply_id,+ confirm_results=results,+ ),+ )+++def _device(env: str) -> int | str | None:+ """A sounddevice index or name from the environment, if given."""+ value = os.environ.get(env)+ if value is None:+ return None+ return int(value) if value.isdigit() else value+++def _ms(seconds: float | None) -> str:+ """Format a latency for the console."""+ return "n/a" if seconds is None else f"{seconds * 1000:.0f}ms"+++if __name__ == "__main__":+ try:+ asyncio.run(main())+ except KeyboardInterrupt:+ print("\nbye")diff --git a/pyproject.toml b/pyproject.tomlindex a490ba158..8ba7f9f98 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -106,6 +106,11 @@ workspace = [tools = ["ripgrep>=14.1,<15",]+# ------------ Realtime voice ------------+realtime = [+ "websockets>=13",+ "sounddevice",+]# ------------ A2A ------------a2a = ["a2a-sdk>=1.1.1",diff --git a/src/agentscope/agent/__init__.py b/src/agentscope/agent/__init__.pyindex 54b852973..b1b1f3052 100644--- a/src/agentscope/agent/__init__.py+++ b/src/agentscope/agent/__init__.py@@ -3,10 +3,14 @@from ._agent import Agentfrom ._a2a_agent import A2AAgentfrom ._config import ContextConfig, InjectionConfig, ModelConfig, ReActConfig+from ._realtime import RealtimeAgent, TurnAggregator, TurnMetrics__all__ = ["Agent","A2AAgent",+ "RealtimeAgent",+ "TurnAggregator",+ "TurnMetrics","ContextConfig","InjectionConfig","ModelConfig",diff --git a/src/agentscope/agent/_realtime/__init__.py b/src/agentscope/agent/_realtime/__init__.pynew file mode 100644index 000000000..2891b12a0--- /dev/null+++ b/src/agentscope/agent/_realtime/__init__.py@@ -0,0 +1,7 @@+# -*- coding: utf-8 -*-+"""The realtime voice agent."""+from ._agent import RealtimeAgent+from ._aggregator import TurnAggregator+from ._metrics import TurnMetrics++__all__ = ["RealtimeAgent", "TurnAggregator", "TurnMetrics"]diff --git a/src/agentscope/agent/_realtime/_agent.py b/src/agentscope/agent/_realtime/_agent.pynew file mode 100644index 000000000..ba3cb228b--- /dev/null+++ b/src/agentscope/agent/_realtime/_agent.py@@ -0,0 +1,1034 @@+# -*- coding: utf-8 -*-+"""The realtime voice agent."""+import asyncio+import base64+import time+import uuid+from dataclasses import dataclass, field+from typing import Any, AsyncIterator++from ...realtime import _events as me+from ._aggregator import TurnAggregator+from ...realtime._base import ModelDisconnectedError, RealtimeModelBase+from ._metrics import TurnMetrics+from ...realtime._transport._base import (+ AudioFrame,+ ControlFrame,+ ControlFrameType,+ TransportBase,+)+from ...realtime._vad import SpeechTransition, VADBase+from ..._logging import logger+from ..._utils._common import _json_loads_with_repair+from ...event import (+ AgentEvent,+ ConfirmResult,+ DataBlockDeltaEvent,+ DataBlockEndEvent,+ DataBlockStartEvent,+ ExternalExecutionResultEvent,+ ModelCallEndEvent,+ ModelCallStartEvent,+ ReplyEndEvent,+ ReplyStartEvent,+ RequireUserConfirmEvent,+ TextBlockDeltaEvent,+ TextBlockEndEvent,+ TextBlockStartEvent,+ ToolCallEndEvent,+ ToolCallStartEvent,+ ToolResultEndEvent,+ ToolResultStartEvent,+ ToolResultTextDeltaEvent,+ UserConfirmResultEvent,+ UserInputAudioStartEvent,+ UserInputTranscriptionEvent,+ UserInterruptEvent,+)+from ...message import (+ Msg,+ TextBlock,+ ToolCallBlock,+ ToolResultBlock,+ ToolResultState,+ UserMsg,+)+from ...permission import PermissionBehavior, PermissionEngine+from ...state import AgentState+from ...tool import ToolChunk, ToolResponse, Toolkit+from ...types import ReplyFinishedReason++# Audio buffered while the model is being reconnected: 10 s at 100 ms chunks.+_BACKLOG_FRAMES = 100+++@dataclass+class _Reply:+ """One in-flight assistant turn, plus the text/audio alignment needed+ to work out what the user actually heard."""++ item_id: str+ text_block_id: str = field(default_factory=lambda: uuid.uuid4().hex)+ audio_block_id: str = field(default_factory=lambda: uuid.uuid4().hex)+ text: str = ""+ audio_ms: float = 0.0+ text_started: bool = False+ audio_started: bool = False+ marks: list[tuple[float, int]] = field(default_factory=list)++ def on_audio(self, pcm: bytes, sample_rate: int) -> None:+ """Account for one chunk of generated audio."""+ self.audio_ms += len(pcm) / (sample_rate * 2) * 1000++ def on_text(self, delta: str) -> None:+ """Record a transcript delta against the audio generated so far."""+ self.text += delta+ self.marks.append((self.audio_ms, len(self.text)))++ def spoken_prefix(self, played_ms: int) -> str:+ """The transcript prefix matching *played_ms* of playback.++ Transcript deltas usually run slightly ahead of the audio they+ describe, so this errs towards keeping one word too many.+ """+ length = 0+ for at_ms, text_len in self.marks:+ if at_ms > played_ms:+ break+ length = text_len+ return self.text[:length]+++class RealtimeAgent:+ """A voice agent: a realtime model on one side, a transport on the+ other, and the turn-taking state machine in between.++ Unlike :class:`~agentscope.agent.Agent` it is bidirectional and has no+ request/reply boundary — audio flows in continuously while events flow+ out of :meth:`run`.++ Three lifetimes are kept apart on purpose. The agent owns the model+ session and the state; a transport is owned by whoever created it; a+ :meth:`run` borrows both for as long as both are alive. So a client+ can drop and reconnect without losing the model session, and the model+ session can time out during a long silence and be re-established on+ the next word without touching the transport.++ The public methods are the only entry point for discrete input. A+ transport carrying a browser's ``ControlFrame`` calls exactly those+ methods rather than reaching into the agent, so one code path handles+ the semantics whether the caller is Python or a browser. Continuous+ audio is separate and always arrives through the transport.++ Example:+ .. code-block:: python++ agent = RealtimeAgent("Friday", "Be brief.", model)+ async with agent: # model session+ async with LocalAudioTransport() as t: # sound card+ async for event in agent.run(t): # this call+ print(event)+ """++ def __init__(+ self,+ name: str,+ sys_prompt: str,+ model: RealtimeModelBase,+ toolkit: Toolkit | None = None,+ state: AgentState | None = None,+ vad: VADBase | None = None,+ aggregator: TurnAggregator | None = None,+ ) -> None:+ """Initialize the realtime agent.++ Args:+ name (`str`):+ Display name stamped on assistant messages and events.+ sys_prompt (`str`):+ System instructions sent to the model on connect, with the+ toolkit's skill instructions appended.+ model (`RealtimeModelBase`):+ The realtime model. Its session is opened by+ :meth:`connect` and lives as long as this agent — not as+ long as any one transport — and is re-established on the+ next user audio if the provider closes it.+ toolkit (`Toolkit | None`, optional):+ Tools the model may call. Executed here, with permission+ checks against ``state.permission_context``.+ state (`AgentState | None`, optional):+ Conversation history, permission rules and tool context.+ A new one is created if omitted.+ vad (`VADBase | None`, optional):+ Server-side voice activity detection over the audio+ arriving from the transport. ``STARTED`` cuts off a reply+ the user speaks over; ``ENDED`` closes the user's turn and+ hands it to the model. When given, the provider's own turn+ detection is switched off so that exactly one source+ decides turn boundaries; when ``None``, the provider+ decides and this agent only reacts to what it reports.+ aggregator (`TurnAggregator | None`, optional):+ Collapses the provider's transcripts into clean user turns:+ merging one that endpointing split, dropping bare+ acknowledgements. Subclass it to change what counts as a+ turn. A default with no backchannel list is used if+ omitted.+ """+ self.name = name+ self.sys_prompt = sys_prompt+ self.model = model+ self.toolkit = toolkit+ self.state = state or AgentState()+ self.vad = vad+ self.aggregator = aggregator or TurnAggregator()++ self._engine = PermissionEngine(self.state.permission_context)+ self._transport: TransportBase | None = None+ self._out: asyncio.Queue = asyncio.Queue()+ self._reply: _Reply | None = None+ self._finished_item = ""+ self._metrics = TurnMetrics()+ self._pending_tools: dict[str, ToolCallBlock] = {}+ self._confirmations: dict[str, asyncio.Future[ConfirmResult]] = {}+ self._tasks: set[asyncio.Task] = set()+ self._barge_lock = asyncio.Lock()++ # Model session: connected flag, the downlink pump that outlives+ # any transport, and reconnect bookkeeping.+ self._connected = False+ self._connected_event = asyncio.Event()+ self._downlink: asyncio.Task | None = None+ self._backlog: list[bytes] = []+ self._retry_at = 0.0+ self._backoff = 1.0++ # ------------------------------------------------------------------+ # Lifecycle: the model session+ # ------------------------------------------------------------------++ async def __aenter__(self) -> "RealtimeAgent":+ """Connect on entry."""+ await self.connect()+ return self++ async def __aexit__(self, *exc: Any) -> None:+ """Disconnect on exit."""+ await self.close()++ async def connect(self) -> None:+ """Open the model session. Safe to call again after the provider+ closed it; a no-op while connected."""+ if self._connected:+ return++ instructions = self.sys_prompt+ tools = None+ if self.toolkit is not None:+ groups = self.state.tool_context.activated_groups+ skills = await self.toolkit.get_skill_instructions(groups)+ if skills:+ instructions = f"{instructions}\n\n{skills}"+ if self.model.card.supports_tools:+ tools = await self.toolkit.get_tool_schemas(groups)++ # TODO(realtime): tools and instructions are sent once, here.+ # Activating a tool group or installing a skill mid-session —+ # ResetTools, the meta tool — therefore has no effect until the+ # next connect, even though the model is told it can do it.+ #+ # Fix: a `RealtimeModelBase.update_session(instructions, tools)`+ # re-sent whenever `state.tool_context.activated_groups` changes.+ # OpenAI, DashScope and xAI accept `session.update` on the open+ # connection; Gemini forbids it and must reconnect with a+ # `sessionResumption` handle plus a new `setup`, which keeps the+ # context. No provider documents whether the update applies+ # retroactively, so treat it as affecting future turns only. Do+ # not let it change `voice`: OpenAI locks it after first audio.+ await self.model.connect(+ context=self.state.context,+ instructions=instructions,+ tools=tools,+ turn_detection_disabled=self.vad is not None,+ )+ if self.vad is not None:+ self.vad.reset()+ self.aggregator.reset()+ self._connected = True+ self._connected_event.set()+ self._backoff = 1.0+ if self._downlink is None:+ self._downlink = asyncio.create_task(+ self._pump_downlink(),+ name="rt-downlink",+ )+ self._downlink.add_done_callback(self._on_downlink_done)++ async def close(self) -> None:+ """Cancel everything in flight and close the model session."""+ for future in self._confirmations.values():+ future.cancel()+ self._confirmations.clear()+ for task in list(self._tasks):+ task.cancel()+ if self._tasks:+ await asyncio.gather(*self._tasks, return_exceptions=True)+ if self._downlink is not None:+ self._downlink.cancel()+ await asyncio.gather(self._downlink, return_exceptions=True)+ self._downlink = None+ self._connected = False+ self._connected_event.clear()+ await self.model.close()++ def _on_downlink_done(self, task: asyncio.Task) -> None:+ """Surface a crashed downlink instead of a silently dead session."""+ if task.cancelled() or task.exception() is None:+ return+ self._connected = False+ self._connected_event.clear()+ logger.error(+ "RealtimeAgent: downlink pump crashed",+ exc_info=task.exception(),+ )++ async def _try_connect(self) -> bool:+ """Reconnect the model with backoff; ``False`` while still down."""+ now = time.monotonic()+ if now < self._retry_at:+ return False+ try:+ await self.connect()+ except Exception as exc: # noqa: BLE001+ self._retry_at = now + self._backoff+ self._backoff = min(self._backoff * 2, 30.0)+ logger.warning(+ "RealtimeAgent: reconnect failed (%s); retrying in %.0fs",+ exc,+ self._backoff,+ )+ return False+ return True++ # ------------------------------------------------------------------+ # Lifecycle: one transport+ # ------------------------------------------------------------------++ async def run(self, transport: TransportBase) -> AsyncIterator[AgentEvent]:+ """Pump *transport* and yield agent events until it ends.++ The transport is borrowed, not owned: it must already be started+ and is not closed here, so the caller can keep it, reuse it or hand+ it on. The stream ends when the transport's input ends — a client+ disconnecting — not when a reply ends; there may be many replies.+ Call again with a new transport to resume the same session.++ Breaking out of the loop early does not run cleanup immediately;+ wrap the generator in :func:`contextlib.aclosing` when that+ matters.+ """+ if self._transport is not None:+ raise RuntimeError("RealtimeAgent.run is already active.")+ # Audio is forwarded as-is in both directions, so the rates must+ # already agree; resampling belongs to the transport, not here.+ pairs = {+ "input": (+ transport.input_sample_rate,+ self.model.input_sample_rate,+ ),+ "output": (+ transport.output_sample_rate,+ self.model.output_sample_rate,+ ),+ }+ if self.vad is not None:+ pairs["vad"] = (self.vad.sample_rate, transport.input_sample_rate)+ for what, (got, want) in pairs.items():+ if got != want:+ raise ValueError(+ f"{what} sample rate mismatch: {got} Hz delivered, "+ f"{want} Hz expected.",+ )++ self._transport = transport+ uplink = asyncio.create_task(+ self._pump_uplink(transport),+ name="rt-up",+ )+ try:+ while not uplink.done():+ getter = asyncio.ensure_future(self._out.get())+ done, _ = await asyncio.wait(+ {getter, uplink},+ return_when=asyncio.FIRST_COMPLETED,+ )+ if getter in done:+ yield getter.result()+ else:+ getter.cancel()+ # The transport is gone: cut off any reply still in flight so+ # the model stops talking to nobody, then hand the caller the+ # events that produced (the interrupted ReplyEnd) before the+ # stream ends rather than leaking them into the next run.+ await self._barge_in()+ while not self._out.empty():+ yield self._out.get_nowait()+ # A transport failure must not look like a clean disconnect.+ if not uplink.cancelled() and uplink.exception() is not None:+ raise uplink.exception() # type: ignore[misc]+ finally:+ if not uplink.done():+ uplink.cancel()+ await asyncio.gather(uplink, return_exceptions=True)+ self._transport = None++ # ------------------------------------------------------------------+ # Discrete input+ # ------------------------------------------------------------------++ async def send(+ self,+ inputs: (+ str+ | Msg+ | UserConfirmResultEvent+ | UserInterruptEvent+ | ExternalExecutionResultEvent+ ),+ ) -> None:+ """Feed the agent anything that is not audio.++ Mirrors :meth:`Agent.reply`'s ``inputs``: a text turn (which cuts+ off any reply in progress first), the outcome of a permission+ prompt, an interrupt, or the result of a tool run elsewhere.++ Raises:+ `NotImplementedError`: For a text turn when the provider takes+ no text input, or for external execution results, which+ are not supported yet.+ """+ match inputs:+ case UserInterruptEvent():+ await self._barge_in()++ case UserConfirmResultEvent():+ for result in inputs.confirm_results:+ future = self._confirmations.get(result.tool_call.id)+ if future and not future.done():+ future.set_result(result)++ case ExternalExecutionResultEvent():+ raise NotImplementedError(+ "External tool execution is not supported yet.",+ )++ case str() | Msg():+ if not self.model.supports_text_input:+ raise NotImplementedError(+ f"{type(self.model).__name__} accepts no text input.",+ )+ msg = (+ UserMsg(name="user", content=inputs)+ if isinstance(inputs, str)+ else inputs+ )+ text = msg.get_text_content() or ""+ await self._barge_in()+ self.state.context.append(msg)+ await self.model.push_text(text)++ async def interrupt(self) -> None:+ """Stop the active reply, as when the user presses stop."""+ await self.send(UserInterruptEvent())++ @property+ def last_turn_metrics(self) -> TurnMetrics:+ """Latency breakdown of the most recent turn."""+ return self._metrics++ # ------------------------------------------------------------------+ # Uplink: transport -> model+ # ------------------------------------------------------------------++ async def _pump_uplink(self, transport: TransportBase) -> None:+ """Forward user audio and control frames until *transport* ends."""+ async for frame in transport.incoming():+ if isinstance(frame, AudioFrame):+ await self._on_audio(frame)+ else:+ await self._on_control(frame)++ async def _on_audio(self, frame: AudioFrame) -> None:+ """Run our VAD if we own it, reconnect if needed, forward audio."""+ speech = self.vad.push(frame.pcm) if self.vad is not None else None+ try:+ await self._forward_audio(frame.pcm, speech)+ except ModelDisconnectedError as exc:+ # The provider closed on us between two frames; the downlink+ # may not have noticed yet. Keep the frame and reconnect on+ # the next one — this is the idle-timeout path, not an error.+ self._mark_disconnected()+ self._backlog.append(frame.pcm)+ logger.info(+ "RealtimeAgent: model session closed (%s); keep talking "+ "and it reconnects on the next audio.",+ exc,+ )++ def _mark_disconnected(self) -> None:+ """Forget the model session so the next audio reconnects."""+ self._connected = False+ self._connected_event.clear()++ async def _forward_audio(+ self,+ pcm: bytes,+ speech: SpeechTransition | None,+ ) -> None:+ """Body of :meth:`_on_audio`; raises on a closed model session."""+ pushed = False+ if not self._connected:+ self._backlog.append(pcm)+ del self._backlog[:-_BACKLOG_FRAMES]+ if not await self._try_connect():+ return+ for buffered in self._backlog:+ await self.model.push_audio(buffered)+ self._backlog.clear()+ pushed = True++ if speech is SpeechTransition.STARTED:+ await self._barge_in()+ elif speech is SpeechTransition.ENDED:+ now = time.monotonic()+ self._metrics.user_speech_end_at = now+ await self.model.commit_turn()+ self._metrics.turn_committed_at = time.monotonic()++ if not pushed:+ await self.model.push_audio(pcm)++ async def _on_control(self, frame: ControlFrame) -> None:+ """Translate one upstream control frame into :meth:`send`."""+ match frame.type:+ case ControlFrameType.TEXT:+ await self.send(frame.data.get("text", ""))+ case ControlFrameType.USER_CONFIRM:+ await self.send(UserConfirmResultEvent(**frame.data))+ case ControlFrameType.INTERRUPT:+ await self.send(UserInterruptEvent())+ case ControlFrameType.EXTERNAL_EXECUTION_RESULT:+ await self.send(ExternalExecutionResultEvent(**frame.data))+ case _:+ logger.debug("RealtimeAgent: ignoring %s frame", frame.type)++ # ------------------------------------------------------------------+ # Barge-in+ # ------------------------------------------------------------------++ async def _barge_in(self) -> None:+ """Cut the reply short and correct both contexts to what was heard.++ Whether a given overlap counts as an interruption is decided+ before this is called — by the provider when it owns turn+ detection, by the VAD's own debounce otherwise.++ Reached from the uplink pump, the downlink pump, :meth:`send` and+ :meth:`run`'s exit, so it is serialised; the losers find the reply+ already closed and return.+ """+ async with self._barge_lock:+ await self._barge_in_locked()++ async def _barge_in_locked(self) -> None:+ """Body of :meth:`_barge_in`, run under the lock."""+ reply = self._reply+ if reply is None:+ return++ spoken, played_ms = (+ "",+ 0,+ ) # nothing reaches the ear without a transport+ if self._transport is not None:+ position = await self._transport.clear_audio()+ if position.item_id and position.item_id != reply.item_id:+ logger.warning(+ "RealtimeAgent: playout reports %s but %s is open; "+ "not truncating.",+ position.item_id,+ reply.item_id,+ )+ return+ played_ms = position.played_ms+ spoken = reply.spoken_prefix(played_ms)++ self._truncate_reply(spoken)+ if self._connected:+ await self.model.truncate(reply.item_id, played_ms, spoken)+ await self.model.cancel_response()+ self._finish_reply(ReplyFinishedReason.INTERRUPTED)++ def _truncate_reply(self, spoken: str) -> None:+ """Rewrite the current reply in context to the part heard.++ Non-text blocks stay: a tool call that already ran belongs in the+ record even though the sentence around it was never heard.+ """+ if not self.state.context:+ return+ tail = self.state.context[-1]+ if tail.role != "assistant" or tail.name != self.name:+ return++ others = (+ []+ if isinstance(tail.content, str)+ else [_ for _ in tail.content if not isinstance(_, TextBlock)]+ )+ if spoken.strip():+ tail.content = [TextBlock(text=spoken), *others]+ elif others:+ tail.content = others+ else:+ self.state.context.pop()++ # ------------------------------------------------------------------+ # Downlink: model -> transport + events (lives with the agent)+ # ------------------------------------------------------------------++ async def _pump_downlink(self) -> None:+ """Translate model events for as long as the agent is open.++ When the provider closes the session the pump does not exit: it+ marks the model disconnected and waits for :meth:`connect` to be+ called again, which the uplink does on the next user audio.+ """+ while True:+ await self._connected_event.wait()+ async for event in self.model.events():+ await self._on_model_event(event)+ self._mark_disconnected()+ self._finish_reply(ReplyFinishedReason.ERROR)+ logger.info(+ "RealtimeAgent: model session ended; keep talking and it "+ "reconnects on the next audio.",+ )++ async def _on_model_event(self, event: me.ModelEvent) -> None:+ """Handle one model event."""+ rate = self.model.output_sample_rate+ match event:+ case me.SpeechStartedEvent():+ self._emit(+ UserInputAudioStartEvent(+ session_id=self.state.session_id,+ item_id=event.item_id,+ ),+ )+ await self._barge_in()++ case me.SpeechEndedEvent():+ # With provider turn detection this is also its commit.+ now = time.monotonic()+ self._metrics.user_speech_end_at = now+ if self.vad is None:+ self._metrics.turn_committed_at = now++ case me.InputTranscriptionEvent():+ self._on_transcription(event)++ case me.ResponseCreatedEvent():+ self._start_reply(event.item_id)++ case me.AudioDeltaEvent():+ if self._transport is None:+ await self._barge_in() # nobody listening+ return+ reply = self._start_reply(event.item_id)+ if reply is None:+ return+ reply.on_audio(event.pcm, rate)+ await self._transport.send_audio(event.pcm, reply.item_id)+ self._emit_audio(reply, event.pcm, rate)+ self._metrics.backend_first_audio_at = (+ self._metrics.backend_first_audio_at or time.monotonic()+ )++ case me.TranscriptDeltaEvent():+ reply = self._start_reply(event.item_id)+ if reply is None:+ return+ reply.on_text(event.delta)+ self._emit_text(reply, event.delta)++ case me.ToolCallEvent():+ if self._start_reply(event.item_id) is not None:+ self._pending_tools[event.tool_call.id] = event.tool_call++ case me.ResponseDoneEvent():+ self._metrics.input_tokens = event.input_tokens+ self._metrics.output_tokens = event.output_tokens+ self._finish_reply(ReplyFinishedReason.COMPLETED)+ self._schedule_tools()++ case me.ModelErrorEvent():+ logger.error(+ "RealtimeAgent: model error %s: %s",+ event.code,+ event.message,+ )+ self._finish_reply(ReplyFinishedReason.ERROR)++ case me.SessionEndedEvent():+ pass # the events() iterator ends right after this++ def _on_transcription(self, event: me.InputTranscriptionEvent) -> None:+ """Record a settled user turn, merging a split one back together."""+ turn = self.aggregator.take(event.text)+ if turn is None:+ logger.debug("RealtimeAgent: dropping %r", event.text)+ return++ if self.aggregator.merges_with_previous() and self._merge_user(turn):+ transcript = self.state.context[-1].get_text_content() or turn+ else:+ self.state.context.append(UserMsg(name="user", content=turn))+ transcript = turn++ self._emit(+ UserInputTranscriptionEvent(+ session_id=self.state.session_id,+ item_id=event.item_id,+ transcript=transcript,+ ),+ )++ def _merge_user(self, text: str) -> bool:+ """Append *text* to the previous user turn that endpointing split.++ The stub assistant message between the two halves is dropped: it+ is whatever the model managed to say before being cut off, which+ answers half a question nobody finished asking.+ """+ context = self.state.context+ if context and context[-1].role == "assistant":+ if not (context[-1].get_text_content() or "").strip():+ context.pop()+ if not context or context[-1].role != "user":+ return False+ previous = context[-1].get_text_content() or ""+ context[-1].content = [TextBlock(text=f"{previous}{text}")]+ return True++ def _start_reply(self, item_id: str) -> _Reply | None:+ """Open a reply for *item_id*, emitting its start events once.++ Returns ``None`` for an item already closed — deltas still in+ flight after a barge-in must not reopen it.+ """+ if self._reply is not None and self._reply.item_id == item_id:+ return self._reply+ if item_id == self._finished_item:+ return None++ self._reply = _Reply(item_id=item_id)+ self.state.reply_id = item_id+ self._metrics = TurnMetrics(+ user_speech_end_at=self._metrics.user_speech_end_at,+ turn_committed_at=self._metrics.turn_committed_at,+ )+ self._emit(+ ReplyStartEvent(+ session_id=self.state.session_id,+ reply_id=item_id,+ name=self.name,+ ),+ )+ self._emit(+ ModelCallStartEvent(+ reply_id=item_id,+ model_name=self.model.model_name,+ ),+ )+ return self._reply++ def _finish_reply(self, reason: ReplyFinishedReason) -> None:+ """Close the open reply, if any."""+ reply = self._reply+ if reply is None:+ return+ if self._transport is not None:+ position = self._transport.playout()+ if position.item_id == reply.item_id:+ self._metrics.first_audio_played_at = position.first_played_at+ if reply.text_started:+ self._emit(+ TextBlockEndEvent(+ reply_id=reply.item_id,+ block_id=reply.text_block_id,+ ),+ )+ if reply.audio_started:+ self._emit(+ DataBlockEndEvent(+ reply_id=reply.item_id,+ block_id=reply.audio_block_id,+ ),+ )+ self._emit(+ ModelCallEndEvent(+ reply_id=reply.item_id,+ input_tokens=self._metrics.input_tokens,+ output_tokens=self._metrics.output_tokens,+ ),+ )+ self._emit(+ ReplyEndEvent(+ session_id=self.state.session_id,+ reply_id=reply.item_id,+ finished_reason=reason,+ ),+ )+ self._finished_item = reply.item_id+ self._reply = None++ def _emit_text(self, reply: _Reply, delta: str) -> None:+ """Emit a transcript delta, opening the block on first use."""+ if not reply.text_started:+ reply.text_started = True+ self._emit(+ TextBlockStartEvent(+ reply_id=reply.item_id,+ block_id=reply.text_block_id,+ ),+ )+ # Grow the tail text block rather than appending one per delta,+ # or the context ends up as dozens of one-word blocks.+ tail = self.state.context[-1] if self.state.context else None+ blocks = (+ tail.content+ if tail is not None and tail.id == reply.item_id+ else None+ )+ if (+ isinstance(blocks, list)+ and blocks+ and isinstance(blocks[-1], TextBlock)+ ):+ blocks[-1].text += delta+ else:+ self.state.append_context(self.name, [TextBlock(text=delta)])+ self._emit(+ TextBlockDeltaEvent(+ reply_id=reply.item_id,+ block_id=reply.text_block_id,+ delta=delta,+ ),+ )++ def _emit_audio(self, reply: _Reply, pcm: bytes, rate: int) -> None:+ """Emit an audio delta, opening the block on first use."""+ media_type = f"audio/pcm;rate={rate}"+ if not reply.audio_started:+ reply.audio_started = True+ self._emit(+ DataBlockStartEvent(+ reply_id=reply.item_id,+ block_id=reply.audio_block_id,+ media_type=media_type,+ ),+ )+ self._emit(+ DataBlockDeltaEvent(+ reply_id=reply.item_id,+ block_id=reply.audio_block_id,+ data=base64.b64encode(pcm).decode("ascii"),+ media_type=media_type,+ ),+ )++ def _emit(self, event: AgentEvent) -> None:+ """Queue one event for :meth:`run`."""+ self._out.put_nowait(event)++ # ------------------------------------------------------------------+ # Tools+ # ------------------------------------------------------------------++ def _schedule_tools(self) -> None:+ """Run the tool calls of the finished reply, then ask for more."""+ if not self._pending_tools or self.toolkit is None:+ return+ calls = list(self._pending_tools.values())+ self._pending_tools.clear()+ task = asyncio.create_task(self._run_tools(calls), name="rt-tools")+ self._tasks.add(task)+ task.add_done_callback(self._tasks.discard)++ async def _run_tools(self, calls: list[ToolCallBlock]) -> None:+ """Execute *calls* in order, then trigger the follow-up response."""+ reply_id = self.state.reply_id+ try:+ for call in calls:+ await self._run_tool(reply_id, call)+ await self.model.request_response()+ except Exception: # noqa: BLE001+ logger.exception("RealtimeAgent: tool execution failed")++ async def _run_tool(self, reply_id: str, call: ToolCallBlock) -> None:+ """Check permission for one call, run it, and report the result."""+ assert self.toolkit is not None+ self._emit(+ ToolCallStartEvent(+ reply_id=reply_id,+ tool_call_id=call.id,+ tool_call_name=call.name,+ ),+ )+ self._emit(ToolCallEndEvent(reply_id=reply_id, tool_call_id=call.id))++ try:+ tool = await self.toolkit.check_tool_available(+ call.name,+ self.state.tool_context.activated_groups,+ )+ tool_input = _json_loads_with_repair(call.input, tool.input_schema)+ except Exception as exc: # noqa: BLE001+ await self._report_tool(reply_id, call, str(exc))+ return++ decision = await self._engine.check_permission(tool, tool_input)+ if decision.behavior in (+ PermissionBehavior.ASK,+ PermissionBehavior.PASSTHROUGH,+ ):+ confirmed = await self._ask_user(reply_id, call, decision)+ if not confirmed:+ await self._report_tool(+ reply_id,+ call,+ f'Tool "{call.name}" denied by user.',+ state=ToolResultState.DENIED,+ )+ return+ elif decision.behavior is PermissionBehavior.DENY:+ await self._report_tool(+ reply_id,+ call,+ decision.message or f'Tool "{call.name}" denied by policy.',+ state=ToolResultState.DENIED,+ )+ return++ self._emit(+ ToolResultStartEvent(+ reply_id=reply_id,+ tool_call_id=call.id,+ tool_call_name=call.name,+ ),+ )+ parts: list[str] = []+ streamed = False+ result_state = ToolResultState.SUCCESS+ try:+ async for chunk in self.toolkit.call_tool(call, self.state):+ texts = [+ b.text for b in chunk.content if isinstance(b, TextBlock)+ ]+ if isinstance(chunk, ToolResponse):+ # The completed result. Chunks before it were display+ # only, so this alone is what the provider receives.+ parts = texts+ result_state = chunk.state+ if not streamed:+ for text in texts:+ self._emit_tool_delta(reply_id, call.id, text)+ break+ if not isinstance(chunk, ToolChunk):+ break+ streamed = True+ for text in texts:+ self._emit_tool_delta(reply_id, call.id, text)+ except Exception: # noqa: BLE001+ logger.exception("RealtimeAgent: tool %s failed", call.name)+ parts = [f"Error executing tool {call.name}."]+ result_state = ToolResultState.ERROR++ await self._report_tool(+ reply_id,+ call,+ "".join(parts) or "Tool executed successfully.",+ state=result_state,+ started=True,+ )++ def _emit_tool_delta(self, reply_id: str, call_id: str, text: str) -> None:+ """Emit one fragment of tool output."""+ self._emit(+ ToolResultTextDeltaEvent(+ reply_id=reply_id,+ tool_call_id=call_id,+ delta=text,+ ),+ )++ async def _ask_user(+ self,+ reply_id: str,+ call: ToolCallBlock,+ decision: Any,+ ) -> bool:+ """Ask the user to confirm *call* and wait for the answer."""+ call.suggested_rules = decision.suggested_rules or []+ self._emit(+ RequireUserConfirmEvent(reply_id=reply_id, tool_calls=[call]),+ )+ future: asyncio.Future[+ ConfirmResult+ ] = asyncio.get_running_loop().create_future()+ self._confirmations[call.id] = future+ try:+ result = await asyncio.wait_for(future, timeout=300)+ except (asyncio.TimeoutError, asyncio.CancelledError):+ return False+ finally:+ self._confirmations.pop(call.id, None)++ for rule in result.rules or []:+ self._engine.add_rule(rule)+ return result.confirmed++ async def _report_tool(+ self,+ reply_id: str,+ call: ToolCallBlock,+ output: str,+ state: ToolResultState = ToolResultState.ERROR,+ started: bool = False,+ ) -> None:+ """Close the tool result lifecycle and send the output back."""+ if not started:+ self._emit(+ ToolResultStartEvent(+ reply_id=reply_id,+ tool_call_id=call.id,+ tool_call_name=call.name,+ ),+ )+ self._emit(+ ToolResultTextDeltaEvent(+ reply_id=reply_id,+ tool_call_id=call.id,+ delta=output,+ ),+ )+ self._emit(+ ToolResultEndEvent(+ reply_id=reply_id,+ tool_call_id=call.id,+ state=state,+ ),+ )+ await self.model.push_tool_result(+ ToolResultBlock(id=call.id, name=call.name, output=output),+ )diff --git a/src/agentscope/agent/_realtime/_aggregator.py b/src/agentscope/agent/_realtime/_aggregator.pynew file mode 100644index 000000000..c94ae1d6a--- /dev/null+++ b/src/agentscope/agent/_realtime/_aggregator.py@@ -0,0 +1,75 @@+# -*- coding: utf-8 -*-+"""Turn aggregation."""+import time++_TRAILING = " 。,,.!?!?、"+++class TurnAggregator:+ """Collapses the transcripts a provider emits into clean user turns.++ Turn boundaries in speech are inferred, and inferred wrongly often+ enough to matter: one sentence gets split across two turns when the+ speaker pauses, and a stray "mm-hmm" gets promoted to a turn of its+ own. Neither is visible to the provider once it has answered, but both+ end up in our own context, where they outlive the call.+ """++ def __init__(+ self,+ merge_window_ms: int = 800,+ backchannels: frozenset[str] = frozenset(),+ min_chars: int = 1,+ ) -> None:+ """Initialize the aggregator.++ Args:+ merge_window_ms (`int`, defaults to `800`):+ A transcript arriving this soon after the previous turn+ continues it rather than starting a new one.+ backchannels (`frozenset[str]`, optional):+ Acknowledgements that never constitute a turn. Language+ specific, so empty by default.+ min_chars (`int`, defaults to `1`):+ Transcripts shorter than this are dropped.+ """+ self.merge_window_ms = merge_window_ms+ self.backchannels = frozenset(backchannels)+ self.min_chars = min_chars+ self._last_at: float | None = None+ self._merges = False++ def take(self, transcript: str) -> str | None:+ """Accept a settled transcript and decide whether it is a turn.++ Args:+ transcript (`str`):+ The transcript the provider settled on.++ Returns:+ `str | None`:+ The turn text, or ``None`` for an empty transcript or a+ bare acknowledgement.+ """+ text = transcript.strip()+ if len(text) < self.min_chars:+ return None+ if text.strip(_TRAILING) in self.backchannels:+ return None++ now = time.monotonic()+ self._merges = (+ self._last_at is not None+ and (now - self._last_at) * 1000 <= self.merge_window_ms+ )+ self._last_at = now+ return text++ def merges_with_previous(self) -> bool:+ """Whether the last :meth:`take` continued the previous turn."""+ return self._merges++ def reset(self) -> None:+ """Forget the previous turn, e.g. on reconnect."""+ self._last_at = None+ self._merges = Falsediff --git a/src/agentscope/agent/_realtime/_metrics.py b/src/agentscope/agent/_realtime/_metrics.pynew file mode 100644index 000000000..43fa62356--- /dev/null+++ b/src/agentscope/agent/_realtime/_metrics.py@@ -0,0 +1,62 @@+# -*- coding: utf-8 -*-+"""Per-turn latency metrics."""+from pydantic import BaseModel, Field+++class TurnMetrics(BaseModel):+ """The four timestamps of one turn, and the delays derived from them.++ All timestamps are :func:`time.monotonic` readings, in seconds.+ """++ user_speech_end_at: float | None = None+ """The user stopped speaking."""++ turn_committed_at: float | None = None+ """Endpointing settled and the turn was handed to the backend."""++ backend_first_audio_at: float | None = None+ """The backend produced its first audio byte."""++ first_audio_played_at: float | None = None+ """That audio actually reached the speaker. Reported by the transport,+ and the only timestamp that reflects what the user perceives."""++ input_tokens: int = Field(default=0, ge=0)+ output_tokens: int = Field(default=0, ge=0)++ @property+ def endpointing_delay(self) -> float | None:+ """Time spent deciding the user had finished."""+ return self._delta(self.user_speech_end_at, self.turn_committed_at)++ @property+ def backend_ttfb(self) -> float | None:+ """Time the backend took to start answering."""+ return self._delta(+ self.turn_committed_at,+ self.backend_first_audio_at,+ )++ @property+ def transport_delay(self) -> float | None:+ """Time between audio leaving the server and reaching the ear."""+ return self._delta(+ self.backend_first_audio_at,+ self.first_audio_played_at,+ )++ @property+ def e2e_latency(self) -> float | None:+ """The user stopped speaking until they heard an answer."""+ return self._delta(+ self.user_speech_end_at,+ self.first_audio_played_at,+ )++ @staticmethod+ def _delta(start: float | None, end: float | None) -> float | None:+ """The gap between two optional timestamps."""+ if start is None or end is None:+ return None+ return end - startdiff --git a/src/agentscope/credential/_base.py b/src/agentscope/credential/_base.pyindex 906beab76..262141706 100644--- a/src/agentscope/credential/_base.py+++ b/src/agentscope/credential/_base.py@@ -9,6 +9,7 @@ from .._utils._common import _generate_idif TYPE_CHECKING:from ..embedding import EmbeddingModelBasefrom ..model import ChatModelBase, ModelCard+ from ..realtime import RealtimeModelBase, RealtimeModelCardfrom ..tts import TTSModelBasefrom ..tts._tts_model_card import TTSModelCard@@ -67,6 +68,23 @@ class CredentialBase(BaseModel):cards.extend(tts_cls.list_models())return cards+ @classmethod+ def get_realtime_model_classes(+ cls,+ ) -> list[Type["RealtimeModelBase"]]:+ """Return the realtime model classes supported by this credential.+ The default returns an empty list (provider does not support+ realtime)."""+ return []++ @classmethod+ def list_realtime_models(cls) -> list["RealtimeModelCard"]:+ """List the candidate realtime models under this credential."""+ cards: list["RealtimeModelCard"] = []+ for rt_cls in cls.get_realtime_model_classes():+ cards.extend(rt_cls.list_models())+ return cards+@classmethoddef list_models(cls) -> list["ModelCard"]:"""List the candidate chat models that are available under thisdiff --git a/src/agentscope/credential/_dashscope.py b/src/agentscope/credential/_dashscope.pyindex 11e6ab59f..9799b5203 100644--- a/src/agentscope/credential/_dashscope.py+++ b/src/agentscope/credential/_dashscope.py@@ -9,6 +9,7 @@ from ._base import CredentialBaseif TYPE_CHECKING:from ..embedding import EmbeddingModelBasefrom ..model import ChatModelBase+ from ..realtime import RealtimeModelBasefrom ..tts import TTSModelBase_DASHSCOPE_BASE_URL = "https://dashscope.aliyuncs.com/compatible-mode/v1"@@ -59,6 +60,18 @@ class DashScopeCredential(CredentialBase):DashScopeCosyVoiceTTSModel,]+ @classmethod+ def get_realtime_model_classes(+ cls,+ ) -> list[Type["RealtimeModelBase"]]:+ """Return the DashScope realtime model classes."""+ from ..realtime import (+ DashScopeAudioRealtimeModel,+ DashScopeRealtimeModel,+ )++ return [DashScopeRealtimeModel, DashScopeAudioRealtimeModel]+@classmethoddef get_embedding_model_class(cls) -> Type["EmbeddingModelBase"]:"""Return the DashScopeEmbeddingModel class."""diff --git a/src/agentscope/event/__init__.py b/src/agentscope/event/__init__.pyindex 51632a731..dd9e40ac8 100644--- a/src/agentscope/event/__init__.py+++ b/src/agentscope/event/__init__.py@@ -35,6 +35,8 @@ from ._event import (ExternalExecutionResultEvent,CustomEvent,AgentEvent,+ UserInputAudioStartEvent,+ UserInputTranscriptionEvent,ConfirmResult,)@@ -73,5 +75,7 @@ __all__ = ["ExternalExecutionResultEvent","CustomEvent","AgentEvent",+ "UserInputAudioStartEvent",+ "UserInputTranscriptionEvent","ConfirmResult",]diff --git a/src/agentscope/event/_event.py b/src/agentscope/event/_event.pyindex 59d5f8d4d..8bce51cbd 100644--- a/src/agentscope/event/_event.py+++ b/src/agentscope/event/_event.py@@ -64,6 +64,9 @@ class EventType(StrEnum):USER_INTERRUPT = "USER_INTERRUPT"EXTERNAL_EXECUTION_RESULT = "EXTERNAL_EXECUTION_RESULT"+ USER_INPUT_AUDIO_START = "USER_INPUT_AUDIO_START"+ USER_INPUT_TRANSCRIPTION = "USER_INPUT_TRANSCRIPTION"+CUSTOM = "CUSTOM"@@ -562,6 +565,34 @@ class CustomEvent(EventBase):"""Arbitrary payload."""+class UserInputAudioStartEvent(EventBase):+ """The user started speaking, as detected by VAD."""++ type: Literal[+ EventType.USER_INPUT_AUDIO_START+ ] = EventType.USER_INPUT_AUDIO_START+ """Event type."""+ session_id: str+ """ID of the session this input belongs to."""+ item_id: str+ """ID of the user input item the provider assigned."""+++class UserInputTranscriptionEvent(EventBase):+ """The settled transcript of one spoken user turn."""++ type: Literal[+ EventType.USER_INPUT_TRANSCRIPTION+ ] = EventType.USER_INPUT_TRANSCRIPTION+ """Event type."""+ session_id: str+ """ID of the session this input belongs to."""+ item_id: str+ """ID of the user input item the provider assigned."""+ transcript: str+ """What the user said."""++AgentEvent: TypeAlias = (ReplyStartEvent| ReplyEndEvent@@ -590,5 +621,7 @@ AgentEvent: TypeAlias = (| UserConfirmResultEvent| UserInterruptEvent| ExternalExecutionResultEvent+ | UserInputAudioStartEvent+ | UserInputTranscriptionEvent| CustomEvent)diff --git a/src/agentscope/realtime/DECISIONS.md b/src/agentscope/realtime/DECISIONS.mdnew file mode 100644index 000000000..d8b734a2b--- /dev/null+++ b/src/agentscope/realtime/DECISIONS.md@@ -0,0 +1,57 @@+# Realtime 设计决策记录++已定的事项不因单个提问重开;改变需要说明新事实。++## 已定++| 决策 | 依据 |+|---|---|+| 三层事实划分:类属性 = adapter 实现;card = 模型是什么;property = 各家归属不同的(采样率、turn detection) | 四家 API 调研:xAI 采样率可调、DashScope 固定 |+| `TruncationSupport` 三态 NONE / SERVER / EXPLICIT | OpenAI(WS)=EXPLICIT、Gemini=SERVER、DashScope/xAI=NONE |+| card 四个上下文限制字段并存,全部可选 | DashScope 按轮数+秒数、OpenAI/Gemini 按 token、同时生效 |+| 公开方法是离散输入的唯一入口,`ControlFrame` 是其线上形态 | livekit / openai SDK / pipecat / vocode 四家一致 |+| `_barge_in` 串行化,三个调用点共用一把锁 | pipecat CHANGELOG:4712 记录的双队列插队 bug |+| 复用 `AgentState`,不另建 `VoiceContext` | `append_context` 已覆盖进行中回合的追加语义 |+| 打断截断留在 `RealtimeAgent._truncate_reply`,不进 `AgentState` | 用户决定,不扩散到 `state/` |+| `SessionConfig` 二分:`chat`(Agent 驱动)/ `voice`(RealtimeAgent 驱动),内层再按 backend 判别 | 分类依据是驱动的运行时类,不是字段形状 |+| `TurnAggregator` 在 M1 做上下文合并那一半 | SERVER 模式无提交权,但脏历史会进持久化/重连/压缩 |+| M1 只做 DashScope,不做 OpenAI | 打断闭环除 provider RPC 外整条链 DashScope 都会跑到 |+| 不复用 `Agent` 实现级联 | 用户决定 |+| VAD 归 agent(`RealtimeAgent(vad=)`),不归 transport;无客户端上报的 speech 字段 | livekit 放 session 层、pipecat 主动从 transport 搬出、无一家采信浏览器 speech |+| 给了 `vad=` 就关 provider 的 turn detection,一个来源判回合;`TurnMode` 枚举删除,由 `vad is None` 推导 | livekit `_resolve_rt_turn_detection_enabled`;pipecat 警告双源发重复帧 |+| transport 由创建者拥有,`run(transport)` 只借用,不 start/close | livekit `start(room=)` 不关 room;三者平级:agent / transport / run |+| 下行泵归 agent(`connect()` 起),上行泵归 `run()`;两次 run 之间模型队列照常消费 | 否则空档里工具结果、模型帧无人处理 |+| 模型会话被 provider 关掉后**不主动重连**,下一帧用户音频触发 `connect()`(可重入,带退避,攒帧) | 静默时挂着 API 连接没必要 |+| 离散输入统一为 `send(inputs)`,镜像 `Agent.reply(inputs=)`;`interrupt()` 为快捷方式 | `send_text` / `send_confirm` 合并;`ExternalExecutionResultEvent` 入口留着,M1 抛 NotImplementedError |+| 流式接口叫 `run`,async generator,起动即迭代 | 与 `reply_stream` 结束条件不同(连接断才结束),不复用其名 |+| `TurnAggregator` 作为实体注入(`aggregator=`),不经 config;pydantic `TurnConfig` 只存在于 app 层 | 它不依赖 agent 内部件,可子类化(pipecat 的 turn strategy 家族) |+| `fade_ms` 归 transport 构造参数 | 淡出由 transport 执行 |+| provider 连接异常由 adapter 翻译成 `ModelDisconnectedError`,agent 只认它、不 import websockets;发送失败 = 断开,攒帧等下一帧重连,上行泵不死 | 真机复现:DashScope 180s 空闲关闭后 `push_audio` 撞上已关 socket,比下行泵察觉更早 |+| `run()` 入口校验 transport / VAD 与 model 的采样率,不一致直接拒绝;agent 不做重采样 | Copilot review;重采样归 transport |+| 工具结果只取 `ToolResponse.content`,`ToolChunk` 仅用于展示 | `ToolResponse` 是完整结果(与 `Agent` 一致),否则流式工具的文本会重复 |+| `RealtimeAgent` / `TurnAggregator` / `TurnMetrics` 放 `agent/_realtime/`;`realtime/` 只放模型侧(model、card、事件、transport、VAD) | 与 `Agent` 同目录;`agent/` 单向依赖 `realtime/` |+| 模型事件统一 `Event` 后缀(`AudioDeltaEvent` 等),基类 `ModelEvent` | 对齐 `agentscope.event` 的命名 |+| 命名保留 `realtime`,不改 `live`;agent 层将来挂级联后端时再评估 `VoiceAgent` | 六家里五家叫 Realtime,M1 的 provider 产品名就是 Qwen-Omni-Realtime |+| DashScope 的 Omni 和 Audio-3.0 两个 S2S API 用两个类,不在一个类里按模型名路由 | `supports_text_input` 等是 per-API 事实;`tts/_dashscope/` 一个 provider 三个类的先例 |+| 反向查找靠 card 上的 `model_type`(产出它的 class 的 `type` tag),服务层建 `{cls.type: cls}` dict,不扫名字 | 同名模型可出现在两个 class 的 card 里(OpenAI chat/responses)|+| `TurnMetrics` 是 agent 的**产出**(四个时刻),不是配置,不进构造函数;`last_turn_metrics` 读取 | 每回合的指标事件(livekit `metrics_collected`)待前端需要时再加 |+| `PlayoutPosition.first_played_at` 由 transport 在音频线程记录,agent 在回合结束时读取 | e2e_latency 的唯一真实来源;随 `LocalAudioTransport` 落地 |+| 用户打字先打断当前回复 | livekit / pipecat 默认;不打断需要额外的待处理状态 |++## 已推迟(保留概念,届时形态如下)++| 事项 | 阶段 | 预期形态 | 为什么现在不做 |+|---|---|---|---|+| filler(工具执行期间的填充语) | M2 | 本地 TTS 播放,不走 provider | DashScope 不收文本输入 |+| 独立的打断判定(叠在 VAD 之上) | M2 | 词数门槛(pipecat)或独立模型(livekit) | SERVER 模式下 provider 已决定;M1 无本地 VAD |+| `update_session(instructions, tools)` | M2 | OpenAI/DashScope/xAI 发 `session.update`;Gemini 重连+resumption handle;不改 `voice` | 见 `_agent.py` connect() 处 TODO |+| `ModelEvent.SessionResumption` | Gemini 接入时 | 随 Gemini adapter 一起回来 | Gemini 唯一的改 instructions 路径 |+| 语义端点检测 | M3 之后 | 先接 pipecat smart-turn ONNX | OpenAI/DashScope 已有 `semantic_vad` |+| 上下文压缩 | 重连功能时 | `summarize(messages) -> str` 灌进新 session | S2S 下压缩本地历史不影响 provider |+| WebSocket transport 兜底 | Safari 支持面确认后 | 同 9 字节头 + 二进制 Opus | 优先 WebTransport |++## 待定++- `SessionConfig` 二分改造单独 PR 的时机+- 模型重连后 UI 提示事件(M1 只记日志)+- `AudioDelta` 事件的 base64 惰性编码(local 场景全部白做)diff --git a/src/agentscope/realtime/__init__.py b/src/agentscope/realtime/__init__.pynew file mode 100644index 000000000..97e1d3def--- /dev/null+++ b/src/agentscope/realtime/__init__.py@@ -0,0 +1,69 @@+# -*- coding: utf-8 -*-+"""Realtime voice sessions in AgentScope.++The model side of a voice session: realtime models, their cards and+events, transports and VAD. The agent that ties them together is+:class:`agentscope.agent.RealtimeAgent`.+"""+from ._base import ModelDisconnectedError, RealtimeModelBase, TruncationSupport+from ._dashscope import (+ DashScopeAudioRealtimeModel,+ DashScopeRealtimeModel,+)+from ._events import (+ AudioDeltaEvent,+ InputTranscriptionEvent,+ ModelErrorEvent,+ ModelEvent,+ ResponseCreatedEvent,+ ResponseDoneEvent,+ SessionEndedEvent,+ SpeechEndedEvent,+ SpeechStartedEvent,+ ToolCallEvent,+ TranscriptDeltaEvent,+)+from ._model_card import RealtimeModelCard+from ._playout import PlayoutPosition+from ._transport import (+ AudioFrame,+ ControlFrame,+ ControlFrameType,+ LocalAudioTransport,+ TransportBase,+ TransportFrame,+)+from ._vad import SpeechTransition, VADBase++__all__ = [+ # Model+ "RealtimeModelBase",+ "DashScopeRealtimeModel",+ "DashScopeAudioRealtimeModel",+ "RealtimeModelCard",+ "TruncationSupport",+ "ModelDisconnectedError",+ # Transport+ "TransportBase",+ "LocalAudioTransport",+ "TransportFrame",+ "AudioFrame",+ "ControlFrame",+ "ControlFrameType",+ "PlayoutPosition",+ # Turn taking+ "VADBase",+ "SpeechTransition",+ # Model events+ "ModelEvent",+ "SessionEndedEvent",+ "SpeechStartedEvent",+ "SpeechEndedEvent",+ "InputTranscriptionEvent",+ "ResponseCreatedEvent",+ "AudioDeltaEvent",+ "TranscriptDeltaEvent",+ "ToolCallEvent",+ "ResponseDoneEvent",+ "ModelErrorEvent",+]diff --git a/src/agentscope/realtime/_base.py b/src/agentscope/realtime/_base.pynew file mode 100644index 000000000..f3d82b2db--- /dev/null+++ b/src/agentscope/realtime/_base.py@@ -0,0 +1,242 @@+# -*- coding: utf-8 -*-+"""The realtime model base class."""+import inspect+from abc import ABC, abstractmethod+from enum import StrEnum+from pathlib import Path+from typing import Any, AsyncIterator++from pydantic import BaseModel++from ._events import ModelEvent+from ._model_card import RealtimeModelCard+from ..credential import CredentialBase+from ..message import Msg, ToolResultBlock+++class ModelDisconnectedError(ConnectionError):+ """The provider closed the session; the next user audio reconnects."""+++class TruncationSupport(StrEnum):+ """How the provider lets an interrupted turn be corrected. A protocol+ property, constant across the models of one API."""++ NONE = "none"+ SERVER = "server"+ """The provider corrects itself; ``truncate`` is a no-op."""++ EXPLICIT = "explicit"+ """The provider accepts an explicit truncate frame."""+++class RealtimeModelBase(ABC):+ """A bidirectional realtime model: audio in, audio and tool calls out.++ Facts live in three places: class attributes for what the adapter+ implements, the model card for what the model is, and properties for+ what is a constant on one provider and a parameter on another.+ """++ class Parameters(BaseModel):+ """Provider-specific tuneables, surfaced to the UI through the+ card's ``parameter_schema``."""++ # ------------------------------------------------------------------+ # Adapter facts — constant across every model of this API+ # ------------------------------------------------------------------++ type: str = ""+ """Identifies the adapter; stamped onto every card it lists so a stored+ config can be mapped back to this class."""++ truncation: TruncationSupport = TruncationSupport.NONE+ """Whether and how an interrupted turn can be corrected."""++ supports_text_input: bool = False+ """Whether a text turn can be injected mid-session."""++ def __init__(+ self,+ model_name: str,+ credential: CredentialBase,+ parameters: "RealtimeModelBase.Parameters | None" = None,+ model_card: RealtimeModelCard | None = None,+ ) -> None:+ """Initialize the realtime model.++ Args:+ model_name (`str`):+ The model name, e.g. ``"qwen3-omni-flash-realtime"``.+ credential (`CredentialBase`):+ The credential used to authenticate against the provider.+ parameters (`RealtimeModelBase.Parameters | None`, optional):+ Provider-specific parameters; defaults are used if omitted.+ model_card (`RealtimeModelCard | None`, optional):+ The model card. When ``None`` it is looked up by+ *model_name* among the cards shipped with this class, so+ that constructing a model by name alone still yields the+ right sample rates and limits.++ Raises:+ `ValueError`: If no card is given and none matches *model_name*.+ """+ self.model_name = model_name+ self.credential = credential+ self.parameters = parameters or self.Parameters()+ self.card = model_card or self._find_card(model_name)++ @classmethod+ def _find_card(cls, model_name: str) -> RealtimeModelCard:+ """Look up the shipped card for *model_name*."""+ for card in cls.list_models():+ if card.name == model_name:+ return card+ raise ValueError(+ f"No realtime model card found for {model_name!r} in "+ f"{cls.__name__}. Pass `model_card` explicitly for a model "+ f"that ships no card.",+ )++ @classmethod+ def list_models(+ cls,+ custom_yaml_dir: str | None = None,+ ) -> list[RealtimeModelCard]:+ """List the candidate models of this adapter.++ Args:+ custom_yaml_dir (`str | None`, optional):+ Directory to scan. Defaults to the ``_models`` directory+ next to the concrete subclass's source file.++ Returns:+ `list[RealtimeModelCard]`: The cards found, oldest error+ skipped with a warning.+ """+ yaml_dir = (+ Path(custom_yaml_dir)+ if custom_yaml_dir+ else Path(inspect.getfile(cls)).parent / "_models"+ )+ cards = RealtimeModelCard.list_from_directory(yaml_dir, cls.Parameters)+ for card in cards:+ card.model_type = cls.type+ return cards++ # ------------------------------------------------------------------+ # Facts that are constant on some providers and tuneable on others+ # ------------------------------------------------------------------++ @property+ def input_sample_rate(self) -> int:+ """The PCM rate :meth:`push_audio` expects, in Hz."""+ return self.card.input_sample_rate++ @property+ def output_sample_rate(self) -> int:+ """The PCM rate of the audio in :class:`AudioDelta`, in Hz."""+ return self.card.output_sample_rate++ @property+ @abstractmethod+ def turn_detection_enabled(self) -> bool:+ """Whether the provider decides turn boundaries. When ``False``+ the caller owns endpointing and must call :meth:`commit_turn`."""++ # ------------------------------------------------------------------+ # Lifecycle+ # ------------------------------------------------------------------++ @abstractmethod+ async def connect(+ self,+ context: list[Msg],+ instructions: str,+ tools: list[dict] | None = None,+ **kwargs: Any,+ ) -> None:+ """Open the session, seeding it with prior *context*.++ Args:+ context (`list[Msg]`):+ Prior conversation, e.g. from ``AgentState.context``.+ instructions (`str`):+ System instructions.+ tools (`list[dict] | None`, optional):+ Tool JSON schemas. Ignored unless the card declares+ ``supports_tools``.+ **kwargs (`Any`):+ Extra provider fields. ``turn_detection_disabled=True``+ asks the provider not to detect turns because the caller+ runs its own VAD; a resumption handle may also be passed.+ """++ @abstractmethod+ async def close(self) -> None:+ """Close the session and release the connection."""++ @abstractmethod+ async def events(self) -> AsyncIterator[ModelEvent]:+ """Yield model events until the session ends."""+ raise NotImplementedError(f"{type(self).__name__}.events")+ yield # pylint: disable=unreachable++ # ------------------------------------------------------------------+ # Input+ # ------------------------------------------------------------------++ @abstractmethod+ async def push_audio(self, pcm: bytes) -> None:+ """Feed user audio: PCM16 mono at :attr:`input_sample_rate`."""++ @abstractmethod+ async def push_text(self, text: str) -> None:+ """Feed a text user turn.++ Raises:+ `NotImplementedError`: If :attr:`supports_text_input` is+ ``False``.+ """++ @abstractmethod+ async def push_tool_result(self, block: ToolResultBlock) -> None:+ """Feed the result of a tool the caller executed."""++ @abstractmethod+ async def commit_turn(self) -> None:+ """Declare the user turn finished.++ Only called when :attr:`turn_detection_enabled` is ``False``.+ """++ # ------------------------------------------------------------------+ # Response control+ # ------------------------------------------------------------------++ @abstractmethod+ async def request_response(self) -> None:+ """Ask the model to start generating a reply."""++ @abstractmethod+ async def cancel_response(self) -> None:+ """Stop the active reply without closing the session."""++ @abstractmethod+ async def truncate(+ self,+ item_id: str,+ played_ms: int,+ played_text: str,+ ) -> None:+ """Rewrite an assistant turn to what the user actually heard. A+ no-op unless :attr:`truncation` is ``EXPLICIT``.++ Args:+ item_id (`str`):+ The interrupted assistant item.+ played_ms (`int`):+ Milliseconds of its audio that reached the speaker.+ played_text (`str`):+ The corresponding prefix of its spoken text.+ """diff --git a/src/agentscope/realtime/_dashscope/__init__.py b/src/agentscope/realtime/_dashscope/__init__.pynew file mode 100644index 000000000..09dbfdfe3--- /dev/null+++ b/src/agentscope/realtime/_dashscope/__init__.py@@ -0,0 +1,6 @@+# -*- coding: utf-8 -*-+"""The DashScope realtime models."""+from ._audio_model import DashScopeAudioRealtimeModel+from ._model import DashScopeRealtimeModel++__all__ = ["DashScopeRealtimeModel", "DashScopeAudioRealtimeModel"]diff --git a/src/agentscope/realtime/_dashscope/_audio_model.py b/src/agentscope/realtime/_dashscope/_audio_model.pynew file mode 100644index 000000000..bf868908b--- /dev/null+++ b/src/agentscope/realtime/_dashscope/_audio_model.py@@ -0,0 +1,126 @@+# -*- coding: utf-8 -*-+"""The DashScope Qwen-Audio realtime model."""+from pathlib import Path+from typing import Any, Literal++from pydantic import Field++from ._model import DashScopeRealtimeModel+from .. import _events as me+from .._base import RealtimeModelBase+from .._model_card import RealtimeModelCard+from ..._logging import logger+++class DashScopeAudioRealtimeModel(DashScopeRealtimeModel):+ """The Qwen-Audio-3.0-Realtime API over WebSocket.++ Same endpoint and framing as Qwen-Omni, but it accepts text turns,+ spells both PCM formats as ``"pcm"``, and offers ``smart_turn``+ endpointing in place of ``semantic_vad``.+ """++ class Parameters(RealtimeModelBase.Parameters):+ """Tuneables surfaced to the UI."""++ voice: str = Field(default="longanqian", title="Voice")+ turn_detection: Literal["server_vad", "smart_turn", "none"] = Field(+ default="server_vad",+ title="Turn Detection",+ description="``none`` hands endpointing to the caller.",+ )+ vad_threshold: float = Field(default=0.5, ge=-1.0, le=1.0)+ vad_silence_duration_ms: int = Field(default=800, ge=200, le=6000)+ voiceprint_audio_urls: list[str] = Field(+ default_factory=list,+ title="Voiceprint Audio URLs",+ description="Speaker references, used by ``smart_turn`` only.",+ )+ max_history_turns: int = Field(default=20, ge=1, le=50)++ type = "dashscope_audio_realtime"+ supports_text_input = True++ parameters: "DashScopeAudioRealtimeModel.Parameters"++ @classmethod+ def list_models(+ cls,+ custom_yaml_dir: str | None = None,+ ) -> list[RealtimeModelCard]:+ """List the Qwen-Audio cards, which live beside the Omni ones."""+ return super().list_models(+ custom_yaml_dir or str(Path(__file__).parent / "_audio_models"),+ )++ async def push_text(self, text: str) -> None:+ """Send a text user turn and ask for the reply."""+ await self._send(+ {+ "type": "conversation.item.create",+ "item": {+ "type": "message",+ "role": "user",+ "content": [{"type": "input_text", "text": text}],+ },+ },+ )+ await self.request_response()++ def _session_update(+ self,+ instructions: str,+ tools: list[dict] | None,+ ) -> dict:+ """Build the ``session.update`` message. ``voice`` only takes+ effect here, on the first update of the session."""+ p = self.parameters+ turn_detection: dict[str, Any] | None = None+ if p.turn_detection == "server_vad":+ turn_detection = {+ "type": "server_vad",+ "threshold": p.vad_threshold,+ "silence_duration_ms": p.vad_silence_duration_ms,+ }+ elif p.turn_detection == "smart_turn":+ urls = p.voiceprint_audio_urls+ turn_detection = {"type": "smart_turn"}+ if urls:+ turn_detection["voiceprint_audio_urls"] = urls++ session: dict[str, Any] = {+ "instructions": instructions,+ "modalities": ["audio", "text"],+ "voice": p.voice,+ "input_audio_format": "pcm",+ "output_audio_format": "pcm",+ "max_history_turns": p.max_history_turns,+ "turn_detection": turn_detection,+ }+ if tools and self.card.supports_tools:+ session["tools"] = tools+ return {"type": "session.update", "session": session}++ def _parse(self, data: dict) -> me.ModelEvent | None:+ """Translate one server frame, dropping the two families that are+ specific to this API before falling back to the Omni parser."""+ match data.get("type", ""):+ # Speech that ``smart_turn`` judged not to be addressed at the+ # model, and the progress of voiceprint registration.+ case "conversation.item.ambient_audio_transcription.delta":+ logger.debug(+ "DashScopeAudioRealtimeModel: ambient speech %r",+ data.get("delta", ""),+ )+ return None++ case (+ "voiceprint_audio_list.in_progress"+ | "voiceprint_audio_list.completed"+ | "voiceprint_audio_list.failed"+ ) as kind:+ logger.debug("DashScopeAudioRealtimeModel: %s", kind)+ return None++ case _:+ return super()._parse(data)diff --git a/src/agentscope/realtime/_dashscope/_audio_models/qwen-audio-3.0-realtime-flash.yaml b/src/agentscope/realtime/_dashscope/_audio_models/qwen-audio-3.0-realtime-flash.yamlnew file mode 100644index 000000000..5582971fd--- /dev/null+++ b/src/agentscope/realtime/_dashscope/_audio_models/qwen-audio-3.0-realtime-flash.yaml@@ -0,0 +1,32 @@+type: realtime_model+name: qwen-audio-3.0-realtime-flash+label: Qwen-Audio-3.0-Realtime-Flash+status: active++input_types:+ - audio/pcm+ - text/plain+output_types:+ - audio/pcm+ - text/plain++input_sample_rate: 16000+output_sample_rate: 24000++supports_tools: true+supports_session_resumption: false++# Turns and cumulative audio seconds both apply; the provider discards+# the oldest turn once either is exceeded.+max_audio_turns: 50+max_audio_duration_s: 300++parameter_overrides:+ voice:+ default: "longanqian"+ enum:+ - "longanqian"+ - "longanlingxin"+ - "longanlingxi"+ - "longanxiaoxin"+ - "longanlufeng"diff --git a/src/agentscope/realtime/_dashscope/_audio_models/qwen-audio-3.0-realtime-plus.yaml b/src/agentscope/realtime/_dashscope/_audio_models/qwen-audio-3.0-realtime-plus.yamlnew file mode 100644index 000000000..c02378a69--- /dev/null+++ b/src/agentscope/realtime/_dashscope/_audio_models/qwen-audio-3.0-realtime-plus.yaml@@ -0,0 +1,32 @@+type: realtime_model+name: qwen-audio-3.0-realtime-plus+label: Qwen-Audio-3.0-Realtime-Plus+status: active++input_types:+ - audio/pcm+ - text/plain+output_types:+ - audio/pcm+ - text/plain++input_sample_rate: 16000+output_sample_rate: 24000++supports_tools: true+supports_session_resumption: false++# Turns and cumulative audio seconds both apply; the provider discards+# the oldest turn once either is exceeded.+max_audio_turns: 50+max_audio_duration_s: 300++parameter_overrides:+ voice:+ default: "longanqian"+ enum:+ - "longanqian"+ - "longanlingxin"+ - "longanlingxi"+ - "longanxiaoxin"+ - "longanlufeng"diff --git a/src/agentscope/realtime/_dashscope/_model.py b/src/agentscope/realtime/_dashscope/_model.pynew file mode 100644index 000000000..9afd6461f--- /dev/null+++ b/src/agentscope/realtime/_dashscope/_model.py@@ -0,0 +1,387 @@+# -*- coding: utf-8 -*-+"""The DashScope realtime model."""+import asyncio+import base64+import json+from typing import Any, AsyncIterator, Literal++from pydantic import Field++from .. import _events as me+from .._base import (+ ModelDisconnectedError,+ RealtimeModelBase,+ TruncationSupport,+)+from .._model_card import RealtimeModelCard+from ..._logging import logger+from ...credential import DashScopeCredential+from ...message import Msg, TextBlock, ToolCallBlock, ToolResultBlock++_REALTIME_URL = "wss://dashscope.aliyuncs.com/api-ws/v1/realtime"+++class DashScopeRealtimeModel(RealtimeModelBase):+ """The Qwen-Omni realtime API over WebSocket.++ The protocol is OpenAI-shaped (``session.update``,+ ``input_audio_buffer.append``, ``response.create``) but narrower: no+ text turns, no ``conversation.item.truncate``, and history that cannot+ be seeded on connect.+ """++ class Parameters(RealtimeModelBase.Parameters):+ """Tuneables surfaced to the UI."""++ voice: str = Field(default="Cherry", title="Voice")+ turn_detection: Literal["server_vad", "semantic_vad", "none"] = Field(+ default="server_vad",+ title="Turn Detection",+ description="``none`` hands endpointing to the caller.",+ )+ vad_threshold: float = Field(default=0.5, ge=0.0, le=1.0)+ vad_silence_duration_ms: int = Field(default=800, ge=0)+ input_audio_transcription: bool = Field(+ default=True,+ title="Transcribe Input",+ description="Whether to transcribe the user's speech.",+ )++ type = "dashscope_omni_realtime"+ truncation = TruncationSupport.NONE+ supports_text_input = False++ def __init__(+ self,+ model_name: str,+ credential: DashScopeCredential,+ parameters: "DashScopeRealtimeModel.Parameters | None" = None,+ model_card: RealtimeModelCard | None = None,+ ) -> None:+ """Initialize the DashScope realtime model.++ Args:+ model_name (`str`):+ The model name, e.g. ``"qwen3-omni-flash-realtime"``.+ credential (`DashScopeCredential`):+ The DashScope credential.+ parameters (`DashScopeRealtimeModel.Parameters | None`, optional):+ Provider parameters; defaults are used if omitted.+ model_card (`RealtimeModelCard | None`, optional):+ The model card, looked up by name if omitted.+ """+ super().__init__(model_name, credential, parameters, model_card)+ self.parameters: DashScopeRealtimeModel.Parameters+ self._ws: Any = None+ self._reader: asyncio.Task | None = None+ self._queue: asyncio.Queue[me.ModelEvent | None] = asyncio.Queue()+ self._item_id = ""+ self._tool_args: dict[str, str] = {}+ self._tool_names: dict[str, str] = {}++ @property+ def turn_detection_enabled(self) -> bool:+ """Whether the provider decides turn boundaries."""+ return self.parameters.turn_detection != "none"++ # ------------------------------------------------------------------+ # Lifecycle+ # ------------------------------------------------------------------++ async def connect(+ self,+ context: list[Msg],+ instructions: str,+ tools: list[dict] | None = None,+ **kwargs: Any,+ ) -> None:+ """Open the WebSocket and send the session config.++ DashScope offers no way to replay *context* into a new session, so+ a non-empty one is logged and dropped rather than silently lost.+ """+ import websockets++ if context:+ logger.warning(+ "DashScopeRealtimeModel: %d prior messages cannot be "+ "seeded into the session; the model starts fresh.",+ len(context),+ )++ if kwargs.get("turn_detection_disabled"):+ self.parameters = self.parameters.model_copy(+ update={"turn_detection": "none"},+ )++ credential: DashScopeCredential = self.credential # type: ignore+ self._ws = await websockets.connect(+ f"{_REALTIME_URL}?model={self.model_name}",+ additional_headers={+ "Authorization": f"Bearer "+ f"{credential.api_key.get_secret_value()}",+ "X-DashScope-DataInspection": "disable",+ },+ )+ self._reader = asyncio.create_task(self._read(), name="dashscope-rt")+ await self._send(self._session_update(instructions, tools))++ async def close(self) -> None:+ """Stop reading and close the WebSocket."""+ if self._reader is not None:+ self._reader.cancel()+ await asyncio.gather(self._reader, return_exceptions=True)+ self._reader = None+ if self._ws is not None:+ await self._ws.close()+ self._ws = None++ async def events(self) -> AsyncIterator[me.ModelEvent]:+ """Yield events until the session ends."""+ while True:+ event = await self._queue.get()+ if event is None:+ return+ yield event++ # ------------------------------------------------------------------+ # Input+ # ------------------------------------------------------------------++ async def push_audio(self, pcm: bytes) -> None:+ """Append PCM16 mono audio at 16 kHz to the input buffer."""+ await self._send(+ {+ "type": "input_audio_buffer.append",+ "audio": base64.b64encode(pcm).decode("ascii"),+ },+ )++ async def push_text(self, text: str) -> None:+ """Unsupported: the API accepts audio and images only."""+ raise NotImplementedError(+ "DashScope realtime accepts no text input.",+ )++ async def push_tool_result(self, block: ToolResultBlock) -> None:+ """Send a ``function_call_output`` item."""+ output = (+ block.output+ if isinstance(block.output, str)+ else "".join(+ _.text for _ in block.output if isinstance(_, TextBlock)+ )+ )+ await self._send(+ {+ "type": "conversation.item.create",+ "item": {+ "type": "function_call_output",+ "call_id": block.id,+ "output": output,+ },+ },+ )++ async def commit_turn(self) -> None:+ """Commit the input buffer as one user turn."""+ await self._send({"type": "input_audio_buffer.commit"})++ # ------------------------------------------------------------------+ # Response control+ # ------------------------------------------------------------------++ async def request_response(self) -> None:+ """Ask for a reply, cancelling one already in flight first."""+ if self._item_id:+ await self.cancel_response()+ await self._send(+ {+ "type": "response.create",+ "response": {"modalities": ["text", "audio"]},+ },+ )++ async def cancel_response(self) -> None:+ """Cancel the reply in flight, if any."""+ if self._item_id:+ await self._send({"type": "response.cancel"})++ async def truncate(+ self,+ item_id: str,+ played_ms: int,+ played_text: str,+ ) -> None:+ """No-op: the protocol has no truncate frame."""++ # ------------------------------------------------------------------+ # Wire+ # ------------------------------------------------------------------++ def _session_update(+ self,+ instructions: str,+ tools: list[dict] | None,+ ) -> dict:+ """Build the ``session.update`` message."""+ p = self.parameters+ session: dict[str, Any] = {+ "instructions": instructions,+ "modalities": ["audio", "text"],+ "voice": p.voice,+ "input_audio_format": f"pcm{self.input_sample_rate // 1000}",+ "output_audio_format": f"pcm{self.output_sample_rate // 1000}",+ "turn_detection": None+ if p.turn_detection == "none"+ else {+ "type": p.turn_detection,+ "threshold": p.vad_threshold,+ "silence_duration_ms": p.vad_silence_duration_ms,+ },+ }+ if p.input_audio_transcription:+ session["input_audio_transcription"] = {+ "model": "gummy-realtime-v1",+ }+ if tools and self.card.supports_tools:+ session["tools"] = tools+ return {"type": "session.update", "session": session}++ async def _send(self, payload: dict) -> None:+ """Send one JSON frame.++ Raises:+ `ModelDisconnectedError`: If the provider has closed the+ session, e.g. after its idle timeout.+ """+ from websockets.exceptions import ConnectionClosed++ if self._ws is None:+ raise ModelDisconnectedError("Not connected.")+ try:+ await self._ws.send(json.dumps(payload, ensure_ascii=False))+ except ConnectionClosed as exc:+ self._ws = None+ raise ModelDisconnectedError(str(exc.rcvd or exc)) from exc++ async def _read(self) -> None:+ """Drain the WebSocket into the event queue until it closes."""+ try:+ async for raw in self._ws:+ if isinstance(raw, bytes):+ raw = raw.decode("utf-8")+ try:+ event = self._parse(json.loads(raw))+ except Exception: # noqa: BLE001+ logger.exception("DashScopeRealtimeModel: bad frame")+ continue+ if event is not None:+ self._queue.put_nowait(event)+ except Exception as exc: # noqa: BLE001+ logger.error("DashScopeRealtimeModel: connection lost: %s", exc)+ finally:+ self._queue.put_nowait(me.SessionEndedEvent(reason="closed"))+ self._queue.put_nowait(None)++ # pylint: disable=too-many-return-statements+ def _parse(self, data: dict) -> me.ModelEvent | None:+ """Translate one server frame into a model event."""+ kind = data.get("type", "")+ item_id = data.get("item_id", "")++ match kind:+ case "response.created":+ self._item_id = data.get("response", {}).get("id", "")+ return me.ResponseCreatedEvent(item_id=self._item_id)++ case "response.done":+ response = data.get("response", {})+ usage = response.get("usage") or {}+ event = me.ResponseDoneEvent(+ item_id=response.get("id") or self._item_id,+ input_tokens=usage.get("input_tokens", 0),+ output_tokens=usage.get("output_tokens", 0),+ )+ self._item_id = ""+ return event++ case "response.audio.delta":+ delta = data.get("delta")+ if not delta:+ return None+ return me.AudioDeltaEvent(+ item_id=self._item_id,+ pcm=base64.b64decode(delta),+ sample_rate=self.output_sample_rate,+ )++ case "response.audio_transcript.delta":+ delta = data.get("delta")+ if not delta:+ return None+ return me.TranscriptDeltaEvent(+ item_id=self._item_id,+ delta=delta,+ )++ case "conversation.item.input_audio_transcription.completed":+ return me.InputTranscriptionEvent(+ item_id=item_id,+ text=data.get("transcript", ""),+ )++ case "input_audio_buffer.speech_started":+ return me.SpeechStartedEvent(+ item_id=item_id,+ at_ms=data.get("audio_start_ms", 0),+ )++ case "input_audio_buffer.speech_stopped":+ return me.SpeechEndedEvent(+ item_id=item_id,+ at_ms=data.get("audio_end_ms", 0),+ )++ case "response.output_item.added" | "response.output_item.done":+ item = data.get("item", {})+ if item.get("type") == "function_call" and item.get("name"):+ self._tool_names[item["call_id"]] = item["name"]+ return None++ case "response.function_call_arguments.delta":+ call_id = data.get("call_id", "")+ self._tool_args[call_id] = self._tool_args.get(+ call_id,+ "",+ ) + data.get("delta", "")+ if data.get("name"):+ self._tool_names[call_id] = data["name"]+ return None++ case "response.function_call_arguments.done":+ # The done frame is authoritative; accumulated deltas are+ # only the fallback for a frame that omits ``arguments``.+ call_id = data.get("call_id", "")+ accumulated = self._tool_args.pop(call_id, "")+ arguments = data.get("arguments") or accumulated+ name = data.get("name") or self._tool_names.pop(call_id, "")+ return me.ToolCallEvent(+ item_id=self._item_id,+ tool_call=ToolCallBlock(+ id=call_id,+ name=name,+ input=arguments,+ ),+ )++ case "error":+ err = data.get("error", {})+ return me.ModelErrorEvent(+ code=err.get("code", ""),+ message=err.get("message", ""),+ )++ case _:+ logger.debug("DashScopeRealtimeModel: ignoring %s", kind)+ return Nonediff --git a/src/agentscope/realtime/_dashscope/_models/qwen-omni-turbo-realtime.yaml b/src/agentscope/realtime/_dashscope/_models/qwen-omni-turbo-realtime.yamlnew file mode 100644index 000000000..3aad9e615--- /dev/null+++ b/src/agentscope/realtime/_dashscope/_models/qwen-omni-turbo-realtime.yaml@@ -0,0 +1,27 @@+type: realtime_model+name: qwen-omni-turbo-realtime+label: Qwen-Omni-Turbo Realtime+status: active++input_types:+ - audio/pcm+ - image/jpeg+output_types:+ - audio/pcm+ - text/plain++input_sample_rate: 16000+output_sample_rate: 24000++# The model page lists "Function Calling: 不支持".+supports_tools: false+supports_session_resumption: false++# The model page states the context in tokens; the realtime guide's+# turn/duration table does not cover this model.+max_context_tokens: 32768+max_session_duration_s: 7200++parameter_overrides:+ voice:+ enum: ["Cherry", "Serena", "Ethan", "Chelsie"]diff --git a/src/agentscope/realtime/_dashscope/_models/qwen3-omni-flash-realtime.yaml b/src/agentscope/realtime/_dashscope/_models/qwen3-omni-flash-realtime.yamlnew file mode 100644index 000000000..8e3296584--- /dev/null+++ b/src/agentscope/realtime/_dashscope/_models/qwen3-omni-flash-realtime.yaml@@ -0,0 +1,27 @@+type: realtime_model+name: qwen3-omni-flash-realtime+label: Qwen3-Omni-Flash Realtime+status: active++input_types:+ - audio/pcm+ - image/jpeg+output_types:+ - audio/pcm+ - text/plain++input_sample_rate: 16000+output_sample_rate: 24000++# Function calling is only offered on the qwen3.5 series.+supports_tools: false+supports_session_resumption: false++# History is bounded by dialog turns, not tokens; the provider discards+# the oldest turn once exceeded.+max_audio_turns: 8+max_session_duration_s: 7200++parameter_overrides:+ voice:+ enum: ["Cherry", "Serena", "Ethan", "Chelsie"]diff --git a/src/agentscope/realtime/_dashscope/_models/qwen3.5-omni-flash-realtime.yaml b/src/agentscope/realtime/_dashscope/_models/qwen3.5-omni-flash-realtime.yamlnew file mode 100644index 000000000..617b97472--- /dev/null+++ b/src/agentscope/realtime/_dashscope/_models/qwen3.5-omni-flash-realtime.yaml@@ -0,0 +1,28 @@+type: realtime_model+name: qwen3.5-omni-flash-realtime+label: Qwen3.5-Omni-Flash Realtime+status: active++input_types:+ - audio/pcm+ - image/jpeg+output_types:+ - audio/pcm+ - text/plain++input_sample_rate: 16000+output_sample_rate: 24000++supports_tools: true+supports_session_resumption: false++# Turns and cumulative audio seconds both apply; the provider discards+# the oldest turn once either is exceeded.+max_audio_turns: 80+max_audio_duration_s: 480+max_session_duration_s: 7200++parameter_overrides:+ voice:+ default: "Tina"+ enum: ["Cherry", "Serena", "Ethan", "Chelsie", "Tina"]diff --git a/src/agentscope/realtime/_dashscope/_models/qwen3.5-omni-plus-realtime.yaml b/src/agentscope/realtime/_dashscope/_models/qwen3.5-omni-plus-realtime.yamlnew file mode 100644index 000000000..8235b8999--- /dev/null+++ b/src/agentscope/realtime/_dashscope/_models/qwen3.5-omni-plus-realtime.yaml@@ -0,0 +1,28 @@+type: realtime_model+name: qwen3.5-omni-plus-realtime+label: Qwen3.5-Omni-Plus Realtime+status: active++input_types:+ - audio/pcm+ - image/jpeg+output_types:+ - audio/pcm+ - text/plain++input_sample_rate: 16000+output_sample_rate: 24000++supports_tools: true+supports_session_resumption: false++# Turns and cumulative audio seconds both apply; the provider discards+# the oldest turn once either is exceeded.+max_audio_turns: 100+max_audio_duration_s: 600+max_session_duration_s: 7200++parameter_overrides:+ voice:+ default: "Tina"+ enum: ["Cherry", "Serena", "Ethan", "Chelsie", "Tina"]diff --git a/src/agentscope/realtime/_events.py b/src/agentscope/realtime/_events.pynew file mode 100644index 000000000..dd22e2862--- /dev/null+++ b/src/agentscope/realtime/_events.py@@ -0,0 +1,84 @@+# -*- coding: utf-8 -*-+"""Events emitted by a realtime model.++These are internal to :mod:`agentscope.realtime`: adapters normalise+vendor frames into them, and :class:`RealtimeAgent` translates them into+the public :class:`~agentscope.event.AgentEvent` stream.+"""+from pydantic import BaseModel, Field++from ..message import ToolCallBlock+++class ModelEvent(BaseModel):+ """Common base class, for type hinting only."""+++class SessionEndedEvent(ModelEvent):+ """The provider session is gone; no further events will arrive."""++ reason: str = ""+++class SpeechStartedEvent(ModelEvent):+ """The provider's own VAD detected the user starting to speak."""++ item_id: str = ""+ at_ms: int = 0+++class SpeechEndedEvent(ModelEvent):+ """The provider's own VAD detected the user stopping."""++ item_id: str = ""+ at_ms: int = 0+++class InputTranscriptionEvent(ModelEvent):+ """The settled transcript of one user turn."""++ item_id: str = ""+ text: str+++class ResponseCreatedEvent(ModelEvent):+ """The provider started producing a reply."""++ item_id: str+++class AudioDeltaEvent(ModelEvent):+ """A chunk of assistant speech."""++ item_id: str+ pcm: bytes+ sample_rate: int+++class TranscriptDeltaEvent(ModelEvent):+ """A chunk of the assistant's spoken text."""++ item_id: str+ delta: str+++class ToolCallEvent(ModelEvent):+ """A complete tool call requested by the provider."""++ item_id: str+ tool_call: ToolCallBlock+++class ResponseDoneEvent(ModelEvent):+ """The provider finished producing a reply."""++ item_id: str+ input_tokens: int = Field(default=0, ge=0)+ output_tokens: int = Field(default=0, ge=0)+++class ModelErrorEvent(ModelEvent):+ """An error reported by the provider."""++ code: str = ""+ message: str = ""diff --git a/src/agentscope/realtime/_model_card.py b/src/agentscope/realtime/_model_card.pynew file mode 100644index 000000000..77eeed6fa--- /dev/null+++ b/src/agentscope/realtime/_model_card.py@@ -0,0 +1,136 @@+# -*- coding: utf-8 -*-+"""The realtime model card."""+import copy+from datetime import datetime+from pathlib import Path+from typing import Literal, Self, Type++import yaml+from pydantic import BaseModel, Field++from .._logging import logger+++class RealtimeModelCard(BaseModel):+ """Describes one realtime model. Everything here varies with the model+ name; what the adapter implements lives on+ :class:`RealtimeModelBase`."""++ type: Literal["realtime_model"] = "realtime_model"++ model_type: str = Field(+ default="",+ description="The adapter class this model belongs to, e.g. "+ "``dashscope_omni_realtime``.",+ )++ name: str = Field(description="The model identifier.")+ label: str = Field(description="Human-readable label for the UI.")++ status: Literal["active", "deprecated", "sunset"] = Field(+ default="active",+ description="Lifecycle status of the model.",+ )+ deprecated_at: datetime | None = Field(+ default=None,+ description="When the model was deprecated, if applicable.",+ )++ input_types: list[str] = Field(+ default_factory=lambda: ["audio/pcm"],+ description="Accepted input media types.",+ )+ output_types: list[str] = Field(+ default_factory=lambda: ["audio/pcm", "text/plain"],+ description="Media types the model produces.",+ )++ input_sample_rate: int = Field(+ default=16000,+ gt=0,+ description="Advertised input PCM sample rate in Hz.",+ )+ output_sample_rate: int = Field(+ default=24000,+ gt=0,+ description="Advertised output PCM sample rate in Hz.",+ )++ supports_tools: bool = Field(+ default=False,+ description="Whether this model accepts tool schemas.",+ )+ supports_session_resumption: bool = Field(+ default=False,+ description="Whether a previous session can be resumed by handle.",+ )++ # Context limits. Providers use different units and several can apply+ # at once, so all are optional; ``None`` means unlimited or unknown.+ max_context_tokens: int | None = Field(default=None, gt=0)+ max_audio_turns: int | None = Field(default=None, gt=0)+ max_audio_duration_s: int | None = Field(default=None, gt=0)+ max_session_duration_s: int | None = Field(default=None, gt=0)++ parameter_schema: dict = Field(default_factory=dict)+ """JSON Schema of the tuneable parameters, surfaced to the UI."""++ parameter_overrides: dict[str, dict] = Field(default_factory=dict)+ """Per-parameter narrowing merged into :attr:`parameter_schema`."""++ @classmethod+ def from_yaml(+ cls,+ yaml_path: str | Path,+ parameter_class: Type[BaseModel],+ ) -> Self:+ """Load a card from YAML, seeding the parameter schema from+ *parameter_class*.++ Args:+ yaml_path (`str | Path`):+ Path to the YAML file.+ parameter_class (`Type[BaseModel]`):+ The adapter's ``Parameters`` class, whose JSON schema seeds+ :attr:`parameter_schema`.++ Returns:+ `RealtimeModelCard`: The loaded card.+ """+ with open(yaml_path, "r", encoding="utf-8") as fh:+ config = yaml.safe_load(fh)++ properties = copy.deepcopy(+ parameter_class.model_json_schema().get("properties", {}),+ )+ overrides = config.pop("parameter_overrides", {}) or {}+ for param, override in overrides.items():+ if override is None or override.get("hidden"):+ properties.pop(param, None)+ elif param in properties:+ properties[param] = {**properties[param], **override}++ return cls(+ **config,+ parameter_schema={"type": "object", "properties": properties},+ parameter_overrides=overrides,+ )++ @classmethod+ def list_from_directory(+ cls,+ yaml_dir: str | Path,+ parameter_class: Type[BaseModel],+ ) -> list["RealtimeModelCard"]:+ """Load every ``*.yaml`` card in *yaml_dir*, skipping broken ones."""+ cards: list["RealtimeModelCard"] = []+ for yaml_file in sorted(Path(yaml_dir).glob("*.yaml")):+ try:+ cards.append(cls.from_yaml(yaml_file, parameter_class))+ except Exception as exc: # noqa: BLE001+ logger.warning(+ "Failed to load realtime model card %s: %s",+ yaml_file,+ exc,+ )+ return cardsdiff --git a/src/agentscope/realtime/_playout.py b/src/agentscope/realtime/_playout.pynew file mode 100644index 000000000..f7753b124--- /dev/null+++ b/src/agentscope/realtime/_playout.py@@ -0,0 +1,18 @@+# -*- coding: utf-8 -*-+"""Playback position of assistant audio."""+from pydantic import BaseModel, Field+++class PlayoutPosition(BaseModel):+ """How much assistant audio actually left the speaker — the only sound+ basis for truncating a barge-in."""++ item_id: str+ """The assistant item this position refers to."""++ played_ms: int = Field(default=0, ge=0)+ """Milliseconds of audio actually played out."""++ first_played_at: float | None = None+ """:func:`time.monotonic` when the first sample of this item reached+ the speaker — the only timestamp that reflects what the user hears."""diff --git a/src/agentscope/realtime/_transport/__init__.py b/src/agentscope/realtime/_transport/__init__.pynew file mode 100644index 000000000..634f7befb--- /dev/null+++ b/src/agentscope/realtime/_transport/__init__.py@@ -0,0 +1,19 @@+# -*- coding: utf-8 -*-+"""Transports connecting a realtime agent to audio endpoints."""+from ._local import LocalAudioTransport+from ._base import (+ AudioFrame,+ ControlFrame,+ ControlFrameType,+ TransportBase,+ TransportFrame,+)++__all__ = [+ "LocalAudioTransport",+ "AudioFrame",+ "ControlFrame",+ "ControlFrameType",+ "TransportBase",+ "TransportFrame",+]diff --git a/src/agentscope/realtime/_transport/_base.py b/src/agentscope/realtime/_transport/_base.pynew file mode 100644index 000000000..a430daa69--- /dev/null+++ b/src/agentscope/realtime/_transport/_base.py@@ -0,0 +1,124 @@+# -*- coding: utf-8 -*-+"""The transport base class."""+from abc import ABC, abstractmethod+from enum import StrEnum+from typing import AsyncIterator++from pydantic import BaseModel++from .._playout import PlayoutPosition+++class AudioFrame(BaseModel):+ """A chunk of upstream user audio."""++ pcm: bytes+ """PCM16 mono at the transport's :attr:`input_sample_rate`."""+++class ControlFrameType(StrEnum):+ """The non-audio things a remote client can say.++ Each one is the wire form of a public :class:`RealtimeAgent` method,+ so that a browser can do what a Python caller does directly. Traffic+ a transport handles for itself — handshakes, playback reports — never+ reaches the agent and has no member here.+ """++ TEXT = "text"+ """A typed user turn."""++ USER_CONFIRM = "user_confirm"+ """The outcome of a tool permission prompt."""++ INTERRUPT = "interrupt"+ """The user pressed stop, as opposed to speaking over the reply."""++ EXTERNAL_EXECUTION_RESULT = "external_execution_result"+ """The result of a tool the client ran itself."""+++class ControlFrame(BaseModel):+ """One upstream control event."""++ type: ControlFrameType+ data: dict = {}+++TransportFrame = AudioFrame | ControlFrame+++class TransportBase(ABC):+ """Connects a :class:`RealtimeAgent` to where audio comes from and+ goes to.++ A transport is a browser connection, a local sound card, or a queue+ the caller drives — the agent cannot tell the difference. It is owned+ by whoever created it, not by the agent; :meth:`RealtimeAgent.run`+ only borrows it.+ """++ input_sample_rate: int+ """The rate of the PCM delivered by :meth:`incoming`."""++ output_sample_rate: int+ """The rate :meth:`send_audio` expects."""++ async def __aenter__(self) -> "TransportBase":+ """Start on entry; the transport's owner is whoever entered."""+ await self.start()+ return self++ async def __aexit__(self, *exc: object) -> None:+ """Close on exit."""+ await self.close()++ @abstractmethod+ async def start(self) -> None:+ """Open the transport and begin capturing."""++ @abstractmethod+ async def close(self) -> None:+ """Release every resource held by the transport."""++ @abstractmethod+ async def incoming(self) -> AsyncIterator[TransportFrame]:+ """Yield upstream frames until the transport closes."""+ raise NotImplementedError(f"{type(self).__name__}.incoming")+ yield # pylint: disable=unreachable++ @abstractmethod+ async def send_audio(self, pcm: bytes, item_id: str) -> None:+ """Queue assistant audio for playback.++ Args:+ pcm (`bytes`):+ PCM16 mono at :attr:`output_sample_rate`.+ item_id (`str`):+ The assistant item this audio belongs to, so that playback+ position can be attributed to it.+ """++ @abstractmethod+ async def send_event(self, event: dict) -> None:+ """Send one control event downstream."""++ @abstractmethod+ async def clear_audio(self) -> PlayoutPosition:+ """Cut playback short and report how much was actually heard.++ Fades out briefly to avoid an audible click, then discards queued+ and in-flight audio.++ Clearing and reporting are one call on purpose: the returned+ position is what the caller must feed to+ :meth:`VoiceBackendBase.truncate`, so the two cannot drift apart+ and the truncation cannot be forgotten.++ Returns:+ `PlayoutPosition`: What the user heard before the cut.+ """++ @abstractmethod+ def playout(self) -> PlayoutPosition:+ """The current playback position, without interrupting."""diff --git a/src/agentscope/realtime/_transport/_local.py b/src/agentscope/realtime/_transport/_local.pynew file mode 100644index 000000000..52339232a--- /dev/null+++ b/src/agentscope/realtime/_transport/_local.py@@ -0,0 +1,202 @@+# -*- coding: utf-8 -*-+"""The local sound-card transport."""+import asyncio+import threading+import time+from typing import Any, AsyncIterator++import numpy as np++from ._base import AudioFrame, TransportBase, TransportFrame+from .._playout import PlayoutPosition+from ..._logging import logger+++class LocalAudioTransport(TransportBase):+ """Microphone in, speaker out, via ``sounddevice``.++ Audio callbacks run on PortAudio's thread, not the event loop: captured+ chunks are handed to the loop with ``call_soon_threadsafe`` and+ playback pulls from a lock-guarded buffer. Playout accounting lives in+ that callback, the same place a browser's AudioWorklet does it.+ """++ def __init__(+ self,+ input_sample_rate: int = 16000,+ output_sample_rate: int = 24000,+ input_device: int | str | None = None,+ output_device: int | str | None = None,+ chunk_ms: int = 100,+ fade_ms: int = 30,+ ) -> None:+ """Initialize the transport.++ Args:+ input_sample_rate (`int`, defaults to `16000`):+ Capture rate; should match the model's input rate.+ output_sample_rate (`int`, defaults to `24000`):+ Playback rate; should match the model's output rate.+ input_device (`int | str | None`, optional):+ ``sounddevice`` input device, default if ``None``.+ output_device (`int | str | None`, optional):+ ``sounddevice`` output device, default if ``None``.+ chunk_ms (`int`, defaults to `100`):+ Capture chunk length handed to :meth:`incoming`.+ fade_ms (`int`, defaults to `30`):+ Fade-out applied by :meth:`clear_audio` to avoid a click.+ """+ self.input_sample_rate = input_sample_rate+ self.output_sample_rate = output_sample_rate+ self._input_device = input_device+ self._output_device = output_device+ self._chunk_frames = input_sample_rate * chunk_ms // 1000+ self._fade_ms = fade_ms+ # Ten seconds of capture; beyond that the oldest chunk is dropped.+ self._max_queued = max(1, 10_000 // chunk_ms)++ self._loop: asyncio.AbstractEventLoop | None = None+ self._in_queue: asyncio.Queue[AudioFrame | None] = asyncio.Queue()+ self._in_stream: Any = None+ self._out_stream: Any = None++ # Playback state, guarded by ``_lock`` (shared with the audio thread).+ self._lock = threading.Lock()+ self._pending = bytearray()+ self._item_id = ""+ self._played_samples = 0+ self._first_played_at: float | None = None++ # ------------------------------------------------------------------+ # Lifecycle+ # ------------------------------------------------------------------++ async def start(self) -> None:+ """Open both streams."""+ import sounddevice as sd++ self._loop = asyncio.get_running_loop()+ self._in_stream = sd.InputStream(+ samplerate=self.input_sample_rate,+ channels=1,+ dtype="int16",+ blocksize=self._chunk_frames,+ device=self._input_device,+ callback=self._on_input,+ )+ self._out_stream = sd.OutputStream(+ samplerate=self.output_sample_rate,+ channels=1,+ dtype="int16",+ device=self._output_device,+ callback=self._on_output,+ )+ self._in_stream.start()+ self._out_stream.start()++ async def close(self) -> None:+ """Stop both streams and end :meth:`incoming`."""+ for stream in (self._in_stream, self._out_stream):+ if stream is not None:+ stream.stop()+ stream.close()+ self._in_stream = self._out_stream = None+ self._in_queue.put_nowait(None)++ # ------------------------------------------------------------------+ # Upstream+ # ------------------------------------------------------------------++ async def incoming(self) -> AsyncIterator[TransportFrame]:+ """Yield captured chunks until :meth:`close`."""+ while True:+ frame = await self._in_queue.get()+ if frame is None:+ return+ yield frame++ def _on_input(+ self,+ indata: Any,+ _frames: int,+ _t: Any,+ status: Any,+ ) -> None:+ """Audio thread: hand one captured chunk to the loop."""+ if status:+ logger.debug("LocalAudioTransport: input %s", status)+ if self._loop is None:+ return+ frame = AudioFrame(pcm=bytes(indata))+ self._loop.call_soon_threadsafe(self._enqueue, frame)++ def _enqueue(self, frame: AudioFrame) -> None:+ """Loop thread: queue a chunk, dropping the oldest when nobody drains+ fast enough, so a stalled consumer never replays stale audio."""+ if self._in_queue.qsize() >= self._max_queued:+ self._in_queue.get_nowait()+ self._in_queue.put_nowait(frame)++ # ------------------------------------------------------------------+ # Downstream+ # ------------------------------------------------------------------++ async def send_audio(self, pcm: bytes, item_id: str) -> None:+ """Queue assistant audio. A new item resets the playout counters."""+ with self._lock:+ if item_id != self._item_id:+ self._item_id = item_id+ self._pending.clear()+ self._played_samples = 0+ self._first_played_at = None+ self._pending.extend(pcm)++ async def send_event(self, event: dict) -> None:+ """No peer to send to; events are consumed via the agent."""++ def _on_output(+ self,+ outdata: Any,+ frames: int,+ _t: Any,+ status: Any,+ ) -> None:+ """Audio thread: fill one block from the buffer, count what played."""+ if status:+ logger.debug("LocalAudioTransport: output %s", status)+ wanted = frames * 2+ with self._lock:+ chunk = bytes(self._pending[:wanted])+ del self._pending[:wanted]+ if chunk:+ self._played_samples += len(chunk) // 2+ if self._first_played_at is None:+ self._first_played_at = time.monotonic()+ n = len(chunk) // 2 # bytes -> int16 samples+ outdata[:n] = np.frombuffer(chunk, dtype=np.int16).reshape(-1, 1)+ outdata[n:] = 0++ async def clear_audio(self) -> PlayoutPosition:+ """Fade the head of the queue out, drop the rest, report the cut."""+ with self._lock:+ position = self._position()+ fade_bytes = self.output_sample_rate * self._fade_ms // 1000 * 2+ head = np.frombuffer(bytes(self._pending[:fade_bytes]), np.int16)+ self._pending.clear()+ if len(head):+ ramp = np.linspace(1.0, 0.0, len(head), dtype=np.float32)+ self._pending.extend((head * ramp).astype(np.int16).tobytes())+ return position++ def playout(self) -> PlayoutPosition:+ """The current playback position."""+ with self._lock:+ return self._position()++ def _position(self) -> PlayoutPosition:+ """Build a position; caller holds the lock."""+ return PlayoutPosition(+ item_id=self._item_id,+ played_ms=self._played_samples * 1000 // self.output_sample_rate,+ first_played_at=self._first_played_at,+ )diff --git a/src/agentscope/realtime/_vad.py b/src/agentscope/realtime/_vad.pynew file mode 100644index 000000000..bbbeba615--- /dev/null+++ b/src/agentscope/realtime/_vad.py@@ -0,0 +1,48 @@+# -*- coding: utf-8 -*-+"""Voice activity detection."""+from abc import ABC, abstractmethod+from enum import StrEnum+++class SpeechTransition(StrEnum):+ """A transition in voice activity, as detected by a VAD. Not an event+ class; the model-side ``SpeechStartedEvent`` is what a provider reports."""++ STARTED = "started"+ ENDED = "ended"+++class VADBase(ABC):+ """Voice activity detection over a stream of PCM16 mono audio.++ Implementations are stateful: ``push`` is called with consecutive+ chunks and returns a transition only on the frame where it happens.+ """++ sample_rate: int+ """The PCM sample rate the implementation expects."""++ @abstractmethod+ def push(self, pcm: bytes) -> SpeechTransition | None:+ """Feed one chunk of audio and report a transition, if any.++ This is an edge detector, not a level query: it returns a value+ only on the chunk where speech starts or stops. Chunk sizes are+ whatever the transport produces, while models want a fixed frame+ (Silero v5 wants 512 samples), so implementations buffer and slice+ internally and report at most one transition per call.++ Args:+ pcm (`bytes`):+ PCM16 mono audio at :attr:`sample_rate`.++ Returns:+ `SpeechTransition | None`:+ The transition detected on this chunk, or ``None``.+ """++ @abstractmethod+ def reset(self) -> None:+ """Drop internal state. Called whenever the audio stream has a+ gap — a reconnect, or the microphone being unmuted — since the+ silence counters carried across it would be stale."""diff --git a/tests/realtime_agent_test.py b/tests/realtime_agent_test.pynew file mode 100644index 000000000..b01b6806f--- /dev/null+++ b/tests/realtime_agent_test.py@@ -0,0 +1,987 @@+# -*- coding: utf-8 -*-+"""Unit tests for RealtimeAgent, driven by a scripted model and a fake+transport — no network, no sound card."""+# pylint: disable=protected-access, unused-argument+import asyncio+from typing import Any, AsyncIterator+from unittest.async_case import IsolatedAsyncioTestCase+from utils import AnyString++from agentscope.agent import RealtimeAgent, TurnAggregator+from agentscope.credential import DashScopeCredential+from agentscope.event import (+ ReplyEndEvent,+ UserInputTranscriptionEvent,+)+from agentscope.realtime import (+ AudioFrame,+ ModelDisconnectedError,+ PlayoutPosition,+ RealtimeModelBase,+ RealtimeModelCard,+ SpeechTransition,+ TransportBase,+ TruncationSupport,+ VADBase,+)+from agentscope.event import (+ ConfirmResult,+ RequireUserConfirmEvent,+ ToolCallEndEvent,+ ToolCallStartEvent,+ ToolResultEndEvent,+ ToolResultStartEvent,+ ToolResultTextDeltaEvent,+ UserConfirmResultEvent,+)+from agentscope.message import Msg, TextBlock, ToolCallBlock, ToolResultBlock+from agentscope.permission import (+ PermissionBehavior,+ PermissionContext,+ PermissionDecision,+)+from agentscope.tool import ToolBase, ToolChunk, ToolResponse, Toolkit+from agentscope.realtime import _events as me++PCM_100MS = b"\x01\x00" * 2400++REPLY_R1 = [+ me.SpeechEndedEvent(item_id="u1"),+ me.InputTranscriptionEvent(item_id="u1", text="讲个故事"),+ me.ResponseCreatedEvent(item_id="r1"),+]+for _word in ["从前", "有座山", "山里", "有座庙", "庙里", "有个", "老和尚"]:+ REPLY_R1 += [+ me.TranscriptDeltaEvent(item_id="r1", delta=_word),+ me.AudioDeltaEvent(item_id="r1", pcm=PCM_100MS, sample_rate=24000),+ ]++REPLY_R2 = [+ me.InputTranscriptionEvent(item_id="u3", text="你好"),+ me.ResponseCreatedEvent(item_id="r2"),+ me.TranscriptDeltaEvent(item_id="r2", delta="你好呀"),+ me.AudioDeltaEvent(item_id="r2", pcm=PCM_100MS, sample_rate=24000),+ me.ResponseDoneEvent(item_id="r2", input_tokens=10, output_tokens=3),+]+++class ScriptedModel(RealtimeModelBase):+ """Plays one event script per session and records every call."""++ truncation = TruncationSupport.NONE+ type = "scripted"++ def __init__(self, scripts: list[list[Any]]) -> None:+ card = RealtimeModelCard(+ name="scripted",+ label="scripted",+ input_sample_rate=16000,+ output_sample_rate=24000,+ )+ super().__init__(+ "scripted",+ DashScopeCredential(api_key="sk-x"),+ model_card=card,+ )+ self.scripts = scripts+ self.calls: list[str] = []+ self.sessions = 0+ self._open = asyncio.Event()+ self._requested = asyncio.Event()++ @property+ def turn_detection_enabled(self) -> bool:+ """The provider owns turn boundaries unless a VAD is given."""+ return True++ async def connect(+ self,+ context: list[Msg],+ instructions: str,+ tools: list[dict] | None = None,+ **kwargs: Any,+ ) -> None:+ """Record a session open and the turn-detection request."""+ self.sessions += 1+ self._open.clear()+ self.calls.append(+ f"connect(session={self.sessions},ctx={len(context)},"+ f"td_off={kwargs.get('turn_detection_disabled')})",+ )++ async def close(self) -> None:+ """Record the close and release a live session."""+ self.calls.append("close")+ self._open.set()++ async def events(self) -> AsyncIterator[me.ModelEvent]:+ """Play the script for the current session."""+ script = self.scripts[self.sessions - 1]+ for event in script:+ if event == "WAIT": # the follow-up reply after a tool call+ await self._requested.wait()+ continue+ yield event+ await asyncio.sleep(0)+ if not script or not isinstance(script[-1], me.SessionEndedEvent):+ await self._open.wait() # a live session stays open++ async def push_audio(self, pcm: bytes) -> None:+ """Count audio pushes."""+ self.calls.append("push_audio")++ async def push_text(self, text: str) -> None:+ """Record a text turn; the agent gates on ``supports_text_input``+ before ever calling this."""+ self.calls.append(f"push_text({text!r})")++ async def push_tool_result(self, block: ToolResultBlock) -> None:+ """Record exactly what the provider would receive."""+ self.calls.append(f"tool_result({block.id},{block.output!r})")++ async def commit_turn(self) -> None:+ """Record the commit."""+ self.calls.append("commit_turn")++ async def request_response(self) -> None:+ """Record the request and let a script waiting on it continue."""+ self.calls.append("request_response")+ self._requested.set()++ async def cancel_response(self) -> None:+ """Record the cancel."""+ self.calls.append("cancel")++ async def truncate(+ self,+ item_id: str,+ played_ms: int,+ played_text: str,+ ) -> None:+ """Record what the agent thinks the user heard."""+ self.calls.append(f"truncate({item_id},{played_ms}ms,{played_text!r})")+++class FakeTransport(TransportBase):+ """Emits ``frames`` chunks of silence, reports 320 ms played."""++ input_sample_rate = 16000+ output_sample_rate = 24000++ def __init__(self, frames: int) -> None:+ self.frames = frames+ self.item = ""+ self.cleared = 0+ self.started = 0+ self.closed = 0++ async def start(self) -> None:+ """Count starts; the owner is the test."""+ self.started += 1++ async def close(self) -> None:+ """Count closes."""+ self.closed += 1++ async def incoming(self) -> AsyncIterator[AudioFrame]:+ """Emit silence on a 20 ms clock."""+ for _ in range(self.frames):+ await asyncio.sleep(0.02)+ yield AudioFrame(pcm=b"\x00" * 3200)++ async def send_audio(self, pcm: bytes, item_id: str) -> None:+ """Remember which item is playing."""+ self.item = item_id++ async def send_event(self, event: dict) -> None:+ """No peer to send to."""++ async def clear_audio(self) -> PlayoutPosition:+ """Count cuts and report the fixed playout position."""+ self.cleared += 1+ return self.playout()++ def playout(self) -> PlayoutPosition:+ """Always 320 ms into the current item."""+ return PlayoutPosition(+ item_id=self.item,+ played_ms=320,+ first_played_at=1.0,+ )+++class EndOnSecondFrameVAD(VADBase):+ """Reports the user stopping on the second chunk."""++ sample_rate = 16000++ def __init__(self) -> None:+ self.seen = 0++ def push(self, pcm: bytes) -> SpeechTransition | None:+ """ENDED on the second chunk, otherwise nothing."""+ self.seen += 1+ return SpeechTransition.ENDED if self.seen == 2 else None++ def reset(self) -> None:+ """Start counting again."""+ self.seen = 0+++class RealtimeAgentTest(IsolatedAsyncioTestCase):+ """Behaviour of the turn-taking state machine."""++ async def _collect(+ self,+ agent: RealtimeAgent,+ transport: FakeTransport,+ ) -> list[tuple[str, Any]]:+ """Run the agent over *transport* and summarise the events."""+ summary: list[tuple[str, Any]] = []+ async with transport:+ async for event in agent.run(transport):+ if isinstance(event, ReplyEndEvent):+ summary.append(("reply_end", event.finished_reason))+ elif isinstance(event, UserInputTranscriptionEvent):+ summary.append(("user", event.transcript))+ return summary++ async def test_barge_in_truncates_to_what_was_heard(self) -> None:+ """A barge-in mid-reply cuts both contexts to the played prefix+ and the duplicate speech_started is swallowed by the lock."""+ script = REPLY_R1 + [+ me.SpeechStartedEvent(item_id="u2"),+ me.SpeechStartedEvent(item_id="u2"),+ ]+ model = ScriptedModel([script])+ agent = RealtimeAgent("Friday", "be brief", model)+ transport = FakeTransport(frames=3)++ async with agent:+ summary = await self._collect(agent, transport)++ self.assertListEqual(+ summary,+ [("user", "讲个故事"), ("reply_end", "interrupted")],+ )+ self.assertEqual(transport.cleared, 1)+ # Audio frames interleave with model events on the transport's+ # clock, so they are counted rather than positioned.+ self.assertListEqual(+ [c for c in model.calls if c != "push_audio"],+ [+ "connect(session=1,ctx=0,td_off=False)",+ "truncate(r1,320ms,'从前有座山山里有座庙')",+ "cancel",+ "close",+ ],+ )+ self.assertEqual(model.calls.count("push_audio"), 3)+ self.assertListEqual(+ [(m.role, m.get_text_content()) for m in agent.state.context],+ [("user", "讲个故事"), ("assistant", "从前有座山山里有座庙")],+ )+ self.assertEqual(agent.last_turn_metrics.first_audio_played_at, 1.0)++ async def test_provider_timeout_reconnects_on_next_audio(self) -> None:+ """When the provider closes the session, nothing reconnects until+ the next user audio, which reconnects with the current context."""+ model = ScriptedModel(+ [+ REPLY_R1 + [me.SessionEndedEvent(reason="idle")],+ REPLY_R2,+ ],+ )+ agent = RealtimeAgent(+ "Friday",+ "be brief",+ model,+ aggregator=TurnAggregator(merge_window_ms=0),+ )++ async with agent:+ # No transport, no audio: the provider times the session out+ # and nothing reconnects.+ await asyncio.sleep(0.1)+ self.assertFalse(agent._connected) # pylint: disable=W0212+ self.assertEqual(model.sessions, 1)++ summary = await self._collect(agent, FakeTransport(frames=2))++ self.assertEqual(model.sessions, 2)+ # The orphaned reply was dropped, so one message carried over.+ self.assertIn("connect(session=2,ctx=1,td_off=False)", model.calls)+ # Events produced while no run was active are delivered first;+ # a reply streamed with nobody listening is cut off exactly once.+ self.assertListEqual(+ summary,+ [+ ("user", "讲个故事"),+ ("reply_end", "interrupted"),+ ("user", "你好"),+ ("reply_end", "completed"),+ ],+ )+ self.assertListEqual(+ [(m.role, m.get_text_content()) for m in agent.state.context],+ [("user", "讲个故事"), ("user", "你好"), ("assistant", "你好呀")],+ )++ async def test_local_vad_owns_turns(self) -> None:+ """Passing a VAD disables provider turn detection and commits the+ turn when the VAD reports the user stopped."""+ model = ScriptedModel([[]])+ agent = RealtimeAgent(+ "Friday",+ "be brief",+ model,+ vad=EndOnSecondFrameVAD(),+ )+ async with agent:+ await self._collect(agent, FakeTransport(frames=3))++ self.assertListEqual(+ model.calls,+ [+ "connect(session=1,ctx=0,td_off=True)",+ "push_audio",+ "commit_turn",+ "push_audio",+ "push_audio",+ "close",+ ],+ )++ async def test_run_exit_cancels_reply_in_flight(self) -> None:+ """The transport ending mid-reply cancels the reply so the model+ does not keep talking to nobody."""+ model = ScriptedModel([REPLY_R1]) # never sends response.done+ agent = RealtimeAgent("Friday", "be brief", model)+ async with agent:+ summary = await self._collect(agent, FakeTransport(frames=1))++ self.assertListEqual(+ summary,+ [("user", "讲个故事"), ("reply_end", "interrupted")],+ )+ self.assertIn("cancel", model.calls)++ async def test_text_rejected_by_audio_only_model(self) -> None:+ """A provider without text input refuses typed turns."""+ agent = RealtimeAgent("Friday", "be brief", ScriptedModel([[]]))+ with self.assertRaises(NotImplementedError):+ await agent.send("hi")++ async def test_backchannel_is_dropped(self) -> None:+ """A bare acknowledgement never becomes a turn."""+ model = ScriptedModel(+ [[me.InputTranscriptionEvent(item_id="u1", text="嗯。")]],+ )+ agent = RealtimeAgent(+ "Friday",+ "be brief",+ model,+ aggregator=TurnAggregator(backchannels=frozenset({"嗯"})),+ )+ async with agent:+ summary = await self._collect(agent, FakeTransport(frames=1))++ self.assertListEqual(summary, [])+ self.assertListEqual(agent.state.context, [])+++class StreamTool(ToolBase):+ """Streams two chunks, then the completed result."""++ name: str = "stream_tool"+ description: str = "streams"+ input_schema: dict[str, Any] = {+ "type": "object",+ "properties": {"q": {"type": "string"}},+ "required": ["q"],+ }+ is_concurrency_safe: bool = True+ is_read_only: bool = True+ is_external_tool: bool = False+ is_mcp: bool = False++ async def check_permissions(+ self,+ tool_input: dict[str, Any],+ context: PermissionContext,+ ) -> PermissionDecision:+ """Run freely."""+ return PermissionDecision(+ behavior=PermissionBehavior.ALLOW,+ decision_reason="test",+ message="test",+ )++ async def __call__(self, q: str, **kwargs: Any) -> Any:+ """Yield chunks then the final response."""+ yield ToolChunk(content=[TextBlock(text=f"{q}-a")])+ yield ToolChunk(content=[TextBlock(text=f"{q}-b")])+ yield ToolResponse(content=[TextBlock(text=f"{q}-final")])+++class AskTool(StreamTool):+ """Requires user confirmation before running. Not read-only, or the+ permission engine's read-only fast path would allow it unasked."""++ name: str = "ask_tool"+ is_read_only: bool = False++ async def check_permissions(+ self,+ tool_input: dict[str, Any],+ context: PermissionContext,+ ) -> PermissionDecision:+ """Always ask."""+ return PermissionDecision(+ behavior=PermissionBehavior.ASK,+ decision_reason="test",+ message="test",+ )+++class BrokenTool(StreamTool):+ """Raises while running."""++ name: str = "broken_tool"++ async def __call__(self, q: str, **kwargs: Any) -> Any:+ """Fail."""+ raise RuntimeError("boom")+ yield # pylint: disable=unreachable+++def _tool_script(name: str) -> list[me.ModelEvent]:+ """A reply that only calls *name* and completes."""+ return [+ me.ResponseCreatedEvent(item_id="r1"),+ me.ToolCallEvent(+ item_id="r1",+ tool_call=ToolCallBlock(id="c1", name=name, input='{"q": "x"}'),+ ),+ me.ResponseDoneEvent(item_id="r1"),+ ]+++class RealtimeAgentToolTest(IsolatedAsyncioTestCase):+ """Tool calls: permission, execution, result delivery."""++ async def _run_tool_scenario(+ self,+ tool: ToolBase,+ confirm: bool | None = None,+ ) -> tuple[list[tuple[str, Any]], ScriptedModel]:+ """Run one tool-calling reply; answer a permission prompt with+ *confirm* if one appears. Returns the tool-related events."""+ model = ScriptedModel([_tool_script(tool.name)])+ agent = RealtimeAgent(+ "Friday",+ "be brief",+ model,+ toolkit=Toolkit(tools=[tool]),+ )+ events: list[tuple[str, Any]] = []+ async with agent:+ transport = FakeTransport(frames=4)+ async with transport:+ async for event in agent.run(transport):+ match event:+ case ToolCallStartEvent():+ events.append(("call_start", event.tool_call_name))+ case ToolCallEndEvent():+ events.append(("call_end", event.tool_call_id))+ case RequireUserConfirmEvent():+ events.append(+ ("ask", [c.name for c in event.tool_calls]),+ )+ await agent.send(+ UserConfirmResultEvent(+ reply_id=event.reply_id,+ confirm_results=[+ ConfirmResult(+ tool_call=event.tool_calls[0],+ confirmed=bool(confirm),+ ),+ ],+ ),+ )+ case ToolResultStartEvent():+ events.append(+ ("result_start", event.tool_call_name),+ )+ case ToolResultTextDeltaEvent():+ events.append(("delta", event.delta))+ case ToolResultEndEvent():+ events.append(("result_end", event.state))+ return events, model++ async def test_streamed_tool_result_is_not_duplicated(self) -> None:+ """Chunks are shown as they come; the provider gets only the+ completed result, once."""+ events, model = await self._run_tool_scenario(StreamTool())++ self.assertListEqual(+ events,+ [+ ("call_start", "stream_tool"),+ ("call_end", "c1"),+ ("result_start", "stream_tool"),+ ("delta", "x-a"),+ ("delta", "x-b"),+ ("result_end", "success"),+ ],+ )+ self.assertListEqual(+ [c for c in model.calls if not c.startswith("push_audio")],+ [+ "connect(session=1,ctx=0,td_off=False)",+ "tool_result(c1,'x-final')",+ "request_response",+ "close",+ ],+ )++ async def test_confirmed_tool_runs(self) -> None:+ """A confirmed permission prompt lets the tool run."""+ events, model = await self._run_tool_scenario(AskTool(), confirm=True)++ self.assertListEqual(+ events,+ [+ ("call_start", "ask_tool"),+ ("call_end", "c1"),+ ("ask", ["ask_tool"]),+ ("result_start", "ask_tool"),+ ("delta", "x-a"),+ ("delta", "x-b"),+ ("result_end", "success"),+ ],+ )+ self.assertIn("tool_result(c1,'x-final')", model.calls)++ async def test_denied_tool_reports_denial(self) -> None:+ """A refused prompt sends a denial to the provider, runs nothing."""+ events, model = await self._run_tool_scenario(AskTool(), confirm=False)++ self.assertListEqual(+ events,+ [+ ("call_start", "ask_tool"),+ ("call_end", "c1"),+ ("ask", ["ask_tool"]),+ ("result_start", "ask_tool"),+ ("delta", 'Tool "ask_tool" denied by user.'),+ ("result_end", "denied"),+ ],+ )+ self.assertIn(+ "tool_result(c1,'Tool \"ask_tool\" denied by user.')",+ model.calls,+ )++ async def test_failing_tool_reports_error(self) -> None:+ """The toolkit turns an exception into an error response, which+ is forwarded as-is."""+ events, model = await self._run_tool_scenario(BrokenTool())++ self.assertListEqual(+ events,+ [+ ("call_start", "broken_tool"),+ ("call_end", "c1"),+ ("result_start", "broken_tool"),+ ("delta", "boom"),+ ("result_end", "error"),+ ],+ )+ self.assertIn("tool_result(c1,'boom')", model.calls)+++class RealtimeAgentFullStreamTest(IsolatedAsyncioTestCase):+ """The complete event stream of a turn that calls a tool and then+ answers with speech, asserted as one structure."""++ async def test_tool_call_then_spoken_reply(self) -> None:+ """User asks → model speaks, calls a tool → tool runs → model+ speaks the answer. Every event, in order."""+ model = ScriptedModel(+ [+ [+ me.SpeechEndedEvent(item_id="u1"),+ me.InputTranscriptionEvent(item_id="u1", text="查天气"),+ me.ResponseCreatedEvent(item_id="r1"),+ me.TranscriptDeltaEvent(item_id="r1", delta="我查一下"),+ me.AudioDeltaEvent(+ item_id="r1",+ pcm=b"\x01\x00",+ sample_rate=24000,+ ),+ me.ToolCallEvent(+ item_id="r1",+ tool_call=ToolCallBlock(+ id="c1",+ name="stream_tool",+ input='{"q": "x"}',+ ),+ ),+ me.ResponseDoneEvent(+ item_id="r1",+ input_tokens=5,+ output_tokens=2,+ ),+ "WAIT",+ me.ResponseCreatedEvent(item_id="r2"),+ me.TranscriptDeltaEvent(item_id="r2", delta="今天晴"),+ me.AudioDeltaEvent(+ item_id="r2",+ pcm=b"\x01\x00",+ sample_rate=24000,+ ),+ me.ResponseDoneEvent(+ item_id="r2",+ input_tokens=9,+ output_tokens=3,+ ),+ ],+ ],+ )+ agent = RealtimeAgent(+ "Friday",+ "be brief",+ model,+ toolkit=Toolkit(tools=[StreamTool()]),+ )+ events = []+ async with agent:+ transport = FakeTransport(frames=6)+ async with transport:+ async for event in agent.run(transport):+ events.append(event.model_dump(mode="json"))++ self.assertListEqual(+ events,+ [+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "USER_INPUT_TRANSCRIPTION",+ "session_id": AnyString(),+ "item_id": "u1",+ "transcript": "查天气",+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "REPLY_START",+ "session_id": AnyString(),+ "reply_id": "r1",+ "name": "Friday",+ "role": "assistant",+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "MODEL_CALL_START",+ "reply_id": "r1",+ "model_name": "scripted",+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "TEXT_BLOCK_START",+ "reply_id": "r1",+ "block_id": AnyString(),+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "TEXT_BLOCK_DELTA",+ "reply_id": "r1",+ "block_id": AnyString(),+ "delta": "我查一下",+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "DATA_BLOCK_START",+ "reply_id": "r1",+ "block_id": AnyString(),+ "media_type": "audio/pcm;rate=24000",+ "name": None,+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "DATA_BLOCK_DELTA",+ "reply_id": "r1",+ "block_id": AnyString(),+ "media_type": "audio/pcm;rate=24000",+ "data": "AQA=",+ "url": None,+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "TEXT_BLOCK_END",+ "reply_id": "r1",+ "block_id": AnyString(),+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "DATA_BLOCK_END",+ "reply_id": "r1",+ "block_id": AnyString(),+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "MODEL_CALL_END",+ "reply_id": "r1",+ "input_tokens": 5,+ "output_tokens": 2,+ "cache_input_tokens": 0,+ "cache_creation_input_tokens": 0,+ "finished_reason": "completed",+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "REPLY_END",+ "session_id": AnyString(),+ "reply_id": "r1",+ "finished_reason": "completed",+ "error": None,+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "TOOL_CALL_START",+ "reply_id": "r1",+ "tool_call_id": "c1",+ "tool_call_name": "stream_tool",+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "TOOL_CALL_END",+ "reply_id": "r1",+ "tool_call_id": "c1",+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "TOOL_RESULT_START",+ "reply_id": "r1",+ "tool_call_id": "c1",+ "tool_call_name": "stream_tool",+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "TOOL_RESULT_TEXT_DELTA",+ "reply_id": "r1",+ "tool_call_id": "c1",+ "delta": "x-a",+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "TOOL_RESULT_TEXT_DELTA",+ "reply_id": "r1",+ "tool_call_id": "c1",+ "delta": "x-b",+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "TOOL_RESULT_END",+ "reply_id": "r1",+ "tool_call_id": "c1",+ "state": "success",+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "REPLY_START",+ "session_id": AnyString(),+ "reply_id": "r2",+ "name": "Friday",+ "role": "assistant",+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "MODEL_CALL_START",+ "reply_id": "r2",+ "model_name": "scripted",+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "TEXT_BLOCK_START",+ "reply_id": "r2",+ "block_id": AnyString(),+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "TEXT_BLOCK_DELTA",+ "reply_id": "r2",+ "block_id": AnyString(),+ "delta": "今天晴",+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "DATA_BLOCK_START",+ "reply_id": "r2",+ "block_id": AnyString(),+ "media_type": "audio/pcm;rate=24000",+ "name": None,+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "DATA_BLOCK_DELTA",+ "reply_id": "r2",+ "block_id": AnyString(),+ "media_type": "audio/pcm;rate=24000",+ "data": "AQA=",+ "url": None,+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "TEXT_BLOCK_END",+ "reply_id": "r2",+ "block_id": AnyString(),+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "DATA_BLOCK_END",+ "reply_id": "r2",+ "block_id": AnyString(),+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "MODEL_CALL_END",+ "reply_id": "r2",+ "input_tokens": 9,+ "output_tokens": 3,+ "cache_input_tokens": 0,+ "cache_creation_input_tokens": 0,+ "finished_reason": "completed",+ },+ {+ "id": AnyString(),+ "created_at": AnyString(),+ "metadata": {},+ "type": "REPLY_END",+ "session_id": AnyString(),+ "reply_id": "r2",+ "finished_reason": "completed",+ "error": None,+ },+ ],+ )+ self.assertListEqual(+ [(m.role, m.get_text_content()) for m in agent.state.context],+ [+ ("user", "查天气"),+ ("assistant", "我查一下"),+ ("assistant", "今天晴"),+ ],+ )+ self.assertListEqual(+ [c for c in model.calls if c != "push_audio"],+ [+ "connect(session=1,ctx=0,td_off=False)",+ "tool_result(c1,'x-final')",+ "request_response",+ "close",+ ],+ )+++class DropsSocketModel(ScriptedModel):+ """Raises on the first push after the provider closed the socket,+ the way a real WebSocket does before the reader notices."""++ def __init__(self) -> None:+ super().__init__([[], []])+ self.dropped = False++ async def push_audio(self, pcm: bytes) -> None:+ """Fail exactly once, then behave."""+ if self.sessions == 1 and not self.dropped:+ self.dropped = True+ raise ModelDisconnectedError("idle for 180 seconds")+ await super().push_audio(pcm)+++class RealtimeAgentDisconnectTest(IsolatedAsyncioTestCase):+ """A send that hits a closed provider socket must not kill the run."""++ async def test_send_on_closed_socket_reconnects_on_next_audio(+ self,+ ) -> None:+ """The failing frame is kept, the run survives, and the next frame+ reconnects and flushes it."""+ model = DropsSocketModel()+ agent = RealtimeAgent("Friday", "be brief", model)+ async with agent:+ transport = FakeTransport(frames=3)+ with self.assertLogs("as", level="INFO") as logs:+ async with transport:+ async for _ in agent.run(transport):+ pass++ self.assertEqual(model.sessions, 2)+ self.assertListEqual(+ [c for c in model.calls if c != "push_audio"],+ [+ "connect(session=1,ctx=0,td_off=False)",+ "connect(session=2,ctx=0,td_off=False)",+ "close",+ ],+ )+ # Three frames captured; the failed one was replayed, so all+ # three reach the provider in the end.+ self.assertEqual(model.calls.count("push_audio"), 3)+ self.assertTrue(+ any("keep talking" in line for line in logs.output),+ logs.output,+ )diff --git a/tests/realtime_aggregator_test.py b/tests/realtime_aggregator_test.pynew file mode 100644index 000000000..83a1f4cda--- /dev/null+++ b/tests/realtime_aggregator_test.py@@ -0,0 +1,56 @@+# -*- coding: utf-8 -*-+"""Unit tests for TurnAggregator."""+import time+import unittest++from agentscope.agent import TurnAggregator+++class TurnAggregatorTest(unittest.TestCase):+ """Collapsing transcripts into turns."""++ def test_continuation_within_window_merges(self) -> None:+ """A transcript arriving inside the merge window continues the+ previous turn; one arriving later starts a new one."""+ aggregator = TurnAggregator(merge_window_ms=300)++ self.assertListEqual(+ [+ aggregator.take("帮我订张机票"),+ aggregator.merges_with_previous(),+ aggregator.take("去上海的"),+ aggregator.merges_with_previous(),+ ],+ ["帮我订张机票", False, "去上海的", True],+ )++ time.sleep(0.4)+ self.assertListEqual(+ [aggregator.take("几点的"), aggregator.merges_with_previous()],+ ["几点的", False],+ )++ def test_backchannels_and_empty_transcripts_are_not_turns(self) -> None:+ """Acknowledgements are matched ignoring trailing punctuation."""+ aggregator = TurnAggregator(backchannels=frozenset({"嗯", "对"}))++ self.assertListEqual(+ [+ aggregator.take("嗯。"),+ aggregator.take("对,"),+ aggregator.take(" "),+ aggregator.take("对的"),+ ],+ [None, None, None, "对的"],+ )++ def test_reset_forgets_previous_turn(self) -> None:+ """After a reset the next transcript never merges."""+ aggregator = TurnAggregator(merge_window_ms=10_000)+ aggregator.take("第一句")+ aggregator.reset()++ self.assertListEqual(+ [aggregator.take("第二句"), aggregator.merges_with_previous()],+ ["第二句", False],+ )diff --git a/tests/realtime_dashscope_test.py b/tests/realtime_dashscope_test.pynew file mode 100644index 000000000..33c1eb012--- /dev/null+++ b/tests/realtime_dashscope_test.py@@ -0,0 +1,413 @@+# -*- coding: utf-8 -*-+"""Unit tests for the DashScope realtime adapters: cards, session config+and frame parsing. Nothing here opens a connection."""+# pylint: disable=protected-access+import base64+import unittest+from unittest.async_case import IsolatedAsyncioTestCase++from agentscope.credential import DashScopeCredential+from agentscope.realtime import (+ DashScopeAudioRealtimeModel,+ DashScopeRealtimeModel,+ ModelDisconnectedError,+ TruncationSupport,+)+from agentscope.realtime import _events as me++CRED = DashScopeCredential(api_key="sk-x")+TRANSCRIPTION_DONE = "conversation.item.input_audio_transcription.completed"+AMBIENT_DELTA = "conversation.item.ambient_audio_transcription.delta"+++class DashScopeCardsTest(unittest.TestCase):+ """Each adapter lists only its own cards, tagged with its type."""++ def test_omni_cards(self) -> None:+ """The Omni adapter lists the four Omni cards."""+ self.assertListEqual(+ [+ (c.name, c.model_type, c.supports_tools, c.max_audio_turns)+ for c in DashScopeRealtimeModel.list_models()+ ],+ [+ (+ "qwen-omni-turbo-realtime",+ "dashscope_omni_realtime",+ False,+ None,+ ),+ (+ "qwen3-omni-flash-realtime",+ "dashscope_omni_realtime",+ False,+ 8,+ ),+ (+ "qwen3.5-omni-flash-realtime",+ "dashscope_omni_realtime",+ True,+ 80,+ ),+ (+ "qwen3.5-omni-plus-realtime",+ "dashscope_omni_realtime",+ True,+ 100,+ ),+ ],+ )++ def test_audio_cards(self) -> None:+ """The Audio adapter lists the two Audio cards."""+ self.assertListEqual(+ [+ (+ c.name,+ c.model_type,+ c.max_audio_turns,+ c.max_audio_duration_s,+ )+ for c in DashScopeAudioRealtimeModel.list_models()+ ],+ [+ (+ "qwen-audio-3.0-realtime-flash",+ "dashscope_audio_realtime",+ 50,+ 300,+ ),+ (+ "qwen-audio-3.0-realtime-plus",+ "dashscope_audio_realtime",+ 50,+ 300,+ ),+ ],+ )++ def test_credential_maps_card_back_to_class(self) -> None:+ """The service-layer lookup: card.model_type -> class, no scan."""+ classes = {c.type: c for c in CRED.get_realtime_model_classes()}+ self.assertDictEqual(+ {+ card.name: classes[card.model_type].__name__+ for card in CRED.list_realtime_models()+ },+ {+ "qwen-audio-3.0-realtime-flash": "DashScopeAudioRealtimeModel",+ "qwen-audio-3.0-realtime-plus": "DashScopeAudioRealtimeModel",+ "qwen-omni-turbo-realtime": "DashScopeRealtimeModel",+ "qwen3-omni-flash-realtime": "DashScopeRealtimeModel",+ "qwen3.5-omni-flash-realtime": "DashScopeRealtimeModel",+ "qwen3.5-omni-plus-realtime": "DashScopeRealtimeModel",+ },+ )++ def test_unknown_model_name_is_rejected(self) -> None:+ """A name with no card fails at construction."""+ with self.assertRaises(ValueError):+ DashScopeRealtimeModel("no-such-model", CRED)+++class DashScopeSessionUpdateTest(unittest.TestCase):+ """The session.update payload each adapter sends on connect."""++ def test_omni_payload(self) -> None:+ """Omni session.update with tools and transcription."""+ model = DashScopeRealtimeModel("qwen3.5-omni-flash-realtime", CRED)+ self.assertDictEqual(+ model._session_update("be nice", [{"type": "function"}]),+ {+ "type": "session.update",+ "session": {+ "instructions": "be nice",+ "modalities": ["audio", "text"],+ "voice": "Cherry",+ "input_audio_format": "pcm16",+ "output_audio_format": "pcm24",+ "turn_detection": {+ "type": "server_vad",+ "threshold": 0.5,+ "silence_duration_ms": 800,+ },+ "input_audio_transcription": {+ "model": "gummy-realtime-v1",+ },+ "tools": [{"type": "function"}],+ },+ },+ )++ def test_omni_withholds_tools_the_card_does_not_support(self) -> None:+ """Tools are not sent for a model whose card declines them."""+ model = DashScopeRealtimeModel("qwen3-omni-flash-realtime", CRED)+ session = model._session_update("x", [{"type": "function"}])["session"]+ self.assertNotIn("tools", session)++ def test_audio_payload_with_smart_turn(self) -> None:+ """Audio session.update with smart_turn and a voiceprint."""+ model = DashScopeAudioRealtimeModel(+ "qwen-audio-3.0-realtime-plus",+ CRED,+ parameters=DashScopeAudioRealtimeModel.Parameters(+ turn_detection="smart_turn",+ voiceprint_audio_urls=["https://x/a.wav"],+ max_history_turns=30,+ ),+ )+ self.assertDictEqual(+ model._session_update("be nice", None),+ {+ "type": "session.update",+ "session": {+ "instructions": "be nice",+ "modalities": ["audio", "text"],+ "voice": "longanqian",+ "input_audio_format": "pcm",+ "output_audio_format": "pcm",+ "max_history_turns": 30,+ "turn_detection": {+ "type": "smart_turn",+ "voiceprint_audio_urls": ["https://x/a.wav"],+ },+ },+ },+ )++ def test_turn_detection_none_hands_endpointing_to_caller(self) -> None:+ """``none`` sends null and marks endpointing as ours."""+ model = DashScopeAudioRealtimeModel(+ "qwen-audio-3.0-realtime-flash",+ CRED,+ parameters=DashScopeAudioRealtimeModel.Parameters(+ turn_detection="none",+ ),+ )+ session = model._session_update("x", None)["session"]+ self.assertIsNone(session["turn_detection"])+ self.assertFalse(model.turn_detection_enabled)++ def test_adapter_facts(self) -> None:+ """Protocol facts differ per adapter, not per model."""+ omni = DashScopeRealtimeModel("qwen3.5-omni-flash-realtime", CRED)+ audio = DashScopeAudioRealtimeModel(+ "qwen-audio-3.0-realtime-plus",+ CRED,+ )+ self.assertListEqual(+ [+ (omni.truncation, omni.supports_text_input),+ (audio.truncation, audio.supports_text_input),+ ],+ [+ (TruncationSupport.NONE, False),+ (TruncationSupport.NONE, True),+ ],+ )+++class DashScopeParseTest(unittest.TestCase):+ """Server frames -> model events."""++ def setUp(self) -> None:+ """Open a response so deltas have an item to attach to."""+ self.model = DashScopeRealtimeModel(+ "qwen3.5-omni-flash-realtime",+ CRED,+ )+ self.model._parse(+ {"type": "response.created", "response": {"id": "r1"}},+ )++ def test_reply_frames(self) -> None:+ """Every reply-side frame maps to one model event."""+ frames = [+ {"type": "response.audio_transcript.delta", "delta": "你好"},+ {+ "type": "response.audio.delta",+ "delta": base64.b64encode(b"\x01\x00").decode(),+ },+ {+ "type": "input_audio_buffer.speech_started",+ "item_id": "u1",+ "audio_start_ms": 120,+ },+ {+ "type": TRANSCRIPTION_DONE,+ "item_id": "u1",+ "transcript": "天气",+ },+ {+ "type": "response.done",+ "response": {+ "id": "r1",+ "usage": {"input_tokens": 10, "output_tokens": 5},+ },+ },+ {"type": "session.updated"},+ ]+ self.assertListEqual(+ [self.model._parse(f) for f in frames],+ [+ me.TranscriptDeltaEvent(item_id="r1", delta="你好"),+ me.AudioDeltaEvent(+ item_id="r1",+ pcm=b"\x01\x00",+ sample_rate=24000,+ ),+ me.SpeechStartedEvent(item_id="u1", at_ms=120),+ me.InputTranscriptionEvent(item_id="u1", text="天气"),+ me.ResponseDoneEvent(+ item_id="r1",+ input_tokens=10,+ output_tokens=5,+ ),+ None,+ ],+ )++ def test_tool_call_done_frame_is_authoritative(self) -> None:+ """Accumulated deltas are only a fallback for a done frame that+ omits ``arguments``."""+ self.model._parse(+ {+ "type": "response.output_item.added",+ "item": {+ "type": "function_call",+ "call_id": "c1",+ "name": "get_weather",+ },+ },+ )+ self.model._parse(+ {+ "type": "response.function_call_arguments.delta",+ "call_id": "c1",+ "delta": '{"city":',+ },+ )+ with_args = self.model._parse(+ {+ "type": "response.function_call_arguments.done",+ "call_id": "c1",+ "arguments": '{"city":"sh"}',+ },+ )+ self.model._parse(+ {+ "type": "response.function_call_arguments.delta",+ "call_id": "c2",+ "name": "f",+ "delta": '{"a":1}',+ },+ )+ without_args = self.model._parse(+ {"type": "response.function_call_arguments.done", "call_id": "c2"},+ )++ self.assertListEqual(+ [+ (with_args.tool_call.name, with_args.tool_call.input),+ (without_args.tool_call.name, without_args.tool_call.input),+ ],+ [("get_weather", '{"city":"sh"}'), ("f", '{"a":1}')],+ )++ def test_audio_specific_frames_are_known_and_dropped(self) -> None:+ """Audio-only frames are recognised and dropped, errors kept."""+ model = DashScopeAudioRealtimeModel(+ "qwen-audio-3.0-realtime-plus",+ CRED,+ )+ self.assertListEqual(+ [+ model._parse(+ {+ "type": AMBIENT_DELTA,+ "delta": "x",+ },+ ),+ model._parse({"type": "voiceprint_audio_list.completed"}),+ model._parse(+ {+ "type": "error",+ "error": {"code": "E1", "message": "boom"},+ },+ ),+ ],+ [None, None, me.ModelErrorEvent(code="E1", message="boom")],+ )+++class DashScopeAudioTextInputTest(IsolatedAsyncioTestCase):+ """push_text sends a text item and asks for the reply."""++ async def test_push_text_wire_shape(self) -> None:+ """A text turn is one item plus a response request."""+ model = DashScopeAudioRealtimeModel(+ "qwen-audio-3.0-realtime-plus",+ CRED,+ )+ sent: list[dict] = []++ async def capture(payload: dict) -> None:+ sent.append(payload)++ model._send = capture # type: ignore[method-assign]+ await model.push_text("你好")++ self.assertListEqual(+ sent,+ [+ {+ "type": "conversation.item.create",+ "item": {+ "type": "message",+ "role": "user",+ "content": [{"type": "input_text", "text": "你好"}],+ },+ },+ {+ "type": "response.create",+ "response": {"modalities": ["text", "audio"]},+ },+ ],+ )+++class DashScopeDisconnectTest(IsolatedAsyncioTestCase):+ """A closed WebSocket surfaces as ModelDisconnectedError."""++ async def test_send_on_closed_socket(self) -> None:+ """websockets' ConnectionClosed becomes the realtime-level error+ and the socket reference is dropped."""+ from websockets.exceptions import ConnectionClosedError+ from websockets.frames import Close++ class ClosedSocket:+ """Raises like a socket the provider already closed."""++ async def send(self, _payload: str) -> None:+ """Fail with the provider's close frame."""+ close = Close(1007, "idle 180s")+ raise ConnectionClosedError(close, close, True)++ model = DashScopeRealtimeModel("qwen3.5-omni-flash-realtime", CRED)+ model._ws = ClosedSocket()++ with self.assertRaises(ModelDisconnectedError) as ctx:+ await model.push_audio(b"\x00\x00")+ self.assertEqual(+ (str(ctx.exception), model._ws),+ (+ "1007 (invalid frame payload data) idle 180s",+ None,+ ),+ )++ async def test_send_before_connect(self) -> None:+ """No socket at all is the same condition."""+ model = DashScopeRealtimeModel("qwen3.5-omni-flash-realtime", CRED)+ with self.assertRaises(ModelDisconnectedError):+ await model.commit_turn()diff --git a/tests/realtime_local_transport_test.py b/tests/realtime_local_transport_test.pynew file mode 100644index 000000000..590d15f46--- /dev/null+++ b/tests/realtime_local_transport_test.py@@ -0,0 +1,145 @@+# -*- coding: utf-8 -*-+"""Unit tests for LocalAudioTransport, driving the PortAudio callbacks+directly — no sound card is opened."""+# pylint: disable=protected-access+import asyncio+from unittest.async_case import IsolatedAsyncioTestCase++import numpy as np++from agentscope.realtime import (+ AudioFrame,+ LocalAudioTransport,+ PlayoutPosition,+)++BLOCK = 2400 # 100 ms at 24 kHz+++def _play(transport: LocalAudioTransport, blocks: int) -> np.ndarray:+ """Run the output callback *blocks* times and return the last block."""+ out = np.zeros((BLOCK, 1), np.int16)+ for _ in range(blocks):+ transport._on_output(out, BLOCK, None, None)+ return out+++class LocalAudioTransportTest(IsolatedAsyncioTestCase):+ """Playback buffering, playout accounting and the interruption fade."""++ async def test_playout_accounting_and_first_played_at(self) -> None:+ """Position advances by what the callback consumed, and the first+ played timestamp is set once."""+ transport = LocalAudioTransport(output_sample_rate=24000)+ await transport.send_audio(+ np.full(24000, 1000, np.int16).tobytes(),+ "r1",+ )++ self.assertEqual(+ transport.playout().model_dump(),+ {"item_id": "r1", "played_ms": 0, "first_played_at": None},+ )+ _play(transport, 5)+ position = transport.playout()+ self.assertEqual(+ (+ position.item_id,+ position.played_ms,+ position.first_played_at is not None,+ ),+ ("r1", 500, True),+ )++ async def test_clear_audio_reports_cut_then_fades(self) -> None:+ """The reported position excludes the fade tail; the tail ramps to+ silence and the remainder of the block is zero-filled."""+ transport = LocalAudioTransport(output_sample_rate=24000, fade_ms=30)+ await transport.send_audio(+ np.full(24000, 1000, np.int16).tobytes(),+ "r1",+ )+ _play(transport, 5)++ cut = await transport.clear_audio()+ tail = np.frombuffer(bytes(transport._pending), np.int16)+ self.assertEqual(+ (+ cut.played_ms,+ len(tail),+ int(tail[0]),+ int(tail[-1]),+ bool(np.all(np.diff(tail) <= 0)),+ ),+ (500, 720, 1000, 0, True),+ )++ out = _play(transport, 1)+ self.assertEqual(+ (+ len(transport._pending),+ int(out[0, 0]),+ int(out[719, 0]),+ int(out[720, 0]),+ int(out[-1, 0]),+ ),+ (0, 1000, 0, 0, 0),+ )++ async def test_new_item_resets_counters(self) -> None:+ """Audio for a new item starts a fresh position."""+ transport = LocalAudioTransport(output_sample_rate=24000)+ await transport.send_audio(+ np.full(24000, 1000, np.int16).tobytes(),+ "r1",+ )+ _play(transport, 3)+ await transport.send_audio(b"\x00\x00" * 10, "r2")++ self.assertEqual(+ transport.playout().model_dump(),+ {"item_id": "r2", "played_ms": 0, "first_played_at": None},+ )++ async def test_partial_block_is_zero_filled(self) -> None:+ """A block larger than what is queued plays the remainder as silence+ and counts only what was queued."""+ transport = LocalAudioTransport(output_sample_rate=24000)+ await transport.send_audio(np.full(600, 7, np.int16).tobytes(), "r1")++ out = _play(transport, 1)+ self.assertEqual(+ (+ int(out[0, 0]),+ int(out[599, 0]),+ int(out[600, 0]),+ transport.playout().played_ms,+ ),+ (7, 7, 0, 25),+ )++ async def test_input_queue_drops_oldest_when_full(self) -> None:+ """Capture never grows past ten seconds; the oldest chunk goes."""+ transport = LocalAudioTransport(input_sample_rate=16000, chunk_ms=100)+ transport._loop = asyncio.get_running_loop()+ for i in range(transport._max_queued + 3):+ transport._enqueue(AudioFrame(pcm=bytes([i % 256]) * 2))++ queued = []+ while not transport._in_queue.empty():+ queued.append(transport._in_queue.get_nowait().pcm[0])+ self.assertEqual(+ (len(queued), queued[0], queued[-1]),+ (transport._max_queued, 3, (transport._max_queued + 2) % 256),+ )++ async def test_playout_position_shape(self) -> None:+ """The position is a plain value object."""+ self.assertEqual(+ PlayoutPosition(+ item_id="x",+ played_ms=1,+ first_played_at=2.0,+ ).model_dump(),+ {"item_id": "x", "played_ms": 1, "first_played_at": 2.0},+ )