Build and run agents you can see, understand and trust.
Python agentchatbotlarge-language-modelsllmllm-agentmcpmulti-agentmulti-modalreact-agentrealtime-agentvoice-agentvoice-assistant
feat(realtime): support OpenAI Realtime API model (#2552)
DavdGaof17846da7018
diff --git a/README.md b/README.mdindex 7390bcb40..6b15d384b 100644--- a/README.md+++ b/README.md@@ -73,6 +73,7 @@ rather than constraining them with strict prompts and opinionated orchestrations## News<!-- BEGIN NEWS -->+- **[2026-09] `INTE`:** OpenAI Realtime API supported in `RealtimeAgent` — `gpt-realtime` models with barge-in, truncation and tool calls.- **[2026-09] `FEAT` `Experimental`:** Realtime voice agent supported — full-duplex speech-to-speech via `RealtimeAgent` on the DashScope Realtime API (Qwen-Omni, Qwen-Audio-3.0), with barge-in and tool calls. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/realtime)- **[2026-09] `FEAT`:** A2A protocol supported — chat with any remote A2A agent via `A2AAgent`. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/a2a) | [Docs](https://docs.agentscope.io/latest/en/building-blocks/a2a)- **[2026-08] `FEAT`:** Pipeline supported — run multiple agents by a fixed logic behind one event stream. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/pipeline) | [Docs](https://docs.agentscope.io/latest/en/building-blocks/pipeline/overview)@@ -82,7 +83,6 @@ rather than constraining them with strict prompts and opinionated orchestrations- **[2026-08] `FEAT`:** Channels supported — connect agents to IM platforms in agent service. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent_service) | [Docs](https://docs.agentscope.io/latest/en/deploy/channel/overview)- **[2026-08] `INTE`:** GitHub MCP Registry and ClawHub supported as built-in hubs. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent_service) | [Docs](https://docs.agentscope.io/latest/en/deploy/hub)- **[2026-08] `FEAT`:** MCP & Skill Hub supported — browse a hub, install into your library, add to a workspace. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent_service) | [Docs](https://docs.agentscope.io/latest/en/deploy/hub)-- **[2026-07] `INTE`:** Daytona-based workspace/sandbox supported. [Docs](https://docs.agentscope.io/latest/en/building-blocks/workspace)<!-- END NEWS -->[More news →](./docs/NEWS.md)diff --git a/README_zh.md b/README_zh.mdindex e8db1e4ad..594146db0 100644--- a/README_zh.md+++ b/README_zh.md@@ -72,6 +72,7 @@ AgentScope 的目标是充分发挥大模型的推理与工具调用能力,## 新闻<!-- BEGIN NEWS -->+- **[2026-09] `集成`:** `RealtimeAgent` 支持 OpenAI Realtime API —— 接入 `gpt-realtime` 系列模型,支持打断、截断与工具调用。- **[2026-09] `功能` `实验性`:** 支持实时语音智能体 —— 通过 `RealtimeAgent` 接入 DashScope Realtime API(Qwen-Omni、Qwen-Audio-3.0),全双工语音对话、打断与工具调用。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/realtime)- **[2026-09] `功能`:** 支持 A2A 协议 —— 通过 `A2AAgent` 与任意远端 A2A 智能体对话。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/a2a) | [文档](https://docs.agentscope.io/latest/zh/building-blocks/a2a)- **[2026-08] `功能`:** 支持流水线 —— 按照固化逻辑运行多智能体,并向外提供统一接口。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/pipeline) | [文档](https://docs.agentscope.io/latest/zh/building-blocks/pipeline/overview)@@ -81,7 +82,6 @@ AgentScope 的目标是充分发挥大模型的推理与工具调用能力,- **[2026-08] `功能`:** 支持消息频道 —— 将智能体接入即时通讯平台。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent_service) | [文档](https://docs.agentscope.io/latest/zh/deploy/channel/overview)- **[2026-08] `集成`:** 内置集成 GitHub MCP Registry 与 ClawHub。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent_service) | [文档](https://docs.agentscope.io/latest/zh/deploy/hub)- **[2026-08] `功能`:** 支持 MCP & Skill Hub —— 浏览 hub、安装到个人库、再添加到工作区。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent_service) | [文档](https://docs.agentscope.io/latest/zh/deploy/hub)-- **[2026-07] `集成`:** 集成 Daytona 工作区/沙箱实现。 [文档](https://docs.agentscope.io/latest/zh/building-blocks/workspace)<!-- END NEWS -->[更多新闻 →](./docs/NEWS_zh.md)diff --git a/docs/NEWS.md b/docs/NEWS.mdindex d1f6f9ea5..d22ffdba9 100644--- a/docs/NEWS.md+++ b/docs/NEWS.md@@ -2,6 +2,7 @@<!-- The first 10 items are automatically synced to README.md and README_zh.md via GitHub Actions. --><!-- To update news in READMEs, modify this file and push to trigger the workflow. -->+- **[2026-09] `INTE`:** OpenAI Realtime API supported in `RealtimeAgent` — `gpt-realtime` models with barge-in, truncation and tool calls.- **[2026-09] `FEAT` `Experimental`:** Realtime voice agent supported — full-duplex speech-to-speech via `RealtimeAgent` on the DashScope Realtime API (Qwen-Omni, Qwen-Audio-3.0), with barge-in and tool calls. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/realtime)- **[2026-09] `FEAT`:** A2A protocol supported — chat with any remote A2A agent via `A2AAgent`. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/a2a) | [Docs](https://docs.agentscope.io/latest/en/building-blocks/a2a)- **[2026-08] `FEAT`:** Pipeline supported — run multiple agents by a fixed logic behind one event stream. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/pipeline) | [Docs](https://docs.agentscope.io/latest/en/building-blocks/pipeline/overview)diff --git a/docs/NEWS_zh.md b/docs/NEWS_zh.mdindex 8f6153206..2444392c6 100644--- a/docs/NEWS_zh.md+++ b/docs/NEWS_zh.md@@ -2,6 +2,7 @@<!-- The first 10 items are automatically synced to README.md and README_zh.md via GitHub Actions. --><!-- To update news in READMEs, modify this file and push to trigger the workflow. -->+- **[2026-09] `集成`:** `RealtimeAgent` 支持 OpenAI Realtime API —— 接入 `gpt-realtime` 系列模型,支持打断、截断与工具调用。- **[2026-09] `功能` `实验性`:** 支持实时语音智能体 —— 通过 `RealtimeAgent` 接入 DashScope Realtime API(Qwen-Omni、Qwen-Audio-3.0),全双工语音对话、打断与工具调用。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/realtime)- **[2026-09] `功能`:** 支持 A2A 协议 —— 通过 `A2AAgent` 与任意远端 A2A 智能体对话。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/a2a) | [文档](https://docs.agentscope.io/latest/zh/building-blocks/a2a)- **[2026-08] `功能`:** 支持流水线 —— 按照固化逻辑运行多智能体,并向外提供统一接口。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/pipeline) | [文档](https://docs.agentscope.io/latest/zh/building-blocks/pipeline/overview)diff --git a/src/agentscope/credential/_openai.py b/src/agentscope/credential/_openai.pyindex ec746ceec..8470df358 100644--- a/src/agentscope/credential/_openai.py+++ b/src/agentscope/credential/_openai.py@@ -9,6 +9,7 @@ from ._base import CredentialBaseif TYPE_CHECKING:from ..embedding import EmbeddingModelBasefrom ..model import ChatModelBase+ from ..realtime import RealtimeModelBasefrom ..tts import TTSModelBase@@ -62,3 +63,10 @@ class OpenAICredential(CredentialBase):from ..tts import OpenAITTSModelreturn [OpenAITTSModel]++ @classmethod+ def get_realtime_model_classes(cls) -> list[Type["RealtimeModelBase"]]:+ """Return the OpenAI realtime model classes."""+ from ..realtime import OpenAIRealtimeModel++ return [OpenAIRealtimeModel]diff --git a/src/agentscope/realtime/__init__.py b/src/agentscope/realtime/__init__.pyindex 85ee2b267..bb5de85fc 100644--- a/src/agentscope/realtime/__init__.py+++ b/src/agentscope/realtime/__init__.py@@ -24,6 +24,7 @@ from ._events import (TranscriptDeltaEvent,)from ._model_card import RealtimeModelCard+from ._openai import OpenAIRealtimeModelfrom ._playout import PlayoutPositionfrom ._transport import (AudioFrame,@@ -41,6 +42,7 @@ __all__ = ["RealtimeModelBase","DashScopeRealtimeModel","DashScopeAudioRealtimeModel",+ "OpenAIRealtimeModel","XAIRealtimeModel","RealtimeModelCard","TruncationSupport",diff --git a/src/agentscope/realtime/_openai/__init__.py b/src/agentscope/realtime/_openai/__init__.pynew file mode 100644index 000000000..eea2c97e7--- /dev/null+++ b/src/agentscope/realtime/_openai/__init__.py@@ -0,0 +1,5 @@+# -*- coding: utf-8 -*-+"""The OpenAI realtime model."""+from ._model import OpenAIRealtimeModel++__all__ = ["OpenAIRealtimeModel"]diff --git a/src/agentscope/realtime/_openai/_model.py b/src/agentscope/realtime/_openai/_model.pynew file mode 100644index 000000000..fc9d21d78--- /dev/null+++ b/src/agentscope/realtime/_openai/_model.py@@ -0,0 +1,436 @@+# -*- coding: utf-8 -*-+"""The OpenAI realtime model."""+import asyncio+import base64+import json+from typing import Any, AsyncIterator, Literal++from pydantic import Field++from .. import _events as me+from .._base import (+ ModelDisconnectedError,+ RealtimeModelBase,+ TruncationSupport,+)+from .._model_card import RealtimeModelCard+from ..._logging import logger+from ...credential import OpenAICredential+from ...message import TextBlock, ToolCallBlock, ToolResultBlock++_OPENAI_BASE_URL = "https://api.openai.com/v1"+++class OpenAIRealtimeModel(RealtimeModelBase):+ """The OpenAI Realtime API over WebSocket.++ Speaks the GA protocol: a ``session`` of type ``realtime`` carrying the+ ``audio.input`` / ``audio.output`` config, and ``response.output_audio``+ frames. PCM is 24 kHz in both directions.+ """++ class Parameters(RealtimeModelBase.Parameters):+ """Tuneables surfaced to the UI."""++ voice: Literal[+ "alloy",+ "ash",+ "ballad",+ "cedar",+ "coral",+ "echo",+ "marin",+ "sage",+ "shimmer",+ "verse",+ ] = Field(default="marin", title="Voice")+ turn_detection: Literal["server_vad", "semantic_vad", "none"] = Field(+ default="server_vad",+ title="Turn Detection",+ description="``none`` hands endpointing to the caller.",+ )+ vad_threshold: float = Field(default=0.5, ge=0.0, le=1.0)+ vad_prefix_padding_ms: int = Field(default=300, ge=0)+ vad_silence_duration_ms: int = Field(default=500, ge=0)+ vad_eagerness: Literal["low", "medium", "high", "auto"] = Field(+ default="auto",+ title="Semantic VAD Eagerness",+ description="How soon ``semantic_vad`` decides the user is done.",+ )+ input_audio_transcription: str = Field(+ default="gpt-4o-mini-transcribe",+ title="Input Transcription Model",+ description="The ASR model transcribing the user's speech; an "+ "empty string turns transcription off.",+ )++ type = "openai_realtime"+ truncation = TruncationSupport.EXPLICIT+ supports_text_input = True++ def __init__(+ self,+ model_name: str,+ credential: OpenAICredential,+ parameters: "OpenAIRealtimeModel.Parameters | None" = None,+ model_card: RealtimeModelCard | None = None,+ ) -> None:+ """Initialize the OpenAI realtime model.++ Args:+ model_name (`str`):+ The model name, e.g. ``"gpt-realtime-2.1"``.+ credential (`OpenAICredential`):+ The OpenAI credential.+ parameters (`OpenAIRealtimeModel.Parameters | None`, optional):+ Provider parameters; defaults are used if omitted.+ model_card (`RealtimeModelCard | None`, optional):+ The model card, looked up by name if omitted.+ """+ super().__init__(model_name, credential, parameters, model_card)+ self.parameters: OpenAIRealtimeModel.Parameters+ self._ws: Any = None+ self._reader: asyncio.Task | None = None+ self._queue: asyncio.Queue[me.ModelEvent | None] = asyncio.Queue()+ self._response_id = ""+ self._item_id = ""++ # ------------------------------------------------------------------+ # Lifecycle+ # ------------------------------------------------------------------++ async def connect(+ self,+ instructions: str,+ tools: list[dict] | None = None,+ **kwargs: Any,+ ) -> None:+ """Open the WebSocket and send the session config."""+ import websockets++ if kwargs.get("turn_detection_disabled"):+ self.parameters = self.parameters.model_copy(+ update={"turn_detection": "none"},+ )++ credential: OpenAICredential = self.credential # type: ignore+ base_url = (credential.base_url or _OPENAI_BASE_URL).rstrip("/")+ headers = {+ "Authorization": f"Bearer "+ f"{credential.api_key.get_secret_value()}",+ }+ if credential.organization:+ headers["OpenAI-Organization"] = credential.organization++ # ``https`` -> ``wss``, ``http`` -> ``ws``.+ url = base_url.replace("http", "ws", 1)+ self._ws = await websockets.connect(+ f"{url}/realtime?model={self.model_name}",+ additional_headers=headers,+ )+ self._reader = asyncio.create_task(self._read(), name="openai-rt")+ await self._send(self._session_update(instructions, tools))++ async def close(self) -> None:+ """Stop reading and close the WebSocket."""+ if self._reader is not None:+ self._reader.cancel()+ await asyncio.gather(self._reader, return_exceptions=True)+ self._reader = None+ if self._ws is not None:+ await self._ws.close()+ self._ws = None++ async def events(self) -> AsyncIterator[me.ModelEvent]:+ """Yield events until the session ends."""+ while True:+ event = await self._queue.get()+ if event is None:+ return+ yield event++ # ------------------------------------------------------------------+ # Input+ # ------------------------------------------------------------------++ async def push_audio(self, pcm: bytes) -> None:+ """Append PCM16 mono audio at 24 kHz to the input buffer."""+ await self._send(+ {+ "type": "input_audio_buffer.append",+ "audio": base64.b64encode(pcm).decode("ascii"),+ },+ )++ async def push_text(self, text: str) -> None:+ """Send a text user turn and ask for the reply."""+ await self._send(+ {+ "type": "conversation.item.create",+ "item": {+ "type": "message",+ "role": "user",+ "content": [{"type": "input_text", "text": text}],+ },+ },+ )+ await self.request_response()++ async def push_tool_result(self, block: ToolResultBlock) -> None:+ """Send a ``function_call_output`` item."""+ output = (+ block.output+ if isinstance(block.output, str)+ else "".join(+ _.text for _ in block.output if isinstance(_, TextBlock)+ )+ )+ await self._send(+ {+ "type": "conversation.item.create",+ "item": {+ "type": "function_call_output",+ "call_id": block.id,+ "output": output,+ },+ },+ )++ async def commit_turn(self) -> None:+ """Commit the input buffer as one user turn."""+ await self._send({"type": "input_audio_buffer.commit"})++ # ------------------------------------------------------------------+ # Response control+ # ------------------------------------------------------------------++ async def request_response(self) -> None:+ """Ask for a reply, cancelling one already in flight first."""+ await self.cancel_response()+ await self._send({"type": "response.create"})++ async def cancel_response(self) -> None:+ """Cancel the reply in flight, if any."""+ if self._response_id:+ await self._send({"type": "response.cancel"})++ async def truncate(+ self,+ item_id: str,+ played_ms: int,+ played_text: str,+ ) -> None:+ """Cut the assistant item back to the audio the user heard. The+ server drops the transcript tail itself, so *played_text* is+ unused."""+ await self._send(+ {+ "type": "conversation.item.truncate",+ "item_id": item_id,+ "content_index": 0,+ "audio_end_ms": played_ms,+ },+ )++ # ------------------------------------------------------------------+ # Wire+ # ------------------------------------------------------------------++ def _session_update(+ self,+ instructions: str,+ tools: list[dict] | None,+ ) -> dict:+ """Build the ``session.update`` message."""+ p = self.parameters+ turn_detection: dict[str, Any] | None = None+ if p.turn_detection == "server_vad":+ turn_detection = {+ "type": "server_vad",+ "threshold": p.vad_threshold,+ "prefix_padding_ms": p.vad_prefix_padding_ms,+ "silence_duration_ms": p.vad_silence_duration_ms,+ }+ elif p.turn_detection == "semantic_vad":+ turn_detection = {+ "type": "semantic_vad",+ "eagerness": p.vad_eagerness,+ }++ audio_input: dict[str, Any] = {+ "format": {"type": "audio/pcm", "rate": self.input_sample_rate},+ "turn_detection": turn_detection,+ }+ if p.input_audio_transcription:+ audio_input["transcription"] = {+ "model": p.input_audio_transcription,+ }++ session: dict[str, Any] = {+ "type": "realtime",+ "instructions": instructions,+ # Audio output always carries its transcript; asking for both+ # modalities at once is rejected.+ "output_modalities": ["audio"],+ "audio": {+ "input": audio_input,+ "output": {+ "format": {+ "type": "audio/pcm",+ "rate": self.output_sample_rate,+ },+ "voice": p.voice,+ },+ },+ }+ if tools and self.card.supports_tools:+ # The toolkit's chat-style wrapper, flattened to the realtime+ # shape: ``type`` beside ``name``/``description``/``parameters``.+ session["tools"] = [+ {"type": "function", **t["function"]}+ for t in tools+ if "function" in t+ ]+ return {"type": "session.update", "session": session}++ async def _send(self, payload: dict) -> None:+ """Send one JSON frame.++ Raises:+ `ModelDisconnectedError`: If the provider has closed the+ session, e.g. after its idle timeout.+ """+ from websockets.exceptions import ConnectionClosed++ if self._ws is None:+ raise ModelDisconnectedError("Not connected.")+ try:+ await self._ws.send(json.dumps(payload, ensure_ascii=False))+ except ConnectionClosed as exc:+ self._ws = None+ raise ModelDisconnectedError(str(exc.rcvd or exc)) from exc++ async def _read(self) -> None:+ """Drain the WebSocket into the event queue until it closes."""+ try:+ async for raw in self._ws:+ if isinstance(raw, bytes):+ raw = raw.decode("utf-8")+ try:+ event = self._parse(json.loads(raw))+ except Exception: # noqa: BLE001+ logger.exception("OpenAIRealtimeModel: bad frame")+ continue+ if event is not None:+ self._queue.put_nowait(event)+ except Exception as exc: # noqa: BLE001+ logger.error("OpenAIRealtimeModel: connection lost: %s", exc)+ finally:+ self._queue.put_nowait(me.SessionEndedEvent(reason="closed"))+ self._queue.put_nowait(None)++ # pylint: disable=too-many-return-statements+ def _parse(self, data: dict) -> me.ModelEvent | None:+ """Translate one server frame into a model event.++ A response is named by its first output item, the one truncation+ and playout refer to; ``response.created`` carries no item yet.+ """+ kind = data.get("type", "")++ match kind:+ case "response.created":+ self._response_id = data.get("response", {}).get("id", "")+ self._item_id = ""+ return None++ case "response.output_item.added":+ if self._item_id:+ return None+ self._item_id = data.get("item", {}).get("id", "")+ return me.ResponseCreatedEvent(item_id=self._item_id)++ case "response.done":+ response = data.get("response", {})+ usage = response.get("usage") or {}+ event: me.ModelEvent = me.ResponseDoneEvent(+ item_id=self._item_id,+ input_tokens=usage.get("input_tokens") or 0,+ output_tokens=usage.get("output_tokens") or 0,+ )+ # ``cancelled`` and ``incomplete`` still delivered a reply;+ # only ``failed`` is an error.+ if response.get("status") == "failed":+ err = (response.get("status_details") or {}).get(+ "error",+ ) or {}+ event = me.ModelErrorEvent(+ code=err.get("code") or "response_failed",+ message=err.get("message", ""),+ )+ self._response_id = ""+ self._item_id = ""+ return event++ # The second name of each pair is the pre-GA one, still emitted+ # by some OpenAI-compatible deployments.+ case "response.output_audio.delta" | "response.audio.delta":+ delta = data.get("delta")+ if not delta:+ return None+ return me.AudioDeltaEvent(+ item_id=self._item_id,+ pcm=base64.b64decode(delta),+ sample_rate=self.output_sample_rate,+ )++ case (+ "response.output_audio_transcript.delta"+ | "response.audio_transcript.delta"+ ):+ delta = data.get("delta")+ if not delta:+ return None+ return me.TranscriptDeltaEvent(+ item_id=self._item_id,+ delta=delta,+ )++ case "response.function_call_arguments.done":+ return me.ToolCallEvent(+ item_id=self._item_id,+ tool_call=ToolCallBlock(+ id=data.get("call_id", ""),+ name=data.get("name", ""),+ input=data.get("arguments") or "",+ ),+ )++ case "conversation.item.input_audio_transcription.completed":+ return me.InputTranscriptionEvent(+ item_id=data.get("item_id", ""),+ text=data.get("transcript", ""),+ )++ case "input_audio_buffer.speech_started":+ return me.SpeechStartedEvent(+ item_id=data.get("item_id", ""),+ at_ms=data.get("audio_start_ms", 0),+ )++ case "input_audio_buffer.speech_stopped":+ return me.SpeechEndedEvent(+ item_id=data.get("item_id", ""),+ at_ms=data.get("audio_end_ms", 0),+ )++ case "error":+ err = data.get("error", {})+ return me.ModelErrorEvent(+ code=err.get("code") or err.get("type", ""),+ message=err.get("message", ""),+ )++ case _:+ logger.debug("OpenAIRealtimeModel: ignoring %s", kind)+ return Nonediff --git a/src/agentscope/realtime/_openai/_models/gpt-realtime-1.5.yaml b/src/agentscope/realtime/_openai/_models/gpt-realtime-1.5.yamlnew file mode 100644index 000000000..f1db38ce1--- /dev/null+++ b/src/agentscope/realtime/_openai/_models/gpt-realtime-1.5.yaml@@ -0,0 +1,19 @@+type: realtime_model+name: gpt-realtime-1.5+label: GPT Realtime 1.5+status: active++input_types:+ - audio/pcm+ - text/plain+output_types:+ - audio/pcm+ - text/plain++input_sample_rate: 24000+output_sample_rate: 24000++supports_tools: true++max_context_tokens: 32000+max_session_duration_s: 3600diff --git a/src/agentscope/realtime/_openai/_models/gpt-realtime-2.1-mini.yaml b/src/agentscope/realtime/_openai/_models/gpt-realtime-2.1-mini.yamlnew file mode 100644index 000000000..fc495be47--- /dev/null+++ b/src/agentscope/realtime/_openai/_models/gpt-realtime-2.1-mini.yaml@@ -0,0 +1,19 @@+type: realtime_model+name: gpt-realtime-2.1-mini+label: GPT Realtime 2.1 Mini+status: active++input_types:+ - audio/pcm+ - text/plain+output_types:+ - audio/pcm+ - text/plain++input_sample_rate: 24000+output_sample_rate: 24000++supports_tools: true++max_context_tokens: 128000+max_session_duration_s: 3600diff --git a/src/agentscope/realtime/_openai/_models/gpt-realtime-2.1.yaml b/src/agentscope/realtime/_openai/_models/gpt-realtime-2.1.yamlnew file mode 100644index 000000000..06e529a54--- /dev/null+++ b/src/agentscope/realtime/_openai/_models/gpt-realtime-2.1.yaml@@ -0,0 +1,19 @@+type: realtime_model+name: gpt-realtime-2.1+label: GPT Realtime 2.1+status: active++input_types:+ - audio/pcm+ - text/plain+output_types:+ - audio/pcm+ - text/plain++input_sample_rate: 24000+output_sample_rate: 24000++supports_tools: true++max_context_tokens: 128000+max_session_duration_s: 3600diff --git a/src/agentscope/realtime/_openai/_models/gpt-realtime-2.yaml b/src/agentscope/realtime/_openai/_models/gpt-realtime-2.yamlnew file mode 100644index 000000000..cf2ab3aad--- /dev/null+++ b/src/agentscope/realtime/_openai/_models/gpt-realtime-2.yaml@@ -0,0 +1,19 @@+type: realtime_model+name: gpt-realtime-2+label: GPT Realtime 2+status: active++input_types:+ - audio/pcm+ - text/plain+output_types:+ - audio/pcm+ - text/plain++input_sample_rate: 24000+output_sample_rate: 24000++supports_tools: true++max_context_tokens: 128000+max_session_duration_s: 3600diff --git a/tests/realtime_openai_test.py b/tests/realtime_openai_test.pynew file mode 100644index 000000000..f0c52e1cc--- /dev/null+++ b/tests/realtime_openai_test.py@@ -0,0 +1,554 @@+# -*- coding: utf-8 -*-+"""Unit tests for the OpenAI realtime adapter: cards, session config and+frame parsing. Nothing here opens a connection."""+# pylint: disable=protected-access+import base64+import unittest+from unittest.async_case import IsolatedAsyncioTestCase+from utils import AnyString++from agentscope.credential import OpenAICredential+from agentscope.realtime import (+ ModelDisconnectedError,+ OpenAIRealtimeModel,+ TruncationSupport,+)+from agentscope.realtime import _events as me+from agentscope.message import ToolResultBlock++CRED = OpenAICredential(api_key="sk-x")+TRANSCRIPTION_DONE = "conversation.item.input_audio_transcription.completed"+++class OpenAICardsTest(unittest.TestCase):+ """The shipped cards and the credential lookup that finds them."""++ def test_cards(self) -> None:+ """Every card is tagged with the adapter type and its limits."""+ self.assertListEqual(+ [+ (+ c.name,+ c.model_type,+ c.supports_tools,+ c.max_context_tokens,+ c.input_sample_rate,+ c.output_sample_rate,+ )+ for c in OpenAIRealtimeModel.list_models()+ ],+ [+ (+ "gpt-realtime-1.5",+ "openai_realtime",+ True,+ 32000,+ 24000,+ 24000,+ ),+ (+ "gpt-realtime-2.1-mini",+ "openai_realtime",+ True,+ 128000,+ 24000,+ 24000,+ ),+ (+ "gpt-realtime-2.1",+ "openai_realtime",+ True,+ 128000,+ 24000,+ 24000,+ ),+ (+ "gpt-realtime-2",+ "openai_realtime",+ True,+ 128000,+ 24000,+ 24000,+ ),+ ],+ )++ def test_credential_maps_card_back_to_class(self) -> None:+ """The service-layer lookup: card.model_type -> class, no scan."""+ classes = {c.type: c for c in CRED.get_realtime_model_classes()}+ self.assertDictEqual(+ {+ card.name: classes[card.model_type].__name__+ for card in CRED.list_realtime_models()+ },+ {+ "gpt-realtime-1.5": "OpenAIRealtimeModel",+ "gpt-realtime-2": "OpenAIRealtimeModel",+ "gpt-realtime-2.1": "OpenAIRealtimeModel",+ "gpt-realtime-2.1-mini": "OpenAIRealtimeModel",+ },+ )++ def test_unknown_model_name_is_rejected(self) -> None:+ """A name with no card fails at construction."""+ with self.assertRaises(ValueError):+ OpenAIRealtimeModel("no-such-model", CRED)++ def test_adapter_facts(self) -> None:+ """Protocol facts are constant across the OpenAI models."""+ model = OpenAIRealtimeModel("gpt-realtime-2.1", CRED)+ self.assertListEqual(+ [+ model.type,+ model.truncation,+ model.supports_text_input,+ model.input_sample_rate,+ model.output_sample_rate,+ ],+ [+ "openai_realtime",+ TruncationSupport.EXPLICIT,+ True,+ 24000,+ 24000,+ ],+ )+++class OpenAISessionUpdateTest(unittest.TestCase):+ """The GA session.update payload sent on connect."""++ def test_server_vad_payload(self) -> None:+ """Server VAD, transcription and tools, in the GA session shape;+ the toolkit's chat-style tool wrapper is flattened."""+ model = OpenAIRealtimeModel("gpt-realtime-2.1", CRED)+ tools = [+ {+ "type": "function",+ "function": {+ "name": "get_weather",+ "description": "Weather by city.",+ "parameters": {"type": "object", "properties": {}},+ },+ },+ ]+ self.assertDictEqual(+ model._session_update("be nice", tools),+ {+ "type": "session.update",+ "session": {+ "type": "realtime",+ "instructions": "be nice",+ "output_modalities": ["audio"],+ "audio": {+ "input": {+ "format": {+ "type": "audio/pcm",+ "rate": 24000,+ },+ "turn_detection": {+ "type": "server_vad",+ "threshold": 0.5,+ "prefix_padding_ms": 300,+ "silence_duration_ms": 500,+ },+ "transcription": {+ "model": "gpt-4o-mini-transcribe",+ },+ },+ "output": {+ "format": {+ "type": "audio/pcm",+ "rate": 24000,+ },+ "voice": "marin",+ },+ },+ "tools": [+ {+ "type": "function",+ "name": "get_weather",+ "description": "Weather by city.",+ "parameters": {+ "type": "object",+ "properties": {},+ },+ },+ ],+ },+ },+ )++ def test_semantic_vad_payload_without_transcription(self) -> None:+ """Semantic VAD takes an eagerness, not a threshold; an empty+ transcription model drops the block entirely."""+ model = OpenAIRealtimeModel(+ "gpt-realtime-1.5",+ CRED,+ parameters=OpenAIRealtimeModel.Parameters(+ voice="cedar",+ turn_detection="semantic_vad",+ vad_eagerness="low",+ input_audio_transcription="",+ ),+ )+ self.assertDictEqual(+ model._session_update("be nice", None),+ {+ "type": "session.update",+ "session": {+ "type": "realtime",+ "instructions": "be nice",+ "output_modalities": ["audio"],+ "audio": {+ "input": {+ "format": {+ "type": "audio/pcm",+ "rate": 24000,+ },+ "turn_detection": {+ "type": "semantic_vad",+ "eagerness": "low",+ },+ },+ "output": {+ "format": {+ "type": "audio/pcm",+ "rate": 24000,+ },+ "voice": "cedar",+ },+ },+ },+ },+ )++ def test_turn_detection_none_hands_endpointing_to_caller(self) -> None:+ """``none`` sends null so the caller commits its own turns."""+ model = OpenAIRealtimeModel(+ "gpt-realtime-2.1",+ CRED,+ parameters=OpenAIRealtimeModel.Parameters(turn_detection="none"),+ )+ self.assertDictEqual(+ model._session_update("x", None),+ {+ "type": "session.update",+ "session": {+ "type": "realtime",+ "instructions": "x",+ "output_modalities": ["audio"],+ "audio": {+ "input": {+ "format": {"type": "audio/pcm", "rate": 24000},+ "turn_detection": None,+ "transcription": {+ "model": "gpt-4o-mini-transcribe",+ },+ },+ "output": {+ "format": {"type": "audio/pcm", "rate": 24000},+ "voice": "marin",+ },+ },+ },+ },+ )++ def test_failed_response_is_an_error(self) -> None:+ """A ``response.done`` with status ``failed`` is not a reply."""+ model = OpenAIRealtimeModel("gpt-realtime-2.1", CRED)+ model._parse({"type": "response.created", "response": {"id": "r"}})+ self.assertEqual(+ model._parse(+ {+ "type": "response.done",+ "response": {+ "id": "r",+ "status": "failed",+ "status_details": {+ "type": "failed",+ "error": {+ "code": "server_error",+ "message": "boom",+ },+ },+ },+ },+ ),+ me.ModelErrorEvent(code="server_error", message="boom"),+ )+++class OpenAIParseTest(unittest.TestCase):+ """Server frames -> model events."""++ def setUp(self) -> None:+ """Open a response so deltas have an item to attach to."""+ self.model = OpenAIRealtimeModel("gpt-realtime-2.1", CRED)++ def test_response_frames(self) -> None:+ """A whole turn: created, first item, deltas, usage, done."""+ frames = [+ {"type": "response.created", "response": {"id": "resp_1"}},+ {+ "type": "response.output_item.added",+ "response_id": "resp_1",+ "item": {"id": "item_1", "type": "message"},+ },+ {+ "type": "response.output_audio_transcript.delta",+ "item_id": "item_1",+ "delta": "hello",+ },+ {+ "type": "response.output_audio.delta",+ "item_id": "item_1",+ "delta": base64.b64encode(b"\x01\x00").decode(),+ },+ {+ "type": "response.done",+ "response": {+ "id": "resp_1",+ "usage": {"input_tokens": 10, "output_tokens": 5},+ },+ },+ {"type": "session.updated"},+ ]+ self.assertListEqual(+ [self.model._parse(f) for f in frames],+ [+ None,+ me.ResponseCreatedEvent(item_id="item_1"),+ me.TranscriptDeltaEvent(item_id="item_1", delta="hello"),+ me.AudioDeltaEvent(+ item_id="item_1",+ pcm=b"\x01\x00",+ sample_rate=24000,+ ),+ me.ResponseDoneEvent(+ item_id="item_1",+ input_tokens=10,+ output_tokens=5,+ ),+ None,+ ],+ )++ def test_pre_ga_audio_names_are_accepted(self) -> None:+ """OpenAI-compatible deployments may still send the beta names."""+ self.model._item_id = "item_1"+ self.assertListEqual(+ [+ self.model._parse(+ {"type": "response.audio_transcript.delta", "delta": "hi"},+ ),+ self.model._parse(+ {+ "type": "response.audio.delta",+ "delta": base64.b64encode(b"\x02\x00").decode(),+ },+ ),+ ],+ [+ me.TranscriptDeltaEvent(item_id="item_1", delta="hi"),+ me.AudioDeltaEvent(+ item_id="item_1",+ pcm=b"\x02\x00",+ sample_rate=24000,+ ),+ ],+ )++ def test_user_turn_frames(self) -> None:+ """The provider's VAD and the settled input transcript."""+ frames = [+ {+ "type": "input_audio_buffer.speech_started",+ "item_id": "user_1",+ "audio_start_ms": 120,+ },+ {+ "type": "input_audio_buffer.speech_stopped",+ "item_id": "user_1",+ "audio_end_ms": 980,+ },+ {+ "type": TRANSCRIPTION_DONE,+ "item_id": "user_1",+ "transcript": "what is the weather",+ },+ {+ "type": "error",+ "error": {+ "type": "invalid_request_error",+ "code": "invalid_value",+ "message": "boom",+ },+ },+ ]+ self.assertListEqual(+ [self.model._parse(f) for f in frames],+ [+ me.SpeechStartedEvent(item_id="user_1", at_ms=120),+ me.SpeechEndedEvent(item_id="user_1", at_ms=980),+ me.InputTranscriptionEvent(+ item_id="user_1",+ text="what is the weather",+ ),+ me.ModelErrorEvent(code="invalid_value", message="boom"),+ ],+ )++ def test_tool_call_frame(self) -> None:+ """The done frame carries the whole call, and it belongs to the+ response's first item, not the function call item."""+ self.model._item_id = "item_1"+ event = self.model._parse(+ {+ "type": "response.function_call_arguments.done",+ "item_id": "item_2",+ "call_id": "call_1",+ "name": "get_weather",+ "arguments": '{"city":"sh"}',+ },+ )+ self.assertDictEqual(+ event.model_dump(),+ {+ "item_id": "item_1",+ "tool_call": {+ "type": "tool_call",+ "id": "call_1",+ "name": "get_weather",+ "input": '{"city":"sh"}',+ "state": "pending",+ "suggested_rules": [],+ "created_at": AnyString(),+ "finished_at": None,+ },+ },+ )+++class OpenAIWireTest(IsolatedAsyncioTestCase):+ """The client frames the adapter writes."""++ def setUp(self) -> None:+ """Capture what would go on the wire."""+ self.model = OpenAIRealtimeModel("gpt-realtime-2.1", CRED)+ self.sent: list[dict] = []++ async def capture(payload: dict) -> None:+ """Record one frame instead of sending it."""+ self.sent.append(payload)++ self.model._send = capture # type: ignore[method-assign]++ async def test_audio_and_commit(self) -> None:+ """Audio is appended base64-encoded, then committed."""+ await self.model.push_audio(b"\x01\x00")+ await self.model.commit_turn()+ self.assertListEqual(+ self.sent,+ [+ {"type": "input_audio_buffer.append", "audio": "AQA="},+ {"type": "input_audio_buffer.commit"},+ ],+ )++ async def test_text_turn(self) -> None:+ """A text turn is one item plus a response request."""+ await self.model.push_text("hello")+ self.assertListEqual(+ self.sent,+ [+ {+ "type": "conversation.item.create",+ "item": {+ "type": "message",+ "role": "user",+ "content": [{"type": "input_text", "text": "hello"}],+ },+ },+ {"type": "response.create"},+ ],+ )++ async def test_tool_result(self) -> None:+ """A tool result is a ``function_call_output`` item."""+ await self.model.push_tool_result(+ ToolResultBlock(+ type="tool_result",+ id="call_1",+ name="get_weather",+ output="sunny",+ ),+ )+ self.assertListEqual(+ self.sent,+ [+ {+ "type": "conversation.item.create",+ "item": {+ "type": "function_call_output",+ "call_id": "call_1",+ "output": "sunny",+ },+ },+ ],+ )++ async def test_barge_in_truncates_then_cancels(self) -> None:+ """A barge-in rewrites the item to the audio heard and cancels+ the response; with none in flight the cancel is skipped."""+ await self.model.truncate("item_1", 1200, "hel")+ await self.model.cancel_response()+ self.model._response_id = "resp_1"+ await self.model.cancel_response()+ self.assertListEqual(+ self.sent,+ [+ {+ "type": "conversation.item.truncate",+ "item_id": "item_1",+ "content_index": 0,+ "audio_end_ms": 1200,+ },+ {"type": "response.cancel"},+ ],+ )+++class OpenAIDisconnectTest(IsolatedAsyncioTestCase):+ """A closed WebSocket surfaces as ModelDisconnectedError."""++ async def test_send_on_closed_socket(self) -> None:+ """websockets' ConnectionClosed becomes the realtime-level error+ and the socket reference is dropped."""+ from websockets.exceptions import ConnectionClosedError+ from websockets.frames import Close++ class ClosedSocket:+ """Raises like a socket the provider already closed."""++ async def send(self, _payload: str) -> None:+ """Fail with the provider's close frame."""+ close = Close(1000, "session expired")+ raise ConnectionClosedError(close, close, True)++ model = OpenAIRealtimeModel("gpt-realtime-2.1", CRED)+ model._ws = ClosedSocket()++ with self.assertRaises(ModelDisconnectedError) as ctx:+ await model.push_audio(b"\x00\x00")+ self.assertEqual(+ (str(ctx.exception), model._ws),+ ("1000 (OK) session expired", None),+ )++ async def test_send_before_connect(self) -> None:+ """No socket at all is the same condition."""+ model = OpenAIRealtimeModel("gpt-realtime-2.1", CRED)+ with self.assertRaises(ModelDisconnectedError):+ await model.commit_turn()