Skip to content

Build and run agents you can see, understand and trust.

feat(realtime): support OpenAI Realtime API model (#2552)

DavdGaof17846da7018
diff --git a/README.md b/README.md
index 7390bcb40..6b15d384b 100644
--- a/README.md
+++ b/README.md
@@ -73,6 +73,7 @@ rather than constraining them with strict prompts and opinionated orchestrations
## News
<!-- BEGIN NEWS -->
+- **[2026-09] `INTE`:** OpenAI Realtime API supported in `RealtimeAgent` — `gpt-realtime` models with barge-in, truncation and tool calls.
- **[2026-09] `FEAT` `Experimental`:** Realtime voice agent supported — full-duplex speech-to-speech via `RealtimeAgent` on the DashScope Realtime API (Qwen-Omni, Qwen-Audio-3.0), with barge-in and tool calls. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/realtime)
- **[2026-09] `FEAT`:** A2A protocol supported — chat with any remote A2A agent via `A2AAgent`. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/a2a) | [Docs](https://docs.agentscope.io/latest/en/building-blocks/a2a)
- **[2026-08] `FEAT`:** Pipeline supported — run multiple agents by a fixed logic behind one event stream. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/pipeline) | [Docs](https://docs.agentscope.io/latest/en/building-blocks/pipeline/overview)
@@ -82,7 +83,6 @@ rather than constraining them with strict prompts and opinionated orchestrations
- **[2026-08] `FEAT`:** Channels supported — connect agents to IM platforms in agent service. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent_service) | [Docs](https://docs.agentscope.io/latest/en/deploy/channel/overview)
- **[2026-08] `INTE`:** GitHub MCP Registry and ClawHub supported as built-in hubs. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent_service) | [Docs](https://docs.agentscope.io/latest/en/deploy/hub)
- **[2026-08] `FEAT`:** MCP & Skill Hub supported — browse a hub, install into your library, add to a workspace. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent_service) | [Docs](https://docs.agentscope.io/latest/en/deploy/hub)
-- **[2026-07] `INTE`:** Daytona-based workspace/sandbox supported. [Docs](https://docs.agentscope.io/latest/en/building-blocks/workspace)
<!-- END NEWS -->
[More news →](./docs/NEWS.md)
diff --git a/README_zh.md b/README_zh.md
index e8db1e4ad..594146db0 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -72,6 +72,7 @@ AgentScope 的目标是充分发挥大模型的推理与工具调用能力,
## 新闻
<!-- BEGIN NEWS -->
+- **[2026-09] `集成`:** `RealtimeAgent` 支持 OpenAI Realtime API —— 接入 `gpt-realtime` 系列模型,支持打断、截断与工具调用。
- **[2026-09] `功能` `实验性`:** 支持实时语音智能体 —— 通过 `RealtimeAgent` 接入 DashScope Realtime API(Qwen-Omni、Qwen-Audio-3.0),全双工语音对话、打断与工具调用。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/realtime)
- **[2026-09] `功能`:** 支持 A2A 协议 —— 通过 `A2AAgent` 与任意远端 A2A 智能体对话。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/a2a) | [文档](https://docs.agentscope.io/latest/zh/building-blocks/a2a)
- **[2026-08] `功能`:** 支持流水线 —— 按照固化逻辑运行多智能体,并向外提供统一接口。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/pipeline) | [文档](https://docs.agentscope.io/latest/zh/building-blocks/pipeline/overview)
@@ -81,7 +82,6 @@ AgentScope 的目标是充分发挥大模型的推理与工具调用能力,
- **[2026-08] `功能`:** 支持消息频道 —— 将智能体接入即时通讯平台。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent_service) | [文档](https://docs.agentscope.io/latest/zh/deploy/channel/overview)
- **[2026-08] `集成`:** 内置集成 GitHub MCP Registry 与 ClawHub。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent_service) | [文档](https://docs.agentscope.io/latest/zh/deploy/hub)
- **[2026-08] `功能`:** 支持 MCP & Skill Hub —— 浏览 hub、安装到个人库、再添加到工作区。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent_service) | [文档](https://docs.agentscope.io/latest/zh/deploy/hub)
-- **[2026-07] `集成`:** 集成 Daytona 工作区/沙箱实现。 [文档](https://docs.agentscope.io/latest/zh/building-blocks/workspace)
<!-- END NEWS -->
[更多新闻 →](./docs/NEWS_zh.md)
diff --git a/docs/NEWS.md b/docs/NEWS.md
index d1f6f9ea5..d22ffdba9 100644
--- a/docs/NEWS.md
+++ b/docs/NEWS.md
@@ -2,6 +2,7 @@
<!-- The first 10 items are automatically synced to README.md and README_zh.md via GitHub Actions. -->
<!-- To update news in READMEs, modify this file and push to trigger the workflow. -->
+- **[2026-09] `INTE`:** OpenAI Realtime API supported in `RealtimeAgent` — `gpt-realtime` models with barge-in, truncation and tool calls.
- **[2026-09] `FEAT` `Experimental`:** Realtime voice agent supported — full-duplex speech-to-speech via `RealtimeAgent` on the DashScope Realtime API (Qwen-Omni, Qwen-Audio-3.0), with barge-in and tool calls. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/realtime)
- **[2026-09] `FEAT`:** A2A protocol supported — chat with any remote A2A agent via `A2AAgent`. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/a2a) | [Docs](https://docs.agentscope.io/latest/en/building-blocks/a2a)
- **[2026-08] `FEAT`:** Pipeline supported — run multiple agents by a fixed logic behind one event stream. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/pipeline) | [Docs](https://docs.agentscope.io/latest/en/building-blocks/pipeline/overview)
diff --git a/docs/NEWS_zh.md b/docs/NEWS_zh.md
index 8f6153206..2444392c6 100644
--- a/docs/NEWS_zh.md
+++ b/docs/NEWS_zh.md
@@ -2,6 +2,7 @@
<!-- The first 10 items are automatically synced to README.md and README_zh.md via GitHub Actions. -->
<!-- To update news in READMEs, modify this file and push to trigger the workflow. -->
+- **[2026-09] `集成`:** `RealtimeAgent` 支持 OpenAI Realtime API —— 接入 `gpt-realtime` 系列模型,支持打断、截断与工具调用。
- **[2026-09] `功能` `实验性`:** 支持实时语音智能体 —— 通过 `RealtimeAgent` 接入 DashScope Realtime API(Qwen-Omni、Qwen-Audio-3.0),全双工语音对话、打断与工具调用。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/realtime)
- **[2026-09] `功能`:** 支持 A2A 协议 —— 通过 `A2AAgent` 与任意远端 A2A 智能体对话。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/a2a) | [文档](https://docs.agentscope.io/latest/zh/building-blocks/a2a)
- **[2026-08] `功能`:** 支持流水线 —— 按照固化逻辑运行多智能体,并向外提供统一接口。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/pipeline) | [文档](https://docs.agentscope.io/latest/zh/building-blocks/pipeline/overview)
diff --git a/src/agentscope/credential/_openai.py b/src/agentscope/credential/_openai.py
index ec746ceec..8470df358 100644
--- a/src/agentscope/credential/_openai.py
+++ b/src/agentscope/credential/_openai.py
@@ -9,6 +9,7 @@ from ._base import CredentialBase
if TYPE_CHECKING:
from ..embedding import EmbeddingModelBase
from ..model import ChatModelBase
+ from ..realtime import RealtimeModelBase
from ..tts import TTSModelBase
@@ -62,3 +63,10 @@ class OpenAICredential(CredentialBase):
from ..tts import OpenAITTSModel
return [OpenAITTSModel]
+
+ @classmethod
+ def get_realtime_model_classes(cls) -> list[Type["RealtimeModelBase"]]:
+ """Return the OpenAI realtime model classes."""
+ from ..realtime import OpenAIRealtimeModel
+
+ return [OpenAIRealtimeModel]
diff --git a/src/agentscope/realtime/__init__.py b/src/agentscope/realtime/__init__.py
index 85ee2b267..bb5de85fc 100644
--- a/src/agentscope/realtime/__init__.py
+++ b/src/agentscope/realtime/__init__.py
@@ -24,6 +24,7 @@ from ._events import (
TranscriptDeltaEvent,
)
from ._model_card import RealtimeModelCard
+from ._openai import OpenAIRealtimeModel
from ._playout import PlayoutPosition
from ._transport import (
AudioFrame,
@@ -41,6 +42,7 @@ __all__ = [
"RealtimeModelBase",
"DashScopeRealtimeModel",
"DashScopeAudioRealtimeModel",
+ "OpenAIRealtimeModel",
"XAIRealtimeModel",
"RealtimeModelCard",
"TruncationSupport",
diff --git a/src/agentscope/realtime/_openai/__init__.py b/src/agentscope/realtime/_openai/__init__.py
new file mode 100644
index 000000000..eea2c97e7
--- /dev/null
+++ b/src/agentscope/realtime/_openai/__init__.py
@@ -0,0 +1,5 @@
+# -*- coding: utf-8 -*-
+"""The OpenAI realtime model."""
+from ._model import OpenAIRealtimeModel
+
+__all__ = ["OpenAIRealtimeModel"]
diff --git a/src/agentscope/realtime/_openai/_model.py b/src/agentscope/realtime/_openai/_model.py
new file mode 100644
index 000000000..fc9d21d78
--- /dev/null
+++ b/src/agentscope/realtime/_openai/_model.py
@@ -0,0 +1,436 @@
+# -*- coding: utf-8 -*-
+"""The OpenAI realtime model."""
+import asyncio
+import base64
+import json
+from typing import Any, AsyncIterator, Literal
+
+from pydantic import Field
+
+from .. import _events as me
+from .._base import (
+ ModelDisconnectedError,
+ RealtimeModelBase,
+ TruncationSupport,
+)
+from .._model_card import RealtimeModelCard
+from ..._logging import logger
+from ...credential import OpenAICredential
+from ...message import TextBlock, ToolCallBlock, ToolResultBlock
+
+_OPENAI_BASE_URL = "https://api.openai.com/v1"
+
+
+class OpenAIRealtimeModel(RealtimeModelBase):
+ """The OpenAI Realtime API over WebSocket.
+
+ Speaks the GA protocol: a ``session`` of type ``realtime`` carrying the
+ ``audio.input`` / ``audio.output`` config, and ``response.output_audio``
+ frames. PCM is 24 kHz in both directions.
+ """
+
+ class Parameters(RealtimeModelBase.Parameters):
+ """Tuneables surfaced to the UI."""
+
+ voice: Literal[
+ "alloy",
+ "ash",
+ "ballad",
+ "cedar",
+ "coral",
+ "echo",
+ "marin",
+ "sage",
+ "shimmer",
+ "verse",
+ ] = Field(default="marin", title="Voice")
+ turn_detection: Literal["server_vad", "semantic_vad", "none"] = Field(
+ default="server_vad",
+ title="Turn Detection",
+ description="``none`` hands endpointing to the caller.",
+ )
+ vad_threshold: float = Field(default=0.5, ge=0.0, le=1.0)
+ vad_prefix_padding_ms: int = Field(default=300, ge=0)
+ vad_silence_duration_ms: int = Field(default=500, ge=0)
+ vad_eagerness: Literal["low", "medium", "high", "auto"] = Field(
+ default="auto",
+ title="Semantic VAD Eagerness",
+ description="How soon ``semantic_vad`` decides the user is done.",
+ )
+ input_audio_transcription: str = Field(
+ default="gpt-4o-mini-transcribe",
+ title="Input Transcription Model",
+ description="The ASR model transcribing the user's speech; an "
+ "empty string turns transcription off.",
+ )
+
+ type = "openai_realtime"
+ truncation = TruncationSupport.EXPLICIT
+ supports_text_input = True
+
+ def __init__(
+ self,
+ model_name: str,
+ credential: OpenAICredential,
+ parameters: "OpenAIRealtimeModel.Parameters | None" = None,
+ model_card: RealtimeModelCard | None = None,
+ ) -> None:
+ """Initialize the OpenAI realtime model.
+
+ Args:
+ model_name (`str`):
+ The model name, e.g. ``"gpt-realtime-2.1"``.
+ credential (`OpenAICredential`):
+ The OpenAI credential.
+ parameters (`OpenAIRealtimeModel.Parameters | None`, optional):
+ Provider parameters; defaults are used if omitted.
+ model_card (`RealtimeModelCard | None`, optional):
+ The model card, looked up by name if omitted.
+ """
+ super().__init__(model_name, credential, parameters, model_card)
+ self.parameters: OpenAIRealtimeModel.Parameters
+ self._ws: Any = None
+ self._reader: asyncio.Task | None = None
+ self._queue: asyncio.Queue[me.ModelEvent | None] = asyncio.Queue()
+ self._response_id = ""
+ self._item_id = ""
+
+ # ------------------------------------------------------------------
+ # Lifecycle
+ # ------------------------------------------------------------------
+
+ async def connect(
+ self,
+ instructions: str,
+ tools: list[dict] | None = None,
+ **kwargs: Any,
+ ) -> None:
+ """Open the WebSocket and send the session config."""
+ import websockets
+
+ if kwargs.get("turn_detection_disabled"):
+ self.parameters = self.parameters.model_copy(
+ update={"turn_detection": "none"},
+ )
+
+ credential: OpenAICredential = self.credential # type: ignore
+ base_url = (credential.base_url or _OPENAI_BASE_URL).rstrip("/")
+ headers = {
+ "Authorization": f"Bearer "
+ f"{credential.api_key.get_secret_value()}",
+ }
+ if credential.organization:
+ headers["OpenAI-Organization"] = credential.organization
+
+ # ``https`` -> ``wss``, ``http`` -> ``ws``.
+ url = base_url.replace("http", "ws", 1)
+ self._ws = await websockets.connect(
+ f"{url}/realtime?model={self.model_name}",
+ additional_headers=headers,
+ )
+ self._reader = asyncio.create_task(self._read(), name="openai-rt")
+ await self._send(self._session_update(instructions, tools))
+
+ async def close(self) -> None:
+ """Stop reading and close the WebSocket."""
+ if self._reader is not None:
+ self._reader.cancel()
+ await asyncio.gather(self._reader, return_exceptions=True)
+ self._reader = None
+ if self._ws is not None:
+ await self._ws.close()
+ self._ws = None
+
+ async def events(self) -> AsyncIterator[me.ModelEvent]:
+ """Yield events until the session ends."""
+ while True:
+ event = await self._queue.get()
+ if event is None:
+ return
+ yield event
+
+ # ------------------------------------------------------------------
+ # Input
+ # ------------------------------------------------------------------
+
+ async def push_audio(self, pcm: bytes) -> None:
+ """Append PCM16 mono audio at 24 kHz to the input buffer."""
+ await self._send(
+ {
+ "type": "input_audio_buffer.append",
+ "audio": base64.b64encode(pcm).decode("ascii"),
+ },
+ )
+
+ async def push_text(self, text: str) -> None:
+ """Send a text user turn and ask for the reply."""
+ await self._send(
+ {
+ "type": "conversation.item.create",
+ "item": {
+ "type": "message",
+ "role": "user",
+ "content": [{"type": "input_text", "text": text}],
+ },
+ },
+ )
+ await self.request_response()
+
+ async def push_tool_result(self, block: ToolResultBlock) -> None:
+ """Send a ``function_call_output`` item."""
+ output = (
+ block.output
+ if isinstance(block.output, str)
+ else "".join(
+ _.text for _ in block.output if isinstance(_, TextBlock)
+ )
+ )
+ await self._send(
+ {
+ "type": "conversation.item.create",
+ "item": {
+ "type": "function_call_output",
+ "call_id": block.id,
+ "output": output,
+ },
+ },
+ )
+
+ async def commit_turn(self) -> None:
+ """Commit the input buffer as one user turn."""
+ await self._send({"type": "input_audio_buffer.commit"})
+
+ # ------------------------------------------------------------------
+ # Response control
+ # ------------------------------------------------------------------
+
+ async def request_response(self) -> None:
+ """Ask for a reply, cancelling one already in flight first."""
+ await self.cancel_response()
+ await self._send({"type": "response.create"})
+
+ async def cancel_response(self) -> None:
+ """Cancel the reply in flight, if any."""
+ if self._response_id:
+ await self._send({"type": "response.cancel"})
+
+ async def truncate(
+ self,
+ item_id: str,
+ played_ms: int,
+ played_text: str,
+ ) -> None:
+ """Cut the assistant item back to the audio the user heard. The
+ server drops the transcript tail itself, so *played_text* is
+ unused."""
+ await self._send(
+ {
+ "type": "conversation.item.truncate",
+ "item_id": item_id,
+ "content_index": 0,
+ "audio_end_ms": played_ms,
+ },
+ )
+
+ # ------------------------------------------------------------------
+ # Wire
+ # ------------------------------------------------------------------
+
+ def _session_update(
+ self,
+ instructions: str,
+ tools: list[dict] | None,
+ ) -> dict:
+ """Build the ``session.update`` message."""
+ p = self.parameters
+ turn_detection: dict[str, Any] | None = None
+ if p.turn_detection == "server_vad":
+ turn_detection = {
+ "type": "server_vad",
+ "threshold": p.vad_threshold,
+ "prefix_padding_ms": p.vad_prefix_padding_ms,
+ "silence_duration_ms": p.vad_silence_duration_ms,
+ }
+ elif p.turn_detection == "semantic_vad":
+ turn_detection = {
+ "type": "semantic_vad",
+ "eagerness": p.vad_eagerness,
+ }
+
+ audio_input: dict[str, Any] = {
+ "format": {"type": "audio/pcm", "rate": self.input_sample_rate},
+ "turn_detection": turn_detection,
+ }
+ if p.input_audio_transcription:
+ audio_input["transcription"] = {
+ "model": p.input_audio_transcription,
+ }
+
+ session: dict[str, Any] = {
+ "type": "realtime",
+ "instructions": instructions,
+ # Audio output always carries its transcript; asking for both
+ # modalities at once is rejected.
+ "output_modalities": ["audio"],
+ "audio": {
+ "input": audio_input,
+ "output": {
+ "format": {
+ "type": "audio/pcm",
+ "rate": self.output_sample_rate,
+ },
+ "voice": p.voice,
+ },
+ },
+ }
+ if tools and self.card.supports_tools:
+ # The toolkit's chat-style wrapper, flattened to the realtime
+ # shape: ``type`` beside ``name``/``description``/``parameters``.
+ session["tools"] = [
+ {"type": "function", **t["function"]}
+ for t in tools
+ if "function" in t
+ ]
+ return {"type": "session.update", "session": session}
+
+ async def _send(self, payload: dict) -> None:
+ """Send one JSON frame.
+
+ Raises:
+ `ModelDisconnectedError`: If the provider has closed the
+ session, e.g. after its idle timeout.
+ """
+ from websockets.exceptions import ConnectionClosed
+
+ if self._ws is None:
+ raise ModelDisconnectedError("Not connected.")
+ try:
+ await self._ws.send(json.dumps(payload, ensure_ascii=False))
+ except ConnectionClosed as exc:
+ self._ws = None
+ raise ModelDisconnectedError(str(exc.rcvd or exc)) from exc
+
+ async def _read(self) -> None:
+ """Drain the WebSocket into the event queue until it closes."""
+ try:
+ async for raw in self._ws:
+ if isinstance(raw, bytes):
+ raw = raw.decode("utf-8")
+ try:
+ event = self._parse(json.loads(raw))
+ except Exception: # noqa: BLE001
+ logger.exception("OpenAIRealtimeModel: bad frame")
+ continue
+ if event is not None:
+ self._queue.put_nowait(event)
+ except Exception as exc: # noqa: BLE001
+ logger.error("OpenAIRealtimeModel: connection lost: %s", exc)
+ finally:
+ self._queue.put_nowait(me.SessionEndedEvent(reason="closed"))
+ self._queue.put_nowait(None)
+
+ # pylint: disable=too-many-return-statements
+ def _parse(self, data: dict) -> me.ModelEvent | None:
+ """Translate one server frame into a model event.
+
+ A response is named by its first output item, the one truncation
+ and playout refer to; ``response.created`` carries no item yet.
+ """
+ kind = data.get("type", "")
+
+ match kind:
+ case "response.created":
+ self._response_id = data.get("response", {}).get("id", "")
+ self._item_id = ""
+ return None
+
+ case "response.output_item.added":
+ if self._item_id:
+ return None
+ self._item_id = data.get("item", {}).get("id", "")
+ return me.ResponseCreatedEvent(item_id=self._item_id)
+
+ case "response.done":
+ response = data.get("response", {})
+ usage = response.get("usage") or {}
+ event: me.ModelEvent = me.ResponseDoneEvent(
+ item_id=self._item_id,
+ input_tokens=usage.get("input_tokens") or 0,
+ output_tokens=usage.get("output_tokens") or 0,
+ )
+ # ``cancelled`` and ``incomplete`` still delivered a reply;
+ # only ``failed`` is an error.
+ if response.get("status") == "failed":
+ err = (response.get("status_details") or {}).get(
+ "error",
+ ) or {}
+ event = me.ModelErrorEvent(
+ code=err.get("code") or "response_failed",
+ message=err.get("message", ""),
+ )
+ self._response_id = ""
+ self._item_id = ""
+ return event
+
+ # The second name of each pair is the pre-GA one, still emitted
+ # by some OpenAI-compatible deployments.
+ case "response.output_audio.delta" | "response.audio.delta":
+ delta = data.get("delta")
+ if not delta:
+ return None
+ return me.AudioDeltaEvent(
+ item_id=self._item_id,
+ pcm=base64.b64decode(delta),
+ sample_rate=self.output_sample_rate,
+ )
+
+ case (
+ "response.output_audio_transcript.delta"
+ | "response.audio_transcript.delta"
+ ):
+ delta = data.get("delta")
+ if not delta:
+ return None
+ return me.TranscriptDeltaEvent(
+ item_id=self._item_id,
+ delta=delta,
+ )
+
+ case "response.function_call_arguments.done":
+ return me.ToolCallEvent(
+ item_id=self._item_id,
+ tool_call=ToolCallBlock(
+ id=data.get("call_id", ""),
+ name=data.get("name", ""),
+ input=data.get("arguments") or "",
+ ),
+ )
+
+ case "conversation.item.input_audio_transcription.completed":
+ return me.InputTranscriptionEvent(
+ item_id=data.get("item_id", ""),
+ text=data.get("transcript", ""),
+ )
+
+ case "input_audio_buffer.speech_started":
+ return me.SpeechStartedEvent(
+ item_id=data.get("item_id", ""),
+ at_ms=data.get("audio_start_ms", 0),
+ )
+
+ case "input_audio_buffer.speech_stopped":
+ return me.SpeechEndedEvent(
+ item_id=data.get("item_id", ""),
+ at_ms=data.get("audio_end_ms", 0),
+ )
+
+ case "error":
+ err = data.get("error", {})
+ return me.ModelErrorEvent(
+ code=err.get("code") or err.get("type", ""),
+ message=err.get("message", ""),
+ )
+
+ case _:
+ logger.debug("OpenAIRealtimeModel: ignoring %s", kind)
+ return None
diff --git a/src/agentscope/realtime/_openai/_models/gpt-realtime-1.5.yaml b/src/agentscope/realtime/_openai/_models/gpt-realtime-1.5.yaml
new file mode 100644
index 000000000..f1db38ce1
--- /dev/null
+++ b/src/agentscope/realtime/_openai/_models/gpt-realtime-1.5.yaml
@@ -0,0 +1,19 @@
+type: realtime_model
+name: gpt-realtime-1.5
+label: GPT Realtime 1.5
+status: active
+
+input_types:
+ - audio/pcm
+ - text/plain
+output_types:
+ - audio/pcm
+ - text/plain
+
+input_sample_rate: 24000
+output_sample_rate: 24000
+
+supports_tools: true
+
+max_context_tokens: 32000
+max_session_duration_s: 3600
diff --git a/src/agentscope/realtime/_openai/_models/gpt-realtime-2.1-mini.yaml b/src/agentscope/realtime/_openai/_models/gpt-realtime-2.1-mini.yaml
new file mode 100644
index 000000000..fc495be47
--- /dev/null
+++ b/src/agentscope/realtime/_openai/_models/gpt-realtime-2.1-mini.yaml
@@ -0,0 +1,19 @@
+type: realtime_model
+name: gpt-realtime-2.1-mini
+label: GPT Realtime 2.1 Mini
+status: active
+
+input_types:
+ - audio/pcm
+ - text/plain
+output_types:
+ - audio/pcm
+ - text/plain
+
+input_sample_rate: 24000
+output_sample_rate: 24000
+
+supports_tools: true
+
+max_context_tokens: 128000
+max_session_duration_s: 3600
diff --git a/src/agentscope/realtime/_openai/_models/gpt-realtime-2.1.yaml b/src/agentscope/realtime/_openai/_models/gpt-realtime-2.1.yaml
new file mode 100644
index 000000000..06e529a54
--- /dev/null
+++ b/src/agentscope/realtime/_openai/_models/gpt-realtime-2.1.yaml
@@ -0,0 +1,19 @@
+type: realtime_model
+name: gpt-realtime-2.1
+label: GPT Realtime 2.1
+status: active
+
+input_types:
+ - audio/pcm
+ - text/plain
+output_types:
+ - audio/pcm
+ - text/plain
+
+input_sample_rate: 24000
+output_sample_rate: 24000
+
+supports_tools: true
+
+max_context_tokens: 128000
+max_session_duration_s: 3600
diff --git a/src/agentscope/realtime/_openai/_models/gpt-realtime-2.yaml b/src/agentscope/realtime/_openai/_models/gpt-realtime-2.yaml
new file mode 100644
index 000000000..cf2ab3aad
--- /dev/null
+++ b/src/agentscope/realtime/_openai/_models/gpt-realtime-2.yaml
@@ -0,0 +1,19 @@
+type: realtime_model
+name: gpt-realtime-2
+label: GPT Realtime 2
+status: active
+
+input_types:
+ - audio/pcm
+ - text/plain
+output_types:
+ - audio/pcm
+ - text/plain
+
+input_sample_rate: 24000
+output_sample_rate: 24000
+
+supports_tools: true
+
+max_context_tokens: 128000
+max_session_duration_s: 3600
diff --git a/tests/realtime_openai_test.py b/tests/realtime_openai_test.py
new file mode 100644
index 000000000..f0c52e1cc
--- /dev/null
+++ b/tests/realtime_openai_test.py
@@ -0,0 +1,554 @@
+# -*- coding: utf-8 -*-
+"""Unit tests for the OpenAI realtime adapter: cards, session config and
+frame parsing. Nothing here opens a connection."""
+# pylint: disable=protected-access
+import base64
+import unittest
+from unittest.async_case import IsolatedAsyncioTestCase
+from utils import AnyString
+
+from agentscope.credential import OpenAICredential
+from agentscope.realtime import (
+ ModelDisconnectedError,
+ OpenAIRealtimeModel,
+ TruncationSupport,
+)
+from agentscope.realtime import _events as me
+from agentscope.message import ToolResultBlock
+
+CRED = OpenAICredential(api_key="sk-x")
+TRANSCRIPTION_DONE = "conversation.item.input_audio_transcription.completed"
+
+
+class OpenAICardsTest(unittest.TestCase):
+ """The shipped cards and the credential lookup that finds them."""
+
+ def test_cards(self) -> None:
+ """Every card is tagged with the adapter type and its limits."""
+ self.assertListEqual(
+ [
+ (
+ c.name,
+ c.model_type,
+ c.supports_tools,
+ c.max_context_tokens,
+ c.input_sample_rate,
+ c.output_sample_rate,
+ )
+ for c in OpenAIRealtimeModel.list_models()
+ ],
+ [
+ (
+ "gpt-realtime-1.5",
+ "openai_realtime",
+ True,
+ 32000,
+ 24000,
+ 24000,
+ ),
+ (
+ "gpt-realtime-2.1-mini",
+ "openai_realtime",
+ True,
+ 128000,
+ 24000,
+ 24000,
+ ),
+ (
+ "gpt-realtime-2.1",
+ "openai_realtime",
+ True,
+ 128000,
+ 24000,
+ 24000,
+ ),
+ (
+ "gpt-realtime-2",
+ "openai_realtime",
+ True,
+ 128000,
+ 24000,
+ 24000,
+ ),
+ ],
+ )
+
+ def test_credential_maps_card_back_to_class(self) -> None:
+ """The service-layer lookup: card.model_type -> class, no scan."""
+ classes = {c.type: c for c in CRED.get_realtime_model_classes()}
+ self.assertDictEqual(
+ {
+ card.name: classes[card.model_type].__name__
+ for card in CRED.list_realtime_models()
+ },
+ {
+ "gpt-realtime-1.5": "OpenAIRealtimeModel",
+ "gpt-realtime-2": "OpenAIRealtimeModel",
+ "gpt-realtime-2.1": "OpenAIRealtimeModel",
+ "gpt-realtime-2.1-mini": "OpenAIRealtimeModel",
+ },
+ )
+
+ def test_unknown_model_name_is_rejected(self) -> None:
+ """A name with no card fails at construction."""
+ with self.assertRaises(ValueError):
+ OpenAIRealtimeModel("no-such-model", CRED)
+
+ def test_adapter_facts(self) -> None:
+ """Protocol facts are constant across the OpenAI models."""
+ model = OpenAIRealtimeModel("gpt-realtime-2.1", CRED)
+ self.assertListEqual(
+ [
+ model.type,
+ model.truncation,
+ model.supports_text_input,
+ model.input_sample_rate,
+ model.output_sample_rate,
+ ],
+ [
+ "openai_realtime",
+ TruncationSupport.EXPLICIT,
+ True,
+ 24000,
+ 24000,
+ ],
+ )
+
+
+class OpenAISessionUpdateTest(unittest.TestCase):
+ """The GA session.update payload sent on connect."""
+
+ def test_server_vad_payload(self) -> None:
+ """Server VAD, transcription and tools, in the GA session shape;
+ the toolkit's chat-style tool wrapper is flattened."""
+ model = OpenAIRealtimeModel("gpt-realtime-2.1", CRED)
+ tools = [
+ {
+ "type": "function",
+ "function": {
+ "name": "get_weather",
+ "description": "Weather by city.",
+ "parameters": {"type": "object", "properties": {}},
+ },
+ },
+ ]
+ self.assertDictEqual(
+ model._session_update("be nice", tools),
+ {
+ "type": "session.update",
+ "session": {
+ "type": "realtime",
+ "instructions": "be nice",
+ "output_modalities": ["audio"],
+ "audio": {
+ "input": {
+ "format": {
+ "type": "audio/pcm",
+ "rate": 24000,
+ },
+ "turn_detection": {
+ "type": "server_vad",
+ "threshold": 0.5,
+ "prefix_padding_ms": 300,
+ "silence_duration_ms": 500,
+ },
+ "transcription": {
+ "model": "gpt-4o-mini-transcribe",
+ },
+ },
+ "output": {
+ "format": {
+ "type": "audio/pcm",
+ "rate": 24000,
+ },
+ "voice": "marin",
+ },
+ },
+ "tools": [
+ {
+ "type": "function",
+ "name": "get_weather",
+ "description": "Weather by city.",
+ "parameters": {
+ "type": "object",
+ "properties": {},
+ },
+ },
+ ],
+ },
+ },
+ )
+
+ def test_semantic_vad_payload_without_transcription(self) -> None:
+ """Semantic VAD takes an eagerness, not a threshold; an empty
+ transcription model drops the block entirely."""
+ model = OpenAIRealtimeModel(
+ "gpt-realtime-1.5",
+ CRED,
+ parameters=OpenAIRealtimeModel.Parameters(
+ voice="cedar",
+ turn_detection="semantic_vad",
+ vad_eagerness="low",
+ input_audio_transcription="",
+ ),
+ )
+ self.assertDictEqual(
+ model._session_update("be nice", None),
+ {
+ "type": "session.update",
+ "session": {
+ "type": "realtime",
+ "instructions": "be nice",
+ "output_modalities": ["audio"],
+ "audio": {
+ "input": {
+ "format": {
+ "type": "audio/pcm",
+ "rate": 24000,
+ },
+ "turn_detection": {
+ "type": "semantic_vad",
+ "eagerness": "low",
+ },
+ },
+ "output": {
+ "format": {
+ "type": "audio/pcm",
+ "rate": 24000,
+ },
+ "voice": "cedar",
+ },
+ },
+ },
+ },
+ )
+
+ def test_turn_detection_none_hands_endpointing_to_caller(self) -> None:
+ """``none`` sends null so the caller commits its own turns."""
+ model = OpenAIRealtimeModel(
+ "gpt-realtime-2.1",
+ CRED,
+ parameters=OpenAIRealtimeModel.Parameters(turn_detection="none"),
+ )
+ self.assertDictEqual(
+ model._session_update("x", None),
+ {
+ "type": "session.update",
+ "session": {
+ "type": "realtime",
+ "instructions": "x",
+ "output_modalities": ["audio"],
+ "audio": {
+ "input": {
+ "format": {"type": "audio/pcm", "rate": 24000},
+ "turn_detection": None,
+ "transcription": {
+ "model": "gpt-4o-mini-transcribe",
+ },
+ },
+ "output": {
+ "format": {"type": "audio/pcm", "rate": 24000},
+ "voice": "marin",
+ },
+ },
+ },
+ },
+ )
+
+ def test_failed_response_is_an_error(self) -> None:
+ """A ``response.done`` with status ``failed`` is not a reply."""
+ model = OpenAIRealtimeModel("gpt-realtime-2.1", CRED)
+ model._parse({"type": "response.created", "response": {"id": "r"}})
+ self.assertEqual(
+ model._parse(
+ {
+ "type": "response.done",
+ "response": {
+ "id": "r",
+ "status": "failed",
+ "status_details": {
+ "type": "failed",
+ "error": {
+ "code": "server_error",
+ "message": "boom",
+ },
+ },
+ },
+ },
+ ),
+ me.ModelErrorEvent(code="server_error", message="boom"),
+ )
+
+
+class OpenAIParseTest(unittest.TestCase):
+ """Server frames -> model events."""
+
+ def setUp(self) -> None:
+ """Open a response so deltas have an item to attach to."""
+ self.model = OpenAIRealtimeModel("gpt-realtime-2.1", CRED)
+
+ def test_response_frames(self) -> None:
+ """A whole turn: created, first item, deltas, usage, done."""
+ frames = [
+ {"type": "response.created", "response": {"id": "resp_1"}},
+ {
+ "type": "response.output_item.added",
+ "response_id": "resp_1",
+ "item": {"id": "item_1", "type": "message"},
+ },
+ {
+ "type": "response.output_audio_transcript.delta",
+ "item_id": "item_1",
+ "delta": "hello",
+ },
+ {
+ "type": "response.output_audio.delta",
+ "item_id": "item_1",
+ "delta": base64.b64encode(b"\x01\x00").decode(),
+ },
+ {
+ "type": "response.done",
+ "response": {
+ "id": "resp_1",
+ "usage": {"input_tokens": 10, "output_tokens": 5},
+ },
+ },
+ {"type": "session.updated"},
+ ]
+ self.assertListEqual(
+ [self.model._parse(f) for f in frames],
+ [
+ None,
+ me.ResponseCreatedEvent(item_id="item_1"),
+ me.TranscriptDeltaEvent(item_id="item_1", delta="hello"),
+ me.AudioDeltaEvent(
+ item_id="item_1",
+ pcm=b"\x01\x00",
+ sample_rate=24000,
+ ),
+ me.ResponseDoneEvent(
+ item_id="item_1",
+ input_tokens=10,
+ output_tokens=5,
+ ),
+ None,
+ ],
+ )
+
+ def test_pre_ga_audio_names_are_accepted(self) -> None:
+ """OpenAI-compatible deployments may still send the beta names."""
+ self.model._item_id = "item_1"
+ self.assertListEqual(
+ [
+ self.model._parse(
+ {"type": "response.audio_transcript.delta", "delta": "hi"},
+ ),
+ self.model._parse(
+ {
+ "type": "response.audio.delta",
+ "delta": base64.b64encode(b"\x02\x00").decode(),
+ },
+ ),
+ ],
+ [
+ me.TranscriptDeltaEvent(item_id="item_1", delta="hi"),
+ me.AudioDeltaEvent(
+ item_id="item_1",
+ pcm=b"\x02\x00",
+ sample_rate=24000,
+ ),
+ ],
+ )
+
+ def test_user_turn_frames(self) -> None:
+ """The provider's VAD and the settled input transcript."""
+ frames = [
+ {
+ "type": "input_audio_buffer.speech_started",
+ "item_id": "user_1",
+ "audio_start_ms": 120,
+ },
+ {
+ "type": "input_audio_buffer.speech_stopped",
+ "item_id": "user_1",
+ "audio_end_ms": 980,
+ },
+ {
+ "type": TRANSCRIPTION_DONE,
+ "item_id": "user_1",
+ "transcript": "what is the weather",
+ },
+ {
+ "type": "error",
+ "error": {
+ "type": "invalid_request_error",
+ "code": "invalid_value",
+ "message": "boom",
+ },
+ },
+ ]
+ self.assertListEqual(
+ [self.model._parse(f) for f in frames],
+ [
+ me.SpeechStartedEvent(item_id="user_1", at_ms=120),
+ me.SpeechEndedEvent(item_id="user_1", at_ms=980),
+ me.InputTranscriptionEvent(
+ item_id="user_1",
+ text="what is the weather",
+ ),
+ me.ModelErrorEvent(code="invalid_value", message="boom"),
+ ],
+ )
+
+ def test_tool_call_frame(self) -> None:
+ """The done frame carries the whole call, and it belongs to the
+ response's first item, not the function call item."""
+ self.model._item_id = "item_1"
+ event = self.model._parse(
+ {
+ "type": "response.function_call_arguments.done",
+ "item_id": "item_2",
+ "call_id": "call_1",
+ "name": "get_weather",
+ "arguments": '{"city":"sh"}',
+ },
+ )
+ self.assertDictEqual(
+ event.model_dump(),
+ {
+ "item_id": "item_1",
+ "tool_call": {
+ "type": "tool_call",
+ "id": "call_1",
+ "name": "get_weather",
+ "input": '{"city":"sh"}',
+ "state": "pending",
+ "suggested_rules": [],
+ "created_at": AnyString(),
+ "finished_at": None,
+ },
+ },
+ )
+
+
+class OpenAIWireTest(IsolatedAsyncioTestCase):
+ """The client frames the adapter writes."""
+
+ def setUp(self) -> None:
+ """Capture what would go on the wire."""
+ self.model = OpenAIRealtimeModel("gpt-realtime-2.1", CRED)
+ self.sent: list[dict] = []
+
+ async def capture(payload: dict) -> None:
+ """Record one frame instead of sending it."""
+ self.sent.append(payload)
+
+ self.model._send = capture # type: ignore[method-assign]
+
+ async def test_audio_and_commit(self) -> None:
+ """Audio is appended base64-encoded, then committed."""
+ await self.model.push_audio(b"\x01\x00")
+ await self.model.commit_turn()
+ self.assertListEqual(
+ self.sent,
+ [
+ {"type": "input_audio_buffer.append", "audio": "AQA="},
+ {"type": "input_audio_buffer.commit"},
+ ],
+ )
+
+ async def test_text_turn(self) -> None:
+ """A text turn is one item plus a response request."""
+ await self.model.push_text("hello")
+ self.assertListEqual(
+ self.sent,
+ [
+ {
+ "type": "conversation.item.create",
+ "item": {
+ "type": "message",
+ "role": "user",
+ "content": [{"type": "input_text", "text": "hello"}],
+ },
+ },
+ {"type": "response.create"},
+ ],
+ )
+
+ async def test_tool_result(self) -> None:
+ """A tool result is a ``function_call_output`` item."""
+ await self.model.push_tool_result(
+ ToolResultBlock(
+ type="tool_result",
+ id="call_1",
+ name="get_weather",
+ output="sunny",
+ ),
+ )
+ self.assertListEqual(
+ self.sent,
+ [
+ {
+ "type": "conversation.item.create",
+ "item": {
+ "type": "function_call_output",
+ "call_id": "call_1",
+ "output": "sunny",
+ },
+ },
+ ],
+ )
+
+ async def test_barge_in_truncates_then_cancels(self) -> None:
+ """A barge-in rewrites the item to the audio heard and cancels
+ the response; with none in flight the cancel is skipped."""
+ await self.model.truncate("item_1", 1200, "hel")
+ await self.model.cancel_response()
+ self.model._response_id = "resp_1"
+ await self.model.cancel_response()
+ self.assertListEqual(
+ self.sent,
+ [
+ {
+ "type": "conversation.item.truncate",
+ "item_id": "item_1",
+ "content_index": 0,
+ "audio_end_ms": 1200,
+ },
+ {"type": "response.cancel"},
+ ],
+ )
+
+
+class OpenAIDisconnectTest(IsolatedAsyncioTestCase):
+ """A closed WebSocket surfaces as ModelDisconnectedError."""
+
+ async def test_send_on_closed_socket(self) -> None:
+ """websockets' ConnectionClosed becomes the realtime-level error
+ and the socket reference is dropped."""
+ from websockets.exceptions import ConnectionClosedError
+ from websockets.frames import Close
+
+ class ClosedSocket:
+ """Raises like a socket the provider already closed."""
+
+ async def send(self, _payload: str) -> None:
+ """Fail with the provider's close frame."""
+ close = Close(1000, "session expired")
+ raise ConnectionClosedError(close, close, True)
+
+ model = OpenAIRealtimeModel("gpt-realtime-2.1", CRED)
+ model._ws = ClosedSocket()
+
+ with self.assertRaises(ModelDisconnectedError) as ctx:
+ await model.push_audio(b"\x00\x00")
+ self.assertEqual(
+ (str(ctx.exception), model._ws),
+ ("1000 (OK) session expired", None),
+ )
+
+ async def test_send_before_connect(self) -> None:
+ """No socket at all is the same condition."""
+ model = OpenAIRealtimeModel("gpt-realtime-2.1", CRED)
+ with self.assertRaises(ModelDisconnectedError):
+ await model.commit_turn()