Skip to content

Build and run agents you can see, understand and trust.

feat(realtime): support realtime voice agent in agentscope (#1056)

qbcede1d7ceefa8
diff --git a/README.md b/README.md
index 9e34c2154..72bace1c6 100644
--- a/README.md
+++ b/README.md
@@ -79,6 +79,7 @@ The AgentScope Ecosystem
## News
<!-- BEGIN NEWS -->
+- **[2026-02] `FEAT`:** Realtime Voice Agent support. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/realtime_voice_agent) | [Multi-Agent Realtime Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/workflows/multiagent_realtime) | [Tutorial](https://doc.agentscope.io/tutorial/task_realtime.html)
- **[2026-01] `COMM`:** Biweekly Meetings launched to share ecosystem updates and development plans - join us! [Details & Schedule](https://github.com/agentscope-ai/agentscope/discussions/1126)
- **[2026-01] `FEAT`:** Database support & memory compression in memory module. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/functionality/short_term_memory/memory_compression) | [Tutorial](https://doc.agentscope.io/tutorial/task_memory.html)
- **[2025-12] `INTG`:** A2A (Agent-to-Agent) protocol support. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/a2a_agent) | [Tutorial](https://doc.agentscope.io/tutorial/task_a2a.html)
@@ -88,7 +89,6 @@ The AgentScope Ecosystem
- **[2025-11] `INTG`:** Agentic RL via Trinity-RFT library. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/tuner/react_agent) | [Trinity-RFT](https://github.com/agentscope-ai/Trinity-RFT)
- **[2025-11] `INTG`:** ReMe for enhanced long-term memory. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/functionality/long_term_memory/reme)
- **[2025-11] `RELS`:** agentscope-samples repository launched and agentscope-runtime upgraded with Docker/K8s deployment and VNC-powered GUI sandboxes. [Samples](https://github.com/agentscope-ai/agentscope-samples) | [Runtime](https://github.com/agentscope-ai/agentscope-runtime)
-- **[2025-11] `DOCS`:** Contributing Guide is online - welcome to contribute! [Guide](./CONTRIBUTING.md)
<!-- END NEWS -->
[More news →](./docs/NEWS.md)
@@ -112,6 +112,7 @@ Welcome to join our community on
- [Example](#example)
- [Hello AgentScope!](#hello-agentscope)
- [Voice Agent](#voice-agent)
+ - [Realtime Voice Agent](#realtime-voice-agent)
- [Human-in-the-loop](#human-in-the-loop)
- [Flexible MCP Usage](#flexible-mcp-usage)
- [Agentic RL](#agentic-rl)
@@ -217,6 +218,16 @@ Create a voice-enabled ReAct agent that can understand and respond with speech,
https://github.com/user-attachments/assets/c5f05254-aff6-4375-90df-85e8da95d5da
+### Realtime Voice Agent
+
+Build a realtime voice agent with web interface that can interact with users via voice input and output.
+
+[Realtime chatbot](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/realtime_voice_agent) | [Realtime Multi-Agent Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/workflows/multiagent_realtime)
+
+https://github.com/user-attachments/assets/1b7b114b-e995-4586-9b3f-d3bb9fcd2558
+
+
+
### Human-in-the-loop
Support realtime interruption in ReActAgent: conversation can be interrupted via cancellation in realtime and resumed
@@ -331,6 +342,7 @@ asyncio.run(multi_agent_conversation())
- [Browser-use Agent](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/browser_agent)
- [Meta Planner Agent](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/meta_planner_agent)
- [A2A Agent](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/a2a_agent)
+- [Realtime Voice Agent](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/realtime_voice_agent)
### Game
@@ -341,6 +353,7 @@ asyncio.run(multi_agent_conversation())
- [Multi-agent Debate](https://github.com/agentscope-ai/agentscope/tree/main/examples/workflows/multiagent_debate)
- [Multi-agent Conversation](https://github.com/agentscope-ai/agentscope/tree/main/examples/workflows/multiagent_conversation)
- [Multi-agent Concurrent](https://github.com/agentscope-ai/agentscope/tree/main/examples/workflows/multiagent_concurrent)
+- [Multi-agent Realtime Conversation](https://github.com/agentscope-ai/agentscope/tree/main/examples/workflows/multiagent_realtime)
### Evaluation
diff --git a/README_zh.md b/README_zh.md
index 29f9585bd..10880c3cf 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -77,6 +77,7 @@ AgentScope 生态
## 📢 新闻
<!-- BEGIN NEWS -->
+- **[2026-02] `功能`:** 支持实时语音交互。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/realtime_voice_agent) | [多智能体实时交互](https://github.com/agentscope-ai/agentscope/tree/main/examples/workflows/multiagent_realtime) | [文档](https://doc.agentscope.io/tutorial/task_realtime.html)
- **[2026-01] `社区`:** AgentScope 双周会议启动,分享生态更新和开发计划 - 欢迎加入![详情与安排](https://github.com/agentscope-ai/agentscope/discussions/1126)
- **[2026-01] `功能`:** 记忆模块新增数据库支持和记忆压缩。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/functionality/short_term_memory/memory_compression) | [教程](https://doc.agentscope.io/tutorial/task_memory.html)
- **[2025-12] `集成`:** A2A(智能体间通信)协议支持。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/a2a_agent) | [教程](https://doc.agentscope.io/zh_CN/tutorial/task_a2a.html)
@@ -86,7 +87,6 @@ AgentScope 生态
- **[2025-11] `集成`:** 通过 Trinity-RFT 库实现智能体强化学习。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/tuner/react_agent) | [Trinity-RFT](https://github.com/agentscope-ai/Trinity-RFT)
- **[2025-11] `集成`:** ReMe 增强长期记忆。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/functionality/long_term_memory/reme)
- **[2025-11] `发布`:** agentscope-samples 样例库上线,agentscope-runtime 升级支持 Docker/K8s 部署和 VNC 图形沙盒。[样例库](https://github.com/agentscope-ai/agentscope-samples) | [Runtime](https://github.com/agentscope-ai/agentscope-runtime)
-- **[2025-11] `文档`:** 贡献指南上线 - 欢迎参与贡献![指南](./CONTRIBUTING_zh.md)
<!-- END NEWS -->
[更多新闻 →](./docs/NEWS_zh.md)
@@ -110,6 +110,7 @@ AgentScope 生态
- [样例](#%E6%A0%B7%E4%BE%8B)
- [Hello AgentScope!](#hello-agentscope)
- [语音智能体](#%E8%AF%AD%E9%9F%B3%E6%99%BA%E8%83%BD%E4%BD%93)
+ - [实时语音智能体](#%E5%AE%9E%E6%97%B6%E8%AF%AD%E9%9F%B3%E6%99%BA%E8%83%BD%E4%BD%93)
- [人机协作](#%E4%BA%BA%E6%9C%BA%E5%8D%8F%E4%BD%9C)
- [灵活的 MCP 控制](#%E7%81%B5%E6%B4%BB%E7%9A%84-mcp-%E6%8E%A7%E5%88%B6)
- [智能体强化学习](#%E6%99%BA%E8%83%BD%E4%BD%93%E5%BC%BA%E5%8C%96%E5%AD%A6%E4%B9%A0)
@@ -210,10 +211,19 @@ asyncio.run(main())
创建支持语音的 ReAct 智能体,能够理解语音并进行语音回复,还可以使用语音交互玩多智能体狼人杀游戏。
-
https://github.com/user-attachments/assets/559af387-fd6f-4f0c-b882-cd4778214801
+### 实时语音智能体
+
+使用 AgentScope 轻松构建实时交互的智能体应用,提供统一的事件接口和工具调用支持。
+
+[实时语音智能体](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/realtime_voice_agent) | [多智能体实时交互](https://github.com/agentscope-ai/agentscope/tree/main/examples/workflows/multiagent_realtime)
+
+https://github.com/user-attachments/assets/d9674ad5-f71d-43d5-a341-5bada318aee0
+
+
+
### 人机协作
在 ReActAgent 中支持实时打断:可以通过取消操作实时中断对话,并通过强大的记忆保留机制无缝恢复。
@@ -327,6 +337,7 @@ asyncio.run(multi_agent_conversation())
- [Browser-use 智能体](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/browser_agent)
- [Meta Planner 智能体](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/meta_planner_agent)
- [A2A 智能体](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/a2a_agent)
+- [实时语音交互智能体](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/realtime_voice_agent)
### 游戏
@@ -337,6 +348,7 @@ asyncio.run(multi_agent_conversation())
- [多智能体辩论](https://github.com/agentscope-ai/agentscope/tree/main/examples/workflows/multiagent_debate)
- [多智能体对话](https://github.com/agentscope-ai/agentscope/tree/main/examples/workflows/multiagent_conversation)
- [多智能体并发](https://github.com/agentscope-ai/agentscope/tree/main/examples/workflows/multiagent_concurrent)
+- [多智能体实时语音交互](https://github.com/agentscope-ai/agentscope/tree/main/examples/workflows/multiagent_realtime)
### 评估
diff --git a/docs/NEWS.md b/docs/NEWS.md
index f5c5d9921..8a8a287d6 100644
--- a/docs/NEWS.md
+++ b/docs/NEWS.md
@@ -2,6 +2,7 @@
<!-- The first 10 items are automatically synced to README.md and README_zh.md via GitHub Actions. -->
<!-- To update news in READMEs, modify this file and push to trigger the workflow. -->
+- **[2026-02] `FEAT`:** Realtime Voice Agent support. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/realtime_voice_agent) | [Multi-Agent Realtime Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/workflows/multiagent_realtime) | [Tutorial](https://doc.agentscope.io/tutorial/task_realtime.html)
- **[2026-01] `COMM`:** Biweekly Meetings launched to share ecosystem updates and development plans - join us! [Details & Schedule](https://github.com/agentscope-ai/agentscope/discussions/1126)
- **[2026-01] `FEAT`:** Database support & memory compression in memory module. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/functionality/short_term_memory/memory_compression) | [Tutorial](https://doc.agentscope.io/tutorial/task_memory.html)
- **[2025-12] `INTG`:** A2A (Agent-to-Agent) protocol support. [Example](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/a2a_agent) | [Tutorial](https://doc.agentscope.io/tutorial/task_a2a.html)
diff --git a/docs/NEWS_zh.md b/docs/NEWS_zh.md
index 7c0d85ffc..cff895b38 100644
--- a/docs/NEWS_zh.md
+++ b/docs/NEWS_zh.md
@@ -2,6 +2,7 @@
<!-- The first 10 items are automatically synced to README.md and README_zh.md via GitHub Actions. -->
<!-- To update news in READMEs, modify this file and push to trigger the workflow. -->
+- **[2026-02] `功能`:** 支持实时语音交互。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/realtime_voice_agent) | [多智能体实时交互](https://github.com/agentscope-ai/agentscope/tree/main/examples/workflows/multiagent_realtime) | [文档](https://doc.agentscope.io/tutorial/task_realtime.html)
- **[2026-01] `社区`:** AgentScope 双周会议启动,分享生态更新和开发计划 - 欢迎加入![详情与安排](https://github.com/agentscope-ai/agentscope/discussions/1126)
- **[2026-01] `功能`:** 记忆模块新增数据库支持和记忆压缩。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/functionality/short_term_memory/memory_compression) | [教程](https://doc.agentscope.io/tutorial/task_memory.html)
- **[2025-12] `集成`:** A2A(智能体间通信)协议支持。[样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/a2a_agent) | [教程](https://doc.agentscope.io/zh_CN/tutorial/task_a2a.html)
@@ -18,4 +19,4 @@
- **[2025-09] `发布`:** AgentScope Runtime 开源 - 支持沙盒化工具执行的高效智能体部署。[GitHub](https://github.com/agentscope-ai/agentscope-runtime)
- **[2025-09] `发布`:** AgentScope Studio 开源。[GitHub](https://github.com/agentscope-ai/agentscope-studio)
- **[2025-08] `文档`:** v1 版本教程上线。[教程](https://doc.agentscope.io/zh_CN/)
-- **[2025-08] `发布`:** AgentScope v1 发布 - 全面拥抱异步执行,诸多新功能和改进。[变更日志](https://github.com/agentscope-ai/agentscope/blob/main/docs/changelog.md)
+- **[2025-08] `发布`:** AgentScope v1 发布 - 全面拥抱异步执行,诸多新功能和改进。[变更日志](https://github.com/agentscope-ai/agentscope/blob/main/docs/changelog.md)
\ No newline at end of file
diff --git a/docs/tutorial/en/index.rst b/docs/tutorial/en/index.rst
index 332af8fa7..e31dab1bb 100644
--- a/docs/tutorial/en/index.rst
+++ b/docs/tutorial/en/index.rst
@@ -57,6 +57,7 @@ Welcome to AgentScope's documentation!
tutorial/task_state
tutorial/task_hook
tutorial/task_a2a
+ tutorial/task_realtime
.. toctree::
:maxdepth: 1
diff --git a/docs/tutorial/en/src/task_realtime.py b/docs/tutorial/en/src/task_realtime.py
new file mode 100644
index 000000000..318dbebda
--- /dev/null
+++ b/docs/tutorial/en/src/task_realtime.py
@@ -0,0 +1,496 @@
+# -*- coding: utf-8 -*-
+"""
+.. _realtime:
+
+Realtime Agent
+====================
+
+The **realtime** agent is designed to handle real-time interactions, such as
+voice conversations or live chat sessions.
+The realtime agent in AgentScope features:
+
+- Integration with OpenAI, DashScope, Gemini, and other realtime model APIs
+- Unified event interface to simplify interactions with different realtime models
+- Support for tool calling capabilities
+- Support for multi-agent interactions
+
+.. note:: The realtime agent is currently under active development. We welcome
+ community contributions, discussions, and feedback! If you're interested in
+ realtime agents, please join our discussion and development.
+
+"""
+
+import asyncio
+import os
+from agentscope.agent import RealtimeAgent
+from agentscope.realtime import (
+ DashScopeRealtimeModel,
+ OpenAIRealtimeModel,
+ GeminiRealtimeModel,
+)
+
+# %%
+# Creating Realtime Models
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# AgentScope currently supports the following realtime model APIs:
+#
+# .. list-table::
+# :header-rows: 1
+# :widths: 15 25 25 15 20
+#
+# * - Provider
+# - Class
+# - Supported Models
+# - Input Modalities
+# - Tool Support
+# * - DashScope
+# - ``DashScopeRealtimeModel``
+# - ``qwen3-omni-flash-realtime``
+# - Text, Audio, Image
+# - No
+# * - OpenAI
+# - ``OpenAIRealtimeModel``
+# - ``gpt-4o-realtime-preview``
+# - Text, Audio
+# - Yes
+# * - Gemini
+# - ``GeminiRealtimeModel``
+# - ``gemini-2.5-flash-native-audio-preview-09-2025``
+# - Text, Audio, Image
+# - Yes
+#
+#
+# Here are examples of initializing different realtime models:
+
+# DashScope realtime model
+dashscope_model = DashScopeRealtimeModel(
+ model_name="qwen3-omni-flash-realtime",
+ api_key=os.getenv("DASHSCOPE_API_KEY"),
+ voice="Cherry", # Options: "Cherry", "Serena", "Ethan", "Chelsie"
+ enable_input_audio_transcription=True,
+)
+
+# OpenAI realtime model
+openai_model = OpenAIRealtimeModel(
+ model_name="gpt-4o-realtime-preview",
+ api_key=os.getenv("OPENAI_API_KEY"),
+ voice="alloy", # Options: "alloy", "echo", "marin", "cedar"
+ enable_input_audio_transcription=True,
+)
+
+# Gemini realtime model
+gemini_model = GeminiRealtimeModel(
+ model_name="gemini-2.5-flash-native-audio-preview-09-2025",
+ api_key=os.getenv("GEMINI_API_KEY"),
+ voice="Puck", # Options: "Puck", "Charon", "Kore", "Fenrir"
+ enable_input_audio_transcription=True,
+)
+
+# %%
+#
+# The realtime model provides the following key methods:
+#
+# .. list-table::
+# :header-rows: 1
+# :widths: 30 70
+#
+# * - Method
+# - Description
+# * - ``connect(outgoing_queue, instructions, tools)``
+# - Establish WebSocket connection to the realtime model API
+# * - ``disconnect()``
+# - Close the WebSocket connection
+# * - ``send(data)``
+# - Send audio/text/image data to the realtime model for processing
+#
+# The ``outgoing_queue`` parameter in ``connect()`` is an asyncio queue used to
+# forward events from the realtime model to the outside (e.g., the agent or frontend).
+#
+#
+# Model Events Interface
+# -----------------------
+#
+# AgentScope provides a unified ``agentscope.realtime.ModelEvents`` interface to simplify
+# interactions with different realtime models. The following events are
+# supported:
+#
+# .. note:: The "session" in ModelEvents refers to the WebSocket connection
+# session between the realtime model and the model API, not the session
+# between the frontend and backend.
+#
+# .. list-table::
+# :header-rows: 1
+# :widths: 40 60
+#
+# * - Event
+# - Description
+# * - ``ModelEvents.ModelSessionCreatedEvent``
+# - Session is successfully created
+# * - ``ModelEvents.ModelSessionEndedEvent``
+# - Session has ended
+# * - ``ModelEvents.ModelResponseCreatedEvent``
+# - Model begins generating a response
+# * - ``ModelEvents.ModelResponseDoneEvent``
+# - Model finished generating a response
+# * - ``ModelEvents.ModelResponseAudioDeltaEvent``
+# - Streaming audio data chunk from the model
+# * - ``ModelEvents.ModelResponseAudioDoneEvent``
+# - Audio response is complete
+# * - ``ModelEvents.ModelResponseAudioTranscriptDeltaEvent``
+# - Streaming transcription chunk of audio response
+# * - ``ModelEvents.ModelResponseAudioTranscriptDoneEvent``
+# - Audio transcription is complete
+# * - ``ModelEvents.ModelResponseToolUseDeltaEvent``
+# - Streaming tool call parameters
+# * - ``ModelEvents.ModelResponseToolUseDoneEvent``
+# - Tool call parameters are complete
+# * - ``ModelEvents.ModelInputTranscriptionDeltaEvent``
+# - Streaming transcription chunk of user input
+# * - ``ModelEvents.ModelInputTranscriptionDoneEvent``
+# - User input transcription is complete
+# * - ``ModelEvents.ModelInputStartedEvent``
+# - Detected start of user audio input (VAD)
+# * - ``ModelEvents.ModelInputDoneEvent``
+# - Detected end of user audio input (VAD)
+# * - ``ModelEvents.ModelErrorEvent``
+# - An error occurred
+#
+#
+#
+# Creating a Realtime Agent
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# The ``RealtimeAgent`` serves as a bridge layer that:
+#
+# - Converts ``ModelEvents`` from realtime models into ``ServerEvents`` for
+# frontend and other agents
+# - Receives ``ClientEvents`` from frontend or other agents and forwards them
+# to the realtime model API
+# - Manages the agent's lifecycle and event queues
+#
+# Server and Client Events
+# -------------------------
+#
+# AgentScope provides unified ``ServerEvents`` and ``ClientEvents`` for
+# communication between backend and frontend:
+#
+# **ServerEvents** (Backend → Frontend):
+#
+# .. list-table::
+# :header-rows: 1
+# :widths: 40 60
+#
+# * - Event
+# - Description
+# * - ``ServerEvents.ServerSessionCreatedEvent``
+# - Session created in backend
+# * - ``ServerEvents.ServerSessionUpdatedEvent``
+# - Session updated in backend
+# * - ``ServerEvents.ServerSessionEndedEvent``
+# - Session ended in backend
+# * - ``ServerEvents.AgentReadyEvent``
+# - Agent is ready to receive inputs
+# * - ``ServerEvents.AgentEndedEvent``
+# - Agent has ended
+# * - ``ServerEvents.AgentResponseCreatedEvent``
+# - Agent starts generating response
+# * - ``ServerEvents.AgentResponseDoneEvent``
+# - Agent finished generating response
+# * - ``ServerEvents.AgentResponseAudioDeltaEvent``
+# - Streaming audio chunk from agent
+# * - ``ServerEvents.AgentResponseAudioDoneEvent``
+# - Audio response complete
+# * - ``ServerEvents.AgentResponseAudioTranscriptDeltaEvent``
+# - Streaming transcription of agent response
+# * - ``ServerEvents.AgentResponseAudioTranscriptDoneEvent``
+# - Transcription complete
+# * - ``ServerEvents.AgentResponseToolUseDeltaEvent``
+# - Streaming tool call data
+# * - ``ServerEvents.AgentResponseToolUseDoneEvent``
+# - Tool call complete
+# * - ``ServerEvents.AgentResponseToolResultEvent``
+# - Tool execution result
+# * - ``ServerEvents.AgentInputTranscriptionDeltaEvent``
+# - Streaming transcription of user input
+# * - ``ServerEvents.AgentInputTranscriptionDoneEvent``
+# - Input transcription complete
+# * - ``ServerEvents.AgentInputStartedEvent``
+# - User audio input started
+# * - ``ServerEvents.AgentInputDoneEvent``
+# - User audio input ended
+# * - ``ServerEvents.AgentErrorEvent``
+# - An error occurred
+#
+# **ClientEvents** (Frontend → Backend):
+#
+# .. list-table::
+# :header-rows: 1
+# :widths: 40 60
+#
+# * - Event
+# - Description
+# * - ``ClientEvents.ClientSessionCreateEvent``
+# - Create a new session with specified configuration
+# * - ``ClientEvents.ClientSessionEndEvent``
+# - End current session
+# * - ``ClientEvents.ClientResponseCreateEvent``
+# - Request agent to generate response immediately
+# * - ``ClientEvents.ClientResponseCancelEvent``
+# - Interrupt agent's current response
+# * - ``ClientEvents.ClientTextAppendEvent``
+# - Append text input
+# * - ``ClientEvents.ClientAudioAppendEvent``
+# - Append audio input
+# * - ``ClientEvents.ClientAudioCommitEvent``
+# - Commit audio input (signal end of input)
+# * - ``ClientEvents.ClientImageAppendEvent``
+# - Append image input
+# * - ``ClientEvents.ClientToolResultEvent``
+# - Send tool execution result
+#
+# Initializing a Realtime Agent
+# ------------------------------
+#
+# Here's how to create and use a realtime agent:
+
+
+async def example_realtime_agent() -> None:
+ """Example of creating and using a realtime agent."""
+ agent = RealtimeAgent(
+ name="Friday",
+ sys_prompt="You are a helpful assistant named Friday.",
+ model=DashScopeRealtimeModel(
+ model_name="qwen3-omni-flash-realtime",
+ api_key=os.getenv("DASHSCOPE_API_KEY"),
+ ),
+ )
+
+ # Create a queue to receive messages from the agent
+ outgoing_queue = asyncio.Queue()
+
+ # The agent is now ready to handle inputs
+ # Handle outgoing messages in a separate task
+ async def handle_agent_messages():
+ while True:
+ event = await outgoing_queue.get()
+ # Process the event (e.g., send to frontend via WebSocket)
+ print(f"Agent event: {event.type}")
+
+ # Start the message handling task
+ asyncio.create_task(handle_agent_messages())
+
+ # Start the agent (establishes connection)
+ await agent.start(outgoing_queue)
+
+ # Stop the agent when done
+ await agent.stop()
+
+
+# %%
+# Starting Realtime Conversation
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+# Now we can set up a realtime conversation between a user and a realtime agent.
+#
+# Here we take FastAPI as an example backend framework to demonstrate how to set up
+# a realtime conversation.
+#
+# **Backend Setup (Server-side):**
+#
+# The backend needs to:
+#
+# 1. Create a WebSocket endpoint to accept frontend connections
+# 2. Create a ``RealtimeAgent`` when the session starts
+# 3. Forward ``ClientEvents`` from frontend to the agent
+# 4. Forward ``ServerEvents`` from agent to the frontend
+#
+# .. code-block:: python
+#
+# from fastapi import FastAPI, WebSocket
+# from agentscope.agent import RealtimeAgent
+# from agentscope.realtime import (
+# DashScopeRealtimeModel,
+# ClientEvents,
+# ServerEvents,
+# )
+#
+# app = FastAPI()
+#
+# @app.websocket("/ws/{user_id}/{session_id}")
+# async def websocket_endpoint(
+# websocket: WebSocket,
+# user_id: str,
+# session_id: str,
+# ):
+# await websocket.accept()
+#
+# # Create queue for agent messages
+# frontend_queue = asyncio.Queue()
+#
+# # Create agent
+# agent = RealtimeAgent(
+# name="Assistant",
+# sys_prompt="You are a helpful assistant.",
+# model=DashScopeRealtimeModel(
+# model_name="qwen3-omni-flash-realtime",
+# api_key=os.getenv("DASHSCOPE_API_KEY"),
+# ),
+# )
+#
+# # Start agent
+# await agent.start(frontend_queue)
+#
+# # Forward messages from agent to frontend
+# async def send_to_frontend():
+# while True:
+# msg = await frontend_queue.get()
+# await websocket.send_json(msg.model_dump())
+#
+# asyncio.create_task(send_to_frontend())
+#
+# # Receive messages from frontend and forward to agent
+# while True:
+# data = await websocket.receive_json()
+# client_event = ClientEvents.from_json(data)
+# await agent.handle_input(client_event)
+#
+# **Frontend Setup (Client-side):**
+#
+# The frontend needs to:
+#
+# 1. Establish WebSocket connection to the backend
+# 2. Send ``CLIENT_SESSION_CREATE`` event to initialize the session
+# 3. Capture audio from microphone and send via ``CLIENT_AUDIO_APPEND`` events
+# 4. Receive and handle ``ServerEvents`` (e.g., play audio, display transcripts)
+#
+# .. code-block:: javascript
+#
+# // Connect to WebSocket
+# const ws = new WebSocket('ws://localhost:8000/ws/user1/session1');
+#
+# ws.onopen = () => {
+# // Create session
+# ws.send(JSON.stringify({
+# type: 'client_session_create',
+# config: {
+# instructions: 'You are a helpful assistant.',
+# user_name: 'User1'
+# }
+# }));
+# };
+#
+# // Handle messages from backend
+# ws.onmessage = (event) => {
+# const data = JSON.parse(event.data);
+# if (data.type === 'response_audio_delta') {
+# // Play audio chunk
+# playAudio(data.delta);
+# }
+# };
+#
+# // Send audio data
+# function sendAudioChunk(audioData) {
+# ws.send(JSON.stringify({
+# type: 'client_audio_append',
+# session_id: 'session1',
+# audio: audioData, // base64 encoded
+# format: { encoding: 'pcm16', sample_rate: 16000 }
+# }));
+# }
+#
+# For a complete working example, see
+# ``examples/agent/realtime_voice_agent/`` in the AgentScope repository.
+
+# %%
+# Multi-Agent Realtime Conversation
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# AgentScope supports multi-agent realtime interactions through the ``ChatRoom``
+# class.
+#
+# Note currently most realtime model APIs only support single-user interactions,
+# but AgentScope's architecture is designed to support multiple agents and users
+# when API capabilities expand.
+#
+# The Realtime ChatRoom
+# ----------------------------
+#
+# AgentScope introduces the ``ChatRoom`` class to manage multiple realtime
+# agents in a shared conversation space. The ChatRoom provides:
+#
+# - Centralized management of multiple ``RealtimeAgent`` instances
+# - Automatic message broadcasting between agents
+# - Unified message queue for frontend communication
+# - Lifecycle management for all agents in the room
+#
+# Using ChatRoom
+# --------------
+#
+# The usage of ``ChatRoom`` is similar to ``RealtimeAgent``:
+#
+
+
+async def example_chat_room() -> None:
+ """Example of using ChatRoom with multiple realtime agents."""
+ from agentscope.pipeline import ChatRoom
+ from agentscope.agent import RealtimeAgent
+ from agentscope.realtime import DashScopeRealtimeModel
+
+ # Create multiple agents
+ agent1 = RealtimeAgent(
+ name="Agent1",
+ sys_prompt="You are Agent1, a helpful assistant.",
+ model=DashScopeRealtimeModel(
+ model_name="qwen3-omni-flash-realtime",
+ api_key=os.getenv("DASHSCOPE_API_KEY"),
+ ),
+ )
+
+ agent2 = RealtimeAgent(
+ name="Agent2",
+ sys_prompt="You are Agent2, a helpful assistant.",
+ model=DashScopeRealtimeModel(
+ model_name="qwen3-omni-flash-realtime",
+ api_key=os.getenv("DASHSCOPE_API_KEY"),
+ ),
+ )
+
+ # Create a chat room with multiple agents
+ chat_room = ChatRoom(agents=[agent1, agent2])
+
+ # Create queue to receive messages from all agents
+ outgoing_queue = asyncio.Queue()
+
+ # Start the chat room
+ await chat_room.start(outgoing_queue)
+
+ # Handle input from frontend
+ # The chat room will broadcast to all agents
+ from agentscope.realtime import ClientEvents
+
+ client_event = ClientEvents.ClientTextAppendEvent(
+ session_id="session1",
+ text="Hello everyone!",
+ )
+ await chat_room.handle_input(client_event)
+
+ # Stop the chat room when done
+ await chat_room.stop()
+
+
+# %%
+# Roadmap
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# The realtime agent feature is currently experimental and under active
+# development. Future plans include:
+#
+# - Support for more realtime model APIs
+# - Enhanced memory management for conversation history
+# - Comprehensive tool calling support across all providers
+# - Multi-user voice interaction support
+# - Improved VAD (Voice Activity Detection) configuration
+# - Better error handling and recovery mechanisms
+#
+# We welcome contributions and feedback from the community to help shape the
+# future of realtime agents in AgentScope!
diff --git a/docs/tutorial/zh_CN/index.rst b/docs/tutorial/zh_CN/index.rst
index 335377b15..251a9d22a 100644
--- a/docs/tutorial/zh_CN/index.rst
+++ b/docs/tutorial/zh_CN/index.rst
@@ -58,6 +58,7 @@ Welcome to AgentScope's documentation!
tutorial/task_state
tutorial/task_hook
tutorial/task_a2a
+ tutorial/task_realtime
.. toctree::
:maxdepth: 1
diff --git a/docs/tutorial/zh_CN/src/task_realtime.py b/docs/tutorial/zh_CN/src/task_realtime.py
new file mode 100644
index 000000000..ca1384c3a
--- /dev/null
+++ b/docs/tutorial/zh_CN/src/task_realtime.py
@@ -0,0 +1,483 @@
+# -*- coding: utf-8 -*-
+"""
+.. _realtime:
+
+实时智能体
+====================
+
+**实时智能体(Realtime Agent)** 用于处理实时交互场景,例如语音对话或实时聊天会话。
+AgentScope 中的实时智能体具有以下特性:
+
+- 集成 OpenAI、DashScope、Gemini 等实时模型 API
+- 统一的事件接口,简化与不同实时模型的交互
+- 支持工具调用能力
+- 支持多智能体交互
+
+.. note:: 实时智能体目前处于活跃开发阶段,欢迎社区贡献、讨论和反馈!如果开发者对实时智能体感兴趣,欢迎加入讨论和开发。
+
+"""
+
+import asyncio
+import os
+from agentscope.agent import RealtimeAgent
+from agentscope.realtime import (
+ DashScopeRealtimeModel,
+ OpenAIRealtimeModel,
+ GeminiRealtimeModel,
+)
+
+# %%
+# 创建实时模型
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# AgentScope 目前支持以下实时模型 API:
+#
+# .. list-table::
+# :header-rows: 1
+# :widths: 15 25 25 15 20
+#
+# * - 提供商
+# - 类名
+# - 支持的模型
+# - 输入模态
+# - 工具支持
+# * - DashScope
+# - ``DashScopeRealtimeModel``
+# - ``qwen3-omni-flash-realtime``
+# - 文本、音频、图像
+# - 否
+# * - OpenAI
+# - ``OpenAIRealtimeModel``
+# - ``gpt-4o-realtime-preview``
+# - 文本、音频
+# - 是
+# * - Gemini
+# - ``GeminiRealtimeModel``
+# - ``gemini-2.5-flash-native-audio-preview-09-2025``
+# - 文本、音频、图像
+# - 是
+#
+#
+# 以下是初始化不同实时模型的示例:
+
+# DashScope 实时模型
+dashscope_model = DashScopeRealtimeModel(
+ model_name="qwen3-omni-flash-realtime",
+ api_key=os.getenv("DASHSCOPE_API_KEY"),
+ voice="Cherry", # 可选项: "Cherry", "Serena", "Ethan", "Chelsie"
+ enable_input_audio_transcription=True,
+)
+
+# OpenAI 实时模型
+openai_model = OpenAIRealtimeModel(
+ model_name="gpt-4o-realtime-preview",
+ api_key=os.getenv("OPENAI_API_KEY"),
+ voice="alloy", # 可选项: "alloy", "echo", "marin", "cedar"
+ enable_input_audio_transcription=True,
+)
+
+# Gemini 实时模型
+gemini_model = GeminiRealtimeModel(
+ model_name="gemini-2.5-flash-native-audio-preview-09-2025",
+ api_key=os.getenv("GEMINI_API_KEY"),
+ voice="Puck", # 可选项: "Puck", "Charon", "Kore", "Fenrir"
+ enable_input_audio_transcription=True,
+)
+
+# %%
+#
+# 实时模型提供以下核心方法:
+#
+# .. list-table::
+# :header-rows: 1
+# :widths: 30 70
+#
+# * - 方法
+# - 描述
+# * - ``connect(outgoing_queue, instructions, tools)``
+# - 建立与实时模型 API 的 WebSocket 连接
+# * - ``disconnect()``
+# - 关闭 WebSocket 连接
+# * - ``send(data)``
+# - 向实时模型发送音频/文本/图像数据进行处理
+#
+# ``connect()`` 方法中的 ``outgoing_queue`` 参数是一个 asyncio 队列,
+# 用于将实时模型的事件转发到外部(例如智能体或前端)。
+#
+#
+# 模型事件接口
+# -----------------------
+#
+# AgentScope 提供统一的 ``agentscope.realtime.ModelEvents`` 接口,
+# 简化与不同实时模型的交互。支持以下事件:
+#
+# .. note:: ModelEvents 中的 "session" 指的是实时模型与模型 API 之间的
+# WebSocket 连接会话,而非前端与后端之间的会话。
+#
+# .. list-table::
+# :header-rows: 1
+# :widths: 40 60
+#
+# * - 事件
+# - 描述
+# * - ``ModelEvents.ModelSessionCreatedEvent``
+# - 会话创建成功
+# * - ``ModelEvents.ModelSessionEndedEvent``
+# - 会话已结束
+# * - ``ModelEvents.ModelResponseCreatedEvent``
+# - 模型开始生成响应
+# * - ``ModelEvents.ModelResponseDoneEvent``
+# - 模型完成响应生成
+# * - ``ModelEvents.ModelResponseAudioDeltaEvent``
+# - 流式音频数据块
+# * - ``ModelEvents.ModelResponseAudioDoneEvent``
+# - 音频响应完成
+# * - ``ModelEvents.ModelResponseAudioTranscriptDeltaEvent``
+# - 流式音频转录文本块
+# * - ``ModelEvents.ModelResponseAudioTranscriptDoneEvent``
+# - 音频转录完成
+# * - ``ModelEvents.ModelResponseToolUseDeltaEvent``
+# - 流式工具调用参数
+# * - ``ModelEvents.ModelResponseToolUseDoneEvent``
+# - 工具调用参数完成
+# * - ``ModelEvents.ModelInputTranscriptionDeltaEvent``
+# - 流式用户输入转录文本块
+# * - ``ModelEvents.ModelInputTranscriptionDoneEvent``
+# - 用户输入转录完成
+# * - ``ModelEvents.ModelInputStartedEvent``
+# - 检测到用户音频输入开始(VAD)
+# * - ``ModelEvents.ModelInputDoneEvent``
+# - 检测到用户音频输入结束(VAD)
+# * - ``ModelEvents.ModelErrorEvent``
+# - 发生错误
+#
+#
+#
+# 创建实时智能体
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# ``RealtimeAgent`` 作为桥接层,负责:
+#
+# - 将实时模型的 ``ModelEvents`` 转换为 ``ServerEvents``,发送给前端和其他智能体
+# - 接收来自前端或其他智能体的 ``ClientEvents``,并转发给实时模型 API
+# - 管理智能体的生命周期和事件队列
+#
+# 服务端和客户端事件
+# -------------------------
+#
+# AgentScope 提供统一的 ``ServerEvents`` 和 ``ClientEvents``,
+# 用于后端与前端之间的通信:
+#
+# **ServerEvents**(后端 → 前端):
+#
+# .. list-table::
+# :header-rows: 1
+# :widths: 40 60
+#
+# * - 事件
+# - 描述
+# * - ``ServerEvents.ServerSessionCreatedEvent``
+# - 后端创建会话
+# * - ``ServerEvents.ServerSessionUpdatedEvent``
+# - 后端更新会话
+# * - ``ServerEvents.ServerSessionEndedEvent``
+# - 后端结束会话
+# * - ``ServerEvents.AgentReadyEvent``
+# - 智能体准备接收输入
+# * - ``ServerEvents.AgentEndedEvent``
+# - 智能体已结束
+# * - ``ServerEvents.AgentResponseCreatedEvent``
+# - 智能体开始生成响应
+# * - ``ServerEvents.AgentResponseDoneEvent``
+# - 智能体完成响应生成
+# * - ``ServerEvents.AgentResponseAudioDeltaEvent``
+# - 智能体流式音频块
+# * - ``ServerEvents.AgentResponseAudioDoneEvent``
+# - 音频响应完成
+# * - ``ServerEvents.AgentResponseAudioTranscriptDeltaEvent``
+# - 智能体响应的流式转录
+# * - ``ServerEvents.AgentResponseAudioTranscriptDoneEvent``
+# - 转录完成
+# * - ``ServerEvents.AgentResponseToolUseDeltaEvent``
+# - 流式工具调用数据
+# * - ``ServerEvents.AgentResponseToolUseDoneEvent``
+# - 工具调用完成
+# * - ``ServerEvents.AgentResponseToolResultEvent``
+# - 工具执行结果
+# * - ``ServerEvents.AgentInputTranscriptionDeltaEvent``
+# - 用户输入的流式转录
+# * - ``ServerEvents.AgentInputTranscriptionDoneEvent``
+# - 输入转录完成
+# * - ``ServerEvents.AgentInputStartedEvent``
+# - 用户音频输入开始
+# * - ``ServerEvents.AgentInputDoneEvent``
+# - 用户音频输入结束
+# * - ``ServerEvents.AgentErrorEvent``
+# - 发生错误
+#
+# **ClientEvents**(前端 → 后端):
+#
+# .. list-table::
+# :header-rows: 1
+# :widths: 40 60
+#
+# * - 事件
+# - 描述
+# * - ``ClientEvents.ClientSessionCreateEvent``
+# - 创建指定配置的新会话
+# * - ``ClientEvents.ClientSessionEndEvent``
+# - 结束当前会话
+# * - ``ClientEvents.ClientResponseCreateEvent``
+# - 请求智能体立即生成响应
+# * - ``ClientEvents.ClientResponseCancelEvent``
+# - 中断智能体的当前响应
+# * - ``ClientEvents.ClientTextAppendEvent``
+# - 追加文本输入
+# * - ``ClientEvents.ClientAudioAppendEvent``
+# - 追加音频输入
+# * - ``ClientEvents.ClientAudioCommitEvent``
+# - 提交音频输入(标志输入结束)
+# * - ``ClientEvents.ClientImageAppendEvent``
+# - 追加图像输入
+# * - ``ClientEvents.ClientToolResultEvent``
+# - 发送工具执行结果
+#
+# 初始化实时智能体
+# ------------------------------
+#
+# 以下是创建和使用实时智能体的示例:
+
+
+async def example_realtime_agent() -> None:
+ """创建和使用实时智能体的示例。"""
+ agent = RealtimeAgent(
+ name="Friday",
+ sys_prompt="你是一个名为 Friday 的助手。",
+ model=DashScopeRealtimeModel(
+ model_name="qwen3-omni-flash-realtime",
+ api_key=os.getenv("DASHSCOPE_API_KEY"),
+ ),
+ )
+
+ # 创建队列接收来自智能体的消息
+ outgoing_queue = asyncio.Queue()
+
+ # 智能体现在已准备好处理输入
+ # 在独立任务中处理输出消息
+ async def handle_agent_messages():
+ while True:
+ event = await outgoing_queue.get()
+ # 处理事件(例如通过 WebSocket 发送到前端)
+ print(f"智能体事件: {event.type}")
+
+ # 启动消息处理任务
+ asyncio.create_task(handle_agent_messages())
+
+ # 启动智能体(建立连接)
+ await agent.start(outgoing_queue)
+
+ # 完成后停止智能体
+ await agent.stop()
+
+
+# %%
+# 启动实时对话
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+# 下面演示如何在用户和实时智能体之间建立实时对话。
+#
+# 这里以 FastAPI 为例,展示如何搭建实时对话的后端框架。
+#
+# **后端设置(服务端):**
+#
+# 后端需要:
+#
+# 1. 创建 WebSocket 端点接受前端连接
+# 2. 在会话开始时创建 ``RealtimeAgent``
+# 3. 将前端的 ``ClientEvents`` 转发给智能体
+# 4. 将智能体的 ``ServerEvents`` 转发给前端
+#
+# .. code-block:: python
+#
+# from fastapi import FastAPI, WebSocket
+# from agentscope.agent import RealtimeAgent
+# from agentscope.realtime import (
+# DashScopeRealtimeModel,
+# ClientEvents,
+# ServerEvents,
+# )
+#
+# app = FastAPI()
+#
+# @app.websocket("/ws/{user_id}/{session_id}")
+# async def websocket_endpoint(
+# websocket: WebSocket,
+# user_id: str,
+# session_id: str,
+# ):
+# await websocket.accept()
+#
+# # 创建智能体消息队列
+# frontend_queue = asyncio.Queue()
+#
+# # 创建智能体
+# agent = RealtimeAgent(
+# name="Assistant",
+# sys_prompt="你是一个有用的助手。",
+# model=DashScopeRealtimeModel(
+# model_name="qwen3-omni-flash-realtime",
+# api_key=os.getenv("DASHSCOPE_API_KEY"),
+# ),
+# )
+#
+# # 启动智能体
+# await agent.start(frontend_queue)
+#
+# # 将智能体消息转发到前端
+# async def send_to_frontend():
+# while True:
+# msg = await frontend_queue.get()
+# await websocket.send_json(msg.model_dump())
+#
+# asyncio.create_task(send_to_frontend())
+#
+# # 接收前端消息并转发给智能体
+# while True:
+# data = await websocket.receive_json()
+# client_event = ClientEvents.from_json(data)
+# await agent.handle_input(client_event)
+#
+# **前端设置(客户端):**
+#
+# 前端需要:
+#
+# 1. 建立与后端的 WebSocket 连接
+# 2. 发送 ``CLIENT_SESSION_CREATE`` 事件初始化会话
+# 3. 捕获麦克风音频,通过 ``CLIENT_AUDIO_APPEND`` 事件发送
+# 4. 接收并处理 ``ServerEvents``(例如播放音频、显示转录文本)
+#
+# .. code-block:: javascript
+#
+# // 连接 WebSocket
+# const ws = new WebSocket('ws://localhost:8000/ws/user1/session1');
+#
+# ws.onopen = () => {
+# // 创建会话
+# ws.send(JSON.stringify({
+# type: 'client_session_create',
+# config: {
+# instructions: '你是一个有用的助手。',
+# user_name: 'User1'
+# }
+# }));
+# };
+#
+# // 处理来自后端的消息
+# ws.onmessage = (event) => {
+# const data = JSON.parse(event.data);
+# if (data.type === 'response_audio_delta') {
+# // 播放音频块
+# playAudio(data.delta);
+# }
+# };
+#
+# // 发送音频数据
+# function sendAudioChunk(audioData) {
+# ws.send(JSON.stringify({
+# type: 'client_audio_append',
+# session_id: 'session1',
+# audio: audioData, // base64 编码
+# format: { encoding: 'pcm16', sample_rate: 16000 }
+# }));
+# }
+#
+# 完整的工作示例请参见 AgentScope 仓库中的
+# ``examples/agent/realtime_voice_agent/``。
+
+# %%
+# 多智能体实时对话
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# AgentScope 通过 ``ChatRoom`` 类支持多智能体实时交互。
+#
+# 请注意目前大多数实时模型 API 仅支持单用户交互,但 AgentScope 的架构设计支持多智能体和多用户,
+# 当 API 能力扩展时即可应用到多智能体场景。
+#
+# 实时聊天室
+# ----------------------------
+#
+# AgentScope 引入 ``ChatRoom`` 类来管理共享对话空间中的多个实时智能体。
+# ChatRoom 提供:
+#
+# - 集中管理多个 ``RealtimeAgent`` 实例
+# - 智能体之间的自动消息广播
+# - 统一的前端通信消息队列
+# - 房间内所有智能体的生命周期管理
+#
+# 使用 ChatRoom
+# --------------
+#
+# ``ChatRoom`` 的用法与 ``RealtimeAgent`` 类似:
+#
+
+
+async def example_chat_room() -> None:
+ """使用 ChatRoom 和多个实时智能体的示例。"""
+ from agentscope.pipeline import ChatRoom
+ from agentscope.agent import RealtimeAgent
+ from agentscope.realtime import DashScopeRealtimeModel
+
+ # 创建多个智能体
+ agent1 = RealtimeAgent(
+ name="Agent1",
+ sys_prompt="你是 Agent1,一个有用的助手。",
+ model=DashScopeRealtimeModel(
+ model_name="qwen3-omni-flash-realtime",
+ api_key=os.getenv("DASHSCOPE_API_KEY"),
+ ),
+ )
+
+ agent2 = RealtimeAgent(
+ name="Agent2",
+ sys_prompt="你是 Agent2,一个有用的助手。",
+ model=DashScopeRealtimeModel(
+ model_name="qwen3-omni-flash-realtime",
+ api_key=os.getenv("DASHSCOPE_API_KEY"),
+ ),
+ )
+
+ # 创建包含多个智能体的聊天室
+ chat_room = ChatRoom(agents=[agent1, agent2])
+
+ # 创建队列接收来自所有智能体的消息
+ outgoing_queue = asyncio.Queue()
+
+ # 启动聊天室
+ await chat_room.start(outgoing_queue)
+
+ # 处理来自前端的输入
+ # 聊天室会广播给所有智能体
+ from agentscope.realtime import ClientEvents
+
+ client_event = ClientEvents.ClientTextAppendEvent(
+ session_id="session1",
+ text="大家好!",
+ )
+ await chat_room.handle_input(client_event)
+
+ # 完成后停止聊天室
+ await chat_room.stop()
+
+
+# %%
+# 发展规划
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# 实时智能体功能目前为实验性质,正在积极开发中。未来计划包括:
+#
+# - 支持更多实时模型 API
+# - 增强对话历史的记忆管理
+# - 多用户语音交互支持
+# - 改进 VAD(语音活动检测)配置
+# - 更好的错误处理和恢复机制
+#
+#
diff --git a/examples/agent/realtime_voice_agent/README.md b/examples/agent/realtime_voice_agent/README.md
new file mode 100644
index 000000000..0e7fd5d78
--- /dev/null
+++ b/examples/agent/realtime_voice_agent/README.md
@@ -0,0 +1,107 @@
+# Realtime Voice Agent Example
+
+This example demonstrates how to build a **real-time voice conversation agent** using AgentScope's RealtimeAgent. The agent supports bidirectional voice streaming, enabling natural voice conversations with low latency and real-time audio transcription.
+
+## Prerequisites
+
+- Python 3.10 or higher
+- Your DashScope API key in an environment variable `DASHSCOPE_API_KEY`
+
+Install the required packages:
+
+```bash
+uv pip install agentscope fastapi uvicorn websockets
+# or
+# pip install agentscope
+```
+
+## Usage
+
+### 1. Start the Server
+
+Run the FastAPI server:
+
+```bash
+cd examples/agent/realtime_voice_agent
+python run_server.py
+```
+
+The server will start on `http://localhost:8000` by default.
+
+### 2. Open the Web Interface
+
+Open your web browser and navigate to:
+
+```
+http://localhost:8000
+```
+
+You will see a web interface with:
+- Configuration panel (instructions and user name)
+- Voice control buttons (Start Recording, Stop Recording, Disconnect)
+- Text input field
+- Message display area
+
+### 3. Start Conversation
+
+1. **Configure the Agent** (optional):
+ - Modify the "Instructions" to customize the agent's behavior
+ - Enter your name in the "User Name" field
+
+2. **Start Voice Recording**:
+ - Click the "🎤 Start Recording" button
+ - Allow microphone access when prompted by your browser
+ - Speak naturally to the agent
+ - The agent will respond with voice and text
+
+3. **Stop Recording**:
+ - Click "⏹️ Stop Recording" to pause voice input
+
+
+## Switching Models
+
+AgentScope supports multiple realtime voice models. By default, this example uses DashScope's `qwen3-omni-flash-realtime` model, but you can easily switch to other providers.
+
+### Supported Models
+
+- **GeminiRealtimeModel**
+- **OpenAIRealtimeModel**
+
+### How to Switch Models
+
+Edit `run_server.py` and replace the model initialization code:
+
+**For OpenAI:**
+
+```python
+from agentscope.realtime import OpenAIRealtimeModel
+
+agent = RealtimeAgent(
+ name="Friday",
+ sys_prompt=sys_prompt,
+ model=OpenAIRealtimeModel(
+ model_name="gpt-4o-realtime-preview",
+ api_key=os.getenv("OPENAI_API_KEY"),
+ voice="alloy", # Options: "alloy", "echo", "marin", "cedar"
+ ),
+)
+```
+
+**For Gemini:**
+
+```python
+from agentscope.realtime import GeminiRealtimeModel
+
+agent = RealtimeAgent(
+ name="Friday",
+ sys_prompt=sys_prompt,
+ model=GeminiRealtimeModel(
+ model_name="gemini-2.5-flash-native-audio-preview-09-2025",
+ api_key=os.getenv("GEMINI_API_KEY"),
+ voice="Puck", # Options: "Puck", "Charon", "Kore", "Fenrir"
+ ),
+)
+```
+
+Don't forget to set the corresponding API key environment variable before starting the server!
+
diff --git a/examples/agent/realtime_voice_agent/chatbot.html b/examples/agent/realtime_voice_agent/chatbot.html
new file mode 100644
index 000000000..1b7d8f92f
--- /dev/null
+++ b/examples/agent/realtime_voice_agent/chatbot.html
@@ -0,0 +1,1228 @@
+<!DOCTYPE html>
+<html>
+<head>
+ <title>Realtime Chatbot with AgentScope</title>
+ <meta charset="UTF-8">
+ <style>
+ * {
+ margin: 0;
+ padding: 0;
+ box-sizing: border-box;
+ }
+
+ body {
+ font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;
+ max-width: 900px;
+ margin: 0 auto;
+ padding: 2rem;
+ background: hsl(0, 0%, 98%);
+ color: hsl(222.2, 84%, 4.9%);
+ line-height: 1.5;
+ min-height: 100vh;
+ display: flex;
+ flex-direction: column;
+ gap: 1.5rem;
+ }
+
+ h1 {
+ font-size: 2rem;
+ font-weight: 600;
+ color: hsl(222.2, 84%, 4.9%);
+ letter-spacing: -0.025em;
+ flex-shrink: 0;
+ }
+
+ #messages {
+ border: 1px solid hsl(214.3, 31.8%, 91.4%);
+ min-height: 300px;
+ flex: 1;
+ overflow-y: auto;
+ padding: 1rem;
+ background: hsl(0, 0%, 100%);
+ border-radius: 0.5rem;
+ box-shadow: 0 1px 3px 0 rgb(0 0 0 / 0.1), 0 1px 2px -1px rgb(0 0 0 / 0.1);
+ }
+
+ #messages::-webkit-scrollbar {
+ width: 8px;
+ }
+
+ #messages::-webkit-scrollbar-track {
+ background: hsl(210, 40%, 96.1%);
+ border-radius: 4px;
+ }
+
+ #messages::-webkit-scrollbar-thumb {
+ background: hsl(215.4, 16.3%, 56.9%);
+ border-radius: 4px;
+ }
+
+ #messages::-webkit-scrollbar-thumb:hover {
+ background: hsl(215.4, 16.3%, 46.9%);
+ }
+
+ input[type="text"] {
+ width: 100%;
+ padding: 0.625rem 0.875rem;
+ font-size: 0.875rem;
+ border: 1px solid hsl(214.3, 31.8%, 91.4%);
+ border-radius: 0.375rem;
+ background: hsl(0, 0%, 100%);
+ color: hsl(222.2, 84%, 4.9%);
+ transition: all 0.15s ease;
+ outline: none;
+ }
+
+ input[type="text"]:focus {
+ border-color: hsl(221.2, 83.2%, 53.3%);
+ box-shadow: 0 0 0 3px hsl(221.2, 83.2%, 53.3%, 0.1);
+ }
+
+ textarea {
+ width: 100%;
+ min-height: 100px;
+ padding: 0.625rem 0.875rem;
+ font-size: 0.875rem;
+ border: 1px solid hsl(214.3, 31.8%, 91.4%);
+ border-radius: 0.375rem;
+ background: hsl(0, 0%, 100%);
+ color: hsl(222.2, 84%, 4.9%);
+ transition: all 0.15s ease;
+ outline: none;
+ resize: vertical;
+ font-family: inherit;
+ line-height: 1.5;
+ }
+
+ textarea:focus {
+ border-color: hsl(221.2, 83.2%, 53.3%);
+ box-shadow: 0 0 0 3px hsl(221.2, 83.2%, 53.3%, 0.1);
+ }
+
+ button {
+ display: inline-flex;
+ align-items: center;
+ justify-content: center;
+ padding: 0.625rem 1rem;
+ font-size: 0.875rem;
+ font-weight: 500;
+ border: 1px solid transparent;
+ border-radius: 0.375rem;
+ cursor: pointer;
+ transition: all 0.15s ease;
+ box-shadow: 0 1px 2px 0 rgb(0 0 0 / 0.05);
+ }
+
+ button:active {
+ transform: scale(0.98);
+ }
+
+ button:focus-visible {
+ outline: 2px solid hsl(221.2, 83.2%, 53.3%);
+ outline-offset: 2px;
+ }
+
+ button:disabled {
+ opacity: 0.5;
+ cursor: not-allowed;
+ }
+
+ /* Primary button style */
+ button.btn-primary {
+ background: hsl(221.2, 83.2%, 53.3%);
+ color: hsl(0, 0%, 100%);
+ border-color: hsl(221.2, 83.2%, 53.3%);
+ }
+
+ button.btn-primary:hover:not(:disabled) {
+ background: hsl(221.2, 83.2%, 45%);
+ border-color: hsl(221.2, 83.2%, 45%);
+ }
+
+ button.btn-primary.recording {
+ background: hsl(0, 84.2%, 60.2%);
+ border-color: hsl(0, 84.2%, 60.2%);
+ animation: pulse 1.5s ease-in-out infinite;
+ }
+
+ button.btn-primary.recording:hover:not(:disabled) {
+ background: hsl(0, 84.2%, 50%);
+ border-color: hsl(0, 84.2%, 50%);
+ }
+
+ /* Secondary button style */
+ button.btn-secondary {
+ background: hsl(0, 0%, 100%);
+ color: hsl(222.2, 47.4%, 11.2%);
+ border-color: hsl(214.3, 31.8%, 91.4%);
+ }
+
+ button.btn-secondary:hover:not(:disabled) {
+ background: hsl(210, 40%, 98%);
+ border-color: hsl(214.3, 31.8%, 81.4%);
+ }
+
+ .message {
+ margin: 0.75rem 0;
+ padding: 0.75rem 1rem;
+ background: hsl(210, 40%, 98%);
+ border-radius: 0.5rem;
+ border: 1px solid hsl(214.3, 31.8%, 91.4%);
+ font-size: 0.875rem;
+ }
+
+ .message strong {
+ color: hsl(222.2, 47.4%, 11.2%);
+ font-weight: 600;
+ }
+
+ @keyframes pulse {
+ 0%, 100% {
+ opacity: 1;
+ box-shadow: 0 0 0 0 hsl(0, 84.2%, 60.2%, 0.7);
+ }
+ 50% {
+ opacity: 0.9;
+ box-shadow: 0 0 0 8px hsl(0, 84.2%, 60.2%, 0);
+ }
+ }
+
+ .controls {
+ display: flex;
+ gap: 0.75rem;
+ align-items: center;
+ flex-shrink: 0;
+ }
+
+ .controls #voiceBtn {
+ flex: 1;
+ }
+
+ .controls button:not(#voiceBtn) {
+ width: 140px;
+ flex-shrink: 0;
+ }
+
+ .text-input-container {
+ display: flex;
+ gap: 0.75rem;
+ align-items: center;
+ flex-shrink: 0;
+ }
+
+ .text-input-container input[type="text"] {
+ flex: 1;
+ }
+
+ .text-input-container button {
+ width: 100px;
+ flex-shrink: 0;
+ }
+
+ .configuration-container {
+ background: hsl(0, 0%, 100%);
+ padding: 1.5rem;
+ border-radius: 0.5rem;
+ border: 1px solid hsl(214.3, 31.8%, 91.4%);
+ box-shadow: 0 1px 3px 0 rgb(0 0 0 / 0.1), 0 1px 2px -1px rgb(0 0 0 / 0.1);
+ flex-shrink: 0;
+ }
+
+ .configuration-container h3 {
+ font-size: 1.125rem;
+ font-weight: 600;
+ margin-bottom: 1rem;
+ color: hsl(222.2, 84%, 4.9%);
+ display: flex;
+ align-items: center;
+ gap: 0.5rem;
+ }
+
+ .config-field {
+ margin-bottom: 1.25rem;
+ }
+
+ .config-field:last-child {
+ margin-bottom: 0;
+ }
+
+ .config-field label {
+ display: block;
+ font-weight: 500;
+ margin-bottom: 0.5rem;
+ color: hsl(222.2, 47.4%, 11.2%);
+ font-size: 0.875rem;
+ }
+
+ .error-message {
+ padding: 0.875rem 1rem;
+ background: hsl(0, 84.2%, 95%);
+ border: 1px solid hsl(0, 84.2%, 85%);
+ border-radius: 0.5rem;
+ display: none;
+ color: hsl(0, 84.2%, 30%);
+ font-size: 0.875rem;
+ font-weight: 500;
+ box-shadow: 0 1px 2px 0 rgb(0 0 0 / 0.05);
+ flex-shrink: 0;
+ }
+
+
+ .model-options {
+ display: flex;
+ flex-direction: row;
+ gap: 0.75rem;
+ }
+
+ .model-option {
+ flex: 1;
+ padding: 0.75rem;
+ border: 2px solid hsl(214.3, 31.8%, 91.4%);
+ border-radius: 0.5rem;
+ cursor: pointer;
+ transition: all 0.15s ease;
+ background: hsl(0, 0%, 100%);
+ display: flex;
+ flex-direction: column;
+ justify-content: center;
+ gap: 0.5rem;
+ }
+
+ .model-option:hover:not(.disabled) {
+ border-color: hsl(221.2, 83.2%, 53.3%);
+ background: hsl(221.2, 83.2%, 98%);
+ }
+
+ .model-option.selected {
+ border-color: hsl(221.2, 83.2%, 53.3%);
+ background: hsl(221.2, 83.2%, 95%);
+ }
+
+ .model-option.disabled {
+ opacity: 0.5;
+ cursor: not-allowed;
+ background: hsl(0, 0%, 98%);
+ }
+
+ .model-option-header {
+ display: flex;
+ align-items: center;
+ gap: 0.5rem;
+ }
+
+ .model-option input[type="radio"] {
+ margin: 0;
+ cursor: pointer;
+ flex-shrink: 0;
+ }
+
+ .model-option.disabled input[type="radio"] {
+ cursor: not-allowed;
+ }
+
+ .model-info {
+ display: flex;
+ flex-direction: column;
+ gap: 0.5rem;
+ flex: 1;
+ }
+
+ .model-name-line {
+ display: flex;
+ align-items: center;
+ gap: 0.5rem;
+ min-height: 1.25rem;
+ }
+
+ .model-name {
+ font-weight: 600;
+ color: hsl(222.2, 84%, 4.9%);
+ }
+
+ .model-unavailable-reason {
+ font-size: 0.625rem;
+ color: hsl(215.4, 16.3%, 56.9%);
+ font-style: italic;
+ white-space: nowrap;
+ }
+
+ .model-tags {
+ display: flex;
+ gap: 0.375rem;
+ flex-wrap: wrap;
+ }
+
+ .model-tag {
+ display: inline-flex;
+ align-items: center;
+ padding: 0.125rem 0.5rem;
+ font-size: 0.75rem;
+ font-weight: 500;
+ border-radius: 0.25rem;
+ background: hsl(214.3, 31.8%, 91.4%);
+ color: hsl(222.2, 47.4%, 11.2%);
+ }
+
+ .model-tag.text {
+ background: hsl(200, 95%, 90%);
+ color: hsl(200, 95%, 30%);
+ }
+
+ .model-tag.audio {
+ background: hsl(280, 85%, 90%);
+ color: hsl(280, 85%, 30%);
+ }
+
+ .model-tag.image {
+ background: hsl(25, 95%, 90%);
+ color: hsl(25, 95%, 30%);
+ }
+
+ .model-tag.tool {
+ background: hsl(142, 71%, 90%);
+ color: hsl(142, 71%, 30%);
+ }
+
+ .tools-label {
+ display: block;
+ font-weight: 500;
+ margin-bottom: 0.5rem;
+ color: hsl(222.2, 47.4%, 11.2%);
+ font-size: 0.875rem;
+ }
+
+ .tools-disabled-hint {
+ font-size: 0.75rem;
+ color: hsl(215.4, 16.3%, 56.9%);
+ font-weight: 400;
+ font-style: italic;
+ margin-left: 0.5rem;
+ }
+
+ .tools-list {
+ display: flex;
+ gap: 0.5rem;
+ }
+
+ .tool-item {
+ flex: 1;
+ display: flex;
+ align-items: center;
+ justify-content: center;
+ padding: 0.5rem 0.75rem;
+ background: hsl(0, 0%, 100%);
+ border: 2px solid hsl(214.3, 31.8%, 91.4%);
+ border-radius: 0.375rem;
+ font-size: 0.875rem;
+ color: hsl(222.2, 47.4%, 11.2%);
+ font-weight: 500;
+ transition: all 0.15s ease;
+ }
+
+ .tool-item.disabled {
+ background: hsl(0, 0%, 96%);
+ color: hsl(215.4, 16.3%, 56.9%);
+ border-color: hsl(214.3, 31.8%, 91.4%);
+ opacity: 0.6;
+ }
+ </style>
+</head>
+<body>
+ <h1>Realtime Chatbot</h1>
+
+ <div class="configuration-container">
+ <h3>⚙️ Configuration</h3>
+
+ <div class="config-field">
+ <label for="instructions">Instructions</label>
+ <textarea id="instructions" placeholder="Enter agent instructions...">You're a helpful assistant named Friday.</textarea>
+ </div>
+
+ <div class="config-field">
+ <label for="agentName">Agent Name</label>
+ <input type="text" id="agentName" placeholder="Enter agent name" value="Friday" />
+ </div>
+
+
+ <div class="config-field">
+ <label>Model Provider</label>
+ <div class="model-options" id="modelOptions">
+ <label class="model-option" data-provider="dashscope">
+ <div class="model-option-header">
+ <input type="radio" name="modelProvider" value="dashscope" checked />
+ <div class="model-info">
+ <div class="model-name-line">
+ <span class="model-name">DashScope</span>
+ <span class="model-unavailable-reason" style="display: none;"></span>
+ </div>
+ <div class="model-tags">
+ <span class="model-tag audio">Audio</span>
+ <span class="model-tag image">Image</span>
+ </div>
+ </div>
+ </div>
+ </label>
+ <label class="model-option" data-provider="gemini">
+ <div class="model-option-header">
+ <input type="radio" name="modelProvider" value="gemini" />
+ <div class="model-info">
+ <div class="model-name-line">
+ <span class="model-name">Gemini</span>
+ <span class="model-unavailable-reason" style="display: none;"></span>
+ </div>
+ <div class="model-tags">
+ <span class="model-tag text">Text</span>
+ <span class="model-tag audio">Audio</span>
+ <span class="model-tag image">Image</span>
+ <span class="model-tag tool">Tool</span>
+ </div>
+ </div>
+ </div>
+ </label>
+ <label class="model-option" data-provider="openai">
+ <div class="model-option-header">
+ <input type="radio" name="modelProvider" value="openai" />
+ <div class="model-info">
+ <div class="model-name-line">
+ <span class="model-name">OpenAI</span>
+ <span class="model-unavailable-reason" style="display: none;"></span>
+ </div>
+ <div class="model-tags">
+ <span class="model-tag text">Text</span>
+ <span class="model-tag audio">Audio</span>
+ <span class="model-tag tool">Tool</span>
+ </div>
+ </div>
+ </div>
+ </label>
+ </div>
+ </div>
+
+ <div class="config-field">
+ <label class="tools-label">
+ Equipped Tools
+ <span class="tools-disabled-hint" id="toolsDisabledHint" style="display: none;">(Not supported by this model)</span>
+ </label>
+ <div class="tools-list" id="toolsList">
+ <div class="tool-item" data-tool="execute_python_code">🐍 execute_python_code</div>
+ <div class="tool-item" data-tool="execute_shell_command">💻 execute_shell_command</div>
+ <div class="tool-item" data-tool="view_text_file">📄 view_text_file</div>
+ </div>
+ </div>
+ </div>
+
+ <div id="errorMessage" class="error-message"></div>
+
+ <div class="controls">
+ <button id="voiceBtn" class="btn-primary" onclick="toggleVoice()">🎤 Start Voice Chat</button>
+ <button class="btn-secondary" onclick="disconnect()">❌ Disconnect</button>
+ </div>
+
+ <div class="text-input-container">
+ <input type="text" id="textInput" placeholder="Type your message here..." />
+ <button id="sendBtn" class="btn-primary" onclick="sendTextMessage()" disabled>📤 Send</button>
+ </div>
+
+
+ <div id="messages"></div>
+
+ <script>
+ let ws = null;
+ let audioContext = null; // For recording, 16kHz
+ let playbackAudioContext = null; // For playback, 24kHz
+ let mediaStream = null;
+ let isRecording = false;
+ let isPlaying = false;
+ let audioPlaybackNode = null;
+ let audioPlaybackQueue = []; // Store decoded Float32Array
+ let audioPlaybackIndex = 0;
+ let sessionId = "session1"; // Session ID
+ let sessionCreated = false; // Track if session has been created
+
+ // Update send button state based on input and recording status
+ function updateSendButtonState() {
+ const textInput = document.getElementById("textInput");
+ const sendBtn = document.getElementById("sendBtn");
+ const hasText = textInput.value.trim().length > 0;
+ sendBtn.disabled = !(hasText && isRecording);
+ }
+
+ // Send text message to backend
+ function sendTextMessage() {
+ const textInput = document.getElementById("textInput");
+ const text = textInput.value.trim();
+
+ if (!text || !isRecording) {
+ return;
+ }
+
+ if (!ws || ws.readyState !== WebSocket.OPEN) {
+ showError("⚠️ WebSocket is not connected!");
+ return;
+ }
+
+ // Send ClientTextAppendEvent
+ ws.send(JSON.stringify({
+ type: "client_text_append",
+ session_id: sessionId,
+ text: text
+ }));
+
+ addMessage("You", text);
+ textInput.value = "";
+ updateSendButtonState();
+ }
+
+ // Add event listener for text input
+ document.addEventListener("DOMContentLoaded", function() {
+ const textInput = document.getElementById("textInput");
+
+ // Update button state on input
+ textInput.addEventListener("input", updateSendButtonState);
+
+ // Send on Enter key
+ textInput.addEventListener("keypress", function(e) {
+ if (e.key === "Enter" && !document.getElementById("sendBtn").disabled) {
+ sendTextMessage();
+ }
+ });
+ });
+
+ // Used to accumulate transcript text
+ let currentTranscript = "";
+ let currentTranscriptElement = null;
+ let currentResponseTranscript = "";
+ let currentResponseTranscriptElement = null;
+
+ // Check available models on page load
+ async function checkAvailableModels() {
+ try {
+ const apiUrl = `${window.location.protocol}//${window.location.host}/api/check-models`;
+
+ const response = await fetch(apiUrl);
+ const availability = await response.json();
+
+ console.log("Model availability:", availability);
+
+ let hasAvailableModel = false;
+
+ // Update UI based on availability
+ const modelOptions = document.querySelectorAll('.model-option');
+ modelOptions.forEach(option => {
+ const provider = option.getAttribute('data-provider');
+ const radio = option.querySelector('input[type="radio"]');
+ const unavailableReason = option.querySelector('.model-unavailable-reason');
+
+ if (!availability[provider]) {
+ option.classList.add('disabled');
+ radio.disabled = true;
+
+ // Show unavailable reason
+ const providerName = provider.charAt(0).toUpperCase() + provider.slice(1);
+ unavailableReason.textContent = `(${providerName.toUpperCase()}_API_KEY not set)`;
+ unavailableReason.style.display = 'inline';
+
+ // If this was the selected option, uncheck it
+ if (radio.checked) {
+ radio.checked = false;
+ }
+ } else {
+ hasAvailableModel = true;
+ unavailableReason.style.display = 'none';
+ }
+ });
+
+ // If no model is selected after checking availability, select first available
+ const selectedRadio = document.querySelector('input[name="modelProvider"]:checked');
+ if (!selectedRadio && hasAvailableModel) {
+ for (const option of modelOptions) {
+ const provider = option.getAttribute('data-provider');
+ if (availability[provider]) {
+ option.querySelector('input[type="radio"]').checked = true;
+ option.classList.add('selected');
+ break;
+ }
+ }
+ }
+
+ // Disable recording button if no model is available
+ const voiceBtn = document.getElementById('voiceBtn');
+ if (!hasAvailableModel) {
+ voiceBtn.disabled = true;
+ showError('⚠️ No model API keys configured. Please set at least one API key to start voice chat.');
+ } else {
+ voiceBtn.disabled = false;
+ }
+
+ // Add click handlers for model options
+ modelOptions.forEach(option => {
+ option.addEventListener('click', function() {
+ if (!this.classList.contains('disabled')) {
+ modelOptions.forEach(opt => opt.classList.remove('selected'));
+ this.classList.add('selected');
+ updateToolsDisplay();
+ }
+ });
+ });
+
+ // Mark initially selected option
+ const currentSelected = document.querySelector('input[name="modelProvider"]:checked');
+ if (currentSelected) {
+ currentSelected.closest('.model-option').classList.add('selected');
+ }
+
+ // Update tools display based on initial selection
+ updateToolsDisplay();
+
+ } catch (error) {
+ console.error("Failed to check model availability:", error);
+ showError("⚠️ Failed to check model availability. Please refresh the page.");
+ }
+ }
+
+ function updateToolsDisplay() {
+ const selectedRadio = document.querySelector('input[name="modelProvider"]:checked');
+ const toolItems = document.querySelectorAll('.tool-item');
+ const toolsDisabledHint = document.getElementById('toolsDisabledHint');
+
+ if (selectedRadio) {
+ const provider = selectedRadio.value;
+ // Enable tools for Gemini and OpenAI, disable for others
+ if (provider === 'gemini' || provider === 'openai') {
+ toolItems.forEach(item => item.classList.remove('disabled'));
+ toolsDisabledHint.style.display = 'none';
+ } else {
+ toolItems.forEach(item => item.classList.add('disabled'));
+ toolsDisabledHint.style.display = 'inline';
+ }
+ } else {
+ toolItems.forEach(item => item.classList.add('disabled'));
+ toolsDisabledHint.style.display = 'inline';
+ }
+ }
+
+ function showError(message) {
+ const errorDiv = document.getElementById("errorMessage");
+ errorDiv.innerText = message;
+ errorDiv.style.display = "block";
+ setTimeout(() => {
+ errorDiv.style.display = "none";
+ }, 5000);
+ }
+
+ async function connect() {
+ const userId = "You";
+ const protocol = window.location.protocol === 'https:' ? 'wss:' : 'ws:';
+ const wsUrl = `${protocol}//${window.location.host}/ws/${userId}/${sessionId}`;
+
+ console.log(`Connecting to WebSocket: ${wsUrl}`);
+ ws = new WebSocket(wsUrl);
+
+ ws.onopen = function() {
+ addMessage("System", "✅ WebSocket connected successfully, ready for voice conversation");
+ };
+
+ ws.onmessage = async function(event) {
+ try {
+ const data = JSON.parse(event.data);
+ console.log("Received message:", data);
+
+ // Handle ServerEvents
+ switch (data.type) {
+ case "server_session_created":
+ sessionCreated = true;
+ addMessage("System", `✅ Session created: ${data.session_id}`);
+ break;
+
+ case "agent_ready":
+ addMessage("System", `🤖 Agent ${data.agent_name} is ready`);
+ break;
+
+ case "agent_response_created":
+ addMessage("System", `💬 Agent ${data.agent_name} started generating response...`);
+ break;
+
+ case "agent_response_audio_delta":
+ // Receive audio data and add to playback queue
+ queueAudioChunk(data.delta);
+ break;
+
+ case "agent_response_audio_done":
+ addMessage("System", "🔊 Audio response completed");
+ break;
+
+ case "agent_response_audio_transcript_delta":
+ // Agent response transcript text
+ appendResponseTranscript(data.agent_name, data.delta || "");
+ break;
+
+ case "agent_response_audio_transcript_done":
+ // Complete Agent response transcript message
+ finishResponseTranscript();
+ break;
+
+ case "agent_input_transcription_delta":
+ // User input transcript text
+ appendTranscript("You", data.delta || "");
+ break;
+
+ case "agent_input_transcription_done":
+ appendTranscript("You", data.transcript || "");
+ // Complete user input transcript message
+ finishTranscript();
+ addMessage("System", `📝 User input recognition completed`);
+ break;
+
+ case "agent_input_started":
+ addMessage("System", "🎤 Voice input started");
+ break;
+
+ case "agent_input_done":
+ addMessage("System", "⏹️ Voice input ended");
+ break;
+
+ case "agent_response_done":
+ addMessage("System", `✅ Response completed (input tokens: ${data.input_tokens}, output tokens: ${data.output_tokens})`);
+ break;
+
+ case "agent_response_tool_use_delta":
+ addMessage("System", `🔧 Tool call: ${data.name}`);
+ break;
+
+ case "agent_response_tool_use_done":
+ // Display tool use with complete information
+ const toolUseInfo = JSON.stringify(data.tool_use, null, 2);
+ addMessage(data.agent_name, `🔧 Tool Use:\n${toolUseInfo}`);
+ break;
+
+ case "agent_response_tool_result":
+ // Display tool result with complete information
+ const toolResultInfo = JSON.stringify(data.tool_result, null, 2);
+ addMessage(data.agent_name, `✅ Tool Result:\n${toolResultInfo}`);
+ break;
+
+ case "agent_error":
+ addMessage("Error", `❌ ${data.error_type}: ${data.message}`);
+ break;
+
+ case "agent_ended":
+ addMessage("System", `👋 Agent ${data.agent_name} has ended`);
+ break;
+
+ case "server_session_ended":
+ addMessage("System", `🔚 Session ${data.session_id} has ended`);
+ break;
+
+ default:
+ console.log("Unhandled event type:", data.type);
+ break;
+ }
+ } catch (e) {
+ console.error("Error processing message:", e);
+ }
+ };
+
+ ws.onclose = function() {
+ addMessage("System", "❌ Disconnected");
+ stopVoice();
+ sessionCreated = false; // Reset session state
+ updateSendButtonState();
+ };
+
+ ws.onerror = function() {
+ addMessage("System", "⚠️ Connection error");
+ };
+ }
+
+ async function toggleVoice() {
+ if (!isRecording) {
+ await startVoice();
+ } else {
+ stopVoice();
+ }
+ }
+
+ async function startVoice() {
+ try {
+ // Check if recording button is disabled
+ const voiceBtn = document.getElementById("voiceBtn");
+ if (voiceBtn.disabled) {
+ showError("⚠️ No model API keys configured. Please set at least one API key to start voice chat.");
+ return;
+ }
+
+ // Validate instructions
+ const instructions = document.getElementById("instructions").value.trim();
+ if (!instructions) {
+ showError("⚠️ Instructions cannot be empty! Please enter instructions before starting voice chat.");
+ return;
+ }
+
+ // Check if WebSocket is connected
+ if (!ws || ws.readyState !== WebSocket.OPEN) {
+ showError("⚠️ WebSocket is not connected! Please wait for connection.");
+ return;
+ }
+
+ // Send session create event if not already created
+ if (!sessionCreated) {
+ const agentName = document.getElementById("agentName").value.trim() || "Friday";
+ const selectedModel = document.querySelector('input[name="modelProvider"]:checked');
+ const modelProvider = selectedModel ? selectedModel.value : "dashscope";
+
+ addMessage("System", "📝 Creating session with instructions...");
+ ws.send(JSON.stringify({
+ type: "client_session_create",
+ config: {
+ instructions: instructions,
+ agent_name: agentName,
+ model_provider: modelProvider
+ }
+ }));
+
+ // Wait for session_created event before proceeding
+ // We'll set a timeout to wait for session creation
+ await new Promise((resolve, reject) => {
+ const timeout = setTimeout(() => {
+ reject(new Error("Session creation timeout"));
+ }, 5000);
+
+ const checkSession = setInterval(() => {
+ if (sessionCreated) {
+ clearTimeout(timeout);
+ clearInterval(checkSession);
+ resolve();
+ }
+ }, 100);
+ });
+ }
+
+ if (!audioContext) {
+ audioContext = new (window.AudioContext || window.webkitAudioContext)({
+ sampleRate: 16000
+ });
+ }
+
+ mediaStream = await navigator.mediaDevices.getUserMedia({
+ audio: {
+ echoCancellation: true,
+ noiseSuppression: true,
+ sampleRate: 16000
+ }
+ });
+
+ const source = audioContext.createMediaStreamSource(mediaStream);
+
+ // Use ScriptProcessorNode to process audio
+ const processor = audioContext.createScriptProcessor(4096, 1, 1);
+
+ let audioChunkCount = 0;
+ processor.onaudioprocess = function(e) {
+ if (!isRecording) return;
+
+ const inputData = e.inputBuffer.getChannelData(0);
+ const pcmData = convertToPCM16(inputData);
+ const base64Audio = arrayBufferToBase64(pcmData);
+
+ if (ws && ws.readyState === WebSocket.OPEN) {
+ audioChunkCount++;
+ if (audioChunkCount % 10 === 0) {
+ console.log(`Sending audio chunk ${audioChunkCount}`);
+ }
+ // Send ClientAudioAppendEvent
+ ws.send(JSON.stringify({
+ type: "client_audio_append",
+ session_id: sessionId,
+ audio: base64Audio,
+ format: {
+ rate: 16000,
+ type: "audio/pcm",
+ }
+ }));
+ }
+ };
+
+ source.connect(processor);
+ const dummyGain = audioContext.createGain();
+ dummyGain.gain.value = 0; // Mute to avoid feedback
+ processor.connect(dummyGain);
+ dummyGain.connect(audioContext.destination);
+
+ isRecording = true;
+ document.getElementById("voiceBtn").classList.add("recording");
+ document.getElementById("voiceBtn").innerText = "🔴 Voice Chat Active";
+ addMessage("System", "🎤 Voice chat started...");
+ updateSendButtonState();
+
+ } catch (err) {
+ console.error("Failed to start recording:", err);
+ if (err.message === "Session creation timeout") {
+ showError("⚠️ Session creation timeout. Please try again.");
+ addMessage("System", "⚠️ Session creation timeout");
+ } else {
+ showError("⚠️ Unable to access microphone: " + err.message);
+ addMessage("System", "⚠️ Unable to access microphone: " + err.message);
+ }
+ }
+ }
+
+ function stopVoice() {
+ isRecording = false;
+
+ if (mediaStream) {
+ mediaStream.getTracks().forEach(track => track.stop());
+ mediaStream = null;
+ }
+
+ // Notify server that recording has stopped - send ClientAudioCommitEvent
+ if (ws && ws.readyState === WebSocket.OPEN) {
+ ws.send(JSON.stringify({
+ type: "client_audio_commit",
+ session_id: sessionId
+ }));
+ }
+
+ document.getElementById("voiceBtn").classList.remove("recording");
+ document.getElementById("voiceBtn").innerText = "🎤 Start Voice Chat";
+ addMessage("System", "⏹️ Voice chat stopped");
+ updateSendButtonState();
+ }
+
+ function convertToPCM16(float32Array) {
+ const int16Array = new Int16Array(float32Array.length);
+ for (let i = 0; i < float32Array.length; i++) {
+ const s = Math.max(-1, Math.min(1, float32Array[i]));
+ int16Array[i] = s < 0 ? s * 0x8000 : s * 0x7FFF;
+ }
+ return int16Array.buffer;
+ }
+
+ function arrayBufferToBase64(buffer) {
+ const bytes = new Uint8Array(buffer);
+ let binary = '';
+ for (let i = 0; i < bytes.byteLength; i++) {
+ binary += String.fromCharCode(bytes[i]);
+ }
+ return btoa(binary);
+ }
+
+ function queueAudioChunk(base64Audio) {
+ try {
+ // Decode base64 audio data and convert to Float32Array
+ const binaryString = atob(base64Audio);
+ const bytes = new Uint8Array(binaryString.length);
+ for (let i = 0; i < binaryString.length; i++) {
+ bytes[i] = binaryString.charCodeAt(i);
+ }
+
+ // Convert to Int16Array (PCM16), then to Float32Array
+ const int16Array = new Int16Array(bytes.buffer);
+ const float32Array = new Float32Array(int16Array.length);
+
+ for (let i = 0; i < int16Array.length; i++) {
+ float32Array[i] = int16Array[i] / 32768.0;
+ }
+
+ // Add decoded audio data to queue
+ audioPlaybackQueue.push(float32Array);
+
+ // If not playing yet, start player
+ if (!isPlaying) {
+ startAudioPlayback();
+ }
+ } catch (err) {
+ console.error("Failed to decode audio chunk:", err);
+ }
+ }
+
+ function startAudioPlayback() {
+ if (isPlaying) return;
+
+ try {
+ // Create separate AudioContext for playback
+ if (!playbackAudioContext) {
+ playbackAudioContext = new (window.AudioContext || window.webkitAudioContext)({
+ sampleRate: 24000
+ });
+ }
+
+ // If AudioContext is suspended (browser policy), resume it
+ if (playbackAudioContext.state === 'suspended') {
+ playbackAudioContext.resume();
+ }
+
+ isPlaying = true;
+ audioPlaybackIndex = 0;
+
+ // Use ScriptProcessorNode for streaming playback
+ const bufferSize = 4096;
+ const processor = playbackAudioContext.createScriptProcessor(bufferSize, 0, 1);
+
+ processor.onaudioprocess = function(e) {
+ const output = e.outputBuffer.getChannelData(0);
+ const samplesNeeded = output.length;
+ let samplesWritten = 0;
+
+ // Get audio data from queue and fill output buffer
+ while (samplesWritten < samplesNeeded && audioPlaybackQueue.length > 0) {
+ const chunk = audioPlaybackQueue[0];
+
+ // Calculate number of samples to read from current chunk
+ const samplesToRead = Math.min(
+ samplesNeeded - samplesWritten,
+ chunk.length - audioPlaybackIndex
+ );
+
+ // Directly copy Float32 data to output
+ for (let i = 0; i < samplesToRead; i++) {
+ output[samplesWritten + i] = chunk[audioPlaybackIndex + i];
+ }
+
+ samplesWritten += samplesToRead;
+ audioPlaybackIndex += samplesToRead;
+
+ // If current chunk is finished, remove it and reset index
+ if (audioPlaybackIndex >= chunk.length) {
+ audioPlaybackQueue.shift();
+ audioPlaybackIndex = 0;
+ }
+ }
+
+ // If queue is empty and no more data, fill with silence
+ if (samplesWritten < samplesNeeded) {
+ for (let i = samplesWritten; i < samplesNeeded; i++) {
+ output[i] = 0;
+ }
+
+ // If queue continues to be empty for a while, stop playback
+ if (audioPlaybackQueue.length === 0) {
+ setTimeout(() => {
+ if (audioPlaybackQueue.length === 0) {
+ stopAudioPlayback();
+ }
+ }, 100);
+ }
+ }
+ };
+
+ processor.connect(playbackAudioContext.destination);
+ audioPlaybackNode = processor;
+
+ } catch (err) {
+ console.error("Failed to start audio playback:", err);
+ isPlaying = false;
+ }
+ }
+
+ function stopAudioPlayback() {
+ if (audioPlaybackNode) {
+ audioPlaybackNode.disconnect();
+ audioPlaybackNode = null;
+ }
+ isPlaying = false;
+ audioPlaybackQueue = [];
+ audioPlaybackIndex = 0;
+ }
+
+
+ function disconnect() {
+ stopVoice();
+ stopAudioPlayback();
+ if (ws) {
+ ws.close();
+ }
+ sessionCreated = false; // Reset session state
+ updateSendButtonState();
+ }
+
+
+ function addMessage(sender, message) {
+ const messagesDiv = document.getElementById("messages");
+ const messageDiv = document.createElement("div");
+ messageDiv.className = "message";
+ const time = new Date().toLocaleTimeString();
+
+ // Check if message contains newlines (like JSON), use <pre> for formatting
+ if (message.includes('\n')) {
+ messageDiv.innerHTML = `<strong>[${time}] ${sender}:</strong><pre style="margin: 0.5rem 0; padding: 0.5rem; background: hsl(210, 40%, 96.1%); border-radius: 0.25rem; overflow-x: auto; font-size: 0.75rem;">${message}</pre>`;
+ } else {
+ messageDiv.innerHTML = `<strong>[${time}] ${sender}:</strong> ${message}`;
+ }
+
+ messagesDiv.insertBefore(messageDiv, messagesDiv.firstChild);
+ messagesDiv.scrollTop = 0;
+ }
+
+ function appendTranscript(sender, text) {
+ const messagesDiv = document.getElementById("messages");
+
+ // If there's no current message element yet, create a new one
+ if (!currentTranscriptElement) {
+ currentTranscript = "";
+ currentTranscriptElement = document.createElement("div");
+ currentTranscriptElement.className = "message";
+ const time = new Date().toLocaleTimeString();
+ currentTranscriptElement.innerHTML = `<strong>[${time}] ${sender}:</strong> <span class="transcript-content"></span>`;
+ messagesDiv.insertBefore(currentTranscriptElement, messagesDiv.firstChild);
+ }
+
+ // Accumulate text
+ currentTranscript += text;
+
+ // Update displayed content
+ const contentSpan = currentTranscriptElement.querySelector('.transcript-content');
+ if (contentSpan) {
+ contentSpan.textContent = currentTranscript;
+ }
+
+ // Scroll to top
+ messagesDiv.scrollTop = 0;
+ }
+
+ function finishTranscript() {
+ // Complete current transcript message, prepare for next one
+ currentTranscript = "";
+ currentTranscriptElement = null;
+ }
+
+ function appendResponseTranscript(sender, text) {
+ const messagesDiv = document.getElementById("messages");
+
+ // If there's no current response message element yet, create a new one
+ if (!currentResponseTranscriptElement) {
+ currentResponseTranscript = "";
+ currentResponseTranscriptElement = document.createElement("div");
+ currentResponseTranscriptElement.className = "message";
+ const time = new Date().toLocaleTimeString();
+ currentResponseTranscriptElement.innerHTML = `<strong>[${time}] ${sender}:</strong> <span class="response-transcript-content"></span>`;
+ messagesDiv.insertBefore(currentResponseTranscriptElement, messagesDiv.firstChild);
+ }
+
+ // Accumulate text
+ currentResponseTranscript += text;
+
+ // Update displayed content
+ const contentSpan = currentResponseTranscriptElement.querySelector('.response-transcript-content');
+ if (contentSpan) {
+ contentSpan.textContent = currentResponseTranscript;
+ }
+
+ // Scroll to top
+ messagesDiv.scrollTop = 0;
+ }
+
+ function finishResponseTranscript() {
+ // Complete current response transcript message, prepare for next one
+ currentResponseTranscript = "";
+ currentResponseTranscriptElement = null;
+ }
+
+
+ // Check available models and auto-connect when page loads
+ window.onload = async function() {
+ await checkAvailableModels();
+ connect();
+ };
+ </script>
+</body>
+</html>
\ No newline at end of file
diff --git a/examples/agent/realtime_voice_agent/run_server.py b/examples/agent/realtime_voice_agent/run_server.py
new file mode 100644
index 000000000..1f4e1e609
--- /dev/null
+++ b/examples/agent/realtime_voice_agent/run_server.py
@@ -0,0 +1,188 @@
+# -*- coding: utf-8 -*-
+"""A test server"""
+import asyncio
+import os
+import traceback
+from pathlib import Path
+
+import uvicorn
+from fastapi import FastAPI, WebSocket
+from fastapi.responses import FileResponse
+
+from agentscope import logger
+from agentscope.agent import RealtimeAgent
+from agentscope.realtime import (
+ DashScopeRealtimeModel,
+ GeminiRealtimeModel,
+ OpenAIRealtimeModel,
+ ClientEvents,
+ ServerEvents,
+ ClientEventType,
+)
+from agentscope.tool import (
+ Toolkit,
+ execute_python_code,
+ execute_shell_command,
+ view_text_file,
+)
+
+app = FastAPI()
+
+
+async def get() -> FileResponse:
+ """Serve the HTML test page."""
+ html_path = Path(__file__).parent / "chatbot.html"
+ return FileResponse(html_path)
+
+
[email protected]("/api/check-models")
+async def check_models() -> dict:
+ """Check which model API keys are available in environment variables."""
+ return {
+ "dashscope": bool(os.getenv("DASHSCOPE_API_KEY")),
+ "gemini": bool(os.getenv("GEMINI_API_KEY")),
+ "openai": bool(os.getenv("OPENAI_API_KEY")),
+ }
+
+
+async def frontend_receive(
+ websocket: WebSocket,
+ frontend_queue: asyncio.Queue,
+) -> None:
+ """Forward the message received from the agent to the frontend."""
+ try:
+ while True:
+ msg: ServerEvents.EventBase = await frontend_queue.get()
+
+ # Send the message as JSON
+ await websocket.send_json(msg.model_dump())
+
+ except Exception as e:
+ print(f"[ERROR] frontend_receive error: {e}")
+ traceback.print_exc()
+
+
[email protected]("/ws/{user_id}/{session_id}")
+async def single_agent_endpoint(
+ websocket: WebSocket,
+ user_id: str,
+ session_id: str,
+) -> None:
+ """WebSocket endpoint for a single realtime agent."""
+ try:
+ await websocket.accept()
+
+ logger.info(
+ "Connected to WebSocket: user_id=%s, session_id=%s",
+ user_id,
+ session_id,
+ )
+
+ # Create the queue to forward messages to the frontend
+ frontend_queue = asyncio.Queue()
+ asyncio.create_task(
+ frontend_receive(websocket, frontend_queue),
+ )
+
+ # Create the realtime agent
+ agent = None
+
+ while True:
+ # Handle the incoming messages from the frontend
+ # i.e. ClientEvents
+ data = await websocket.receive_json()
+
+ client_event = ClientEvents.from_json(data)
+
+ if isinstance(
+ client_event,
+ ClientEvents.ClientSessionCreateEvent,
+ ):
+ # Create the agent by the given session arguments
+ instructions = client_event.config.get(
+ "instructions",
+ "You're a helpful assistant.",
+ )
+ agent_name = client_event.config.get("agent_name", "Friday")
+ model_provider = client_event.config.get(
+ "model_provider",
+ "dashscope",
+ )
+
+ sys_prompt = instructions
+
+ # Create toolkit with tools for models that support them
+ toolkit = None
+ if model_provider in ["gemini", "openai"]:
+ toolkit = Toolkit()
+ toolkit.register_tool_function(execute_python_code)
+ toolkit.register_tool_function(execute_shell_command)
+ toolkit.register_tool_function(view_text_file)
+
+ # Create the appropriate model based on provider
+ if model_provider == "dashscope":
+ model = DashScopeRealtimeModel(
+ model_name="qwen3-omni-flash-realtime",
+ api_key=os.getenv("DASHSCOPE_API_KEY"),
+ )
+ elif model_provider == "gemini":
+ model = GeminiRealtimeModel(
+ model_name=(
+ "gemini-2.5-flash-native-audio-preview-09-2025"
+ ),
+ api_key=os.getenv("GEMINI_API_KEY"),
+ )
+ elif model_provider == "openai":
+ model = OpenAIRealtimeModel(
+ model_name="gpt-4o-realtime-preview",
+ api_key=os.getenv("OPENAI_API_KEY"),
+ )
+ else:
+ raise ValueError(
+ f"Unsupported model provider: {model_provider}",
+ )
+
+ # Create the agent
+ agent = RealtimeAgent(
+ name=agent_name,
+ sys_prompt=sys_prompt,
+ model=model,
+ toolkit=toolkit,
+ )
+
+ await agent.start(frontend_queue)
+
+ # Send session_created event to frontend
+ await websocket.send_json(
+ ServerEvents.ServerSessionCreatedEvent(
+ session_id=session_id,
+ ).model_dump(),
+ )
+ print(
+ f"Session created successfully: {session_id}",
+ )
+
+ elif client_event.type == ClientEventType.CLIENT_SESSION_END:
+ # End the session with the agent
+ if agent:
+ await agent.stop()
+ agent = None
+
+ else:
+ await agent.handle_input(client_event)
+
+ except Exception as e:
+ print(f"[ERROR] WebSocket endpoint error: {e}")
+ traceback.print_exc()
+ raise
+
+
+if __name__ == "__main__":
+ uvicorn.run(
+ "run_server:app",
+ host="localhost",
+ port=8000,
+ reload=True,
+ log_level="info",
+ )
diff --git a/examples/agent/voice_agent/README.md b/examples/agent/voice_agent/README.md
index 2b92da2be..e73d34c6e 100644
--- a/examples/agent/voice_agent/README.md
+++ b/examples/agent/voice_agent/README.md
@@ -17,4 +17,4 @@ Run the following commands to set up and run the example:
```bash
python main.py
-```
+```
\ No newline at end of file
diff --git a/examples/workflows/multiagent_realtime/README.md b/examples/workflows/multiagent_realtime/README.md
new file mode 100644
index 000000000..048d94266
--- /dev/null
+++ b/examples/workflows/multiagent_realtime/README.md
@@ -0,0 +1,160 @@
+# Multi-Agent Realtime Voice Interaction Example
+
+This example demonstrates how to use AgentScope's `ChatRoom` class to create a multi-agent real-time voice interaction system where two AI agents can have autonomous conversations without user input.
+
+## Features
+
+- 🗣️ **Real-time Voice Interaction**: Two agents communicate through voice in real-time
+- 🤖 **Autonomous Conversation**: Agents converse with each other without user intervention
+- ⚙️ **Customizable Configuration**: Configure agent names and instructions through the web interface
+- 🎨 **Modern UI**: Clean, shadcn-inspired interface for easy interaction
+- 📊 **Live Transcript**: See the conversation transcripts in real-time
+
+## Architecture
+
+The example uses:
+- **Backend**: FastAPI server with WebSocket support
+- **Frontend**: HTML5 with Web Audio API for audio playback
+- **AgentScope Components**:
+ - `ChatRoom`: Manages multiple `RealtimeAgent` instances
+ - `RealtimeAgent`: Handles real-time voice interaction with AI models
+ - `DashScopeRealtimeModel`: DashScope's Qwen3-Omni realtime model
+
+## Prerequisites
+
+1. **Python Dependencies**:
+ ```bash
+ pip install agentscope[dashscope]
+ pip install fastapi uvicorn
+ ```
+
+2. **DashScope API Key**:
+ - Set your DashScope API key as an environment variable:
+ ```bash
+ export DASHSCOPE_API_KEY="your-api-key-here"
+ ```
+
+## Usage
+
+1. **Start the Server**:
+ ```bash
+ python run_server.py
+ ```
+
+2. **Open the Web Interface**:
+ - Navigate to `http://localhost:8000` in your web browser
+
+3. **Configure Agents**:
+ - Set names and instructions for both Agent 1 and Agent 2
+ - Example configurations:
+ - **Agent 1 (Alice)**: "You are Alice, a cheerful and optimistic person who loves to share stories and ask questions. Keep your responses brief and conversational."
+ - **Agent 2 (Bob)**: "You are Bob, a thoughtful and analytical person who enjoys deep conversations. Keep your responses brief and conversational."
+
+4. **Start the Conversation**:
+ - Click the "▶️ Start Conversation" button
+ - The agents will begin conversing autonomously
+ - You'll see transcripts and system messages in the message panel
+ - Audio playback will stream in real-time
+
+5. **Stop the Conversation**:
+ - Click the "⏹️ Stop Conversation" button when you want to end the session
+
+## How It Works
+
+### Backend Flow
+
+1. **WebSocket Connection**: Client connects via WebSocket to `/ws/{user_id}/{session_id}`
+2. **Session Creation**:
+ - Client sends `client_session_create` event with agent configurations
+ - Server creates two `RealtimeAgent` instances with specified names and instructions
+ - Server creates a `ChatRoom` with both agents
+ - Server starts the chat room and returns `session_created` event
+3. **Message Broadcasting**:
+ - `ChatRoom` automatically broadcasts messages between agents
+ - All events (audio, transcripts, etc.) are forwarded to the frontend
+4. **Session End**: Client sends `client_session_end` event to stop the conversation
+
+### Frontend Flow
+
+1. **WebSocket Setup**: Establishes connection and waits for server events
+2. **Session Management**: Sends configuration and manages conversation state
+3. **Audio Playback**:
+ - Receives base64-encoded PCM16 audio chunks
+ - Decodes and queues audio data
+ - Uses Web Audio API `ScriptProcessorNode` for streaming playback at 24kHz
+4. **Transcript Display**: Shows real-time transcripts from both agents
+
+## Key Components
+
+### ChatRoom
+
+The `ChatRoom` class manages multiple `RealtimeAgent` instances:
+- Establishes connections for all agents
+- Broadcasts messages between agents automatically
+- Forwards events to the frontend
+- Handles lifecycle management (start/stop)
+
+### RealtimeAgent
+
+Each `RealtimeAgent`:
+- Connects to the DashScope realtime API
+- Processes audio input from other agents
+- Generates voice responses
+- Emits events for transcripts, audio, and status updates
+
+## Customization
+
+### Changing the Model
+
+To use a different model, modify the `DashScopeRealtimeModel` configuration in `run_server.py`:
+
+```python
+model=DashScopeRealtimeModel(
+ model_name="your-model-name",
+ api_key=os.getenv("DASHSCOPE_API_KEY"),
+)
+```
+
+### Adding More Agents
+
+To add more agents, modify the agent creation section in `run_server.py`:
+
+```python
+agent3 = RealtimeAgent(
+ name=agent3_name,
+ sys_prompt=agent3_instructions,
+ model=DashScopeRealtimeModel(
+ model_name="qwen3-omni-flash-realtime",
+ api_key=os.getenv("DASHSCOPE_API_KEY"),
+ ),
+)
+
+chat_room = ChatRoom(agents=[agent1, agent2, agent3])
+```
+
+And update the frontend to include configuration fields for the additional agents.
+
+## Troubleshooting
+
+### No Audio Playback
+- Ensure your browser supports Web Audio API
+- Check browser console for audio-related errors
+- Verify the audio format matches the expected PCM16 at 24kHz
+
+### Connection Issues
+- Verify your DashScope API key is set correctly
+- Check that port 8000 is not blocked by firewall
+- Review server logs for error messages
+
+### Agents Not Responding
+- Ensure both agent configurations have valid instructions
+- Check that the instructions encourage conversational behavior
+- Review the console logs for API errors
+
+## References
+
+- [AgentScope Documentation](https://modelscope.github.io/agentscope/)
+- [DashScope API Documentation](https://help.aliyun.com/zh/model-studio/)
+- [FastAPI Documentation](https://fastapi.tiangolo.com/)
+- [Web Audio API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API)
+
diff --git a/examples/workflows/multiagent_realtime/multi_agent.html b/examples/workflows/multiagent_realtime/multi_agent.html
new file mode 100644
index 000000000..2e3893cf4
--- /dev/null
+++ b/examples/workflows/multiagent_realtime/multi_agent.html
@@ -0,0 +1,1273 @@
+<!DOCTYPE html>
+<html>
+<head>
+ <title>Multi-Agent Realtime Voice Interaction</title>
+ <meta charset="UTF-8">
+ <style>
+ * {
+ margin: 0;
+ padding: 0;
+ box-sizing: border-box;
+ }
+
+ body {
+ font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;
+ max-width: 1200px;
+ margin: 0 auto;
+ padding: 2rem;
+ background: hsl(0, 0%, 98%);
+ color: hsl(222.2, 84%, 4.9%);
+ line-height: 1.5;
+ }
+
+ h1 {
+ font-size: 2rem;
+ font-weight: 600;
+ margin-bottom: 0.5rem;
+ color: hsl(222.2, 84%, 4.9%);
+ letter-spacing: -0.025em;
+ }
+
+ .subtitle {
+ font-size: 0.875rem;
+ color: hsl(215.4, 16.3%, 46.9%);
+ margin-bottom: 1.5rem;
+ }
+
+ #messages {
+ border: 1px solid hsl(214.3, 31.8%, 91.4%);
+ height: 400px;
+ overflow-y: auto;
+ padding: 1rem;
+ margin: 1.5rem 0;
+ background: hsl(0, 0%, 100%);
+ border-radius: 0.5rem;
+ box-shadow: 0 1px 3px 0 rgb(0 0 0 / 0.1), 0 1px 2px -1px rgb(0 0 0 / 0.1);
+ }
+
+ #messages::-webkit-scrollbar {
+ width: 8px;
+ }
+
+ #messages::-webkit-scrollbar-track {
+ background: hsl(210, 40%, 96.1%);
+ border-radius: 4px;
+ }
+
+ #messages::-webkit-scrollbar-thumb {
+ background: hsl(215.4, 16.3%, 56.9%);
+ border-radius: 4px;
+ }
+
+ #messages::-webkit-scrollbar-thumb:hover {
+ background: hsl(215.4, 16.3%, 46.9%);
+ }
+
+ input[type="text"] {
+ width: 100%;
+ padding: 0.625rem 0.875rem;
+ font-size: 0.875rem;
+ border: 1px solid hsl(214.3, 31.8%, 91.4%);
+ border-radius: 0.375rem;
+ background: hsl(0, 0%, 100%);
+ color: hsl(222.2, 84%, 4.9%);
+ transition: all 0.15s ease;
+ outline: none;
+ }
+
+ input[type="text"]:focus {
+ border-color: hsl(221.2, 83.2%, 53.3%);
+ box-shadow: 0 0 0 3px hsl(221.2, 83.2%, 53.3%, 0.1);
+ }
+
+ textarea {
+ width: 100%;
+ min-height: 80px;
+ padding: 0.625rem 0.875rem;
+ font-size: 0.875rem;
+ border: 1px solid hsl(214.3, 31.8%, 91.4%);
+ border-radius: 0.375rem;
+ background: hsl(0, 0%, 100%);
+ color: hsl(222.2, 84%, 4.9%);
+ transition: all 0.15s ease;
+ outline: none;
+ resize: vertical;
+ font-family: inherit;
+ line-height: 1.5;
+ }
+
+ textarea:focus {
+ border-color: hsl(221.2, 83.2%, 53.3%);
+ box-shadow: 0 0 0 3px hsl(221.2, 83.2%, 53.3%, 0.1);
+ }
+
+ button {
+ display: inline-flex;
+ align-items: center;
+ justify-content: center;
+ padding: 0.625rem 1rem;
+ font-size: 0.875rem;
+ font-weight: 500;
+ border: none;
+ border-radius: 0.375rem;
+ cursor: pointer;
+ transition: all 0.15s ease;
+ background: hsl(222.2, 47.4%, 11.2%);
+ color: hsl(210, 40%, 98%);
+ box-shadow: 0 1px 2px 0 rgb(0 0 0 / 0.05);
+ }
+
+ button:hover:not(:disabled) {
+ background: hsl(222.2, 47.4%, 15%);
+ }
+
+ button:active:not(:disabled) {
+ transform: scale(0.98);
+ }
+
+ button:focus-visible {
+ outline: 2px solid hsl(221.2, 83.2%, 53.3%);
+ outline-offset: 2px;
+ }
+
+ button:disabled {
+ opacity: 0.5;
+ cursor: not-allowed;
+ }
+
+ button.primary {
+ background: hsl(221.2, 83.2%, 53.3%);
+ color: hsl(0, 0%, 100%);
+ }
+
+ button.primary:hover:not(:disabled) {
+ background: hsl(221.2, 83.2%, 48%);
+ }
+
+ button.destructive {
+ background: hsl(0, 84.2%, 60.2%);
+ color: hsl(0, 0%, 100%);
+ }
+
+ button.destructive:hover:not(:disabled) {
+ background: hsl(0, 84.2%, 55%);
+ }
+
+ button.active {
+ background: hsl(142.1, 76.2%, 36.3%);
+ animation: pulse 1.5s ease-in-out infinite;
+ }
+
+ @keyframes pulse {
+ 0%, 100% {
+ opacity: 1;
+ box-shadow: 0 0 0 0 hsl(142.1, 76.2%, 36.3%, 0.7);
+ }
+ 50% {
+ opacity: 0.9;
+ box-shadow: 0 0 0 8px hsl(142.1, 76.2%, 36.3%, 0);
+ }
+ }
+
+ .message {
+ margin: 0.75rem 0;
+ padding: 0.75rem 1rem;
+ background: hsl(210, 40%, 98%);
+ border-radius: 0.5rem;
+ border: 1px solid hsl(214.3, 31.8%, 91.4%);
+ font-size: 0.875rem;
+ }
+
+ .message strong {
+ color: hsl(222.2, 47.4%, 11.2%);
+ font-weight: 600;
+ }
+
+ .controls {
+ display: flex;
+ flex-wrap: wrap;
+ gap: 0.75rem;
+ margin: 1.5rem 0;
+ }
+
+ .configuration-container {
+ background: hsl(0, 0%, 100%);
+ padding: 1.5rem;
+ border-radius: 0.5rem;
+ margin: 1.5rem 0;
+ border: 1px solid hsl(214.3, 31.8%, 91.4%);
+ box-shadow: 0 1px 3px 0 rgb(0 0 0 / 0.1), 0 1px 2px -1px rgb(0 0 0 / 0.1);
+ }
+
+ .configuration-container h3 {
+ font-size: 1.125rem;
+ font-weight: 600;
+ margin-bottom: 1rem;
+ color: hsl(222.2, 84%, 4.9%);
+ display: flex;
+ align-items: center;
+ gap: 0.5rem;
+ }
+
+ .agents-grid {
+ display: grid;
+ grid-template-columns: 1fr 1fr;
+ gap: 1.5rem;
+ margin-bottom: 1.5rem;
+ }
+
+ .agent-config {
+ padding: 1rem;
+ background: hsl(210, 40%, 98%);
+ border-radius: 0.5rem;
+ border: 1px solid hsl(214.3, 31.8%, 91.4%);
+ }
+
+ .agent-config h4 {
+ font-size: 1rem;
+ font-weight: 600;
+ margin-bottom: 0.75rem;
+ color: hsl(222.2, 47.4%, 11.2%);
+ }
+
+ .config-field {
+ margin-bottom: 1rem;
+ }
+
+ .config-field:last-child {
+ margin-bottom: 0;
+ }
+
+ .config-field label {
+ display: block;
+ font-weight: 500;
+ margin-bottom: 0.5rem;
+ color: hsl(222.2, 47.4%, 11.2%);
+ font-size: 0.875rem;
+ }
+
+ .error-message {
+ padding: 0.875rem 1rem;
+ background: hsl(0, 84.2%, 95%);
+ border: 1px solid hsl(0, 84.2%, 85%);
+ border-radius: 0.5rem;
+ margin: 1rem 0;
+ display: none;
+ color: hsl(0, 84.2%, 30%);
+ font-size: 0.875rem;
+ font-weight: 500;
+ box-shadow: 0 1px 2px 0 rgb(0 0 0 / 0.05);
+ }
+
+ .model-options {
+ display: flex;
+ flex-direction: row;
+ gap: 0.75rem;
+ }
+
+ .model-option {
+ flex: 1;
+ padding: 0.75rem;
+ border: 2px solid hsl(214.3, 31.8%, 91.4%);
+ border-radius: 0.5rem;
+ cursor: pointer;
+ transition: all 0.15s ease;
+ background: hsl(0, 0%, 100%);
+ display: flex;
+ flex-direction: column;
+ justify-content: center;
+ gap: 0.5rem;
+ }
+
+ .model-option:hover:not(.disabled) {
+ border-color: hsl(221.2, 83.2%, 53.3%);
+ background: hsl(221.2, 83.2%, 98%);
+ }
+
+ .model-option.selected {
+ border-color: hsl(221.2, 83.2%, 53.3%);
+ background: hsl(221.2, 83.2%, 95%);
+ }
+
+ .model-option.disabled {
+ opacity: 0.5;
+ cursor: not-allowed;
+ background: hsl(0, 0%, 98%);
+ }
+
+ .model-option-header {
+ display: flex;
+ align-items: center;
+ gap: 0.5rem;
+ }
+
+ .model-option input[type="radio"] {
+ margin: 0;
+ cursor: pointer;
+ flex-shrink: 0;
+ }
+
+ .model-option.disabled input[type="radio"] {
+ cursor: not-allowed;
+ }
+
+ .model-info {
+ display: flex;
+ flex-direction: column;
+ gap: 0.5rem;
+ flex: 1;
+ }
+
+ .model-name-line {
+ display: flex;
+ align-items: center;
+ gap: 0.5rem;
+ min-height: 1.25rem;
+ }
+
+ .model-name {
+ font-weight: 600;
+ color: hsl(222.2, 84%, 4.9%);
+ }
+
+ .model-unavailable-reason {
+ font-size: 0.625rem;
+ color: hsl(215.4, 16.3%, 56.9%);
+ font-style: italic;
+ white-space: nowrap;
+ }
+
+ @media (max-width: 768px) {
+ .agents-grid {
+ grid-template-columns: 1fr;
+ }
+ }
+
+
+
+ .audio-mode-selector {
+ margin-bottom: 1rem;
+ }
+
+ .audio-mode-selector label {
+ display: block;
+ font-weight: 500;
+ margin-bottom: 0.5rem;
+ color: hsl(222.2, 47.4%, 11.2%);
+ font-size: 0.875rem;
+ }
+
+ .preset-group {
+ display: flex;
+ gap: 1rem;
+ flex-wrap: wrap;
+ }
+
+ .preset-card {
+ flex: 1;
+ min-width: 200px;
+ padding: 1rem 1.25rem;
+ background: hsl(0, 0%, 100%);
+ border: 2px solid hsl(214.3, 31.8%, 91.4%);
+ border-radius: 0.5rem;
+ cursor: pointer;
+ transition: all 0.2s ease;
+ text-align: center;
+ font-weight: 500;
+ font-size: 0.875rem;
+ color: hsl(222.2, 84%, 4.9%);
+ user-select: none;
+ }
+
+ .preset-card:hover {
+ border-color: hsl(221.2, 83.2%, 53.3%);
+ background: hsl(210, 40%, 98%);
+ transform: translateY(-2px);
+ box-shadow: 0 4px 6px -1px rgb(0 0 0 / 0.1), 0 2px 4px -2px rgb(0 0 0 / 0.1);
+ }
+
+ .preset-card.selected {
+ background: hsl(221.2, 83.2%, 53.3%);
+ border-color: hsl(221.2, 83.2%, 53.3%);
+ color: hsl(0, 0%, 100%);
+ box-shadow: 0 4px 6px -1px rgb(0 0 0 / 0.2), 0 2px 4px -2px rgb(0 0 0 / 0.2);
+ }
+
+ .preset-card.selected:hover {
+ background: hsl(221.2, 83.2%, 48%);
+ border-color: hsl(221.2, 83.2%, 48%);
+ transform: translateY(-2px);
+ }
+ </style>
+</head>
+<body>
+ <h1>🗣️ Multi-Agent Realtime Voice Interaction</h1>
+ <p class="subtitle">Two AI agents having a real-time voice conversation using AgentScope ChatRoom</p>
+
+ <div class="configuration-container">
+ <h3>⚙️ Agent Configuration</h3>
+
+
+ <div class="audio-mode-selector">
+ <label>🎭 Debate Preset</label>
+ <div class="preset-group">
+ <div class="preset-card" data-preset="ai" onclick="selectPreset('ai')">
+ 🤖 AI Threat v.s. AI Optimism (en)
+ </div>
+ <div class="preset-card" data-preset="human" onclick="selectPreset('human')">
+ 👥 人性本恶 v.s. 人性本善 (zh)
+ </div>
+ </div>
+ </div>
+
+ <div class="config-field">
+ <label>Model Provider</label>
+ <div class="model-options" id="modelOptions">
+ <label class="model-option" data-provider="dashscope">
+ <div class="model-option-header">
+ <input type="radio" name="modelProvider" value="dashscope" checked />
+ <div class="model-info">
+ <div class="model-name-line">
+ <span class="model-name">DashScope</span>
+ <span class="model-unavailable-reason" style="display: none;"></span>
+ </div>
+ </div>
+ </div>
+ </label>
+ <label class="model-option" data-provider="gemini">
+ <div class="model-option-header">
+ <input type="radio" name="modelProvider" value="gemini" />
+ <div class="model-info">
+ <div class="model-name-line">
+ <span class="model-name">Gemini</span>
+ <span class="model-unavailable-reason" style="display: none;"></span>
+ </div>
+ </div>
+ </div>
+ </label>
+ <label class="model-option" data-provider="openai">
+ <div class="model-option-header">
+ <input type="radio" name="modelProvider" value="openai" />
+ <div class="model-info">
+ <div class="model-name-line">
+ <span class="model-name">OpenAI</span>
+ <span class="model-unavailable-reason" style="display: none;"></span>
+ </div>
+ </div>
+ </div>
+ </label>
+ </div>
+ </div>
+
+ <div class="agents-grid">
+ <div class="agent-config">
+ <h4>🤖 Agent 1</h4>
+ <div class="config-field">
+ <label for="agent1Name">Name</label>
+ <input type="text" id="agent1Name" placeholder="Enter agent 1 name" value="Elon Musk" />
+ </div>
+ <div class="config-field">
+ <label for="agent1Instructions">Instructions</label>
+ <textarea id="agent1Instructions" placeholder="Enter agent 1 instructions...">You're Elon Musk arguing with Mark Zuckerberg about whether AI is humanity's greatest threat. You believe it absolutely is - we're building superintelligence without safeguards, risking autonomous weapons, mass job loss, and losing control to machines smarter than us. Be bold, direct, and warn that optimists like Zuck are dangerously naive. Use your trademark sarcasm and urgency. You must keep each response under 100 words.</textarea>
+ </div>
+ </div>
+
+ <div class="agent-config">
+ <h4>🤖 Agent 2</h4>
+ <div class="config-field">
+ <label for="agent2Name">Name</label>
+ <input type="text" id="agent2Name" placeholder="Enter agent 2 name" value="Mark Zuckerberg" />
+ </div>
+ <div class="config-field">
+ <label for="agent2Instructions">Instructions</label>
+ <textarea id="agent2Instructions" placeholder="Enter agent 2 instructions...">You're Mark Zuckerberg arguing with Elon Musk about whether AI is humanity's greatest threat. You believe it's not - climate change, pandemics, and inequality are bigger dangers, while AI is already saving lives and solving problems. Be calm, rational, and data-driven. Counter Elon's fearmongering with practical AI benefits and argue responsible development beats fear-based thinking.</textarea>
+ </div>
+ </div>
+ </div>
+ </div>
+
+ <div id="errorMessage" class="error-message"></div>
+
+ <div class="controls">
+ <button id="startBtn" class="primary" onclick="startConversation()">▶️ Start Conversation</button>
+ <button id="stopBtn" class="destructive" onclick="stopConversation()" disabled>⏹️ Stop Conversation</button>
+ </div>
+
+ <div id="messages"></div>
+
+ <script>
+ let ws = null;
+ let playbackAudioContext = null; // For playback, 24kHz
+
+ // Audio queue management: sequential playback
+ let globalAudioQueue = []; // Array of { agentId, agentName, chunks, messageElement }
+ let currentlyPlaying = null; // Currently playing agent info
+ let isPlayingGlobal = false;
+
+ let sessionId = "session1"; // Session ID
+ let sessionCreated = false; // Track if session has been created
+ let isConversationActive = false;
+
+ // Used to accumulate transcript text
+ let currentResponseTranscripts = {}; // Store by agent_id
+
+
+ // Silent audio monitoring for continuous audio stream
+ let lastAudioReceivedTime = null;
+ let silentAudioCheckInterval = null;
+ const SILENT_AUDIO_CHECK_INTERVAL = 2000; // Check every 1000ms (1 second)
+ const SILENT_AUDIO_THRESHOLD = 2000; // Send silent audio if no audio received for 200ms
+
+ // Debate presets
+ const debatePresets = {
+ ai: {
+ agent1: {
+ name: "Elon Musk",
+ instructions: "You're Elon Musk arguing with Mark Zuckerberg about whether AI is humanity's greatest threat. You believe it absolutely is - we're building superintelligence without safeguards, risking autonomous weapons, mass job loss, and losing control to machines smarter than us. Be bold, direct, and warn that optimists like Zuck are dangerously naive. Use your trademark sarcasm and urgency. You must keep each response under 100 words."
+ },
+ agent2: {
+ name: "Mark Zuckerberg",
+ instructions: "You're Mark Zuckerberg arguing with Elon Musk about whether AI is humanity's greatest threat. You believe it's not - climate change, pandemics, and inequality are bigger dangers, while AI is already saving lives and solving problems. Be calm, rational, and data-driven. Counter Elon's fearmongering with practical AI benefits and argue responsible development beats fear-based thinking."
+ }
+ },
+ human: {
+ agent1: {
+ name: "郭德纲",
+ instructions: "你是郭德纲,你要和于谦进行一场关于\"人性本恶\"的辩论。你坚定地认为人性本恶,人生来就有自私、贪婪等恶的一面,只有通过教育、道德约束和社会规范才能引导人向善。用你幽默风趣的语言风格,结合相声的表达方式,用生动的例子和机智的话语来论证你的观点。每次回应控制在100字以内。"
+ },
+ agent2: {
+ name: "于谦",
+ instructions: "你是于谦,你要和郭德纲进行一场关于\"人性本善\"的辩论。你坚信人性本善,人生来就有善良、同情等美好品质,恶是后天环境影响造成的。用你朴实、真诚的语言风格,举出人性光辉的例子来反驳郭德纲的观点。保持相声捧哏的风格,既要论证有力,又要配合好逗哏。每次回应控制在100字以内。"
+ }
+ }
+ };
+
+ // Track currently selected preset
+ let currentPreset = null;
+
+ function selectPreset(presetType) {
+ if (!debatePresets[presetType]) return;
+
+ // Update selected state
+ currentPreset = presetType;
+ document.querySelectorAll('.preset-card').forEach(card => {
+ card.classList.remove('selected');
+ });
+ document.querySelector(`.preset-card[data-preset="${presetType}"]`).classList.add('selected');
+
+ // Load preset values
+ const preset = debatePresets[presetType];
+ document.getElementById("agent1Name").value = preset.agent1.name;
+ document.getElementById("agent1Instructions").value = preset.agent1.instructions;
+ document.getElementById("agent2Name").value = preset.agent2.name;
+ document.getElementById("agent2Instructions").value = preset.agent2.instructions;
+
+ console.log(`Loaded ${presetType} debate preset`);
+ }
+
+ function clearPresetSelection() {
+ currentPreset = null;
+ document.querySelectorAll('.preset-card').forEach(card => {
+ card.classList.remove('selected');
+ });
+ }
+
+ // Monitor input changes to clear preset selection
+ function setupInputMonitoring() {
+ const inputs = [
+ document.getElementById("agent1Name"),
+ document.getElementById("agent1Instructions"),
+ document.getElementById("agent2Name"),
+ document.getElementById("agent2Instructions")
+ ];
+
+ inputs.forEach(input => {
+ input.addEventListener('input', () => {
+ // Check if current values match any preset
+ if (currentPreset && debatePresets[currentPreset]) {
+ const preset = debatePresets[currentPreset];
+ const agent1Name = document.getElementById("agent1Name").value;
+ const agent1Instructions = document.getElementById("agent1Instructions").value;
+ const agent2Name = document.getElementById("agent2Name").value;
+ const agent2Instructions = document.getElementById("agent2Instructions").value;
+
+ // If values don't match preset, clear selection
+ if (agent1Name !== preset.agent1.name ||
+ agent1Instructions !== preset.agent1.instructions ||
+ agent2Name !== preset.agent2.name ||
+ agent2Instructions !== preset.agent2.instructions) {
+ clearPresetSelection();
+ }
+ }
+ });
+ });
+ }
+
+ function showError(message) {
+ const errorDiv = document.getElementById("errorMessage");
+ errorDiv.innerText = message;
+ errorDiv.style.display = "block";
+ setTimeout(() => {
+ errorDiv.style.display = "none";
+ }, 5000);
+ }
+
+ function generateSilentAudio(durationMs, sampleRate) {
+ // Calculate number of samples needed
+ const numSamples = Math.floor(sampleRate * durationMs / 1000);
+ // Create Int16Array filled with zeros (silence)
+ const int16Array = new Int16Array(numSamples);
+ return int16Array.buffer;
+ }
+
+ function arrayBufferToBase64(buffer) {
+ const bytes = new Uint8Array(buffer);
+ let binary = '';
+ for (let i = 0; i < bytes.byteLength; i++) {
+ binary += String.fromCharCode(bytes[i]);
+ }
+ return btoa(binary);
+ }
+
+
+ function startSilentAudioMonitoring() {
+ // Initialize timestamp
+ lastAudioReceivedTime = Date.now();
+
+ // Start periodic check
+ silentAudioCheckInterval = setInterval(() => {
+ checkAndSendSilentAudio();
+ }, SILENT_AUDIO_CHECK_INTERVAL);
+
+ console.log("Started silent audio monitoring");
+ }
+
+ function stopSilentAudioMonitoring() {
+ if (silentAudioCheckInterval) {
+ clearInterval(silentAudioCheckInterval);
+ silentAudioCheckInterval = null;
+ }
+ lastAudioReceivedTime = null;
+ console.log("Stopped silent audio monitoring");
+ }
+
+ function checkAndSendSilentAudio() {
+ // Check if session is active
+ if (!isConversationActive || !ws || ws.readyState !== WebSocket.OPEN) {
+ return;
+ }
+
+ // Check if any audio is playing
+ if (isPlayingGlobal) {
+ // If audio is playing, update timestamp (playing also counts as audio activity)
+ lastAudioReceivedTime = Date.now();
+ return;
+ }
+
+ // Calculate time since last audio received
+ const now = Date.now();
+ const timeSinceLastAudio = now - lastAudioReceivedTime;
+
+ // If threshold exceeded, send silent audio
+ if (timeSinceLastAudio >= SILENT_AUDIO_THRESHOLD) {
+ sendSilentAudio(timeSinceLastAudio);
+ // Update timestamp
+ lastAudioReceivedTime = now;
+ }
+ }
+
+ function sendSilentAudio(durationMs) {
+ if (!ws || ws.readyState !== WebSocket.OPEN) {
+ return;
+ }
+
+ // Limit maximum duration to avoid sending too large data
+ const maxDuration = 2000; // Max 1 second
+ const actualDuration = Math.min(durationMs, maxDuration);
+
+ // Generate silent audio
+ const silentBuffer = generateSilentAudio(actualDuration, 16000);
+ const base64Audio = arrayBufferToBase64(silentBuffer);
+
+ // Send audio data
+ ws.send(JSON.stringify({
+ type: "client_audio_append",
+ session_id: sessionId,
+ audio: base64Audio,
+ format: {
+ rate: 16000,
+ type: "audio/pcm"
+ }
+ }));
+
+ console.log(`Sent ${actualDuration}ms silent audio to backend`);
+ }
+
+
+ async function connect() {
+ const protocol = window.location.protocol === 'https:' ? 'wss:' : 'ws:';
+ const host = window.location.hostname;
+ const port = window.location.port || (window.location.protocol === 'https:' ? '443' : '80');
+ const wsUrl = `${protocol}//${host}:${port}/ws/user1/${sessionId}`;
+
+ console.log(`Connecting to WebSocket: ${wsUrl}`);
+ ws = new WebSocket(wsUrl);
+
+ ws.onopen = function(event) {
+ addMessage("System", "✅ Connected to server");
+ };
+
+ ws.onmessage = async function(event) {
+ try {
+ const data = JSON.parse(event.data);
+ console.log("Received message:", data);
+
+ // Handle ServerEvents
+ switch (data.type) {
+ case "server_session_created":
+ sessionCreated = true;
+ addMessage("System", `✅ Chat room created: ${data.session_id}`);
+ break;
+
+ case "agent_ready":
+ addMessage("System", `🤖 Agent ${data.agent_name} is ready`);
+ break;
+
+ case "agent_response_created":
+ // Initialize audio buffer for this response (no message display)
+ if (data.agent_id) {
+ initializeAudioBuffer(data.agent_id, data.agent_name);
+ }
+ break;
+
+ case "agent_response_audio_delta":
+ // Update last audio received timestamp
+ lastAudioReceivedTime = Date.now();
+
+ // Receive audio data
+ console.log(`Received audio delta from agent: ${data.agent_name} (${data.agent_id})`);
+ if (data.agent_id) {
+ queueAudioChunk(data.agent_id, data.agent_name, data.delta);
+ }
+ break;
+
+ case "agent_response_audio_done":
+ // Mark audio complete (no message display)
+ if (data.agent_id) {
+ markAudioComplete(data.agent_id);
+ }
+ break;
+
+ case "agent_response_audio_transcript_delta":
+ // Agent response transcript text
+ appendResponseTranscript(
+ data.agent_name,
+ data.delta || "",
+ data.agent_id
+ );
+ break;
+
+ case "agent_response_audio_transcript_done":
+ // Complete Agent response transcript message
+ finishResponseTranscript(data.agent_id);
+ break;
+
+ case "agent_response_done":
+ // Response completed (no message display)
+ break;
+
+ case "agent_error":
+ addMessage("Error", `❌ ${data.error_type}: ${data.message}`);
+ break;
+
+ case "agent_ended":
+ addMessage("System", `👋 Agent ${data.agent_name} has ended`);
+ break;
+
+ case "server_session_ended":
+ addMessage("System", `🔚 Session ${data.session_id} has ended`);
+ break;
+
+ default:
+ console.log("Unhandled event type:", data.type);
+ break;
+ }
+ } catch (e) {
+ console.error("Error processing message:", e);
+ }
+ };
+
+ ws.onclose = function(event) {
+ addMessage("System", "❌ Disconnected");
+ stopSilentAudioMonitoring();
+ sessionCreated = false;
+ isConversationActive = false;
+ updateButtons();
+ };
+
+ ws.onerror = function(error) {
+ addMessage("System", "⚠️ Connection error");
+ };
+ }
+
+ async function startConversation() {
+ try {
+ // Validate inputs
+ const agent1Name = document.getElementById("agent1Name").value.trim();
+ const agent1Instructions = document.getElementById("agent1Instructions").value.trim();
+ const agent2Name = document.getElementById("agent2Name").value.trim();
+ const agent2Instructions = document.getElementById("agent2Instructions").value.trim();
+
+ if (!agent1Name || !agent1Instructions || !agent2Name || !agent2Instructions) {
+ showError("⚠️ All fields are required! Please fill in all agent configurations.");
+ return;
+ }
+
+ // Check if WebSocket is connected
+ if (!ws || ws.readyState !== WebSocket.OPEN) {
+ showError("⚠️ WebSocket is not connected! Please wait for connection.");
+ return;
+ }
+
+ // Disable start button
+ isConversationActive = true;
+ updateButtons();
+
+ // Get selected model provider
+ const selectedModel = document.querySelector('input[name="modelProvider"]:checked');
+ const modelProvider = selectedModel ? selectedModel.value : "dashscope";
+
+ // Send session create event
+ addMessage("System", "📝 Creating chat room...");
+ ws.send(JSON.stringify({
+ type: "client_session_create",
+ config: {
+ agent1_name: agent1Name,
+ agent1_instructions: agent1Instructions,
+ agent2_name: agent2Name,
+ agent2_instructions: agent2Instructions,
+ model_provider: modelProvider
+ }
+ }));
+
+ // Wait for session_created event
+ await new Promise((resolve, reject) => {
+ const timeout = setTimeout(() => {
+ reject(new Error("Session creation timeout"));
+ }, 10000);
+
+ const checkSession = setInterval(() => {
+ if (sessionCreated) {
+ clearTimeout(timeout);
+ clearInterval(checkSession);
+ resolve();
+ }
+ }, 100);
+ });
+
+ addMessage("System", "🎉 Chat room created! Agents are now conversing...");
+
+ // Start silent audio monitoring
+ startSilentAudioMonitoring();
+
+ } catch (err) {
+ console.error("Failed to start conversation:", err);
+ if (err.message === "Session creation timeout") {
+ showError("⚠️ Session creation timeout. Please try again.");
+ addMessage("System", "⚠️ Session creation timeout");
+ } else {
+ showError("⚠️ Failed to start conversation: " + err.message);
+ addMessage("System", "⚠️ Failed to start conversation: " + err.message);
+ }
+ isConversationActive = false;
+ updateButtons();
+ }
+ }
+
+ function stopConversation() {
+ // Stop silent audio monitoring
+ stopSilentAudioMonitoring();
+
+ if (ws && ws.readyState === WebSocket.OPEN) {
+ ws.send(JSON.stringify({
+ type: "client_session_end",
+ session_id: sessionId
+ }));
+ }
+
+ stopAllAudioPlayback();
+ sessionCreated = false;
+ isConversationActive = false;
+ updateButtons();
+ addMessage("System", "⏹️ Conversation stopped");
+ }
+
+ function updateButtons() {
+ document.getElementById("startBtn").disabled = isConversationActive;
+ document.getElementById("stopBtn").disabled = !isConversationActive;
+
+ if (isConversationActive) {
+ document.getElementById("startBtn").classList.remove("primary");
+ document.getElementById("stopBtn").classList.add("active");
+ } else {
+ document.getElementById("startBtn").classList.add("primary");
+ document.getElementById("stopBtn").classList.remove("active");
+ }
+ }
+
+ // ==================== Audio Queue Management ====================
+
+ function initializeAudioBuffer(agentId, agentName) {
+ // Find if this agent already has an entry in the queue
+ let queueEntry = globalAudioQueue.find(entry => entry.agentId === agentId && !entry.isComplete);
+
+ if (!queueEntry) {
+ queueEntry = {
+ agentId: agentId,
+ agentName: agentName,
+ chunks: [],
+ messageElement: null,
+ isComplete: false,
+ playbackIndex: 0
+ };
+ globalAudioQueue.push(queueEntry);
+ console.log(`Initialized audio buffer for agent: ${agentName} (${agentId})`);
+ }
+ }
+
+ function queueAudioChunk(agentId, agentName, base64Audio) {
+ try {
+ // Decode audio chunk
+ const float32Array = decodeAudioChunk(base64Audio);
+
+ // Find or create queue entry for this agent
+ let queueEntry = globalAudioQueue.find(entry => entry.agentId === agentId && !entry.isComplete);
+
+ if (!queueEntry) {
+ queueEntry = {
+ agentId: agentId,
+ agentName: agentName,
+ chunks: [],
+ messageElement: null,
+ isComplete: false,
+ playbackIndex: 0
+ };
+ globalAudioQueue.push(queueEntry);
+ }
+
+ queueEntry.chunks.push(float32Array);
+ console.log(`Queued audio chunk for ${agentName}, total chunks: ${queueEntry.chunks.length}`);
+
+
+ // If not currently playing, start playback
+ if (!isPlayingGlobal) {
+ playNextInQueue();
+ }
+ } catch (err) {
+ console.error("Failed to queue audio chunk:", err);
+ }
+ }
+
+ function markAudioComplete(agentId) {
+ const queueEntry = globalAudioQueue.find(entry => entry.agentId === agentId && !entry.isComplete);
+ if (queueEntry) {
+ queueEntry.isComplete = true;
+ console.log(`Marked audio complete for agent: ${agentId}`);
+ }
+ }
+
+ function playNextInQueue() {
+ if (isPlayingGlobal) return;
+
+ // Find the first entry with audio chunks
+ const nextEntry = globalAudioQueue.find(entry => entry.chunks.length > 0);
+
+ if (!nextEntry) {
+ console.log("No audio in queue to play");
+ return;
+ }
+
+ console.log(`Starting playback for agent: ${nextEntry.agentName}`);
+ currentlyPlaying = nextEntry;
+ isPlayingGlobal = true;
+
+ // Highlight the corresponding message
+ if (nextEntry.messageElement) {
+ nextEntry.messageElement.classList.add('playing-audio');
+ }
+
+
+ startPlayback(nextEntry);
+ }
+
+ function startPlayback(queueEntry) {
+ try {
+ if (!playbackAudioContext) {
+ playbackAudioContext = new (window.AudioContext || window.webkitAudioContext)({
+ sampleRate: 24000
+ });
+ }
+
+ if (playbackAudioContext.state === 'suspended') {
+ playbackAudioContext.resume();
+ }
+
+ const bufferSize = 4096;
+ const processor = playbackAudioContext.createScriptProcessor(bufferSize, 0, 1);
+
+ processor.onaudioprocess = function(e) {
+ const output = e.outputBuffer.getChannelData(0);
+ const samplesNeeded = output.length;
+ let samplesWritten = 0;
+
+ while (samplesWritten < samplesNeeded && queueEntry.chunks.length > 0) {
+ const chunk = queueEntry.chunks[0];
+ const samplesToRead = Math.min(
+ samplesNeeded - samplesWritten,
+ chunk.length - queueEntry.playbackIndex
+ );
+
+ for (let i = 0; i < samplesToRead; i++) {
+ output[samplesWritten + i] = chunk[queueEntry.playbackIndex + i];
+ }
+
+ samplesWritten += samplesToRead;
+ queueEntry.playbackIndex += samplesToRead;
+
+ if (queueEntry.playbackIndex >= chunk.length) {
+ queueEntry.chunks.shift();
+ queueEntry.playbackIndex = 0;
+ }
+ }
+
+ // Fill remaining with silence
+ for (let i = samplesWritten; i < samplesNeeded; i++) {
+ output[i] = 0;
+ }
+
+ // Check if this agent's audio is complete
+ if (queueEntry.chunks.length === 0 && queueEntry.isComplete) {
+ setTimeout(() => {
+ if (queueEntry.chunks.length === 0 && queueEntry.isComplete) {
+ finishCurrentPlayback();
+ }
+ }, 100);
+ }
+ };
+
+ processor.connect(playbackAudioContext.destination);
+ queueEntry.node = processor;
+
+ } catch (err) {
+ console.error("Failed to start playback:", err);
+ finishCurrentPlayback();
+ }
+ }
+
+ function finishCurrentPlayback() {
+ if (!currentlyPlaying) return;
+
+ console.log(`Finished playback for agent: ${currentlyPlaying.agentName}`);
+
+ // Cleanup
+ if (currentlyPlaying.node) {
+ currentlyPlaying.node.disconnect();
+ currentlyPlaying.node = null;
+ }
+
+ // Remove highlight
+ if (currentlyPlaying.messageElement) {
+ currentlyPlaying.messageElement.classList.remove('playing-audio');
+ }
+
+ // Remove from queue
+ const index = globalAudioQueue.indexOf(currentlyPlaying);
+ if (index > -1) {
+ globalAudioQueue.splice(index, 1);
+ }
+
+ currentlyPlaying = null;
+ isPlayingGlobal = false;
+
+ // Play next in queue
+ setTimeout(() => playNextInQueue(), 100);
+ }
+
+
+ // ==================== Common Audio Functions ====================
+
+ function decodeAudioChunk(base64Audio) {
+ const binaryString = atob(base64Audio);
+ const bytes = new Uint8Array(binaryString.length);
+ for (let i = 0; i < binaryString.length; i++) {
+ bytes[i] = binaryString.charCodeAt(i);
+ }
+
+ const int16Array = new Int16Array(bytes.buffer);
+ const float32Array = new Float32Array(int16Array.length);
+
+ for (let i = 0; i < int16Array.length; i++) {
+ float32Array[i] = int16Array[i] / 32768.0;
+ }
+
+ return float32Array;
+ }
+
+ function stopAllAudioPlayback() {
+ // Stop sequential playback
+ if (currentlyPlaying && currentlyPlaying.node) {
+ currentlyPlaying.node.disconnect();
+ currentlyPlaying.node = null;
+ if (currentlyPlaying.messageElement) {
+ currentlyPlaying.messageElement.classList.remove('playing-audio');
+ }
+ }
+ currentlyPlaying = null;
+ isPlayingGlobal = false;
+ globalAudioQueue = [];
+ }
+
+ function addMessage(sender, message) {
+ const messagesDiv = document.getElementById("messages");
+ const messageDiv = document.createElement("div");
+ messageDiv.className = "message";
+ const time = new Date().toLocaleTimeString();
+ messageDiv.innerHTML = `<strong>[${time}] ${sender}:</strong> ${message}`;
+ messagesDiv.insertBefore(messageDiv, messagesDiv.firstChild);
+ messagesDiv.scrollTop = 0;
+ }
+
+ function appendResponseTranscript(agentName, text, agentId) {
+ const messagesDiv = document.getElementById("messages");
+
+ // If there's no current response message element for this agent, create one
+ if (!currentResponseTranscripts[agentId]) {
+ currentResponseTranscripts[agentId] = {
+ text: "",
+ element: document.createElement("div")
+ };
+
+ const transcript = currentResponseTranscripts[agentId];
+ transcript.element.className = "message";
+ const time = new Date().toLocaleTimeString();
+ transcript.element.innerHTML = `<strong>[${time}] ${agentName}:</strong> <span class="response-transcript-content"></span>`;
+ messagesDiv.insertBefore(transcript.element, messagesDiv.firstChild);
+
+ // Link this message element to the audio queue entry
+ const queueEntry = globalAudioQueue.find(entry => entry.agentId === agentId && !entry.isComplete);
+ if (queueEntry) {
+ queueEntry.messageElement = transcript.element;
+ }
+ }
+
+ // Accumulate text
+ currentResponseTranscripts[agentId].text += text;
+
+ // Update displayed content
+ const contentSpan = currentResponseTranscripts[agentId].element.querySelector('.response-transcript-content');
+ if (contentSpan) {
+ contentSpan.textContent = currentResponseTranscripts[agentId].text;
+ }
+
+ // Scroll to top
+ messagesDiv.scrollTop = 0;
+ }
+
+ function finishResponseTranscript(agentId) {
+ // Complete current response transcript message for this agent
+ if (currentResponseTranscripts[agentId]) {
+ delete currentResponseTranscripts[agentId];
+ }
+ }
+
+ async function checkModelAvailability() {
+ try {
+ const response = await fetch('/model_availability');
+ const availability = await response.json();
+ console.log("Model availability:", availability);
+
+ const modelOptions = document.querySelectorAll('.model-option');
+ let hasAvailableModel = false;
+
+ modelOptions.forEach(option => {
+ const provider = option.getAttribute('data-provider');
+ const radio = option.querySelector('input[type="radio"]');
+ const unavailableReason = option.querySelector('.model-unavailable-reason');
+
+ if (!availability[provider]) {
+ // Mark as disabled
+ option.classList.add('disabled');
+ radio.disabled = true;
+
+ // Show unavailable reason
+ const providerName = provider.charAt(0).toUpperCase() + provider.slice(1);
+ unavailableReason.textContent = `(${providerName.toUpperCase()}_API_KEY not set)`;
+ unavailableReason.style.display = 'inline';
+
+ // If this was the selected option, uncheck it
+ if (radio.checked) {
+ radio.checked = false;
+ }
+ } else {
+ hasAvailableModel = true;
+ unavailableReason.style.display = 'none';
+ }
+ });
+
+ // If no model is selected after checking availability, select first available
+ const selectedRadio = document.querySelector('input[name="modelProvider"]:checked');
+ if (!selectedRadio && hasAvailableModel) {
+ for (const option of modelOptions) {
+ const provider = option.getAttribute('data-provider');
+ if (availability[provider]) {
+ option.querySelector('input[type="radio"]').checked = true;
+ option.classList.add('selected');
+ break;
+ }
+ }
+ }
+
+ // Disable start button if no model is available
+ const startBtn = document.getElementById('startBtn');
+ if (!hasAvailableModel) {
+ startBtn.disabled = true;
+ showError('⚠️ No model API keys configured. Please set at least one API key to start conversation.');
+ } else {
+ startBtn.disabled = false;
+ }
+
+ // Add click handlers for model options
+ modelOptions.forEach(option => {
+ option.addEventListener('click', function() {
+ if (!this.classList.contains('disabled')) {
+ modelOptions.forEach(opt => opt.classList.remove('selected'));
+ this.classList.add('selected');
+ }
+ });
+ });
+
+ // Mark initially selected option
+ const currentSelected = document.querySelector('input[name="modelProvider"]:checked');
+ if (currentSelected) {
+ currentSelected.closest('.model-option').classList.add('selected');
+ }
+
+ } catch (error) {
+ console.error("Failed to check model availability:", error);
+ showError("⚠️ Failed to check model availability. Please refresh the page.");
+ }
+ }
+
+ // Auto-connect when page loads
+ window.onload = function() {
+ connect();
+ setupInputMonitoring();
+ // Set default preset to AI debate
+ selectPreset('ai');
+ // Check model availability
+ checkModelAvailability();
+ };
+ </script>
+</body>
+</html>
+
diff --git a/examples/workflows/multiagent_realtime/run_server.py b/examples/workflows/multiagent_realtime/run_server.py
new file mode 100644
index 000000000..5c62ab84f
--- /dev/null
+++ b/examples/workflows/multiagent_realtime/run_server.py
@@ -0,0 +1,220 @@
+# -*- coding: utf-8 -*-
+"""A multi-agent realtime voice interaction server using ChatRoom."""
+import asyncio
+import os
+import traceback
+from pathlib import Path
+
+import uvicorn
+from fastapi import FastAPI, WebSocket
+from fastapi.responses import FileResponse
+
+from agentscope import logger
+from agentscope.agent import RealtimeAgent
+from agentscope.message import TextBlock
+from agentscope.pipeline import ChatRoom
+from agentscope.realtime import (
+ ClientEvents,
+ ServerEvents,
+ ClientEventType,
+ DashScopeRealtimeModel,
+ GeminiRealtimeModel,
+ OpenAIRealtimeModel,
+)
+
+app = FastAPI()
+
+
+async def get() -> FileResponse:
+ """Serve the HTML test page."""
+ html_path = Path(__file__).parent / "multi_agent.html"
+ return FileResponse(html_path)
+
+
[email protected]("/model_availability")
+async def model_availability() -> dict:
+ """Check which model API keys are available in environment variables."""
+ return {
+ "dashscope": bool(os.getenv("DASHSCOPE_API_KEY")),
+ "gemini": bool(os.getenv("GEMINI_API_KEY")),
+ "openai": bool(os.getenv("OPENAI_API_KEY")),
+ }
+
+
+async def frontend_receive(
+ websocket: WebSocket,
+ frontend_queue: asyncio.Queue,
+) -> None:
+ """Forward the message received from the agents to the frontend."""
+ try:
+ while True:
+ msg: ServerEvents.EventBase = await frontend_queue.get()
+
+ # Send the message as JSON
+ await websocket.send_json(msg.model_dump())
+
+ except Exception as e:
+ print(f"[ERROR] frontend_receive error: {e}")
+ traceback.print_exc()
+
+
[email protected]("/ws/{user_id}/{session_id}")
+async def multi_agent_endpoint(
+ websocket: WebSocket,
+ user_id: str,
+ session_id: str,
+) -> None:
+ """WebSocket endpoint for multi-agent realtime voice interaction."""
+ try:
+ await websocket.accept()
+
+ logger.info(
+ "Connected to WebSocket: user_id=%s, session_id=%s",
+ user_id,
+ session_id,
+ )
+
+ # Create the queue to forward messages to the frontend
+ frontend_queue = asyncio.Queue()
+ asyncio.create_task(
+ frontend_receive(websocket, frontend_queue),
+ )
+
+ # Chat room and agents
+ chat_room = None
+
+ while True:
+ # Handle the incoming messages from the frontend
+ # i.e. ClientEvents
+ data = await websocket.receive_json()
+
+ client_event = ClientEvents.from_json(data)
+
+ if isinstance(
+ client_event,
+ ClientEvents.ClientSessionCreateEvent,
+ ):
+ # Create agents by the given session arguments
+ agent1_name = client_event.config.get("agent1_name", "Agent1")
+ agent1_instructions = client_event.config.get(
+ "agent1_instructions",
+ "You are a helpful assistant.",
+ )
+
+ agent2_name = client_event.config.get("agent2_name", "Agent2")
+ agent2_instructions = client_event.config.get(
+ "agent2_instructions",
+ "You are a helpful assistant.",
+ )
+
+ model_provider = client_event.config.get(
+ "model_provider",
+ "dashscope",
+ )
+
+ # Create the appropriate model based on provider
+ if model_provider == "dashscope":
+ model1 = DashScopeRealtimeModel(
+ model_name="qwen3-omni-flash-realtime",
+ api_key=os.getenv("DASHSCOPE_API_KEY"),
+ voice="Dylan",
+ enable_input_audio_transcription=False,
+ )
+ model2 = DashScopeRealtimeModel(
+ model_name="qwen3-omni-flash-realtime",
+ api_key=os.getenv("DASHSCOPE_API_KEY"),
+ voice="Peter",
+ enable_input_audio_transcription=False,
+ )
+
+ elif model_provider == "gemini":
+ model1 = GeminiRealtimeModel(
+ model_name=(
+ "gemini-2.5-flash-native-audio-preview-09-2025"
+ ),
+ api_key=os.getenv("GEMINI_API_KEY"),
+ voice="Puck",
+ )
+ model2 = GeminiRealtimeModel(
+ model_name=(
+ "gemini-2.5-flash-native-audio-preview-09-2025"
+ ),
+ api_key=os.getenv("GEMINI_API_KEY"),
+ voice="Charon",
+ )
+
+ elif model_provider == "openai":
+ model1 = OpenAIRealtimeModel(
+ model_name="gpt-4o-realtime-preview",
+ api_key=os.getenv("OPENAI_API_KEY"),
+ voice="alloy",
+ )
+ model2 = OpenAIRealtimeModel(
+ model_name="gpt-4o-realtime-preview",
+ api_key=os.getenv("OPENAI_API_KEY"),
+ voice="echo",
+ )
+ else:
+ raise ValueError(
+ f"Unsupported model provider: {model_provider}",
+ )
+
+ # Create the first agent
+ agent1 = RealtimeAgent(
+ name=agent1_name,
+ sys_prompt=agent1_instructions,
+ model=model1,
+ )
+
+ # Create the second agent
+ agent2 = RealtimeAgent(
+ name=agent2_name,
+ sys_prompt=agent2_instructions,
+ model=model2,
+ )
+
+ # Create chat room with both agents
+ chat_room = ChatRoom(agents=[agent1, agent2])
+
+ await chat_room.start(frontend_queue)
+
+ # Send session_created event to frontend
+ await websocket.send_json(
+ ServerEvents.ServerSessionCreatedEvent(
+ session_id=session_id,
+ ).model_dump(),
+ )
+
+ await agent1.model.send(
+ TextBlock(
+ type="text",
+ text="<system>Now you can talk.</system>",
+ ),
+ )
+
+ elif client_event.type == ClientEventType.CLIENT_SESSION_END:
+ # End the session with the chat room
+ if chat_room:
+ await chat_room.stop()
+ chat_room = None
+
+ else:
+ # Forward other events to the chat room
+ if chat_room:
+ await chat_room.handle_input(client_event)
+
+ except Exception as e:
+ print(f"[ERROR] WebSocket endpoint error: {e}")
+ traceback.print_exc()
+ raise
+
+
+if __name__ == "__main__":
+ uvicorn.run(
+ "run_server:app",
+ host="localhost",
+ port=8000,
+ reload=True,
+ log_level="info",
+ )
diff --git a/pyproject.toml b/pyproject.toml
index 17db9a330..0ac39f771 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -51,6 +51,8 @@ a2a = [
"nacos-sdk-python>=3.0.0",
]
+# ------------ Realtime -------------
+realtime = ["websockets>=14.0", "scipy"]
# ------------ Model APIs ------------
gemini = ["google-genai"]
@@ -143,6 +145,7 @@ full = [
"agentscope[memory]",
"agentscope[rag]",
"agentscope[evaluate]",
+ "agentscope[realtime]",
]
# ------------ Development ------------
diff --git a/src/agentscope/__init__.py b/src/agentscope/__init__.py
index 092828bc2..5efecd6c9 100644
--- a/src/agentscope/__init__.py
+++ b/src/agentscope/__init__.py
@@ -40,7 +40,6 @@ _config = _ConfigCls(
),
)
-
from . import exception
from . import module
from . import message
@@ -57,6 +56,7 @@ from . import pipeline
from . import tracing
from . import rag
from . import a2a
+from . import realtime
from ._logging import (
logger,
diff --git a/src/agentscope/_utils/_common.py b/src/agentscope/_utils/_common.py
index e43afd132..5b2601340 100644
--- a/src/agentscope/_utils/_common.py
+++ b/src/agentscope/_utils/_common.py
@@ -13,6 +13,7 @@ import uuid
from datetime import datetime
from typing import Any, Callable, Type, Dict
+import numpy as np
import requests
from docstring_parser import parse
from json_repair import repair_json
@@ -424,3 +425,50 @@ def _parse_tool_function(
func_json_schema["function"]["description"] = func_description
return func_json_schema
+
+
+def _resample_pcm_delta(
+ pcm_base64: str,
+ sample_rate: int,
+ target_rate: int,
+) -> str:
+ """Resampling the input pcm base64 data into the target rate.
+
+ Args:
+ pcm_base64 (`str`):
+ The input base64 audio data in pcm format.
+ sample_rate (`int`):
+ The sampling rate of the input data.
+ target_rate (`int`):
+ The target rate of the input data.
+
+ Returns:
+ `str`:
+ The resampling base64 audio data in the required sampling
+ rate.
+ """
+ pcm_data = base64.b64decode(pcm_base64)
+
+ # Into numpy array first
+ audio_array = np.frombuffer(pcm_data, dtype=np.int16)
+
+ # return directly if the same
+ if sample_rate == target_rate:
+ return pcm_base64
+
+ # compute the number of samples
+ num_samples = int(len(audio_array) * target_rate / sample_rate)
+
+ from scipy import signal
+
+ # Use scipy to resample
+ resampled_audio = signal.resample(audio_array, num_samples)
+
+ # Turn it back into bytes
+ resampled_audio = np.clip(resampled_audio, -32768, 32767).astype(np.int16)
+
+ # into base64
+ resampled_bytes = resampled_audio.tobytes()
+ resampled_base64 = base64.b64encode(resampled_bytes).decode("utf-8")
+
+ return resampled_base64
diff --git a/src/agentscope/agent/__init__.py b/src/agentscope/agent/__init__.py
index e7f02083f..e63c4c420 100644
--- a/src/agentscope/agent/__init__.py
+++ b/src/agentscope/agent/__init__.py
@@ -11,6 +11,7 @@ from ._user_input import (
)
from ._user_agent import UserAgent
from ._a2a_agent import A2AAgent
+from ._realtime_agent import RealtimeAgent
__all__ = [
@@ -23,4 +24,5 @@ __all__ = [
"StudioUserInput",
"UserAgent",
"A2AAgent",
+ "RealtimeAgent",
]
diff --git a/src/agentscope/agent/_realtime_agent.py b/src/agentscope/agent/_realtime_agent.py
new file mode 100644
index 000000000..e1474d16c
--- /dev/null
+++ b/src/agentscope/agent/_realtime_agent.py
@@ -0,0 +1,360 @@
+# -*- coding: utf-8 -*-
+"""The realtime agent class."""
+import asyncio
+from asyncio import Queue
+
+import shortuuid
+
+from .._logging import logger
+from .._utils._common import _resample_pcm_delta
+from ..message import (
+ AudioBlock,
+ Base64Source,
+ TextBlock,
+ ImageBlock,
+ ToolUseBlock,
+ ToolResultBlock,
+)
+from ..module import StateModule
+from ..realtime import (
+ ModelEvents,
+ RealtimeModelBase,
+ ServerEvents,
+ ClientEvents,
+)
+from ..tool import Toolkit
+
+
+class RealtimeAgent(StateModule):
+ """The realtime agent class. Different from the `AgentBase` class,
+ this class is designed for real-time interaction scenarios, such as
+ realtime chat, voice assistants, etc.
+
+ Example:
+ This realtime agent requires a queue to handle outgoing messages to
+ the frontend and other agents, and its lifecycle is managed by the
+ `start` and `stop` methods.
+
+ .. code-block:: python
+ :caption: An example of using the RealtimeAgent class.
+
+ from agentscope.agent import RealtimeAgent
+ from agentscope.realtime import DashScopeRealtimeModel
+ import asyncio
+
+ agent = RealtimeAgent(
+ name="Friday",
+ sys_prompt="You are a helpful assistant.",
+ model=DashScopeRealtimeModel(
+ model_name="qwen3-omni-flash-realtime",
+ api_key=os.getenv("DASHSCOPE_API_KEY"),
+ )
+ )
+
+ queue = asyncio.Queue()
+ await agent.start(queue)
+
+ # handle the outgoing messages from the agent in another asyncio
+ # task
+ ...
+
+ await agent.stop()
+
+ """
+
+ def __init__(
+ self,
+ name: str,
+ sys_prompt: str,
+ model: RealtimeModelBase,
+ toolkit: Toolkit | None = None,
+ ) -> None:
+ """Initialize the RealtimeAgent class.
+
+ Args:
+ name (`str`):
+ The name of the agent.
+ sys_prompt (`str`):
+ The system prompt of the agent.
+ model (`RealtimeModelBase`):
+ The realtime model used by the agent.
+ toolkit (`Toolkit | None`, optional):
+ A `Toolkit` object that contains the tool functions. If not
+ provided, a default empty `Toolkit` will be created.
+ """
+ super().__init__()
+
+ self.id = shortuuid.uuid()
+ self.name = name
+ self.sys_prompt = sys_prompt
+ self.model = model
+ self.toolkit = toolkit
+
+ # A queue to handle the incoming events from other agents or the
+ # frontend.
+ self._incoming_queue = Queue()
+ self._external_event_handling_task = None
+
+ # The queue to gather model responses.
+ self._model_response_queue = Queue()
+ self._model_response_handling_task = None
+
+ async def start(self, outgoing_queue: Queue) -> None:
+ """Establish a connection for real-time interaction.
+
+ Args:
+ outgoing_queue (`Queue`):
+ The queue to push messages to the frontend and other agents.
+ """
+ # Start the realtime model connection.
+ await self.model.connect(
+ self._model_response_queue,
+ instructions=self.sys_prompt,
+ tools=self.toolkit.get_json_schemas() if self.toolkit else None,
+ )
+
+ # Start the forwarding loop.
+ self._external_event_handling_task = asyncio.create_task(
+ self._forward_loop(),
+ )
+
+ # Start the response handling loop.
+ self._model_response_handling_task = asyncio.create_task(
+ self._model_response_loop(outgoing_queue),
+ )
+
+ async def stop(self) -> None:
+ """Close the connection."""
+
+ if not self._external_event_handling_task.done():
+ self._external_event_handling_task.cancel()
+
+ await self.model.disconnect()
+
+ async def _forward_loop(self) -> None:
+ """The loop to forward messages from other agents or the frontend to
+ the realtime model for processing.
+
+ outside ==> agent ==> realtime model
+ """
+ logger.info(
+ "Agent '%s' begins the loops to receive external events",
+ self.name,
+ )
+
+ while True:
+ event = await self._incoming_queue.get()
+
+ match event:
+ # Only handle the events that we need
+ case ServerEvents.AgentResponseAudioDeltaEvent() as event:
+ # Convert the sample rate to the required format by the
+ # model
+ receive_rate = event.format.rate
+ if self.model.input_sample_rate != receive_rate:
+ delta = _resample_pcm_delta(
+ event.delta,
+ receive_rate,
+ self.model.input_sample_rate,
+ )
+
+ else:
+ delta = event.delta
+
+ await self.model.send(
+ AudioBlock(
+ type="audio",
+ source=Base64Source(
+ type="base64",
+ media_type=event.format.type,
+ data=delta,
+ ),
+ ),
+ )
+
+ case ServerEvents.AgentResponseAudioDoneEvent():
+ # Send a silence audio block to indicate the end of audio
+ pass
+
+ case ClientEvents.ClientAudioAppendEvent() as event:
+ # Construct media_type from format info
+ # format contains: {"sample_rate": 16000, "encoding":
+ # "pcm16"}
+ # encoding = event.format.get("encoding", "pcm16")
+ # media_type = (
+ # f"audio/{encoding.replace('16', '')}"
+ # if "pcm" in encoding
+ # else "audio/pcm"
+ # )
+
+ await self.model.send(
+ AudioBlock(
+ type="audio",
+ source=Base64Source(
+ type="base64",
+ media_type=event.format.type,
+ data=event.audio,
+ ),
+ ),
+ )
+
+ case ClientEvents.ClientTextAppendEvent() as event:
+ await self.model.send(
+ TextBlock(
+ type="text",
+ text=event.text,
+ ),
+ )
+ case ClientEvents.ClientImageAppendEvent() as event:
+ # Construct media_type from format info
+ media_type = event.format.get("type", "image/jpeg")
+
+ await self.model.send(
+ ImageBlock(
+ type="image",
+ source=Base64Source(
+ type="base64",
+ media_type=media_type,
+ data=event.image,
+ ),
+ ),
+ )
+
+ async def _model_response_loop(self, outgoing_queue: Queue) -> None:
+ """The loop to handle model responses and forward them to the
+ frontend and other agents.
+
+ realtime model ==> agent ==> outside
+
+ Args:
+ outgoing_queue (`Queue`):
+ The queue to push messages to the frontend and other agents.
+ """
+ while True:
+ model_event = await self._model_response_queue.get()
+
+ agent_kwargs = {"agent_id": self.id, "agent_name": self.name}
+
+ agent_event = None
+ match model_event:
+ # The events that can be converted from model events to agent
+ # events directly
+ case (
+ ModelEvents.ModelResponseCreatedEvent()
+ | ModelEvents.ModelResponseDoneEvent()
+ | ModelEvents.ModelResponseAudioDeltaEvent()
+ | ModelEvents.ModelResponseAudioDoneEvent()
+ | ModelEvents.ModelResponseAudioTranscriptDeltaEvent()
+ | ModelEvents.ModelResponseAudioTranscriptDoneEvent()
+ | ModelEvents.ModelResponseToolUseDeltaEvent()
+ | ModelEvents.ModelInputTranscriptionDeltaEvent()
+ | ModelEvents.ModelInputTranscriptionDoneEvent()
+ | ModelEvents.ModelInputStartedEvent()
+ | ModelEvents.ModelInputDoneEvent()
+ | ModelEvents.ModelErrorEvent()
+ ) as event:
+ # Directly map the model event to agent event
+ agent_event = ServerEvents.from_model_event(
+ event,
+ **agent_kwargs,
+ )
+
+ # The events that need special handling
+ case ModelEvents.ModelSessionCreatedEvent():
+ # Send the agent ready event to the outside.
+ agent_event = ServerEvents.AgentReadyEvent(**agent_kwargs)
+
+ case ModelEvents.ModelSessionEndedEvent():
+ # Send the agent session ended event to the outside.
+ agent_event = ServerEvents.AgentEndedEvent(**agent_kwargs)
+
+ # The tool use done that requires executing the tool
+ # Such event may generate multiple outgoing events:
+ # 1. Tool use done event
+ # 2. Tool result event
+ case ModelEvents.ModelResponseToolUseDoneEvent() as event:
+ # Send the tool use done event immediately
+ done_event = ServerEvents.AgentResponseToolUseDoneEvent(
+ response_id=event.response_id,
+ item_id=event.item_id,
+ tool_use=event.tool_use,
+ **agent_kwargs,
+ )
+
+ # Directly put the done event to the outgoing queue
+ await outgoing_queue.put(done_event)
+
+ # Then execute the tool call using accumulated arguments
+ if self.toolkit:
+ # Execute the tool call asynchronously
+ asyncio.create_task(
+ self._acting(
+ tool_use=event.tool_use,
+ outgoing_queue=outgoing_queue,
+ ),
+ )
+
+ case _:
+ logger.debug(
+ "Unknown model event type: %s",
+ type(model_event),
+ )
+
+ if agent_event is not None:
+ # Put the processed response to the outgoing queue.
+ await outgoing_queue.put(agent_event)
+
+ async def handle_input(
+ self,
+ event: ClientEvents.EventBase | ServerEvents.EventBase,
+ ) -> None:
+ """Handle the input message from the frontend or the other agents.
+
+ Args:
+ event (`ClientEvents.EventBase | ServerEvents.EventBase`):
+ The input event from the frontend or other agents.
+ """
+ await self._incoming_queue.put(event)
+
+ async def _acting(
+ self,
+ tool_use: ToolUseBlock,
+ outgoing_queue: Queue,
+ ) -> None:
+ """Execute the tool call and send the result back to the outside (
+ frontend or other agents).
+
+ Args:
+ tool_use (`ToolUseBlock`):
+ The tool use block containing the tool call information.
+ outgoing_queue (`Queue`):
+ The queue to push messages to the frontend and other agents.
+ """
+ if not self.toolkit:
+ return
+
+ res = await self.toolkit.call_tool_function(tool_use)
+
+ last_chunk = None
+ async for chunk in res:
+ last_chunk = chunk
+
+ if last_chunk:
+ tool_result_block = ToolResultBlock(
+ type="tool_result",
+ id=tool_use.get("id"),
+ name=tool_use.get("name"),
+ output=last_chunk.content,
+ )
+
+ # Send the tool result back to the model
+ await self.model.send(tool_result_block)
+
+ # Also send event to frontend/other agents
+ outgoing_event = ServerEvents.AgentResponseToolResultEvent(
+ tool_result=tool_result_block,
+ agent_id=self.id,
+ agent_name=self.name,
+ )
+
+ await outgoing_queue.put(outgoing_event)
diff --git a/src/agentscope/pipeline/__init__.py b/src/agentscope/pipeline/__init__.py
index 9ab9b581e..b5338a93d 100644
--- a/src/agentscope/pipeline/__init__.py
+++ b/src/agentscope/pipeline/__init__.py
@@ -9,6 +9,7 @@ from ._functional import (
fanout_pipeline,
stream_printing_messages,
)
+from ._chat_room import ChatRoom
__all__ = [
"MsgHub",
@@ -17,4 +18,5 @@ __all__ = [
"FanoutPipeline",
"fanout_pipeline",
"stream_printing_messages",
+ "ChatRoom",
]
diff --git a/src/agentscope/pipeline/_chat_room.py b/src/agentscope/pipeline/_chat_room.py
new file mode 100644
index 000000000..e01c1e7b7
--- /dev/null
+++ b/src/agentscope/pipeline/_chat_room.py
@@ -0,0 +1,97 @@
+# -*- coding: utf-8 -*-
+"""The Voice chat room"""
+import asyncio
+from asyncio import Queue
+
+from ..agent import RealtimeAgent
+from ..realtime import ClientEvents, ServerEvents
+
+
+class ChatRoom:
+ """The chat room abstraction to broadcast messages among multiple realtime
+ agents, and handle the messages from the frontend.
+ """
+
+ def __init__(self, agents: list[RealtimeAgent]) -> None:
+ """Initialize the ChatRoom class.
+
+ Args:
+ agents (`list[RealtimeAgent]`):
+ The list of agents participating in the chat room.
+ """
+ self.agents = agents
+
+ # The queue used to gather messages from all agents and push them to
+ # the frontend.
+ self._queue = Queue()
+
+ self._task = None
+
+ async def start(self, outgoing_queue: Queue) -> None:
+ """Establish connections for all agents in the chat room.
+
+ Args:
+ outgoing_queue (`Queue`):
+ The queue to push messages to the frontend, which will be used
+ by all agents to push their messages.
+ """
+
+ for agent in self.agents:
+ await agent.start(self._queue)
+
+ # Start the forwarding loop.
+ self._task = asyncio.create_task(self._forward_loop(outgoing_queue))
+
+ async def _forward_loop(self, outgoing_queue: Queue) -> None:
+ """The loop to forward messages from all agents to the frontend and
+ the other agents.
+
+ Args:
+ outgoing_queue (`Queue`):
+ The queue to push messages to the frontend.
+ """
+
+ while True:
+ # Obtain the message from the client frontend
+ event = await self._queue.get()
+
+ # Only push ServerEvents to the frontend, not ClientEvents
+ # to avoid echoing client messages back
+ if isinstance(event, ClientEvents.EventBase):
+ # Push the message to the frontend queue.
+ for agent in self.agents:
+ await agent.handle_input(event)
+
+ elif isinstance(event, ServerEvents.EventBase):
+ # Broadcast the message to all agents except the sender.
+ # Use create_task instead of gather to avoid blocking
+
+ # Forward the agent/server events to the frontend
+ await outgoing_queue.put(event)
+
+ # Broadcast to other agents
+ sender_id = getattr(event, "agent_id", None)
+ if sender_id:
+ for agent in self.agents:
+ if agent.id != sender_id:
+ await agent.handle_input(event)
+
+ async def stop(self) -> None:
+ """Close connections for all agents in the chat room."""
+
+ for agent in self.agents:
+ await agent.stop()
+
+ # Close the forwarding loop.
+ if not self._task.done():
+ self._task.cancel()
+
+ async def handle_input(self, event: ClientEvents.EventBase) -> None:
+ """Handle input message from the frontend and distribute it to all
+ agents in the chat room.
+
+ Args:
+ event (`ClientEvents.EventBase`):
+ The event from the frontend.
+ """
+ await self._queue.put(event)
diff --git a/src/agentscope/realtime/__init__.py b/src/agentscope/realtime/__init__.py
new file mode 100644
index 000000000..ee5c23129
--- /dev/null
+++ b/src/agentscope/realtime/__init__.py
@@ -0,0 +1,28 @@
+# -*- coding: utf-8 -*-
+"""The realtime module in AgentScope, providing realtime models and events."""
+
+from ._events import (
+ ModelEvents,
+ ModelEventType,
+ ServerEvents,
+ ServerEventType,
+ ClientEvents,
+ ClientEventType,
+)
+from ._base import RealtimeModelBase
+from ._dashscope_realtime_model import DashScopeRealtimeModel
+from ._openai_realtime_model import OpenAIRealtimeModel
+from ._gemini_realtime_model import GeminiRealtimeModel
+
+__all__ = [
+ "ModelEventType",
+ "ModelEvents",
+ "ServerEventType",
+ "ServerEvents",
+ "ClientEventType",
+ "ClientEvents",
+ "RealtimeModelBase",
+ "DashScopeRealtimeModel",
+ "OpenAIRealtimeModel",
+ "GeminiRealtimeModel",
+]
diff --git a/src/agentscope/realtime/_base.py b/src/agentscope/realtime/_base.py
new file mode 100644
index 000000000..2bea54786
--- /dev/null
+++ b/src/agentscope/realtime/_base.py
@@ -0,0 +1,173 @@
+# -*- coding: utf-8 -*-
+"""The realtime model base class."""
+import asyncio
+import json
+from abc import abstractmethod
+from asyncio import Queue
+from typing import Any
+
+import websockets
+from websockets import ClientConnection
+
+from ._events import ModelEvents
+from ..message import AudioBlock, TextBlock, ImageBlock, ToolResultBlock
+
+
+class RealtimeModelBase:
+ """The realtime model base class."""
+
+ model_name: str
+ """The model name"""
+
+ support_input_modalities: list[str]
+ """The supported input modalities of the DashScope realtime model."""
+
+ websocket_url: str
+ """The websocket URL of the realtime model API."""
+
+ websocket_headers: dict[str, str]
+ """The websocket headers of the realtime model API."""
+
+ input_sample_rate: int
+ """The input audio sample rate."""
+
+ output_sample_rate: int
+ """The output audio sample rate."""
+
+ def __init__(
+ self,
+ model_name: str,
+ ) -> None:
+ """Initialize the RealtimeModelBase class.
+
+ Args:
+ model_name (`str`):
+ The model name.
+ """
+
+ self.model_name = model_name
+
+ # The incoming queue to handle the data returned from the realtime
+ # model API.
+ self._incoming_queue = Queue()
+ self._incoming_task = None
+
+ self._websocket: ClientConnection | None = None
+
+ @abstractmethod
+ async def send(
+ self,
+ data: AudioBlock | TextBlock | ImageBlock | ToolResultBlock,
+ ) -> None:
+ """Send data to the realtime model for processing.
+
+ Args:
+ data (`AudioBlock | TextBlock | ImageBlock | ToolResultBlock`):
+ The data to be sent to the realtime model.
+ """
+
+ async def connect(
+ self,
+ outgoing_queue: Queue,
+ instructions: str,
+ tools: list[dict] | None = None,
+ ) -> None:
+ """Establish a connection to the realtime model.
+
+ Args:
+ outgoing_queue (`Queue`):
+ The queue to push the model responses to the outside.
+ instructions (`str`):
+ The instructions to guide the realtime model's behavior.
+ tools (`list[dict]`, *optional*):
+ The list of tools JSON schemas.
+ """
+
+ self._websocket = await websockets.connect(
+ self.websocket_url,
+ additional_headers=self.websocket_headers,
+ )
+
+ self._incoming_task = asyncio.create_task(
+ self._receive_model_event_loop(outgoing_queue),
+ )
+
+ # Updating the session with instructions and other configurations
+ session_config = self._build_session_config(instructions, tools)
+ await self._websocket.send(
+ json.dumps(session_config, ensure_ascii=False),
+ )
+
+ @abstractmethod
+ def _build_session_config(
+ self,
+ instructions: str,
+ tools: list[dict] | None,
+ **kwargs: Any,
+ ) -> dict:
+ """Build the session configuration message to initialize or update
+ the realtime model session.
+
+ Args:
+ instructions (`str`):
+ The instructions to guide the realtime model's behavior.
+ tools (`list[dict]`, optional):
+ The list of tools available to the realtime model.
+ **kwargs (`Any`):
+ Additional keyword arguments for session configuration.
+
+ Returns:
+ `dict`:
+ The session configuration message.
+ """
+
+ async def disconnect(self) -> None:
+ """Close the connection to the realtime model."""
+ # TODO: session ended
+
+ if self._incoming_task and not self._incoming_task.done():
+ self._incoming_task.cancel()
+
+ if self._websocket:
+ await self._websocket.close()
+
+ async def _receive_model_event_loop(self, outgoing_queue: Queue) -> None:
+ """The loop to receive and handle the model responses.
+
+ Args:
+ outgoing_queue (`Queue`):
+ The queue to push the model responses to the outside.
+ """
+
+ async for message in self._websocket:
+ if isinstance(message, bytes):
+ message = message.decode("utf-8")
+
+ # Parse the message into ModelEvent instance(s)
+ events = await self.parse_api_message(message)
+
+ if events is None:
+ continue
+
+ if isinstance(events, ModelEvents.EventBase):
+ events = [events]
+
+ for event in events:
+ # Send the event to the outgoing queue
+ await outgoing_queue.put(event)
+
+ @abstractmethod
+ async def parse_api_message(
+ self,
+ message: str,
+ ) -> ModelEvents.EventBase | list[ModelEvents.EventBase] | None:
+ """Parse the message received from the realtime model API.
+
+ Args:
+ message (`str`):
+ The message received from the realtime model API.
+
+ Returns:
+ `ModelEvents.EventBase | list[ModelEvents.EventBase] | None`:
+ The unified model event(s) in agentscope format.
+ """
diff --git a/src/agentscope/realtime/_dashscope_realtime_model.py b/src/agentscope/realtime/_dashscope_realtime_model.py
new file mode 100644
index 000000000..f319b7b44
--- /dev/null
+++ b/src/agentscope/realtime/_dashscope_realtime_model.py
@@ -0,0 +1,406 @@
+# -*- coding: utf-8 -*-
+"""The dashscope realtime model class."""
+import json
+from typing import Literal, Any
+
+import shortuuid
+
+from ._events import ModelEvents
+from ._base import RealtimeModelBase
+from .._logging import logger
+from .._utils._common import _get_bytes_from_web_url
+from ..message import AudioBlock, TextBlock, ImageBlock, ToolResultBlock
+
+
+class DashScopeRealtimeModel(RealtimeModelBase):
+ """The DashScope realtime model class.
+
+ TODO:
+ - Support non-VAD mode
+ - Support update session config during the session
+ """
+
+ support_input_modalities: list[str] = ["audio", "image"]
+ """The supported input modalities of the DashScope realtime model."""
+
+ support_tools: bool = False
+ """The DashScope Realtime API doesn't support tools yet (last updated in
+ 20260129)."""
+
+ websocket_url: str = (
+ "wss://dashscope.aliyuncs.com/api-ws/v1/realtime?model={model_name}"
+ )
+ """The websocket URL of the DashScope realtime model API."""
+
+ websocket_headers: dict[str, str] = {
+ "Authorization": "Bearer {api_key}",
+ "X-DashScope-DataInspection": "disable",
+ }
+ """The websocket headers of the DashScope realtime model API."""
+
+ input_sample_rate: int
+ """The input audio sample rate."""
+
+ output_sample_rate: int
+ """The output audio sample rate."""
+
+ def __init__(
+ self,
+ model_name: str,
+ api_key: str,
+ voice: str
+ | Literal["Cherry", "Serena", "Ethan", "Chelsie"] = "Cherry",
+ enable_input_audio_transcription: bool = True,
+ ) -> None:
+ """Initialize the DashScopeRealtimeModel class.
+
+ Args:
+ model_name (`str`):
+ The model name, e.g. "qwen3-omni-flash-realtime".
+ api_key (`str`):
+ The API key for authentication.
+ voice (`str | Literal["Cherry", "Serena", "Ethan", "Chelsie"]`, \
+ defaults to `"Cherry"`):
+ The voice to be used for text-to-speech.
+ enable_input_audio_transcription (`bool`, defaults to `True`):
+ Whether to enable input audio transcription.
+ """
+ super().__init__(model_name)
+
+ self.voice = voice
+ self.enable_input_audio_transcription = (
+ enable_input_audio_transcription
+ )
+
+ # The dashscope realtime API requires 16kHz input sample rate
+ # for all models.
+ self.input_sample_rate = 16000
+
+ # The output sample rate depends on the model.
+ # For "qwen3-omni-flash-realtime" models, it's 24kHz.
+ # For others, it's 16kHz.
+ if model_name.startswith("qwen3-omni-flash-realtime"):
+ self.output_sample_rate = 24000
+ else:
+ self.output_sample_rate = 16000
+
+ # Set the model name in the websocket URL.
+ self.websocket_url = self.websocket_url.format(model_name=model_name)
+
+ # Set the API key in the websocket headers.
+ self.websocket_headers["Authorization"] = self.websocket_headers[
+ "Authorization"
+ ].format(api_key=api_key)
+
+ # Record the response ID for the current session.
+ self._response_id = ""
+
+ def _build_session_config(
+ self,
+ instructions: str,
+ tools: list[dict] | None,
+ **kwargs: Any,
+ ) -> dict:
+ """Build the session configuration."""
+ session_config: dict = {
+ "instructions": instructions,
+ # The output modalities of the model
+ "modalities": ["audio", "text"],
+ "input_audio_format": "pcm" + str(self.input_sample_rate // 1000),
+ "output_audio_format": "pcm"
+ + str(self.output_sample_rate // 1000),
+ "voice": self.voice,
+ **kwargs,
+ }
+
+ # Input audio transcription
+ if self.enable_input_audio_transcription:
+ session_config["input_audio_transcription"] = {
+ "model": "gummy-realtime-v1",
+ }
+
+ # By default, we enable the VAD capability
+ # TODO: support none-VAD mode
+ session_config["turn_detection"] = {
+ "type": "server_vad",
+ "threshold": 0.5,
+ "silence_duration_ms": 800,
+ }
+
+ return {
+ "type": "session.update",
+ "session": session_config,
+ }
+
+ async def send(
+ self,
+ data: AudioBlock | TextBlock | ImageBlock | ToolResultBlock,
+ ) -> None:
+ """Send the data to the DashScope realtime model for processing.
+
+ .. note:: The DashScope Realtime API currently only supports audio and
+ image data input.
+
+ Args:
+ data (`AudioBlock | TextBlock | ImageBlock | ToolResultBlock`):
+ The data to be sent to the DashScope realtime model.
+ """
+ from websockets import State
+
+ if not self._websocket or self._websocket.state != State.OPEN:
+ raise RuntimeError(
+ f"WebSocket is not connected for model {self.model_name}. "
+ "Call the `connect` method first.",
+ )
+
+ # Type checking
+ assert (
+ isinstance(data, dict) and "type" in data
+ ), "Data must be a dict with a 'type' field."
+
+ # The source must be base64 for audio data
+ data_type = data.get("type")
+
+ if data_type not in self.support_input_modalities:
+ logger.warning(
+ "DashScope Realtime API does not support %s data input. "
+ "Supported modalities are: %s",
+ data_type,
+ ", ".join(self.support_input_modalities),
+ )
+ return
+
+ # Process the data based on its type
+ if data_type == "image":
+ to_send_message = await self._parse_image_data(
+ ImageBlock(
+ type="image",
+ source=data.get("source"),
+ ),
+ )
+
+ elif data_type == "audio":
+ to_send_message = await self._parse_audio_data(
+ AudioBlock(
+ type="audio",
+ source=data.get("source"),
+ ),
+ )
+
+ elif data_type == "text":
+ # TODO: The following code doesn't work and cannot support text
+ # input yet.
+ to_send_message = json.dumps(
+ {
+ "event_id": shortuuid.uuid(),
+ "type": "response.create",
+ "response": {
+ "instructions": data.get("text", ""),
+ },
+ },
+ ensure_ascii=False,
+ )
+
+ else:
+ raise ValueError(
+ f"Unsupported data type: {data_type}",
+ )
+
+ await self._websocket.send(to_send_message)
+
+ async def parse_api_message(
+ self,
+ message: str,
+ ) -> ModelEvents.EventBase | list[ModelEvents.EventBase] | None:
+ """Parse the message received from the DashScope realtime model API.
+
+ Args:
+ message (`str`):
+ The message received from the DashScope realtime model API.
+
+ Returns:
+ `ModelEvents.EventBase | list[ModelEvents.EventBase] | None`:
+ The unified model event(s) in agentscope format.
+ """
+ try:
+ data = json.loads(message)
+ except json.decoder.JSONDecodeError:
+ return None
+
+ if not isinstance(data, dict):
+ return None
+
+ model_event = None
+ match data.get("type", ""):
+ # ================ Session related events ================
+ case "session.created":
+ model_event = ModelEvents.ModelSessionCreatedEvent(
+ session_id=data.get("session", {}).get("id", ""),
+ )
+
+ case "session.updated":
+ # TODO: handle the session updated event
+ pass
+
+ # ================ Response related events ================
+ case "response.created":
+ self._response_id = data.get("response", {}).get("id", "")
+ model_event = ModelEvents.ModelResponseCreatedEvent(
+ response_id=self._response_id,
+ )
+
+ case "response.done":
+ response = data.get("response", {})
+ response_id = response.get("id", "") or self._response_id
+ usage = response.get("usage", {})
+ model_event = ModelEvents.ModelResponseDoneEvent(
+ response_id=response_id,
+ input_tokens=usage.get("input_tokens", 0),
+ output_tokens=usage.get("output_tokens", 0),
+ )
+ # clear the response id
+ self._response_id = ""
+
+ case "response.audio.delta":
+ audio_data = data.get("delta", "")
+ if audio_data:
+ model_event = ModelEvents.ModelResponseAudioDeltaEvent(
+ response_id=self._response_id,
+ item_id=data.get("item_id", ""),
+ delta=audio_data,
+ format={
+ "type": "audio/pcm",
+ "rate": self.output_sample_rate,
+ },
+ )
+
+ case "response.audio.done":
+ model_event = ModelEvents.ModelResponseAudioDoneEvent(
+ response_id=self._response_id,
+ item_id=data.get("item_id", ""),
+ )
+
+ # ================ Transcription related events ================
+
+ case "response.audio_transcript.delta":
+ transcript_data = data.get("delta", "")
+ if transcript_data:
+ model_event = (
+ ModelEvents.ModelResponseAudioTranscriptDeltaEvent(
+ response_id=self._response_id,
+ delta=transcript_data,
+ item_id=data.get("item_id", ""),
+ )
+ )
+
+ case "response.audio_transcript.done":
+ model_event = (
+ ModelEvents.ModelResponseAudioTranscriptDoneEvent(
+ response_id=self._response_id,
+ item_id=data.get("item_id", ""),
+ )
+ )
+
+ case "conversation.item.input_audio_transcription.completed":
+ transcript_data = data.get("transcript", "")
+ if transcript_data:
+ model_event = ModelEvents.ModelInputTranscriptionDoneEvent(
+ transcript=transcript_data,
+ item_id=data.get("item_id", ""),
+ )
+
+ # ================= VAD related events =================
+ case "input_audio_buffer.speech_started":
+ model_event = ModelEvents.ModelInputStartedEvent(
+ item_id=data.get("item_id", ""),
+ audio_start_ms=data.get("audio_start_ms", 0),
+ )
+
+ case "input_audio_buffer.speech_stopped":
+ model_event = ModelEvents.ModelInputDoneEvent(
+ item_id=data.get("item_id", ""),
+ audio_end_ms=data.get("audio_end_ms", 0),
+ )
+
+ # ================= Error events =================
+ case "error":
+ error = data.get("error", {})
+ model_event = ModelEvents.ModelErrorEvent(
+ error_type=error.get("type", "unknown"),
+ code=error.get("code", "unknown"),
+ message=error.get("message", "An unknown error occurred."),
+ )
+
+ # ================= Unknown events =================
+ case _:
+ logger.debug(
+ "Unknown DashScope realtime model event type: %s",
+ data.get("type", None),
+ )
+
+ return model_event
+
+ async def _parse_image_data(self, block: ImageBlock) -> str:
+ """Parse the image data block to the format required by the DashScope
+ realtime model API.
+
+ Args:
+ block (`ImageBlock`):
+ The image data block.
+
+ Returns:
+ `str`: The parsed message to be sent to the DashScope realtime
+ model API.
+ """
+ if block["source"]["type"] == "base64":
+ return json.dumps(
+ {
+ "type": "input_image_buffer.append",
+ "image": block["source"]["data"],
+ },
+ )
+
+ if block["source"]["type"] == "url":
+ image = _get_bytes_from_web_url(block["source"]["url"])
+ return json.dumps(
+ {
+ "type": "input_image_url.append",
+ "image_url": image,
+ },
+ )
+
+ raise ValueError(
+ f"Unsupported image source type: {block['source']['type']}",
+ )
+
+ async def _parse_audio_data(self, block: AudioBlock) -> str:
+ """Parse the audio data block to the format required by the DashScope
+ realtime model API.
+
+ Args:
+ block (`AudioBlock`):
+ The audio data block.
+
+ Returns:
+ `str`: The parsed message to be sent to the DashScope realtime
+ model API.
+ """
+ source_type = block["source"]["type"]
+
+ if source_type == "base64":
+ audio_data = block["source"]["data"]
+
+ elif source_type == "url":
+ audio_data = _get_bytes_from_web_url(block["source"]["url"])
+
+ else:
+ raise ValueError(
+ f"Unsupported audio source type: {source_type}",
+ )
+
+ return json.dumps(
+ {
+ "type": "input_audio_buffer.append",
+ "audio": audio_data,
+ },
+ )
diff --git a/src/agentscope/realtime/_events/__init__.py b/src/agentscope/realtime/_events/__init__.py
new file mode 100644
index 000000000..f353f06a5
--- /dev/null
+++ b/src/agentscope/realtime/_events/__init__.py
@@ -0,0 +1,15 @@
+# -*- coding: utf-8 -*-
+"""The events in the realtime module."""
+
+from ._model_event import ModelEvents, ModelEventType
+from ._client_event import ClientEvents, ClientEventType
+from ._server_event import ServerEvents, ServerEventType
+
+__all__ = [
+ "ModelEventType",
+ "ModelEvents",
+ "ClientEventType",
+ "ClientEvents",
+ "ServerEventType",
+ "ServerEvents",
+]
diff --git a/src/agentscope/realtime/_events/_client_event.py b/src/agentscope/realtime/_events/_client_event.py
new file mode 100644
index 000000000..9487927e0
--- /dev/null
+++ b/src/agentscope/realtime/_events/_client_event.py
@@ -0,0 +1,198 @@
+# -*- coding: utf-8 -*-
+"""The client events for web-to-backend communication."""
+from enum import Enum
+from typing import List
+
+from pydantic import BaseModel
+
+from ._utils import AudioFormat
+from ...message import TextBlock, AudioBlock, ImageBlock, VideoBlock
+
+
+class ClientEventType(str, Enum):
+ """Types of client events for web-to-backend communication."""
+
+ # ============== Session control ================
+ CLIENT_SESSION_CREATE = "client_session_create"
+ """The user creates a new session in the frontend."""
+
+ CLIENT_SESSION_END = "client_session_end"
+ """The user ends the current session in the frontend."""
+
+ # ============== Response control ================
+ CLIENT_RESPONSE_CREATE = "client_response_create"
+ """The user requests the agent to generate a response immediately."""
+
+ CLIENT_RESPONSE_CANCEL = "client_response_cancel"
+ """The user interrupts the agent's current response generation."""
+
+ CLIENT_IMAGE_APPEND = "client_image_append"
+ """The user appends an image input to the current session."""
+
+ CLIENT_TEXT_APPEND = "client_text_append"
+ """The user appends a text input to the current session."""
+
+ CLIENT_AUDIO_APPEND = "client_audio_append"
+ """The user appends an audio input to the current session."""
+
+ CLIENT_AUDIO_COMMIT = "client_audio_commit"
+ """The user commits the audio input to signal end of input."""
+
+ CLIENT_TOOL_RESULT = "client_tool_result"
+ """The tool result executed in the frontend is sent back to the backend."""
+
+
+class ClientEvents:
+ """Realtime client events."""
+
+ class EventBase(BaseModel):
+ """The base class for all client events, used to unify the type
+ hinting."""
+
+ class ClientSessionCreateEvent(EventBase):
+ """Session create event in the frontend"""
+
+ type: ClientEventType = ClientEventType.CLIENT_SESSION_CREATE
+ """The event type."""
+
+ config: dict
+ """The session config."""
+
+ class ClientSessionEndEvent(EventBase):
+ """Session end event in the frontend"""
+
+ type: ClientEventType = ClientEventType.CLIENT_SESSION_END
+ """The event type."""
+
+ session_id: str
+ """The session ID."""
+
+ class ClientResponseCreateEvent(EventBase):
+ """Response create event in the frontend"""
+
+ type: ClientEventType = ClientEventType.CLIENT_RESPONSE_CREATE
+ """The event type."""
+
+ session_id: str
+ """The session ID."""
+
+ class ClientResponseCancelEvent(EventBase):
+ """Response cancel event in the frontend"""
+
+ type: ClientEventType = ClientEventType.CLIENT_RESPONSE_CANCEL
+ """The event type."""
+
+ session_id: str
+ """The session ID."""
+
+ class ClientImageAppendEvent(EventBase):
+ """Image append event in the frontend"""
+
+ type: ClientEventType = ClientEventType.CLIENT_IMAGE_APPEND
+ """The event type."""
+
+ session_id: str
+ """The session ID."""
+
+ data: str
+ """The image data, encoded as base64 string."""
+
+ format: dict
+ """The image format information."""
+
+ class ClientTextAppendEvent(EventBase):
+ """Text append event in the frontend"""
+
+ type: ClientEventType = ClientEventType.CLIENT_TEXT_APPEND
+ """The event type."""
+
+ session_id: str
+ """The session ID."""
+
+ text: str
+ """The text data."""
+
+ class ClientAudioAppendEvent(EventBase):
+ """Audio append event in the frontend"""
+
+ type: ClientEventType = ClientEventType.CLIENT_AUDIO_APPEND
+ """The event type."""
+
+ session_id: str
+ """The session ID."""
+
+ audio: str
+ """The audio data, encoded as base64 string."""
+
+ format: AudioFormat
+ """The audio format information."""
+
+ class ClientAudioCommitEvent(EventBase):
+ """Audio commit event in the frontend"""
+
+ type: ClientEventType = ClientEventType.CLIENT_AUDIO_COMMIT
+ """The event type."""
+
+ session_id: str
+ """The session ID."""
+
+ class ClientToolResultEvent(EventBase):
+ """Tool result event in the frontend"""
+
+ type: ClientEventType = ClientEventType.CLIENT_TOOL_RESULT
+ """The event type."""
+
+ session_id: str
+ """The session ID."""
+
+ id: str
+ """The tool call ID."""
+
+ name: str
+ """The tool name."""
+
+ output: str | List[TextBlock | ImageBlock | AudioBlock | VideoBlock]
+ """The tool result."""
+
+ MAPPING = {
+ ClientEventType.CLIENT_SESSION_CREATE: ClientSessionCreateEvent,
+ ClientEventType.CLIENT_SESSION_END: ClientSessionEndEvent,
+ ClientEventType.CLIENT_RESPONSE_CREATE: ClientResponseCreateEvent,
+ ClientEventType.CLIENT_RESPONSE_CANCEL: ClientResponseCancelEvent,
+ ClientEventType.CLIENT_IMAGE_APPEND: ClientImageAppendEvent,
+ ClientEventType.CLIENT_TEXT_APPEND: ClientTextAppendEvent,
+ ClientEventType.CLIENT_AUDIO_APPEND: ClientAudioAppendEvent,
+ ClientEventType.CLIENT_AUDIO_COMMIT: ClientAudioCommitEvent,
+ ClientEventType.CLIENT_TOOL_RESULT: ClientToolResultEvent,
+ }
+
+ @classmethod
+ def from_json(cls, json_data: dict) -> EventBase:
+ """Parse the client event from JSON data and return the corresponding
+ event instance.
+
+ Args:
+ json_data (`dict`):
+ The JSON data, which must contain the "type" field.
+
+ Raises:
+ `ValueError`:
+ If the event type is unknown.
+
+ Returns:
+ `ClientEvents.EventBase`:
+ The corresponding client event instance.
+ """
+ if not isinstance(json_data, dict) or "type" not in json_data:
+ raise ValueError(
+ f"Invalid JSON data for ClientEvent: {json_data}",
+ )
+
+ event_type = json_data["type"]
+
+ if event_type not in cls.MAPPING:
+ raise ValueError(f"Unknown ClientEvent type: {event_type}")
+
+ # Obtain the event class from the mapping
+ event_class = cls.MAPPING[event_type]
+ return event_class(**json_data)
diff --git a/src/agentscope/realtime/_events/_model_event.py b/src/agentscope/realtime/_events/_model_event.py
new file mode 100644
index 000000000..45316c686
--- /dev/null
+++ b/src/agentscope/realtime/_events/_model_event.py
@@ -0,0 +1,343 @@
+# -*- coding: utf-8 -*-
+"""The unified event from realtime model APIs in AgentScope, which will be
+consumed by the realtime agents."""
+from enum import Enum
+from typing import Literal
+
+from pydantic import BaseModel
+
+from ._utils import AudioFormat
+from ...message import ToolUseBlock
+
+
+class ModelEventType(str, Enum):
+ """Types of model events from the API."""
+
+ # API session lifecycle
+ MODEL_SESSION_CREATED = "model_session_created"
+ """The realtime API session has been created."""
+
+ MODEL_SESSION_ENDED = "model_session_ended"
+ """The realtime API session has ended."""
+
+ # ============= MODEL GENERATED EVENTS =============
+
+ MODEL_RESPONSE_CREATED = "model_response_created"
+ """The realtime model begins generating a response."""
+
+ MODEL_RESPONSE_DONE = "model_response_done"
+ """The realtime model has finished generating a response."""
+
+ # ============= MODEL RESPONSE CONTENT EVENTS =============
+
+ MODEL_RESPONSE_AUDIO_DELTA = "model_response_audio_delta"
+ """The realtime model response audio delta event."""
+
+ MODEL_RESPONSE_AUDIO_DONE = "model_response_audio_done"
+ """The realtime model response audio done event."""
+
+ MODEL_RESPONSE_AUDIO_TRANSCRIPT_DELTA = (
+ "model_response_audio_transcript_delta"
+ )
+ """The realtime model response audio transcript delta event."""
+
+ MODEL_RESPONSE_AUDIO_TRANSCRIPT_DONE = (
+ "model_response_audio_transcript_done"
+ )
+ """The realtime model response audio transcript done event."""
+
+ MODEL_RESPONSE_TOOL_USE_DELTA = "model_response_tool_use_delta"
+ """The realtime model response tool use delta event."""
+
+ MODEL_RESPONSE_TOOL_USE_DONE = "model_response_tool_use_done"
+ """The realtime model response tool use done event."""
+
+ # Input transcription
+ MODEL_INPUT_TRANSCRIPTION_DELTA = "model_input_transcription_delta"
+ """The input transcription delta event."""
+
+ MODEL_INPUT_TRANSCRIPTION_DONE = "model_input_transcription_done"
+ """The input transcription done event."""
+
+ # Input detection (VAD)
+ MODEL_INPUT_STARTED = "model_input_started"
+ """The input has started event."""
+
+ MODEL_INPUT_DONE = "model_input_done"
+ """The input has done event."""
+
+ # ============= ERROR EVENTS =============
+
+ MODEL_ERROR = "model_error"
+ """An error event from the realtime model API."""
+
+ # ============ WebSocket Events ============
+
+ # WebSocket events (if used)
+ MODEL_WEBSOCKET_CONNECT = "model_websocket_connect"
+ """The model WebSocket has connected."""
+
+ MODEL_WEBSOCKET_DISCONNECT = "model_websocket_disconnect"
+ """The model WebSocket has disconnected."""
+
+
+class ModelEvents:
+ """The realtime model events that will be consumed by the realtime
+ agents"""
+
+ class EventBase(BaseModel):
+ """The base class for all model events, used to unify the type
+ hinting."""
+
+ class ModelSessionCreatedEvent(EventBase):
+ """Realtime model session created event.
+
+ .. note:: This session is the connection between the realtime API and
+ the client, not the conversation session.
+ """
+
+ session_id: str
+ """The session ID."""
+
+ type: Literal[
+ ModelEventType.MODEL_SESSION_CREATED
+ ] = ModelEventType.MODEL_SESSION_CREATED
+ """The event type."""
+
+ class ModelSessionEndedEvent(EventBase):
+ """Session ended event.
+
+ .. note:: This session is the connection between the realtime API and
+ the client, not the conversation session.
+ """
+
+ session_id: str
+ """The session ID."""
+
+ reason: str
+ """The reason for session end."""
+
+ type: Literal[
+ ModelEventType.MODEL_SESSION_ENDED
+ ] = ModelEventType.MODEL_SESSION_ENDED
+ """The event type."""
+
+ class ModelResponseCreatedEvent(EventBase):
+ """The realtime model begins generating a response."""
+
+ response_id: str
+ """The response ID."""
+
+ type: Literal[
+ ModelEventType.MODEL_RESPONSE_CREATED
+ ] = ModelEventType.MODEL_RESPONSE_CREATED
+ """The event type."""
+
+ class ModelResponseDoneEvent(EventBase):
+ """Model response done event."""
+
+ response_id: str
+ """The response ID."""
+
+ input_tokens: int
+ """The number of input tokens."""
+
+ output_tokens: int
+ """The number of output tokens."""
+
+ metadata: dict[str, str] = {}
+ """Additional metadata."""
+
+ type: Literal[
+ ModelEventType.MODEL_RESPONSE_DONE
+ ] = ModelEventType.MODEL_RESPONSE_DONE
+ """The event type."""
+
+ class ModelResponseAudioDeltaEvent(EventBase):
+ """Model response audio delta event."""
+
+ response_id: str
+ """The response ID."""
+
+ item_id: str
+ """The conversation item ID."""
+
+ delta: str
+ """The audio chunk data, encoded in base64."""
+
+ format: AudioFormat
+ """The audio format information."""
+
+ type: Literal[
+ ModelEventType.MODEL_RESPONSE_AUDIO_DELTA
+ ] = ModelEventType.MODEL_RESPONSE_AUDIO_DELTA
+ """The event type."""
+
+ class ModelResponseAudioDoneEvent(EventBase):
+ """Model response audio done event."""
+
+ response_id: str
+ """The response ID."""
+
+ item_id: str
+ """The conversation item ID."""
+
+ type: Literal[
+ ModelEventType.MODEL_RESPONSE_AUDIO_DONE
+ ] = ModelEventType.MODEL_RESPONSE_AUDIO_DONE
+ """The event type."""
+
+ class ModelResponseAudioTranscriptDeltaEvent(EventBase):
+ """Model response audio transcript delta event."""
+
+ response_id: str
+ """The response ID."""
+
+ item_id: str
+ """The conversation item ID."""
+
+ delta: str
+ """The transcript chunk data."""
+
+ type: Literal[
+ ModelEventType.MODEL_RESPONSE_AUDIO_TRANSCRIPT_DELTA
+ ] = ModelEventType.MODEL_RESPONSE_AUDIO_TRANSCRIPT_DELTA
+ """The event type."""
+
+ class ModelResponseAudioTranscriptDoneEvent(EventBase):
+ """Model response audio transcript done event."""
+
+ response_id: str
+ """The response ID."""
+
+ item_id: str
+ """The conversation item ID."""
+
+ type: Literal[
+ ModelEventType.MODEL_RESPONSE_AUDIO_TRANSCRIPT_DONE
+ ] = ModelEventType.MODEL_RESPONSE_AUDIO_TRANSCRIPT_DONE
+ """The event type."""
+
+ class ModelResponseToolUseDeltaEvent(EventBase):
+ """Model response tool use delta event."""
+
+ response_id: str
+ """The response ID."""
+
+ item_id: str
+ """The response item ID."""
+
+ tool_use: ToolUseBlock
+ """The tool use block delta, the arguments are accumulated in the
+ `raw_input` field."""
+
+ type: Literal[
+ ModelEventType.MODEL_RESPONSE_TOOL_USE_DELTA
+ ] = ModelEventType.MODEL_RESPONSE_TOOL_USE_DELTA
+ """The event type."""
+
+ class ModelResponseToolUseDoneEvent(EventBase):
+ """Model response tool use done event."""
+
+ response_id: str
+ """The response ID."""
+
+ item_id: str
+ """The response item ID."""
+
+ tool_use: ToolUseBlock
+ """The complete tool use block."""
+
+ type: Literal[
+ ModelEventType.MODEL_RESPONSE_TOOL_USE_DONE
+ ] = ModelEventType.MODEL_RESPONSE_TOOL_USE_DONE
+ """The event type."""
+
+ class ModelInputTranscriptionDeltaEvent(EventBase):
+ """Input transcription delta event."""
+
+ item_id: str
+ """The conversation item ID."""
+
+ delta: str
+ """The transcription delta."""
+
+ type: Literal[
+ ModelEventType.MODEL_INPUT_TRANSCRIPTION_DELTA
+ ] = ModelEventType.MODEL_INPUT_TRANSCRIPTION_DELTA
+ """The event type."""
+
+ class ModelInputTranscriptionDoneEvent(EventBase):
+ """Input transcription done event."""
+
+ transcript: str
+ """The complete transcription."""
+
+ item_id: str
+ """The conversation item ID."""
+
+ input_tokens: int | None = None
+ """The number of input tokens."""
+
+ output_tokens: int | None = None
+ """The number of output tokens."""
+
+ type: Literal[
+ ModelEventType.MODEL_INPUT_TRANSCRIPTION_DONE
+ ] = ModelEventType.MODEL_INPUT_TRANSCRIPTION_DONE
+ """The event type."""
+
+ class ModelInputStartedEvent(EventBase):
+ """Input started event."""
+
+ item_id: str
+ """The conversation item ID."""
+
+ audio_start_ms: int
+ """The audio start time in milliseconds."""
+
+ type: Literal[
+ ModelEventType.MODEL_INPUT_STARTED
+ ] = ModelEventType.MODEL_INPUT_STARTED
+ """The event type."""
+
+ class ModelInputDoneEvent(EventBase):
+ """Input done event."""
+
+ item_id: str
+ """The conversation item ID."""
+
+ audio_end_ms: int
+ """The audio end time in milliseconds."""
+
+ type: Literal[
+ ModelEventType.MODEL_INPUT_DONE
+ ] = ModelEventType.MODEL_INPUT_DONE
+ """The event type."""
+
+ class ModelErrorEvent(EventBase):
+ """Error event."""
+
+ error_type: str
+ """The error type."""
+
+ code: str
+ """The error code."""
+
+ message: str
+ """The error message."""
+
+ type: Literal[ModelEventType.MODEL_ERROR] = ModelEventType.MODEL_ERROR
+ """The event type."""
+
+ class WebsocketConnectEvent(EventBase):
+ """WebSocket connect event."""
+
+ type: Literal[ModelEventType.MODEL_WEBSOCKET_CONNECT]
+ """The event type."""
+
+ class WebsocketDisconnectEvent(EventBase):
+ """WebSocket disconnect event."""
+
+ type: Literal[ModelEventType.MODEL_WEBSOCKET_DISCONNECT]
+ """The event type."""
diff --git a/src/agentscope/realtime/_events/_server_event.py b/src/agentscope/realtime/_events/_server_event.py
new file mode 100644
index 000000000..caf231f64
--- /dev/null
+++ b/src/agentscope/realtime/_events/_server_event.py
@@ -0,0 +1,524 @@
+# -*- coding: utf-8 -*-
+"""The websocket events generated from the realtime agent and backend."""
+from enum import Enum
+from typing import Literal
+
+from pydantic import BaseModel
+
+from ._utils import AudioFormat
+from ._model_event import ModelEvents
+from ...message import ToolUseBlock, ToolResultBlock
+
+
+class ServerEventType(str, Enum):
+ """Types of agent events for backend-to-web communication."""
+
+ # Session lifecycle
+ SERVER_SESSION_CREATED = "server_session_created"
+ """The session between the web frontend and backend is created."""
+
+ SERVER_SESSION_UPDATED = "server_session_updated"
+ """The session between the web frontend and backend is updated."""
+
+ SERVER_SESSION_ENDED = "server_session_ended"
+ """The session between the web frontend and backend is ended."""
+
+ # ============== AGENT LIFECYCLE EVENTS ================
+
+ AGENT_READY = "agent_ready"
+ """The agent is created and ready to receive inputs."""
+
+ AGENT_ENDED = "agent_ended"
+ """The agent ended."""
+
+ # ============== AGENT RESPONSE EVENTS =================
+
+ # Response events
+ AGENT_RESPONSE_CREATED = "agent_response_created"
+ """The agent starts generating a response."""
+
+ AGENT_RESPONSE_DONE = "agent_response_done"
+ """The agent finished generating a response."""
+
+ # ============== Response content events =================
+
+ AGENT_RESPONSE_AUDIO_DELTA = "agent_response_audio_delta"
+ """The agent's response audio data delta."""
+
+ AGENT_RESPONSE_AUDIO_DONE = "agent_response_audio_done"
+ """The agent's response audio data is complete."""
+
+ AGENT_RESPONSE_AUDIO_TRANSCRIPT_DELTA = (
+ "agent_response_audio_transcript_delta"
+ )
+ """The agent's response audio transcription delta."""
+
+ AGENT_RESPONSE_AUDIO_TRANSCRIPT_DONE = (
+ "agent_response_audio_transcript_done"
+ )
+ """The agent's response audio transcription is complete."""
+
+ AGENT_RESPONSE_TOOL_USE_DELTA = "agent_response_tool_use_delta"
+ """The agent's response tool use data delta."""
+
+ AGENT_RESPONSE_TOOL_USE_DONE = "agent_response_tool_use_done"
+ """The agent's response tool use data is complete."""
+
+ AGENT_RESPONSE_TOOL_RESULT = "agent_response_tool_result"
+ """The tool execution result."""
+
+ # ============== INPUT AUDIO TRANSCRIPTION EVENTS =================
+
+ AGENT_INPUT_TRANSCRIPTION_DELTA = "agent_input_transcription_delta"
+ """The input audio transcription delta."""
+
+ AGENT_INPUT_TRANSCRIPTION_DONE = "agent_input_transcription_done"
+ """The input audio transcription is complete."""
+
+ # Input detection
+ AGENT_INPUT_STARTED = "agent_input_started"
+ """Detected the start of user input audio."""
+
+ AGENT_INPUT_DONE = "agent_input_done"
+ """Detected the end of user input audio."""
+
+ # ============== ERROR EVENTS =================
+
+ AGENT_ERROR = "agent_error"
+ """An error occurred in the backend or agent."""
+
+
+class ServerEvents:
+ """Realtime server events."""
+
+ class EventBase(BaseModel):
+ """The base class for all server events, used to unify the type
+ hinting."""
+
+ class ServerSessionCreatedEvent(EventBase):
+ """Server session created event in the backend"""
+
+ session_id: str
+ """The session ID."""
+
+ type: Literal[
+ ServerEventType.SERVER_SESSION_CREATED
+ ] = ServerEventType.SERVER_SESSION_CREATED
+ """The event type."""
+
+ class ServerSessionUpdatedEvent(EventBase):
+ """Server session updated event in the backend"""
+
+ session_id: str
+ """The session ID."""
+
+ type: Literal[
+ ServerEventType.SERVER_SESSION_UPDATED
+ ] = ServerEventType.SERVER_SESSION_UPDATED
+ """The event type."""
+
+ class ServerSessionEndedEvent(EventBase):
+ """Server Session ended event in the backend"""
+
+ session_id: str
+ """The session ID."""
+
+ type: Literal[
+ ServerEventType.SERVER_SESSION_ENDED
+ ] = ServerEventType.SERVER_SESSION_ENDED
+ """The event type."""
+
+ class AgentReadyEvent(EventBase):
+ """Agent ready event in the backend"""
+
+ agent_id: str
+ """The agent ID."""
+
+ agent_name: str
+ """The agent name."""
+
+ type: Literal[
+ ServerEventType.AGENT_READY
+ ] = ServerEventType.AGENT_READY
+ """The event type."""
+
+ class AgentEndedEvent(EventBase):
+ """Agent ended event in the backend"""
+
+ agent_id: str
+ """The agent ID."""
+
+ agent_name: str
+ """The agent name."""
+
+ type: Literal[
+ ServerEventType.AGENT_ENDED
+ ] = ServerEventType.AGENT_ENDED
+ """The event type."""
+
+ class AgentResponseCreatedEvent(EventBase):
+ """Response created event in the backend"""
+
+ response_id: str
+ """The response ID."""
+
+ agent_id: str
+ """The agent ID."""
+
+ agent_name: str
+ """The agent name."""
+
+ type: Literal[
+ ServerEventType.AGENT_RESPONSE_CREATED
+ ] = ServerEventType.AGENT_RESPONSE_CREATED
+ """The event type."""
+
+ class AgentResponseDoneEvent(EventBase):
+ """Response done event in the backend"""
+
+ response_id: str
+ """The response ID."""
+
+ input_tokens: int
+ """The number of input tokens used."""
+
+ output_tokens: int
+ """The number of output tokens used."""
+
+ metadata: dict[str, str] = {}
+ """Additional metadata about the response."""
+
+ agent_id: str
+ """The agent ID."""
+
+ agent_name: str
+ """The agent name."""
+
+ type: Literal[
+ ServerEventType.AGENT_RESPONSE_DONE
+ ] = ServerEventType.AGENT_RESPONSE_DONE
+ """The event type."""
+
+ class AgentResponseAudioDeltaEvent(EventBase):
+ """Response audio delta event in the backend"""
+
+ response_id: str
+ """The response ID."""
+
+ item_id: str
+ """The response item ID."""
+
+ delta: str
+ """The audio chunk data, encoded as base64 string."""
+
+ format: AudioFormat
+ """The audio format information."""
+
+ agent_id: str
+ """The agent ID."""
+
+ agent_name: str
+ """The agent name."""
+
+ type: Literal[
+ ServerEventType.AGENT_RESPONSE_AUDIO_DELTA
+ ] = ServerEventType.AGENT_RESPONSE_AUDIO_DELTA
+ """The event type."""
+
+ class AgentResponseAudioDoneEvent(EventBase):
+ """Response audio done event in the backend"""
+
+ response_id: str
+ """The response ID."""
+
+ item_id: str
+ """The response item ID."""
+
+ agent_id: str
+ """The agent ID."""
+
+ agent_name: str
+ """The agent name."""
+
+ type: Literal[
+ ServerEventType.AGENT_RESPONSE_AUDIO_DONE
+ ] = ServerEventType.AGENT_RESPONSE_AUDIO_DONE
+
+ class AgentResponseAudioTranscriptDeltaEvent(EventBase):
+ """Response audio transcript delta event in the backend"""
+
+ response_id: str
+ """The response ID."""
+
+ item_id: str
+ """The response item ID."""
+
+ delta: str
+ """The transcript chunk data."""
+
+ agent_id: str
+ """The agent ID."""
+
+ agent_name: str
+ """The agent name."""
+
+ type: Literal[
+ ServerEventType.AGENT_RESPONSE_AUDIO_TRANSCRIPT_DELTA
+ ] = ServerEventType.AGENT_RESPONSE_AUDIO_TRANSCRIPT_DELTA
+ """The event type."""
+
+ class AgentResponseAudioTranscriptDoneEvent(EventBase):
+ """Response audio transcript done event in the backend"""
+
+ response_id: str
+ """The response ID."""
+
+ item_id: str
+ """The response item ID."""
+
+ agent_id: str
+ """The agent ID."""
+
+ agent_name: str
+ """The agent name."""
+
+ type: Literal[
+ ServerEventType.AGENT_RESPONSE_AUDIO_TRANSCRIPT_DONE
+ ] = ServerEventType.AGENT_RESPONSE_AUDIO_TRANSCRIPT_DONE
+ """The event type."""
+
+ class AgentResponseToolUseDeltaEvent(EventBase):
+ """Response tool use delta event in the backend"""
+
+ response_id: str
+ """The response ID."""
+
+ item_id: str
+ """The response item ID."""
+
+ tool_use: ToolUseBlock
+ """The tool use block delta, the arguments are accumulated in the
+ `raw_input` field."""
+
+ agent_id: str
+ """The agent ID."""
+
+ agent_name: str
+ """The agent name."""
+
+ type: Literal[
+ ServerEventType.AGENT_RESPONSE_TOOL_USE_DELTA
+ ] = ServerEventType.AGENT_RESPONSE_TOOL_USE_DELTA
+ """The event type."""
+
+ class AgentResponseToolUseDoneEvent(EventBase):
+ """Response tool use done event in the backend"""
+
+ response_id: str
+ """The response ID."""
+
+ item_id: str
+ """The response item ID."""
+
+ tool_use: ToolUseBlock
+ """The complete tool use block."""
+
+ agent_id: str
+ """The agent ID."""
+
+ agent_name: str
+ """The agent name."""
+
+ type: Literal[
+ ServerEventType.AGENT_RESPONSE_TOOL_USE_DONE
+ ] = ServerEventType.AGENT_RESPONSE_TOOL_USE_DONE
+ """The event type."""
+
+ class AgentResponseToolResultEvent(EventBase):
+ """Response tool result event"""
+
+ tool_result: ToolResultBlock
+ """The tool result block."""
+
+ agent_id: str
+ """The agent ID."""
+
+ agent_name: str
+ """The agent name."""
+
+ type: Literal[
+ ServerEventType.AGENT_RESPONSE_TOOL_RESULT
+ ] = ServerEventType.AGENT_RESPONSE_TOOL_RESULT
+ """The event type."""
+
+ class AgentInputTranscriptionDeltaEvent(EventBase):
+ """Input transcription delta event in the backend"""
+
+ item_id: str
+ """The conversation item ID."""
+
+ delta: str
+ """The transcription chunk data."""
+
+ agent_id: str
+ """The agent ID."""
+
+ agent_name: str
+ """The agent name."""
+
+ type: Literal[
+ ServerEventType.AGENT_INPUT_TRANSCRIPTION_DELTA
+ ] = ServerEventType.AGENT_INPUT_TRANSCRIPTION_DELTA
+ """The event type."""
+
+ class AgentInputTranscriptionDoneEvent(EventBase):
+ """Input transcription done event in the backend"""
+
+ transcript: str
+ """The complete transcription text."""
+
+ item_id: str
+ """The conversation item ID."""
+
+ input_tokens: int | None = None
+ """The number of input tokens."""
+
+ output_tokens: int | None = None
+ """The number of output tokens."""
+
+ agent_id: str
+ """The agent ID."""
+
+ agent_name: str
+ """The agent name."""
+
+ type: Literal[
+ ServerEventType.AGENT_INPUT_TRANSCRIPTION_DONE
+ ] = ServerEventType.AGENT_INPUT_TRANSCRIPTION_DONE
+ """The event type."""
+
+ class AgentInputStartedEvent(EventBase):
+ """Input started event in the backend"""
+
+ item_id: str
+ """The conversation item ID."""
+
+ audio_start_ms: int
+ """The audio start time in milliseconds."""
+
+ agent_id: str
+ """The agent ID."""
+
+ agent_name: str
+ """The agent name."""
+
+ type: Literal[
+ ServerEventType.AGENT_INPUT_STARTED
+ ] = ServerEventType.AGENT_INPUT_STARTED
+ """The event type."""
+
+ class AgentInputDoneEvent(EventBase):
+ """Input done event in the backend"""
+
+ item_id: str
+ """The conversation item ID."""
+
+ audio_end_ms: int
+ """The audio end time in milliseconds."""
+
+ agent_id: str
+ """The agent ID."""
+
+ agent_name: str
+ """The agent name."""
+
+ type: Literal[
+ ServerEventType.AGENT_INPUT_DONE
+ ] = ServerEventType.AGENT_INPUT_DONE
+ """The event type."""
+
+ class AgentErrorEvent(EventBase):
+ """Error event in the backend"""
+
+ error_type: str
+ """The error type."""
+
+ code: str
+ """The error code."""
+
+ message: str
+ """The error message."""
+
+ agent_id: str
+ """The agent ID."""
+
+ agent_name: str
+ """The agent name."""
+
+ type: Literal[
+ ServerEventType.AGENT_ERROR
+ ] = ServerEventType.AGENT_ERROR
+ """The event type."""
+
+ @classmethod
+ def from_model_event(
+ cls,
+ model_event: ModelEvents.ModelResponseCreatedEvent
+ | ModelEvents.ModelResponseDoneEvent
+ | ModelEvents.ModelResponseAudioDeltaEvent
+ | ModelEvents.ModelResponseAudioDoneEvent
+ | ModelEvents.ModelResponseAudioTranscriptDeltaEvent
+ | ModelEvents.ModelResponseAudioTranscriptDoneEvent
+ | ModelEvents.ModelResponseToolUseDeltaEvent
+ | ModelEvents.ModelResponseToolUseDoneEvent
+ | ModelEvents.ModelInputTranscriptionDeltaEvent
+ | ModelEvents.ModelInputTranscriptionDoneEvent
+ | ModelEvents.ModelInputStartedEvent
+ | ModelEvents.ModelInputDoneEvent
+ | ModelEvents.ModelErrorEvent,
+ agent_id: str,
+ agent_name: str,
+ ) -> EventBase:
+ """Convert a model event to a server event quickly with
+ 1) replace the "model_" prefix with "agent_" in the type field; 2) add
+ agent_id and agent_name fields
+
+ Args:
+ model_event (`ModelEvents.EventBase`):
+ The model event to convert.
+ agent_id (`str`):
+ The agent ID.
+ agent_name (`str`):
+ The agent name.
+
+ Returns:
+ `ServerEvents.EventBase`:
+ The converted server event.
+ """
+ # Obtain the corresponding agent event class
+ cls_name = model_event.__class__.__name__.replace("Model", "Agent")
+ agent_event_cls = getattr(cls, cls_name)
+
+ # The data dict of the model event
+ model_event_dict = model_event.model_dump()
+
+ # 1) Replace the "model_" prefix with "agent_" in the type field
+ if "type" in model_event_dict:
+ model_event_dict["type"] = model_event_dict["type"].replace(
+ "model_",
+ "agent_",
+ )
+
+ try:
+ # 2) Add agent_id and agent_name fields
+ model_event_dict["agent_id"] = agent_id
+ model_event_dict["agent_name"] = agent_name
+ agent_event = agent_event_cls.model_validate(model_event_dict)
+
+ except Exception as e:
+ raise RuntimeError(
+ f"Failed to convert model event {model_event} to agent "
+ f"event {agent_event_cls}: {e}",
+ ) from e
+
+ return agent_event
diff --git a/src/agentscope/realtime/_events/_utils.py b/src/agentscope/realtime/_events/_utils.py
new file mode 100644
index 000000000..90fca3222
--- /dev/null
+++ b/src/agentscope/realtime/_events/_utils.py
@@ -0,0 +1,15 @@
+# -*- coding: utf-8 -*-
+"""The utils for realtime events."""
+from pydantic import BaseModel, ConfigDict
+
+
+class AudioFormat(BaseModel):
+ """The audio format class"""
+
+ model_config = ConfigDict(extra="allow")
+
+ type: str
+ """The audio type, e.g., 'audio/pcm'"""
+
+ rate: int
+ """The audio sample rate, e.g., 16000"""
diff --git a/src/agentscope/realtime/_gemini_realtime_model.py b/src/agentscope/realtime/_gemini_realtime_model.py
new file mode 100644
index 000000000..b396cdbcb
--- /dev/null
+++ b/src/agentscope/realtime/_gemini_realtime_model.py
@@ -0,0 +1,662 @@
+# -*- coding: utf-8 -*-
+"""The Gemini realtime model class."""
+import json
+from typing import Literal, Any
+
+import shortuuid
+
+from ._events import ModelEvents
+from ._base import RealtimeModelBase
+from .._logging import logger
+from .._utils._common import _get_bytes_from_web_url
+from ..message import (
+ AudioBlock,
+ ImageBlock,
+ TextBlock,
+ ToolResultBlock,
+ ToolUseBlock,
+)
+
+
+class GeminiRealtimeModel(RealtimeModelBase):
+ """The Gemini realtime model class."""
+
+ support_input_modalities: list[str] = [
+ "audio",
+ "text",
+ "image",
+ "tool_result",
+ ]
+ """The supported input modalities of the Gemini realtime model."""
+
+ websocket_url: str = (
+ "wss://generativelanguage.googleapis.com/ws/"
+ "google.ai.generativelanguage.v1alpha.GenerativeService."
+ "BidiGenerateContent?key={api_key}"
+ )
+ """The websocket URL of the Gemini realtime model API."""
+
+ websocket_headers: dict[str, str] = {
+ "Content-Type": "application/json",
+ }
+ """The websocket headers of the Gemini realtime model API."""
+
+ input_sample_rate: int
+ """The input audio sample rate."""
+
+ output_sample_rate: int
+ """The output audio sample rate."""
+
+ def __init__(
+ self,
+ model_name: str,
+ api_key: str,
+ voice: Literal["Puck", "Charon", "Kore", "Fenrir"] | str = "Puck",
+ enable_input_audio_transcription: bool = True,
+ ) -> None:
+ """Initialize the GeminiRealtimeModel class.
+
+ Args:
+ model_name (`str`):
+ The model name, e.g. "gemini-2.5-flash-native-audio-preview
+ -09-2025". Refers to `official documentation
+ <https://ai.google.dev/gemini-api/docs/live?hl=zh-cn&example=mic-stream>`_
+ for more details.
+ api_key (`str`):
+ The Gemini API key for authentication.
+ voice (`Literal["Puck", "Charon", "Kore", "Fenrir"] str`,
+ defaults to `"Puck"`):
+ The voice to be used for text-to-speech.
+ enable_input_audio_transcription (`bool`, defaults to `True`):
+ Whether to enable input audio transcription.
+ """
+ super().__init__(model_name)
+
+ self.voice = voice
+ self.enable_input_audio_transcription = (
+ enable_input_audio_transcription
+ )
+
+ # The Gemini realtime API uses 16kHz input and 24kHz output.
+ self.input_sample_rate = 16000
+ self.output_sample_rate = 24000
+
+ # Set the API key in the websocket URL.
+ self.websocket_url = self.websocket_url.format(api_key=api_key)
+
+ # Response tracking state.
+ # Note: Unlike DashScope/OpenAI which send explicit `response.created`
+ # events, Gemini does not. We generate response IDs ourselves using
+ # short UUID to ensure uniqueness.
+ self._response_id: str | None = None
+
+ def _build_session_config(
+ self,
+ instructions: str,
+ tools: list[dict] | None,
+ **kwargs: Any,
+ ) -> dict:
+ """Build Gemini setup message.
+
+ Gemini Live API requires a "setup" message as the first message
+ to configure the session.
+
+ Args:
+ instructions (`str`):
+ The system instructions for the model.
+ tools (`list[dict]`):
+ The list of tool JSON schemas.
+ **kwargs:
+ Additional configuration parameters.
+
+ Returns:
+ `dict`:
+ The session configuration dict.
+ """
+ # Model configuration
+ session_config: dict = {
+ "model": f"models/{self.model_name}",
+ "systemInstruction": {
+ "parts": [{"text": instructions}],
+ },
+ "outputAudioTranscription": {},
+ }
+
+ # Audio transcription configuration
+ if self.enable_input_audio_transcription:
+ session_config["inputAudioTranscription"] = {}
+
+ # Generation configuration
+ generation_config: dict = {
+ "responseModalities": ["AUDIO"],
+ **kwargs,
+ }
+
+ # Voice configuration
+ if self.voice:
+ generation_config["speechConfig"] = {
+ "voiceConfig": {
+ "prebuiltVoiceConfig": {"voiceName": self.voice},
+ },
+ }
+
+ session_config["generationConfig"] = generation_config
+
+ # Tools configuration
+ if tools:
+ session_config["tools"] = self._format_toolkit_schema(tools)
+
+ setup_msg = {"setup": session_config}
+ return setup_msg
+
+ def _format_toolkit_schema(
+ self,
+ schemas: list[dict[str, Any]],
+ ) -> list[dict[str, Any]]:
+ """Format the tools JSON schema into Gemini format.
+
+ Args:
+ schemas (`list[dict[str, Any]]`):
+ The tool schemas.
+
+ Returns:
+ `list[dict[str, Any]]`:
+ The formatted tools for Gemini.
+ """
+ function_declarations = []
+ for schema in schemas:
+ if "function" not in schema:
+ continue
+ func = schema["function"].copy()
+ function_declarations.append(func)
+
+ return [{"function_declarations": function_declarations}]
+
+ async def send(
+ self,
+ data: AudioBlock | TextBlock | ImageBlock | ToolResultBlock,
+ ) -> None:
+ """Send the data to the Gemini realtime model for processing.
+
+ Args:
+ data (`AudioBlock | TextBlock | ImageBlock | ToolResultBlock`):
+ The data to be sent to the Gemini realtime model.
+ """
+ from websockets import State
+
+ if not self._websocket or self._websocket.state != State.OPEN:
+ raise RuntimeError(
+ f"WebSocket is not connected for model {self.model_name}. "
+ "Call the `connect` method first.",
+ )
+
+ # Type checking
+ assert (
+ isinstance(data, dict) and "type" in data
+ ), "Data must be a dict with a 'type' field."
+
+ # The source must be base64 for audio data
+ data_type = data.get("type")
+
+ if data_type not in self.support_input_modalities:
+ logger.warning(
+ "Gemini Realtime API does not support %s data input. "
+ "Supported modalities are: %s",
+ data_type,
+ ", ".join(self.support_input_modalities),
+ )
+ return
+
+ # Process the data based on its type
+ if data_type == "image":
+ to_send_message = await self._parse_image_data(
+ ImageBlock(
+ type="image",
+ source=data.get("source"),
+ ),
+ )
+
+ elif data_type == "audio":
+ to_send_message = await self._parse_audio_data(
+ AudioBlock(
+ type="audio",
+ source=data.get("source"),
+ ),
+ )
+
+ elif data_type == "text":
+ to_send_message = await self._parse_text_data(
+ TextBlock(
+ type="text",
+ text=data.get("text"),
+ ),
+ )
+
+ elif data_type == "tool_result":
+ to_send_message = await self._parse_tool_result_data(
+ ToolResultBlock(
+ type="tool_result",
+ name=data.get("name"),
+ output=data.get("output"),
+ id=data.get("id"),
+ ),
+ )
+
+ else:
+ raise RuntimeError(f"Unsupported data type {data_type}")
+
+ if to_send_message:
+ await self._websocket.send(to_send_message)
+
+ async def parse_api_message(
+ self,
+ message: str,
+ ) -> ModelEvents.EventBase | list[ModelEvents.EventBase] | None:
+ """Parse the message received from the Gemini realtime model API.
+
+ Args:
+ message (`str`):
+ The message received from the Gemini realtime model API.
+
+ Returns:
+ `ModelEvents.EventBase | list[ModelEvents.EventBase] | None`:
+ The unified model event(s) in agentscope format.
+ """
+ try:
+ data = json.loads(message)
+ except json.decoder.JSONDecodeError:
+ return None
+
+ if not isinstance(data, dict):
+ return None
+
+ model_event = None
+
+ # ================ Setup related events ================
+ if "setupComplete" in data:
+ model_event = ModelEvents.ModelSessionCreatedEvent(
+ session_id="gemini_session",
+ )
+
+ # ================ Server content events ================
+ elif "serverContent" in data:
+ model_event = await self._parse_server_content(
+ data["serverContent"],
+ )
+
+ # ================ Tool call events ================
+ elif "toolCall" in data:
+ model_event = await self._parse_tool_call(data["toolCall"])
+
+ # ================ Tool call cancellation ================
+ elif "toolCallCancellation" in data:
+ # Tool call was cancelled
+ # This effectively ends the current response.
+ logger.info(
+ "Tool call cancelled: %s",
+ data["toolCallCancellation"],
+ )
+ response_id = self._response_id or ""
+ self._response_id = None # Clear response ID
+ model_event = ModelEvents.ModelResponseDoneEvent(
+ response_id=response_id,
+ input_tokens=0,
+ output_tokens=0,
+ )
+
+ # ================ Error events ================
+ elif "error" in data:
+ error = data["error"]
+ model_event = ModelEvents.ModelErrorEvent(
+ error_type=error.get("status", "unknown"),
+ code=str(error.get("code", "unknown")),
+ message=error.get("message", "An unknown error occurred."),
+ )
+
+ else:
+ logger.debug(
+ "Unknown Gemini realtime model message keys: %s",
+ list(data.keys()),
+ )
+
+ return model_event
+
+ def _ensure_response_id(self) -> str:
+ """Ensure a response ID exists, creating one if necessary.
+
+ Gemini doesn't send explicit response.created events, so we generate
+ the response ID on first audio/text chunk using short UUID.
+
+ Returns:
+ `str`: The current response ID.
+ """
+ if not self._response_id:
+ self._response_id = f"resp_{shortuuid.uuid()}"
+ # After the check above, _response_id is guaranteed to be non-None
+ assert self._response_id is not None
+ return self._response_id
+
+ def _parse_model_turn(
+ self,
+ model_turn: dict,
+ ) -> ModelEvents.EventBase | None:
+ """Parse the modelTurn content from Gemini API.
+
+ Args:
+ model_turn (`dict`):
+ The modelTurn dictionary containing parts with audio/text.
+
+ Returns:
+ `ModelEvents.EventBase | None`:
+ The parsed model event, or None if no valid content found.
+ """
+ parts = model_turn.get("parts", [])
+
+ for part in parts:
+ # Check for audio data
+ if "inlineData" in part:
+ event = self._parse_inline_data(part["inlineData"])
+ if event:
+ return event
+
+ # Check for text data
+ if "text" in part:
+ text_data = part["text"]
+ if text_data:
+ response_id = self._ensure_response_id()
+ return ModelEvents.ModelResponseAudioTranscriptDeltaEvent(
+ response_id=response_id,
+ delta=text_data,
+ item_id="",
+ )
+
+ return None
+
+ def _parse_inline_data(
+ self,
+ inline_data: dict,
+ ) -> ModelEvents.EventBase | None:
+ """Parse inline data (audio) from a model turn part.
+
+ Args:
+ inline_data (`dict`):
+ The inlineData dictionary containing mimeType and data.
+
+ Returns:
+ `ModelEvents | None`:
+ Audio delta event if valid audio data, None otherwise.
+ """
+ mime_type = inline_data.get("mimeType", "")
+ if not mime_type.startswith("audio/"):
+ return None
+
+ audio_data = inline_data.get("data", "")
+ if not audio_data:
+ return None
+
+ response_id = self._ensure_response_id()
+ return ModelEvents.ModelResponseAudioDeltaEvent(
+ response_id=response_id,
+ item_id="",
+ delta=audio_data,
+ format={
+ "type": "audio/pcm",
+ "rate": self.output_sample_rate,
+ },
+ )
+
+ async def _parse_server_content(
+ self,
+ server_content: dict,
+ ) -> ModelEvents.EventBase | None:
+ """Parse the serverContent message from Gemini API.
+
+ Args:
+ server_content (`dict`):
+ The serverContent dictionary from the API response.
+
+ Returns:
+ `ModelEvents.EventBase | None`:
+ The unified model event in agentscope format.
+ """
+ model_event = None
+
+ # Handle model turn (response with audio/text)
+ if "modelTurn" in server_content:
+ model_event = self._parse_model_turn(server_content["modelTurn"])
+
+ # Handle output transcription
+ elif "outputTranscription" in server_content:
+ text = server_content["outputTranscription"].get("text", "")
+ if text:
+ model_event = (
+ ModelEvents.ModelResponseAudioTranscriptDeltaEvent(
+ response_id=self._response_id or "",
+ delta=text,
+ item_id="",
+ )
+ )
+
+ # Handle input transcription
+ elif "inputTranscription" in server_content:
+ text = server_content["inputTranscription"].get("text", "")
+ if text:
+ model_event = ModelEvents.ModelInputTranscriptionDoneEvent(
+ transcript=text,
+ item_id="",
+ )
+
+ # Handle generation complete (response done)
+ elif "generationComplete" in server_content:
+ response_id = self._response_id or ""
+ self._response_id = None
+ model_event = ModelEvents.ModelResponseDoneEvent(
+ response_id=response_id,
+ input_tokens=0,
+ output_tokens=0,
+ )
+
+ # Handle turn complete
+ elif "turnComplete" in server_content:
+ logger.debug("Gemini: turnComplete received")
+ # turnComplete without generationComplete means interrupted
+ if self._response_id:
+ response_id = self._response_id
+ self._response_id = None
+ model_event = ModelEvents.ModelResponseDoneEvent(
+ response_id=response_id,
+ input_tokens=0,
+ output_tokens=0,
+ )
+
+ # Handle interrupted
+ elif "interrupted" in server_content:
+ logger.debug("Gemini: response interrupted")
+
+ return model_event
+
+ async def _parse_tool_call(
+ self,
+ tool_call: dict,
+ ) -> list[ModelEvents.EventBase] | None:
+ """Parse the tool call message from Gemini API.
+
+ Args:
+ tool_call (`dict`):
+ The toolCall dictionary from the API response.
+
+ Returns:
+ `list[ModelEvents.EventBase] | None`:
+ The unified model event(s) in agentscope format.
+ """
+ function_calls = tool_call.get("functionCalls", [])
+ if not function_calls:
+ return None
+
+ events = []
+ for func_call in function_calls:
+ name = func_call.get("name", "")
+ call_id = func_call.get("id", "")
+ args = func_call.get("args", {})
+
+ # Return the accumulated arguments instead of just the delta
+ model_event = ModelEvents.ModelResponseToolUseDoneEvent(
+ response_id=self._response_id or "",
+ item_id="",
+ tool_use=ToolUseBlock(
+ type="tool_use",
+ id=call_id,
+ name=name,
+ input=args,
+ raw_input=json.dumps(args, ensure_ascii=False),
+ ),
+ )
+ events.append(model_event)
+
+ return events if events else None
+
+ async def _parse_image_data(self, block: ImageBlock) -> str | None:
+ """Parse the image data block to the format required by the Gemini
+ realtime model API.
+
+ Args:
+ block (`ImageBlock`):
+ The image data block.
+
+ Returns:
+ `str | None`: The parsed message to be sent to the Gemini realtime
+ model API.
+ """
+ source = block.get("source", {})
+ source_type = source.get("type", "")
+ # media_type is in Base64Source, use default for URLSource
+ media_type = source.get("media_type", "image/jpeg")
+
+ if source_type == "base64":
+ image_data = source.get("data", "")
+ elif source_type == "url":
+ image_data = _get_bytes_from_web_url(str(source.get("url", "")))
+ else:
+ raise ValueError(f"Unsupported image source type: {source_type}")
+
+ return json.dumps(
+ {
+ "realtimeInput": {
+ "video": {
+ "mimeType": media_type,
+ "data": image_data,
+ },
+ },
+ },
+ )
+
+ async def _parse_audio_data(self, block: AudioBlock) -> str:
+ """Parse the audio data block to the format required by the Gemini
+ realtime model API.
+
+ Args:
+ block (`AudioBlock`):
+ The audio data block.
+
+ Returns:
+ `str`: The parsed message to be sent to the Gemini realtime
+ model API.
+ """
+ source = block.get("source", {})
+ source_type = source.get("type", "")
+
+ if source_type == "base64":
+ audio_data = source.get("data", "")
+ elif source_type == "url":
+ audio_data = _get_bytes_from_web_url(str(source.get("url", "")))
+ else:
+ raise ValueError(f"Unsupported audio source type: {source_type}")
+
+ return json.dumps(
+ {
+ "realtimeInput": {
+ "audio": {
+ "mimeType": f"audio/pcm;rate={self.input_sample_rate}",
+ "data": audio_data,
+ },
+ },
+ },
+ )
+
+ async def _parse_text_data(self, block: TextBlock) -> str:
+ """Parse the text data block to the format required by the Gemini
+ realtime model API.
+
+ Args:
+ block (`TextBlock`):
+ The text data block.
+
+ Returns:
+ `str`: The parsed message to be sent to the Gemini realtime
+ model API.
+ """
+ text = block.get("text", "")
+
+ return json.dumps(
+ {
+ "clientContent": {
+ "turns": [
+ {
+ "role": "user",
+ "parts": [{"text": text}],
+ },
+ ],
+ # TODO: should be set to False?
+ "turnComplete": True,
+ },
+ },
+ )
+
+ async def _parse_tool_result_data(self, block: ToolResultBlock) -> str:
+ """Parse the tool result data block to the format required by the
+ Gemini realtime model API.
+
+ Args:
+ block (`ToolResultBlock`):
+ The tool result data block.
+
+ Returns:
+ `str`: The parsed message to be sent to the Gemini realtime
+ model API.
+ """
+ tool_id = block.get("id", "")
+ tool_name = block.get("name", "")
+ output = block.get("output", "")
+
+ # Extract text from list of blocks (most common case)
+ if isinstance(output, list):
+ text = "".join(
+ str(item.get("text", ""))
+ if isinstance(item, dict) and item.get("type") == "text"
+ else str(item)
+ for item in output
+ )
+ result_obj = {"result": text}
+ elif isinstance(output, str):
+ try:
+ result_obj = json.loads(output)
+ except json.JSONDecodeError:
+ result_obj = {"result": output}
+ else:
+ result_obj = (
+ output if isinstance(output, dict) else {"result": str(output)}
+ )
+
+ return json.dumps(
+ {
+ "toolResponse": {
+ "functionResponses": [
+ {
+ "id": tool_id,
+ "name": tool_name,
+ "response": result_obj,
+ },
+ ],
+ },
+ },
+ )
diff --git a/src/agentscope/realtime/_openai_realtime_model.py b/src/agentscope/realtime/_openai_realtime_model.py
new file mode 100644
index 000000000..e75a4fa27
--- /dev/null
+++ b/src/agentscope/realtime/_openai_realtime_model.py
@@ -0,0 +1,464 @@
+# -*- coding: utf-8 -*-
+"""The OpenAI realtime model class."""
+import json
+from typing import Literal, Any
+
+from ._events import ModelEvents
+from ._base import RealtimeModelBase
+from .._logging import logger
+from .._utils._common import _get_bytes_from_web_url, _json_loads_with_repair
+from ..message import (
+ AudioBlock,
+ TextBlock,
+ ImageBlock,
+ ToolResultBlock,
+ ToolUseBlock,
+)
+
+
+class OpenAIRealtimeModel(RealtimeModelBase):
+ """The OpenAI realtime model class."""
+
+ support_input_modalities: list[str] = ["audio", "text"]
+ """The supported input modalities of the OpenAI realtime model."""
+
+ support_tools: bool = True
+ """The OpenAI realtime model supports tools API."""
+
+ websocket_url: str = "wss://api.openai.com/v1/realtime?model={model_name}"
+ """The websocket URL of the OpenAI realtime model API."""
+
+ websocket_headers: dict[str, str] = {
+ "Authorization": "Bearer {api_key}",
+ "OpenAI-Beta": "realtime=v1",
+ }
+ """The websocket headers of the OpenAI realtime model API."""
+
+ input_sample_rate: int
+ """The input audio sample rate."""
+
+ output_sample_rate: int
+ """The output audio sample rate."""
+
+ def __init__(
+ self,
+ model_name: str,
+ api_key: str,
+ voice: Literal["alloy", "echo", "marin", "cedar"] | str = "alloy",
+ enable_input_audio_transcription: bool = True,
+ ) -> None:
+ """Initialize the OpenAIRealtimeModel class.
+
+ Args:
+ model_name (`str`):
+ The model name, e.g. "gpt-4o-realtime-preview".
+ api_key (`str`):
+ The API key for authentication.
+ voice (`Literal["alloy", "echo", "marin", "cedar"] | str`, \
+ defaults to `"alloy"`):
+ The voice to be used for text-to-speech.
+ enable_input_audio_transcription (`bool`, defaults to `True`):
+ Whether to enable input audio transcription.
+ """
+ super().__init__(model_name)
+
+ self.voice = voice
+ self.enable_input_audio_transcription = (
+ enable_input_audio_transcription
+ )
+
+ # The OpenAI realtime API uses 24kHz for both input and output.
+ self.input_sample_rate = 24000
+ self.output_sample_rate = 24000
+
+ # Set the model name in the websocket URL.
+ self.websocket_url = self.websocket_url.format(model_name=model_name)
+
+ # Set the API key in the websocket headers.
+ self.websocket_headers["Authorization"] = self.websocket_headers[
+ "Authorization"
+ ].format(api_key=api_key)
+
+ # Record the response ID for the current session.
+ self._response_id = ""
+
+ # Tool arguments accumulator for tracking tool call parameters
+ self._tool_args_accumulator: dict[str, str] = {}
+
+ def _build_session_config(
+ self,
+ instructions: str,
+ tools: list[dict] | None,
+ **kwargs: Any,
+ ) -> dict:
+ """Build the session configuration for the OpenAI realtime model."""
+
+ session_config: dict[str, Any] = {
+ "type": "realtime",
+ "output_modalities": ["audio"],
+ "audio": {
+ "input": {
+ "turn_detection": {
+ "type": "server_vad",
+ "create_response": True,
+ },
+ },
+ "output": {
+ "voice": self.voice,
+ },
+ },
+ "instructions": instructions,
+ **kwargs,
+ }
+
+ # Input audio transcription
+ if self.enable_input_audio_transcription:
+ session_config["audio"]["input"]["transcription"] = {
+ "model": "whisper-1",
+ }
+
+ # Tools configuration
+ if tools:
+ # TODO: format the tools
+ session_config["tools"] = tools
+
+ return {
+ "type": "session.update",
+ "session": session_config,
+ }
+
+ async def send(
+ self,
+ data: AudioBlock | TextBlock | ImageBlock | ToolResultBlock,
+ ) -> None:
+ """Send the data to the OpenAI realtime model for processing.
+
+ Args:
+ data (`AudioBlock | TextBlock | ImageBlock | ToolResultBlock`):
+ The data to be sent to the OpenAI realtime model.
+ """
+ from websockets import State
+
+ if not self._websocket or self._websocket.state != State.OPEN:
+ raise RuntimeError(
+ f"WebSocket is not connected for model {self.model_name}. "
+ "Call the `connect` method first.",
+ )
+
+ # Type checking
+ assert (
+ isinstance(data, dict) and "type" in data
+ ), "Data must be a dict with a 'type' field."
+
+ # The source must be base64 for audio data
+ data_type = data.get("type")
+
+ if data_type not in self.support_input_modalities:
+ logger.warning(
+ "OpenAI Realtime API does not support %s data input. "
+ "Supported modalities are: %s",
+ data_type,
+ ", ".join(self.support_input_modalities),
+ )
+ return
+
+ # Process the data based on its type
+ if data_type == "audio":
+ to_send_message = await self._parse_audio_data(
+ AudioBlock(
+ type="audio",
+ source=data.get("source"),
+ ),
+ )
+
+ elif data_type == "text":
+ to_send_message = await self._parse_text_data(
+ TextBlock(
+ type="text",
+ text=data.get("text"),
+ ),
+ )
+
+ elif data_type == "tool_result":
+ to_send_message = await self._parse_tool_result_data(
+ ToolResultBlock(
+ type="tool_result",
+ id=data.get("id"),
+ output=data.get("output"),
+ name=data.get("name"),
+ ),
+ )
+
+ else:
+ raise RuntimeError(f"Unsupported data type {data_type}")
+
+ await self._websocket.send(to_send_message)
+
+ async def parse_api_message(
+ self,
+ message: str,
+ ) -> ModelEvents.EventBase | list[ModelEvents.EventBase] | None:
+ """Parse the message received from the OpenAI realtime model API.
+
+ Args:
+ message (`str`):
+ The message received from the OpenAI realtime model API.
+
+ Returns:
+ `ModelEvents.EventBase | list[ModelEvents.EventBase] | None`:
+ The unified model event(s) in agentscope format.
+ """
+ try:
+ data = json.loads(message)
+ except json.decoder.JSONDecodeError:
+ return None
+
+ if not isinstance(data, dict):
+ return None
+
+ model_event = None
+ match data.get("type", ""):
+ # ================ Session related events ================
+ case "session.created":
+ model_event = ModelEvents.ModelSessionCreatedEvent(
+ session_id=data.get("session", {}).get("id", ""),
+ )
+
+ case "session.updated":
+ # TODO: handle the session updated event
+ pass
+
+ # ================ Response related events ================
+ case "response.created":
+ # TODO: @qbc we create one shortuuid here?
+ self._response_id = data.get("response", {}).get("id", "")
+ model_event = ModelEvents.ModelResponseCreatedEvent(
+ response_id=self._response_id,
+ )
+
+ case "response.done":
+ response = data.get("response", {})
+ response_id = response.get("id", self._response_id)
+ usage = response.get("usage", {})
+ model_event = ModelEvents.ModelResponseDoneEvent(
+ response_id=response_id,
+ input_tokens=usage.get("input_tokens", 0),
+ output_tokens=usage.get("output_tokens", 0),
+ )
+ # clear the response id
+ self._response_id = ""
+
+ case "response.output_audio.delta":
+ audio_data = data.get("delta", "")
+ if audio_data:
+ model_event = ModelEvents.ModelResponseAudioDeltaEvent(
+ response_id=self._response_id,
+ item_id=data.get("item_id", ""),
+ delta=audio_data,
+ format={
+ "type": "audio/pcm",
+ "rate": self.output_sample_rate,
+ },
+ )
+
+ case "response.output_audio.done":
+ model_event = ModelEvents.ModelResponseAudioDoneEvent(
+ response_id=self._response_id,
+ item_id=data.get("item_id", ""),
+ )
+
+ # ================ Transcription related events ================
+ case "response.output_audio_transcript.delta":
+ transcript_data = data.get("delta", "")
+ if transcript_data:
+ model_event = (
+ ModelEvents.ModelResponseAudioTranscriptDeltaEvent(
+ response_id=self._response_id,
+ delta=transcript_data,
+ item_id=data.get("item_id", ""),
+ )
+ )
+
+ case "response.output_audio_transcript.done":
+ model_event = (
+ ModelEvents.ModelResponseAudioTranscriptDoneEvent(
+ response_id=self._response_id,
+ item_id=data.get("item_id", ""),
+ )
+ )
+
+ case "response.function_call_arguments.delta":
+ arguments_delta = data.get("delta")
+ call_id = data.get("call_id", "")
+ if arguments_delta:
+ # Accumulate arguments
+ if call_id not in self._tool_args_accumulator:
+ self._tool_args_accumulator[call_id] = ""
+ self._tool_args_accumulator[call_id] += arguments_delta
+
+ # Return the accumulated arguments instead of just the
+ # delta
+ model_event = ModelEvents.ModelResponseToolUseDeltaEvent(
+ response_id=self._response_id,
+ item_id=data.get("item_id", ""),
+ tool_use=ToolUseBlock(
+ type="tool_use",
+ id=call_id,
+ name=data.get("name", ""),
+ input={},
+ raw_input=self._tool_args_accumulator[call_id],
+ ),
+ )
+ # TODO: This handles only one tool call at a time. For
+ # parallel tool calls, we might need to reconsider the
+ # event handling mechanism.
+
+ case "response.function_call_arguments.done":
+ call_id = data.get("call_id", "")
+ current_input = self._tool_args_accumulator[call_id]
+ model_event = ModelEvents.ModelResponseToolUseDoneEvent(
+ response_id=self._response_id,
+ item_id=data.get("item_id", ""),
+ tool_use=ToolUseBlock(
+ type="tool_use",
+ id=call_id,
+ name=data.get("name", ""),
+ input=_json_loads_with_repair(current_input),
+ raw_input=current_input,
+ ),
+ )
+ # Clear the accumulator for this call_id when done
+ if call_id in self._tool_args_accumulator:
+ del self._tool_args_accumulator[call_id]
+
+ case "conversation.item.input_audio_transcription.delta":
+ delta = data.get("delta", "")
+ if delta:
+ model_event = (
+ ModelEvents.ModelInputTranscriptionDeltaEvent(
+ item_id=data.get("item_id", ""),
+ delta=delta,
+ )
+ )
+
+ case "conversation.item.input_audio_transcription.completed":
+ transcript_data = data.get("transcript", "")
+ if transcript_data:
+ model_event = ModelEvents.ModelInputTranscriptionDoneEvent(
+ transcript=transcript_data,
+ item_id=data.get("item_id", ""),
+ )
+
+ # ================= VAD related events =================
+ case "input_audio_buffer.speech_started":
+ model_event = ModelEvents.ModelInputStartedEvent(
+ item_id=data.get("item_id", ""),
+ audio_start_ms=data.get("audio_start_ms", 0),
+ )
+
+ case "input_audio_buffer.speech_stopped":
+ model_event = ModelEvents.ModelInputDoneEvent(
+ item_id=data.get("item_id", ""),
+ audio_end_ms=data.get("audio_end_ms", 0),
+ )
+
+ # ================= Error events =================
+ case "error":
+ error = data.get("error", {})
+ model_event = ModelEvents.ModelErrorEvent(
+ error_type=error.get("type", "unknown"),
+ code=error.get("code", "unknown"),
+ message=error.get("message", "An unknown error occurred."),
+ )
+
+ # ================= Unknown events =================
+ case _:
+ logger.debug(
+ "Unknown OpenAI realtime model event type: %s",
+ data.get("type", None),
+ )
+
+ return model_event
+
+ async def _parse_audio_data(self, block: AudioBlock) -> str:
+ """Parse the audio data block to the format required by the OpenAI
+ realtime model API.
+
+ Args:
+ block (`AudioBlock`):
+ The audio data block.
+
+ Returns:
+ `str`: The parsed message to be sent to the OpenAI realtime
+ model API.
+ """
+ if block["source"]["type"] == "base64":
+ audio_data = block["source"]["data"]
+
+ elif block["source"]["type"] == "url":
+ audio_data = _get_bytes_from_web_url(block["source"]["url"])
+
+ else:
+ raise ValueError(
+ f"Unsupported audio source type: {block['source']['type']}",
+ )
+
+ return json.dumps(
+ {
+ "type": "input_audio_buffer.append",
+ "audio": audio_data,
+ },
+ )
+
+ async def _parse_text_data(self, block: TextBlock) -> str:
+ """Parse the text data block to the format required by the OpenAI
+ realtime model API.
+
+ Args:
+ block (`TextBlock`):
+ The text data block.
+
+ Returns:
+ `str`: The parsed message to be sent to the OpenAI realtime
+ model API.
+ """
+ text = block.get("text", "")
+
+ return json.dumps(
+ {
+ "type": "conversation.item.create",
+ "item": {
+ "type": "message",
+ "role": "user",
+ "content": [
+ {
+ "type": "input_text",
+ "text": text,
+ },
+ ],
+ },
+ },
+ )
+
+ async def _parse_tool_result_data(self, block: ToolResultBlock) -> str:
+ """Parse the tool result data block to the format required by the
+ OpenAI realtime model API.
+
+ Args:
+ block (`ToolResultBlock`):
+ The tool result data block.
+
+ Returns:
+ `str`: The parsed message to be sent to the OpenAI realtime
+ model API.
+ """
+ return json.dumps(
+ {
+ "type": "conversation.item.create",
+ "item": {
+ "type": "function_call_output",
+ "call_id": block.get("id"),
+ "output": block.get("output"),
+ },
+ },
+ )
diff --git a/tests/realtime_dashscope_test.py b/tests/realtime_dashscope_test.py
new file mode 100644
index 000000000..208df27d1
--- /dev/null
+++ b/tests/realtime_dashscope_test.py
@@ -0,0 +1,383 @@
+# -*- coding: utf-8 -*-
+# pylint: disable=protected-access
+"""Unit tests for DashScope Realtime Model class."""
+import json
+from unittest.async_case import IsolatedAsyncioTestCase
+from unittest.mock import AsyncMock, patch
+
+from agentscope.realtime import DashScopeRealtimeModel, ModelEvents
+from agentscope.message import (
+ AudioBlock,
+ ImageBlock,
+ Base64Source,
+ URLSource,
+)
+
+
+class TestDashScopeRealtimeModelParseAPIMessage(IsolatedAsyncioTestCase):
+ """Test parsing API messages from DashScope realtime model."""
+
+ def setUp(self) -> None:
+ """Set up test fixtures."""
+ self.model = DashScopeRealtimeModel(
+ model_name="qwen3-omni-flash-realtime",
+ api_key="test_api_key",
+ voice="Cherry",
+ )
+
+ async def test_parse_session_created_event(self) -> None:
+ """Test parsing session.created event."""
+ message = json.dumps(
+ {
+ "type": "session.created",
+ "session": {
+ "id": "session_123",
+ },
+ },
+ )
+
+ event = await self.model.parse_api_message(message)
+
+ self.assertIsInstance(event, ModelEvents.ModelSessionCreatedEvent)
+ self.assertEqual(event.session_id, "session_123")
+ self.assertEqual(event.type, "model_session_created")
+
+ async def test_parse_response_created_event(self) -> None:
+ """Test parsing response.created event."""
+ message = json.dumps(
+ {
+ "type": "response.created",
+ "response": {
+ "id": "resp_456",
+ },
+ },
+ )
+
+ event = await self.model.parse_api_message(message)
+
+ self.assertIsInstance(event, ModelEvents.ModelResponseCreatedEvent)
+ self.assertEqual(event.response_id, "resp_456")
+ self.assertEqual(event.type, "model_response_created")
+ # Check that response_id is stored internally
+ self.assertEqual(self.model._response_id, "resp_456")
+
+ async def test_parse_response_done_event(self) -> None:
+ """Test parsing response.done event."""
+ # Set up the internal response_id
+ self.model._response_id = "resp_789"
+
+ message = json.dumps(
+ {
+ "type": "response.done",
+ "response": {
+ "id": "resp_789",
+ "usage": {
+ "input_tokens": 100,
+ "output_tokens": 50,
+ },
+ },
+ },
+ )
+
+ event = await self.model.parse_api_message(message)
+
+ self.assertIsInstance(event, ModelEvents.ModelResponseDoneEvent)
+ self.assertEqual(event.response_id, "resp_789")
+ self.assertEqual(event.input_tokens, 100)
+ self.assertEqual(event.output_tokens, 50)
+ self.assertEqual(event.type, "model_response_done")
+ # Check that response_id is cleared
+ self.assertEqual(self.model._response_id, "")
+
+ async def test_parse_response_audio_delta_event(self) -> None:
+ """Test parsing response.audio.delta event."""
+ self.model._response_id = "resp_audio_1"
+
+ message = json.dumps(
+ {
+ "type": "response.audio.delta",
+ "item_id": "item_audio_1",
+ "delta": "base64_audio_data_chunk",
+ },
+ )
+
+ event = await self.model.parse_api_message(message)
+
+ self.assertIsInstance(event, ModelEvents.ModelResponseAudioDeltaEvent)
+ self.assertEqual(event.response_id, "resp_audio_1")
+ self.assertEqual(event.item_id, "item_audio_1")
+ self.assertEqual(event.delta, "base64_audio_data_chunk")
+ self.assertEqual(event.format.type, "audio/pcm")
+ self.assertEqual(event.format.rate, 24000)
+ self.assertEqual(event.type, "model_response_audio_delta")
+
+ async def test_parse_response_audio_done_event(self) -> None:
+ """Test parsing response.audio.done event."""
+ self.model._response_id = "resp_audio_2"
+
+ message = json.dumps(
+ {
+ "type": "response.audio.done",
+ "item_id": "item_audio_2",
+ },
+ )
+
+ event = await self.model.parse_api_message(message)
+
+ self.assertIsInstance(event, ModelEvents.ModelResponseAudioDoneEvent)
+ self.assertEqual(event.response_id, "resp_audio_2")
+ self.assertEqual(event.item_id, "item_audio_2")
+ self.assertEqual(event.type, "model_response_audio_done")
+
+ async def test_parse_response_audio_transcript_delta_event(self) -> None:
+ """Test parsing response.audio_transcript.delta event."""
+ self.model._response_id = "resp_transcript_1"
+
+ message = json.dumps(
+ {
+ "type": "response.audio_transcript.delta",
+ "item_id": "item_transcript_1",
+ "delta": "Hello ",
+ },
+ )
+
+ event = await self.model.parse_api_message(message)
+
+ self.assertIsInstance(
+ event,
+ ModelEvents.ModelResponseAudioTranscriptDeltaEvent,
+ )
+ self.assertEqual(event.response_id, "resp_transcript_1")
+ self.assertEqual(event.item_id, "item_transcript_1")
+ self.assertEqual(event.delta, "Hello ")
+ self.assertEqual(
+ event.type,
+ "model_response_audio_transcript_delta",
+ )
+
+ async def test_parse_response_audio_transcript_done_event(self) -> None:
+ """Test parsing response.audio_transcript.done event."""
+ self.model._response_id = "resp_transcript_2"
+
+ message = json.dumps(
+ {
+ "type": "response.audio_transcript.done",
+ "item_id": "item_transcript_2",
+ },
+ )
+
+ event = await self.model.parse_api_message(message)
+
+ self.assertIsInstance(
+ event,
+ ModelEvents.ModelResponseAudioTranscriptDoneEvent,
+ )
+ self.assertEqual(event.response_id, "resp_transcript_2")
+ self.assertEqual(event.item_id, "item_transcript_2")
+ self.assertEqual(event.type, "model_response_audio_transcript_done")
+
+ async def test_parse_input_audio_transcription_completed_event(
+ self,
+ ) -> None:
+ """Test parsing conversation.item.input_audio_transcription.completed
+ event."""
+ message = json.dumps(
+ {
+ "type": "conversation.item.input_audio_transcription."
+ "completed",
+ "item_id": "item_input_1",
+ "transcript": "Hello world",
+ },
+ )
+
+ event = await self.model.parse_api_message(message)
+
+ self.assertIsInstance(
+ event,
+ ModelEvents.ModelInputTranscriptionDoneEvent,
+ )
+ self.assertEqual(event.item_id, "item_input_1")
+ self.assertEqual(event.transcript, "Hello world")
+ self.assertEqual(event.type, "model_input_transcription_done")
+
+ async def test_parse_input_audio_buffer_speech_started_event(
+ self,
+ ) -> None:
+ """Test parsing input_audio_buffer.speech_started event."""
+ message = json.dumps(
+ {
+ "type": "input_audio_buffer.speech_started",
+ "item_id": "item_vad_1",
+ "audio_start_ms": 1000,
+ },
+ )
+
+ event = await self.model.parse_api_message(message)
+
+ self.assertIsInstance(event, ModelEvents.ModelInputStartedEvent)
+ self.assertEqual(event.item_id, "item_vad_1")
+ self.assertEqual(event.audio_start_ms, 1000)
+ self.assertEqual(event.type, "model_input_started")
+
+ async def test_parse_input_audio_buffer_speech_stopped_event(
+ self,
+ ) -> None:
+ """Test parsing input_audio_buffer.speech_stopped event."""
+ message = json.dumps(
+ {
+ "type": "input_audio_buffer.speech_stopped",
+ "item_id": "item_vad_2",
+ "audio_end_ms": 5000,
+ },
+ )
+
+ event = await self.model.parse_api_message(message)
+
+ self.assertIsInstance(event, ModelEvents.ModelInputDoneEvent)
+ self.assertEqual(event.item_id, "item_vad_2")
+ self.assertEqual(event.audio_end_ms, 5000)
+ self.assertEqual(event.type, "model_input_done")
+
+ async def test_parse_error_event(self) -> None:
+ """Test parsing error event."""
+ message = json.dumps(
+ {
+ "type": "error",
+ "error": {
+ "type": "invalid_request",
+ "code": "400",
+ "message": "Invalid request format",
+ },
+ },
+ )
+
+ event = await self.model.parse_api_message(message)
+
+ self.assertIsInstance(event, ModelEvents.ModelErrorEvent)
+ self.assertEqual(event.error_type, "invalid_request")
+ self.assertEqual(event.code, "400")
+ self.assertEqual(event.message, "Invalid request format")
+ self.assertEqual(event.type, "model_error")
+
+
+class TestDashScopeRealtimeModelSend(IsolatedAsyncioTestCase):
+ """Test sending data to DashScope realtime model."""
+
+ def setUp(self) -> None:
+ """Set up test fixtures."""
+ from websockets import State
+
+ self.model = DashScopeRealtimeModel(
+ model_name="qwen3-omni-flash-realtime",
+ api_key="test_api_key",
+ voice="Cherry",
+ )
+ # Mock the websocket
+ self.mock_websocket = AsyncMock()
+ self.mock_websocket.state = State.OPEN
+ self.model._websocket = self.mock_websocket
+
+ async def test_send_audio_base64(self) -> None:
+ """Test sending audio data with base64 source."""
+ audio_data = AudioBlock(
+ type="audio",
+ source=Base64Source(
+ type="base64",
+ media_type="audio/wav",
+ data="base64_encoded_audio_data",
+ ),
+ )
+
+ await self.model.send(audio_data)
+
+ # Verify websocket.send was called
+ self.mock_websocket.send.assert_called_once()
+
+ # Parse the send message
+ sent_message = self.mock_websocket.send.call_args[0][0]
+ sent_data = json.loads(sent_message)
+
+ self.assertEqual(sent_data["type"], "input_audio_buffer.append")
+ self.assertEqual(sent_data["audio"], "base64_encoded_audio_data")
+
+ async def test_send_image_base64(self) -> None:
+ """Test sending image data with base64 source."""
+ image_data = ImageBlock(
+ type="image",
+ source=Base64Source(
+ type="base64",
+ media_type="image/png",
+ data="base64_encoded_image_data",
+ ),
+ )
+
+ await self.model.send(image_data)
+
+ # Verify websocket.send was called
+ self.mock_websocket.send.assert_called_once()
+
+ # Parse the send message
+ sent_message = self.mock_websocket.send.call_args[0][0]
+ sent_data = json.loads(sent_message)
+
+ self.assertEqual(sent_data["type"], "input_image_buffer.append")
+ self.assertEqual(sent_data["image"], "base64_encoded_image_data")
+
+ async def test_send_image_url(self) -> None:
+ """Test sending image data with URL source."""
+ image_data = ImageBlock(
+ type="image",
+ source=URLSource(
+ type="url",
+ url="https://example.com/image.jpg",
+ ),
+ )
+
+ with patch(
+ "agentscope.realtime._dashscope_realtime_model."
+ "_get_bytes_from_web_url",
+ ) as mock_get_bytes:
+ mock_get_bytes.return_value = "fetched_image_bytes"
+
+ await self.model.send(image_data)
+
+ # Verify URL was fetched
+ mock_get_bytes.assert_called_once_with(
+ "https://example.com/image.jpg",
+ )
+
+ # Verify websocket.send was called
+ self.mock_websocket.send.assert_called_once()
+
+ # Parse the sent message
+ sent_message = self.mock_websocket.send.call_args[0][0]
+ sent_data = json.loads(sent_message)
+
+ self.assertEqual(sent_data["type"], "input_image_url.append")
+ self.assertEqual(sent_data["image_url"], "fetched_image_bytes")
+
+ # async def test_send_text(self) -> None:
+ # """Test sending text data."""
+ # text_data = TextBlock(
+ # type="text",
+ # text="Hello, how are you?",
+ # )
+ #
+ # with patch("shortuuid.uuid") as mock_uuid:
+ # mock_uuid.return_value = "test_uuid_123"
+ #
+ # await self.model.send(text_data)
+ #
+ # # Verify websocket.send was called
+ # self.mock_websocket.send.assert_called_once()
+ #
+ # # Parse the sent message
+ # sent_message = self.mock_websocket.send.call_args[0][0]
+ # sent_data = json.loads(sent_message)
+ #
+ # self.assertEqual(sent_data["event_id"], "test_uuid_123")
+ # self.assertEqual(sent_data["type"], "response.create")
+ # self.assertEqual(
+ # sent_data["response"]["instructions"],
+ # "Hello, how are you?",
+ # )
diff --git a/tests/realtime_event_test.py b/tests/realtime_event_test.py
new file mode 100644
index 000000000..c115116de
--- /dev/null
+++ b/tests/realtime_event_test.py
@@ -0,0 +1,519 @@
+# -*- coding: utf-8 -*-
+"""The realtime event test unittests."""
+from unittest import IsolatedAsyncioTestCase
+
+from agentscope.realtime import ModelEvents, ServerEvents, ClientEvents
+from agentscope.message import ToolUseBlock
+
+
+class TestServerEventsFromModelEvent(IsolatedAsyncioTestCase):
+ """Test ServerEvents.from_model_event method."""
+
+ def setUp(self) -> None:
+ """Set up test fixtures."""
+ self.agent_id = "test_agent_123"
+ self.agent_name = "TestAgent"
+
+ async def test_model_response_created_event(self) -> None:
+ """Test converting ModelResponseCreatedEvent to
+ AgentResponseCreatedEvent."""
+ model_event = ModelEvents.ModelResponseCreatedEvent(
+ response_id="resp_001",
+ )
+
+ server_event = ServerEvents.from_model_event(
+ model_event,
+ agent_id=self.agent_id,
+ agent_name=self.agent_name,
+ )
+
+ self.assertIsInstance(
+ server_event,
+ ServerEvents.AgentResponseCreatedEvent,
+ )
+ self.assertEqual(server_event.response_id, "resp_001")
+ self.assertEqual(server_event.agent_id, self.agent_id)
+ self.assertEqual(server_event.agent_name, self.agent_name)
+ self.assertEqual(server_event.type, "agent_response_created")
+
+ async def test_model_response_done_event(self) -> None:
+ """Test converting ModelResponseDoneEvent to AgentResponseDoneEvent."""
+ model_event = ModelEvents.ModelResponseDoneEvent(
+ response_id="resp_002",
+ input_tokens=100,
+ output_tokens=50,
+ metadata={"key": "value"},
+ )
+
+ server_event = ServerEvents.from_model_event(
+ model_event,
+ agent_id=self.agent_id,
+ agent_name=self.agent_name,
+ )
+
+ self.assertIsInstance(
+ server_event,
+ ServerEvents.AgentResponseDoneEvent,
+ )
+ self.assertEqual(server_event.response_id, "resp_002")
+ self.assertEqual(server_event.input_tokens, 100)
+ self.assertEqual(server_event.output_tokens, 50)
+ self.assertEqual(server_event.metadata, {"key": "value"})
+ self.assertEqual(server_event.agent_id, self.agent_id)
+ self.assertEqual(server_event.agent_name, self.agent_name)
+ self.assertEqual(server_event.type, "agent_response_done")
+
+ async def test_model_response_audio_delta_event(self) -> None:
+ """Test converting ModelResponseAudioDeltaEvent to
+ AgentResponseAudioDeltaEvent."""
+ model_event = ModelEvents.ModelResponseAudioDeltaEvent(
+ response_id="resp_003",
+ item_id="item_001",
+ delta="base64_audio_data",
+ format={"type": "audio/pcm", "rate": 16000},
+ )
+
+ server_event = ServerEvents.from_model_event(
+ model_event,
+ agent_id=self.agent_id,
+ agent_name=self.agent_name,
+ )
+
+ self.assertIsInstance(
+ server_event,
+ ServerEvents.AgentResponseAudioDeltaEvent,
+ )
+ self.assertEqual(server_event.response_id, "resp_003")
+ self.assertEqual(server_event.item_id, "item_001")
+ self.assertEqual(server_event.delta, "base64_audio_data")
+ self.assertEqual(server_event.format.type, "audio/pcm")
+ self.assertEqual(server_event.format.rate, 16000)
+ self.assertEqual(server_event.agent_id, self.agent_id)
+ self.assertEqual(server_event.agent_name, self.agent_name)
+
+ async def test_model_response_audio_done_event(self) -> None:
+ """Test converting ModelResponseAudioDoneEvent to
+ AgentResponseAudioDoneEvent."""
+ model_event = ModelEvents.ModelResponseAudioDoneEvent(
+ response_id="resp_004",
+ item_id="item_002",
+ )
+
+ server_event = ServerEvents.from_model_event(
+ model_event,
+ agent_id=self.agent_id,
+ agent_name=self.agent_name,
+ )
+
+ self.assertIsInstance(
+ server_event,
+ ServerEvents.AgentResponseAudioDoneEvent,
+ )
+ self.assertEqual(server_event.response_id, "resp_004")
+ self.assertEqual(server_event.item_id, "item_002")
+ self.assertEqual(server_event.agent_id, self.agent_id)
+ self.assertEqual(server_event.agent_name, self.agent_name)
+
+ async def test_model_response_audio_transcript_delta_event(self) -> None:
+ """Test converting ModelResponseAudioTranscriptDeltaEvent."""
+ model_event = ModelEvents.ModelResponseAudioTranscriptDeltaEvent(
+ response_id="resp_005",
+ item_id="item_003",
+ delta="Hello ",
+ )
+
+ server_event = ServerEvents.from_model_event(
+ model_event,
+ agent_id=self.agent_id,
+ agent_name=self.agent_name,
+ )
+
+ self.assertIsInstance(
+ server_event,
+ ServerEvents.AgentResponseAudioTranscriptDeltaEvent,
+ )
+ self.assertEqual(server_event.response_id, "resp_005")
+ self.assertEqual(server_event.item_id, "item_003")
+ self.assertEqual(server_event.delta, "Hello ")
+ self.assertEqual(server_event.agent_id, self.agent_id)
+ self.assertEqual(server_event.agent_name, self.agent_name)
+
+ async def test_model_response_audio_transcript_done_event(self) -> None:
+ """Test converting ModelResponseAudioTranscriptDoneEvent."""
+ model_event = ModelEvents.ModelResponseAudioTranscriptDoneEvent(
+ response_id="resp_006",
+ item_id="item_004",
+ )
+
+ server_event = ServerEvents.from_model_event(
+ model_event,
+ agent_id=self.agent_id,
+ agent_name=self.agent_name,
+ )
+
+ self.assertIsInstance(
+ server_event,
+ ServerEvents.AgentResponseAudioTranscriptDoneEvent,
+ )
+ self.assertEqual(server_event.response_id, "resp_006")
+ self.assertEqual(server_event.item_id, "item_004")
+ self.assertEqual(server_event.agent_id, self.agent_id)
+ self.assertEqual(server_event.agent_name, self.agent_name)
+
+ async def test_model_response_tool_use_delta_event(self) -> None:
+ """Test converting ModelResponseToolUseDeltaEvent."""
+ tool_use = ToolUseBlock(
+ type="tool_use",
+ id="tool_001",
+ name="get_weather",
+ input={"location": "San"},
+ )
+
+ model_event = ModelEvents.ModelResponseToolUseDeltaEvent(
+ response_id="resp_007",
+ item_id="item_005",
+ tool_use=tool_use,
+ )
+
+ server_event = ServerEvents.from_model_event(
+ model_event,
+ agent_id=self.agent_id,
+ agent_name=self.agent_name,
+ )
+
+ self.assertIsInstance(
+ server_event,
+ ServerEvents.AgentResponseToolUseDeltaEvent,
+ )
+ self.assertEqual(server_event.response_id, "resp_007")
+ self.assertEqual(server_event.item_id, "item_005")
+ self.assertEqual(server_event.tool_use["id"], "tool_001")
+ self.assertEqual(server_event.tool_use["name"], "get_weather")
+ self.assertEqual(server_event.agent_id, self.agent_id)
+ self.assertEqual(server_event.agent_name, self.agent_name)
+
+ async def test_model_response_tool_use_done_event(self) -> None:
+ """Test converting ModelResponseToolUseDoneEvent."""
+ tool_use = ToolUseBlock(
+ type="tool_use",
+ id="tool_002",
+ name="get_weather",
+ input={"location": "San Francisco"},
+ )
+
+ model_event = ModelEvents.ModelResponseToolUseDoneEvent(
+ response_id="resp_008",
+ item_id="item_006",
+ tool_use=tool_use,
+ )
+
+ server_event = ServerEvents.from_model_event(
+ model_event,
+ agent_id=self.agent_id,
+ agent_name=self.agent_name,
+ )
+
+ self.assertIsInstance(
+ server_event,
+ ServerEvents.AgentResponseToolUseDoneEvent,
+ )
+ self.assertEqual(server_event.response_id, "resp_008")
+ self.assertEqual(server_event.item_id, "item_006")
+ self.assertEqual(server_event.tool_use["id"], "tool_002")
+ self.assertEqual(server_event.tool_use["name"], "get_weather")
+ self.assertEqual(server_event.agent_id, self.agent_id)
+ self.assertEqual(server_event.agent_name, self.agent_name)
+
+ async def test_model_input_transcription_delta_event(self) -> None:
+ """Test converting ModelInputTranscriptionDeltaEvent."""
+ model_event = ModelEvents.ModelInputTranscriptionDeltaEvent(
+ item_id="item_007",
+ delta="How are ",
+ )
+
+ server_event = ServerEvents.from_model_event(
+ model_event,
+ agent_id=self.agent_id,
+ agent_name=self.agent_name,
+ )
+
+ self.assertIsInstance(
+ server_event,
+ ServerEvents.AgentInputTranscriptionDeltaEvent,
+ )
+ self.assertEqual(server_event.item_id, "item_007")
+ self.assertEqual(server_event.delta, "How are ")
+ self.assertEqual(server_event.agent_id, self.agent_id)
+ self.assertEqual(server_event.agent_name, self.agent_name)
+
+ async def test_model_input_transcription_done_event(self) -> None:
+ """Test converting ModelInputTranscriptionDoneEvent."""
+ model_event = ModelEvents.ModelInputTranscriptionDoneEvent(
+ transcript="How are you?",
+ item_id="item_008",
+ input_tokens=10,
+ output_tokens=5,
+ )
+
+ server_event = ServerEvents.from_model_event(
+ model_event,
+ agent_id=self.agent_id,
+ agent_name=self.agent_name,
+ )
+
+ self.assertIsInstance(
+ server_event,
+ ServerEvents.AgentInputTranscriptionDoneEvent,
+ )
+ self.assertEqual(server_event.transcript, "How are you?")
+ self.assertEqual(server_event.item_id, "item_008")
+ self.assertEqual(server_event.input_tokens, 10)
+ self.assertEqual(server_event.output_tokens, 5)
+ self.assertEqual(server_event.agent_id, self.agent_id)
+ self.assertEqual(server_event.agent_name, self.agent_name)
+
+ async def test_model_input_started_event(self) -> None:
+ """Test converting ModelInputStartedEvent."""
+ model_event = ModelEvents.ModelInputStartedEvent(
+ item_id="item_009",
+ audio_start_ms=1000,
+ )
+
+ server_event = ServerEvents.from_model_event(
+ model_event,
+ agent_id=self.agent_id,
+ agent_name=self.agent_name,
+ )
+
+ self.assertIsInstance(
+ server_event,
+ ServerEvents.AgentInputStartedEvent,
+ )
+ self.assertEqual(server_event.item_id, "item_009")
+ self.assertEqual(server_event.audio_start_ms, 1000)
+ self.assertEqual(server_event.agent_id, self.agent_id)
+ self.assertEqual(server_event.agent_name, self.agent_name)
+
+ async def test_model_input_done_event(self) -> None:
+ """Test converting ModelInputDoneEvent."""
+ model_event = ModelEvents.ModelInputDoneEvent(
+ item_id="item_010",
+ audio_end_ms=5000,
+ )
+
+ server_event = ServerEvents.from_model_event(
+ model_event,
+ agent_id=self.agent_id,
+ agent_name=self.agent_name,
+ )
+
+ self.assertIsInstance(
+ server_event,
+ ServerEvents.AgentInputDoneEvent,
+ )
+ self.assertEqual(server_event.item_id, "item_010")
+ self.assertEqual(server_event.audio_end_ms, 5000)
+ self.assertEqual(server_event.agent_id, self.agent_id)
+ self.assertEqual(server_event.agent_name, self.agent_name)
+
+ async def test_model_error_event(self) -> None:
+ """Test converting ModelErrorEvent."""
+ model_event = ModelEvents.ModelErrorEvent(
+ error_type="rate_limit_error",
+ code="429",
+ message="Rate limit exceeded",
+ )
+
+ server_event = ServerEvents.from_model_event(
+ model_event,
+ agent_id=self.agent_id,
+ agent_name=self.agent_name,
+ )
+
+ self.assertIsInstance(
+ server_event,
+ ServerEvents.AgentErrorEvent,
+ )
+ self.assertEqual(server_event.error_type, "rate_limit_error")
+ self.assertEqual(server_event.code, "429")
+ self.assertEqual(server_event.message, "Rate limit exceeded")
+ self.assertEqual(server_event.agent_id, self.agent_id)
+ self.assertEqual(server_event.agent_name, self.agent_name)
+ self.assertEqual(server_event.type, "agent_error")
+
+
+class TestClientEventsFromJson(IsolatedAsyncioTestCase):
+ """Test ClientEvents.from_json method."""
+
+ async def test_client_session_create_event(self) -> None:
+ """Test parsing ClientSessionCreateEvent from JSON."""
+ json_data = {
+ "type": "client_session_create",
+ "config": {
+ "instructions": "You are a helpful assistant.",
+ "user_name": "TestUser",
+ },
+ }
+
+ event = ClientEvents.from_json(json_data)
+
+ self.assertIsInstance(event, ClientEvents.ClientSessionCreateEvent)
+ self.assertEqual(event.type, "client_session_create")
+ self.assertEqual(
+ event.config["instructions"],
+ "You are a helpful assistant.",
+ )
+ self.assertEqual(event.config["user_name"], "TestUser")
+
+ async def test_client_session_end_event(self) -> None:
+ """Test parsing ClientSessionEndEvent from JSON."""
+ json_data = {
+ "type": "client_session_end",
+ "session_id": "session_123",
+ }
+
+ event = ClientEvents.from_json(json_data)
+
+ self.assertIsInstance(event, ClientEvents.ClientSessionEndEvent)
+ self.assertEqual(event.type, "client_session_end")
+ self.assertEqual(event.session_id, "session_123")
+
+ async def test_client_response_create_event(self) -> None:
+ """Test parsing ClientResponseCreateEvent from JSON."""
+ json_data = {
+ "type": "client_response_create",
+ "session_id": "session_456",
+ }
+
+ event = ClientEvents.from_json(json_data)
+
+ self.assertIsInstance(event, ClientEvents.ClientResponseCreateEvent)
+ self.assertEqual(event.type, "client_response_create")
+ self.assertEqual(event.session_id, "session_456")
+
+ async def test_client_response_cancel_event(self) -> None:
+ """Test parsing ClientResponseCancelEvent from JSON."""
+ json_data = {
+ "type": "client_response_cancel",
+ "session_id": "session_789",
+ }
+
+ event = ClientEvents.from_json(json_data)
+
+ self.assertIsInstance(event, ClientEvents.ClientResponseCancelEvent)
+ self.assertEqual(event.type, "client_response_cancel")
+ self.assertEqual(event.session_id, "session_789")
+
+ async def test_client_image_append_event(self) -> None:
+ """Test parsing ClientImageAppendEvent from JSON."""
+ json_data = {
+ "type": "client_image_append",
+ "session_id": "session_001",
+ "data": "base64_image_data",
+ "format": {"mime_type": "image/png"},
+ }
+
+ event = ClientEvents.from_json(json_data)
+
+ self.assertIsInstance(event, ClientEvents.ClientImageAppendEvent)
+ self.assertEqual(event.type, "client_image_append")
+ self.assertEqual(event.session_id, "session_001")
+ self.assertEqual(event.data, "base64_image_data")
+ self.assertEqual(event.format, {"mime_type": "image/png"})
+
+ async def test_client_text_append_event(self) -> None:
+ """Test parsing ClientTextAppendEvent from JSON."""
+ json_data = {
+ "type": "client_text_append",
+ "session_id": "session_002",
+ "text": "Hello, how are you?",
+ }
+
+ event = ClientEvents.from_json(json_data)
+
+ self.assertIsInstance(event, ClientEvents.ClientTextAppendEvent)
+ self.assertEqual(event.type, "client_text_append")
+ self.assertEqual(event.session_id, "session_002")
+ self.assertEqual(event.text, "Hello, how are you?")
+
+ async def test_client_audio_append_event(self) -> None:
+ """Test parsing ClientAudioAppendEvent from JSON."""
+ json_data = {
+ "type": "client_audio_append",
+ "session_id": "session_003",
+ "audio": "base64_audio_data",
+ "format": {"type": "audio/pcm", "rate": 16000},
+ }
+
+ event = ClientEvents.from_json(json_data)
+
+ self.assertIsInstance(event, ClientEvents.ClientAudioAppendEvent)
+ self.assertEqual(event.type, "client_audio_append")
+ self.assertEqual(event.session_id, "session_003")
+ self.assertEqual(event.audio, "base64_audio_data")
+ self.assertEqual(event.format.type, "audio/pcm")
+ self.assertEqual(event.format.rate, 16000)
+
+ async def test_client_audio_commit_event(self) -> None:
+ """Test parsing ClientAudioCommitEvent from JSON."""
+ json_data = {
+ "type": "client_audio_commit",
+ "session_id": "session_004",
+ }
+
+ event = ClientEvents.from_json(json_data)
+
+ self.assertIsInstance(event, ClientEvents.ClientAudioCommitEvent)
+ self.assertEqual(event.type, "client_audio_commit")
+ self.assertEqual(event.session_id, "session_004")
+
+ async def test_client_tool_result_event(self) -> None:
+ """Test parsing ClientToolResultEvent from JSON."""
+ json_data = {
+ "type": "client_tool_result",
+ "session_id": "session_005",
+ "id": "tool_call_123",
+ "name": "get_weather",
+ "output": "The weather is sunny, 25°C",
+ }
+
+ event = ClientEvents.from_json(json_data)
+
+ self.assertIsInstance(event, ClientEvents.ClientToolResultEvent)
+ self.assertEqual(event.type, "client_tool_result")
+ self.assertEqual(event.session_id, "session_005")
+ self.assertEqual(event.id, "tool_call_123")
+ self.assertEqual(event.name, "get_weather")
+ self.assertEqual(event.output, "The weather is sunny, 25°C")
+
+ async def test_invalid_json_data_no_type(self) -> None:
+ """Test parsing invalid JSON data without type field."""
+ json_data = {
+ "session_id": "session_006",
+ }
+
+ with self.assertRaises(ValueError) as context:
+ ClientEvents.from_json(json_data)
+
+ self.assertIn("Invalid JSON data", str(context.exception))
+
+ async def test_invalid_json_data_not_dict(self) -> None:
+ """Test parsing invalid JSON data that is not a dict."""
+ json_data = "not a dict"
+
+ with self.assertRaises(ValueError) as context:
+ ClientEvents.from_json(json_data)
+
+ self.assertIn("Invalid JSON data", str(context.exception))
+
+ async def test_unknown_event_type(self) -> None:
+ """Test parsing JSON with unknown event type."""
+ json_data = {
+ "type": "unknown_event_type",
+ "session_id": "session_007",
+ }
+
+ with self.assertRaises(ValueError) as context:
+ ClientEvents.from_json(json_data)
+
+ self.assertIn("Unknown ClientEvent type", str(context.exception))