Build and run agents you can see, understand and trust.
Python agentchatbotlarge-language-modelsllmllm-agentmcpmulti-agentmulti-modalreact-agentrealtime-agentvoice-agentvoice-assistant
Support DashScope omni and GPT audio models in ReActAgent class (#759)
qbce361db06d562
diff --git a/README.md b/README.mdindex 50d8f06b1..d089cf60b 100644--- a/README.md+++ b/README.md@@ -103,6 +103,7 @@ Quick overview of important features in **AgentScope 1.0**:| ... | | |## 📢 News+- **[2025-09]** **Voice agent** is online! `ReActAgent` supports Qwen-Omni and GPT-Audio natively now, check our [new example](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/voice_agent) and [roadmap](https://github.com/agentscope-ai/agentscope/issues/773).- **[2025-09]** A new powerful 📋**Plan** module is online now! Check out the [tutorial](https://doc.agentscope.io/tutorial/task_plan.html) for more details.- **[2025-09]** **AgentScope Runtime** is open-sourced now! Enabling effective agent deployment with sandboxed tool execution for production-ready AI applications. Check out the [GitHub repo](https://github.com/agentscope-ai/agentscope-runtime).- **[2025-09]** **AgentScope Studio** is open-sourced now! Check out the [GitHub repo](https://github.com/agentscope-ai/agentscope-studio).@@ -335,6 +336,7 @@ as_studio- [Long-Term Memory](https://doc.agentscope.io/tutorial/task_long_term_memory.html)- [Agent](https://doc.agentscope.io/tutorial/task_agent.html)- [Pipeline](https://doc.agentscope.io/tutorial/task_pipeline.html)+ - [Plan](https://doc.agentscope.io/tutorial/task_plan.html)- [State/Session Management](https://doc.agentscope.io/tutorial/task_state.html)- [Agent Hooks](https://doc.agentscope.io/tutorial/task_hook.html)- [MCP](https://doc.agentscope.io/tutorial/task_mcp.html)@@ -361,6 +363,7 @@ as_studio- [Structured Output](https://github.com/agentscope-ai/agentscope/tree/main/examples/functionality/structured_output)- [Long-Term Memory](https://github.com/agentscope-ai/agentscope/tree/main/examples/functionality/long_term_memory)- [Session with SQLite](https://github.com/agentscope-ai/agentscope/tree/main/examples/functionality/session_with_sqlite)+ - [Voice Agent](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/voice_agent)## ⚖️ Licensediff --git a/README_zh.md b/README_zh.mdindex ca67dacf2..769d9b032 100644--- a/README_zh.md+++ b/README_zh.md@@ -104,6 +104,7 @@ AgentScope v1.0 新功能概览:| ... | | |## 📢 新闻+- **[2025-09]** **Voice agent** 已上线!`ReActAgent` 已原生支持 Qwen-Omni 和 GPT-Audio 系列模型,欢迎查看 [样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/voice_agent) 和 [roadmap](https://github.com/agentscope-ai/agentscope/issues/773)。- **[2025-09]** 一个全新功能强大的 📋**Plan** 模块已经上线 AgentScope!查看[文档](https://doc.agentscope.io/zh_CN/tutorial/task_plan.html)了解更多详情。- **[2025-09]** **AgentScope Runtime** 现已开源!支持沙盒化工具执行的高效智能体部署,助力打造生产级AI应用。查看 [GitHub 仓库](https://github.com/agentscope-ai/agentscope-runtime)。- **[2025-09]** **AgentScope Studio** 现已开源!查看 [GitHub 仓库](https://github.com/agentscope-ai/agentscope-studio)。@@ -337,6 +338,7 @@ as_studio- [长期记忆](https://doc.agentscope.io/zh_CN/tutorial/task_long_term_memory.html)- [智能体](https://doc.agentscope.io/zh_CN/tutorial/task_agent.html)- [管道(Pipeline)](https://doc.agentscope.io/zh_CN/tutorial/task_pipeline.html)+ - [计划](https://doc.agentscope.io/zh_CN/tutorial/task_plan.html)- [状态/会话管理](https://doc.agentscope.io/zh_CN/tutorial/task_state.html)- [智能体钩子函数](https://doc.agentscope.io/zh_CN/tutorial/task_hook.html)- [MCP](https://doc.agentscope.io/zh_CN/tutorial/task_mcp.html)@@ -363,6 +365,7 @@ as_studio- [结构化输出](https://github.com/agentscope-ai/agentscope/tree/main/examples/functionality/structured_output)- [长期记忆](https://github.com/agentscope-ai/agentscope/tree/main/examples/functionality/long_term_memory)- [基于DB的会话管理](https://github.com/agentscope-ai/agentscope/tree/main/examples/functionality/session_with_sqlite)+ - [语音智能体](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/voice_agent)## ⚖️ 许可diff --git a/examples/agent/voice_agent/README.md b/examples/agent/voice_agent/README.mdnew file mode 100644index 000000000..2b92da2be--- /dev/null+++ b/examples/agent/voice_agent/README.md@@ -0,0 +1,20 @@+# Voice Agent++> This is experimental functionality in AgentScope.++This example demonstrates how to create a voice agent using AgentScope with Qwen-Omni model, featuring both text and audio output capabilities.++> **Note**:+> - Qwen-Omni may not generate tool calls when the audio output is enabled.+> - This example supports DashScope `Qwen-Omni` and OpenAI `GPT-4o Audio` models. You can change model by modifying the `model` parameter in `main.py`.+> - We haven't tested vLLM yet. Contributions are welcome!++## Quick Start++Ensure you have installed agentscope and set ``DASHSCOPE_API_KEY`` in your environment variables.++Run the following commands to set up and run the example:++```bash+python main.py+```diff --git a/examples/agent/voice_agent/main.py b/examples/agent/voice_agent/main.pynew file mode 100644index 000000000..085bb0851--- /dev/null+++ b/examples/agent/voice_agent/main.py@@ -0,0 +1,50 @@+# -*- coding: utf-8 -*-+"""+A ReAct agent example that demonstrates audio output capability.+Note: When audio output is enabled, tool calling functionality may be disabled.+"""+import asyncio+import os+from agentscope.agent import ReActAgent, UserAgent+from agentscope.formatter import OpenAIChatFormatter++from agentscope.memory import InMemoryMemory+from agentscope.model import OpenAIChatModel+++async def main() -> None:+ """The main entry point for the ReAct audio agent example."""++ agent = ReActAgent(+ name="Friday",+ sys_prompt="You are a helpful assistant",+ model=OpenAIChatModel(+ model_name="qwen-omni-turbo",+ client_args={+ "base_url": "https://dashscope.aliyuncs.com/"+ "compatible-mode/v1",+ },+ api_key=os.getenv("DASHSCOPE_API_KEY"),+ stream=True,+ # More options can be found in the DashScope API docs:+ # https://help.aliyun.com/zh/model-studio/qwen-omni+ generate_kwargs={+ "modalities": ["text", "audio"],+ "audio": {"voice": "Cherry", "format": "wav"},+ },+ ),+ formatter=OpenAIChatFormatter(),+ memory=InMemoryMemory(),+ )++ user = UserAgent("Bob")++ msg = None+ while True:+ msg = await user(msg)+ if msg.get_text_content() == "exit":+ break+ msg = await agent(msg)+++asyncio.run(main())diff --git a/setup.py b/setup.pyindex f30e1ba71..a24492229 100644--- a/setup.py+++ b/setup.py@@ -31,6 +31,7 @@ minimal_requires = ["python-socketio","shortuuid","tiktoken",+ "sounddevice",]extra_requires = [diff --git a/src/agentscope/agent/_agent_base.py b/src/agentscope/agent/_agent_base.pyindex 291e4c4fa..dac803bd2 100644--- a/src/agentscope/agent/_agent_base.py+++ b/src/agentscope/agent/_agent_base.py@@ -5,13 +5,21 @@ import jsonfrom asyncio import Taskfrom collections import OrderedDictfrom typing import Callable, Any-+import base64import shortuuid+import numpy as npfrom ._agent_meta import _AgentMetafrom .._logging import loggerfrom ..module import StateModule-from ..message import Msg+from ..message import (+ Msg,+ AudioBlock,+ ToolUseBlock,+ ToolResultBlock,+ ImageBlock,+ VideoBlock,+)from ..types import AgentHookTypes@@ -145,7 +153,9 @@ class AgentBase(StateModule, metaclass=_AgentMeta):self._instance_pre_observe_hooks = OrderedDict()self._instance_post_observe_hooks = OrderedDict()- # The prefix used in streaming printing+ # The prefix used in streaming printing, which will save the+ # accumulated text and audio streaming data for each message id.+ # e.g. {"text": "xxx", "audio": (stream_obj, "{base64_data}")}self._stream_prefix = {}# The subscribers that will receive the reply message by their@@ -190,41 +200,182 @@ class AgentBase(StateModule, metaclass=_AgentMeta):if self._disable_console_output:return+ # The accumulated textual content to print, including the text blocks+ # and the thinking blocksthinking_and_text_to_print = []- for block in msg.get_content_blocks():- prefix = self._stream_prefix.get(msg.id, "")- if block["type"] in ["text", "thinking"]:- block_type = block["type"]- format_prefix = "" if block_type == "text" else "(thinking)"- thinking_and_text_to_print.append(- f"{msg.name}{format_prefix}: {block[block_type]}",+ for block in msg.get_content_blocks():+ if block["type"] == "audio":+ self._process_audio_block(msg.id, block)++ elif block["type"] == "text":+ self._print_text_block(+ msg.id,+ name_prefix=msg.name,+ text_content=block["text"],+ thinking_and_text_to_print=thinking_and_text_to_print,)- to_print = "\n".join(thinking_and_text_to_print)- if len(to_print) > len(prefix):- print(to_print[len(prefix) :], end="")- self._stream_prefix[msg.id] = to_print+ elif block["type"] == "thinking":+ self._print_text_block(+ msg.id,+ name_prefix=f"{msg.name}(thinking)",+ text_content=block["thinking"],+ thinking_and_text_to_print=thinking_and_text_to_print,+ )elif last:- if prefix:- if not prefix.endswith("\n"):- print(- "\n"- + json.dumps(block, indent=4, ensure_ascii=False),- )- else:- print(json.dumps(block, indent=4, ensure_ascii=False))- else:- print(- f"{msg.name}: "- f"{json.dumps(block, indent=4, ensure_ascii=False)}",- )+ self._print_last_block(block, msg)++ # Clean up resources if this is the last message in streamingif last and msg.id in self._stream_prefix:- last_prefix = self._stream_prefix.pop(msg.id)- if not last_prefix.endswith("\n"):+ if "audio" in self._stream_prefix[msg.id]:+ player, _ = self._stream_prefix[msg.id]["audio"]+ # Close the miniaudio player+ player.close()+ stream_prefix = self._stream_prefix.pop(msg.id)+ if "text" in stream_prefix and not stream_prefix["text"].endswith(+ "\n",+ ):print()+ def _process_audio_block(+ self,+ msg_id: str,+ audio_block: AudioBlock,+ ) -> None:+ """Process audio block content.++ Args:+ msg_id (`str`):+ The unique identifier of the message+ audio_block (`AudioBlock`):+ The audio content block+ """+ if "source" not in audio_block:+ raise ValueError(+ "The audio block must contain the 'source' field.",+ )++ if audio_block["source"]["type"] == "url":+ # TODO: maybe download and play the audio from the URL?+ print(json.dumps(audio_block, indent=4, ensure_ascii=False))++ elif audio_block["source"]["type"] == "base64":+ data = audio_block["source"]["data"]++ if msg_id not in self._stream_prefix:+ self._stream_prefix[msg_id] = {}++ audio_prefix = self._stream_prefix[msg_id].get("audio", None)++ import sounddevice as sd++ # The player and the prefix data is cached for streaming audio+ if audio_prefix:+ player, audio_prefix_data = audio_prefix+ else:+ player = sd.OutputStream(+ samplerate=24000,+ channels=1,+ dtype=np.float32,+ blocksize=1024,+ latency="low",+ )+ player.start()+ audio_prefix_data = ""++ # play the audio data+ new_audio_data = data[len(audio_prefix_data) :]+ if new_audio_data:+ audio_bytes = base64.b64decode(new_audio_data)+ audio_np = np.frombuffer(audio_bytes, dtype=np.int16)+ audio_float = audio_np.astype(np.float32) / 32768.0++ # Write to the audio output stream+ player.write(audio_float)++ # save the player and the prefix data+ self._stream_prefix[msg_id]["audio"] = (+ player,+ data,+ )++ else:+ raise ValueError(+ "Unsupported audio source type: "+ f"{audio_block['source']['type']}",+ )++ def _print_text_block(+ self,+ msg_id: str,+ name_prefix: str,+ text_content: str,+ thinking_and_text_to_print: list[str],+ ) -> None:+ """Print the text block and thinking block content.++ Args:+ msg_id (`str`):+ The unique identifier of the message+ name_prefix (`str`):+ The prefix for the message, e.g. "{name}: " for text block and+ "{name}(thinking): " for thinking block.+ text_content (`str`):+ The textual content to be printed.+ thinking_and_text_to_print (`list[str]`):+ A list of textual content to be printed together. Here we+ gather the text and thinking blocks to print them together.+ """+ thinking_and_text_to_print.append(+ f"{name_prefix}: {text_content}",+ )+ # The accumulated text and thinking blocks to print+ to_print = "\n".join(thinking_and_text_to_print)++ # The text prefix that has been printed+ if msg_id not in self._stream_prefix:+ self._stream_prefix[msg_id] = {}++ text_prefix = self._stream_prefix[msg_id].get("text", "")++ # Only print when there is new text content+ if len(to_print) > len(text_prefix):+ print(to_print[len(text_prefix) :], end="")++ # Save the printed text prefix+ self._stream_prefix[msg_id]["text"] = to_print++ def _print_last_block(+ self,+ block: ToolUseBlock | ToolResultBlock | ImageBlock | VideoBlock,+ msg: Msg,+ ) -> None:+ """Process and print the last content block, and the block type+ is not audio, text, or thinking.++ Args:+ block (`ToolUseBlock | ToolResultBlock | ImageBlock | VideoBlock`):+ The content block to be printed+ msg (`Msg`):+ The message object+ """+ text_prefix = self._stream_prefix.get(msg.id, {}).get("text", "")++ if text_prefix:+ # Add a newline to separate from previous text content+ print_newline = "" if text_prefix.endswith("\n") else "\n"+ print(+ f"{print_newline}"+ f"{json.dumps(block, indent=4, ensure_ascii=False)}",+ )+ else:+ print(+ f"{msg.name}:"+ f" {json.dumps(block, indent=4, ensure_ascii=False)}",+ )+async def __call__(self, *args: Any, **kwargs: Any) -> Msg:"""Call the reply function with the given arguments."""self._reply_id = shortuuid.uuid()diff --git a/src/agentscope/model/_model_response.py b/src/agentscope/model/_model_response.pyindex 67d96fb06..4726328ac 100644--- a/src/agentscope/model/_model_response.py+++ b/src/agentscope/model/_model_response.py@@ -7,7 +7,12 @@ from typing import Literal, Sequencefrom ._model_usage import ChatUsagefrom .._utils._common import _get_timestampfrom .._utils._mixin import DictMixin-from ..message import TextBlock, ToolUseBlock, ThinkingBlock+from ..message import (+ TextBlock,+ ToolUseBlock,+ ThinkingBlock,+ AudioBlock,+)from ..types import JSONSerializableObject@@ -15,7 +20,7 @@ from ..types import JSONSerializableObjectclass ChatResponse(DictMixin):"""The response of chat models."""- content: Sequence[TextBlock | ToolUseBlock | ThinkingBlock]+ content: Sequence[TextBlock | ToolUseBlock | ThinkingBlock | AudioBlock]"""The content of the chat response, which can include text blocks,tool use blocks, or thinking blocks."""diff --git a/src/agentscope/model/_openai_model.py b/src/agentscope/model/_openai_model.pyindex f767dd47f..777f2a1ba 100644--- a/src/agentscope/model/_openai_model.py+++ b/src/agentscope/model/_openai_model.py@@ -19,7 +19,13 @@ from ._model_base import ChatModelBasefrom ._model_usage import ChatUsagefrom .._logging import loggerfrom .._utils._common import _json_loads_with_repair-from ..message import ToolUseBlock, TextBlock, ThinkingBlock+from ..message import (+ ToolUseBlock,+ TextBlock,+ ThinkingBlock,+ AudioBlock,+ Base64Source,+)from ..tracing import trace_llmfrom ..types import JSONSerializableObject@@ -241,9 +247,12 @@ class OpenAIChatModel(ChatModelBase):usage, res = None, Nonetext = ""thinking = ""+ audio = ""tool_calls = OrderedDict()metadata = None- contents: List[TextBlock | ToolUseBlock | ThinkingBlock] = []+ contents: List[+ TextBlock | ToolUseBlock | ThinkingBlock | AudioBlock+ ] = []async with response as stream:async for item in stream:@@ -278,6 +287,17 @@ class OpenAIChatModel(ChatModelBase):)text += choice.delta.content or ""+ if (+ hasattr(choice.delta, "audio")+ and "data" in choice.delta.audio+ ):+ audio += choice.delta.audio["data"]+ if (+ hasattr(choice.delta, "audio")+ and "transcript" in choice.delta.audio+ ):+ text += choice.delta.audio["transcript"]+for tool_call in choice.delta.tool_calls or []:if tool_call.index in tool_calls:if tool_call.function.arguments is not None:@@ -303,6 +323,22 @@ class OpenAIChatModel(ChatModelBase):),)+ if audio:+ media_type = self.generate_kwargs.get("audio", {}).get(+ "format",+ "wav",+ )+ contents.append(+ AudioBlock(+ type="audio",+ source=Base64Source(+ data=audio,+ media_type=f"audio/{media_type}",+ type="base64",+ ),+ ),+ )+if text:contents.append(TextBlock(@@ -362,7 +398,9 @@ class OpenAIChatModel(ChatModelBase):If `structured_model` is not `None`, the expected structured outputwill be stored in the metadata of the `ChatResponse`."""- content_blocks: List[TextBlock | ToolUseBlock | ThinkingBlock] = []+ content_blocks: List[+ TextBlock | ToolUseBlock | ThinkingBlock | AudioBlock+ ] = []metadata = Noneif response.choices:@@ -385,6 +423,29 @@ class OpenAIChatModel(ChatModelBase):text=response.choices[0].message.content,),)+ if choice.message.audio:+ media_type = self.generate_kwargs.get("audio", {}).get(+ "format",+ "mp3",+ )+ content_blocks.append(+ AudioBlock(+ type="audio",+ source=Base64Source(+ data=choice.message.audio.data,+ media_type=f"audio/{media_type}",+ type="base64",+ ),+ ),+ )++ if choice.message.audio.transcript:+ content_blocks.append(+ TextBlock(+ type="text",+ text=choice.message.audio.transcript,+ ),+ )for tool_call in choice.message.tool_calls or []:content_blocks.append(diff --git a/tests/model_openai_test.py b/tests/model_openai_test.pyindex 8f7d1f5b6..668e3577d 100644--- a/tests/model_openai_test.py+++ b/tests/model_openai_test.py@@ -261,6 +261,7 @@ class TestOpenAIChatModel(IsolatedAsyncioTestCase):message.content = contentmessage.reasoning_content = Nonemessage.tool_calls = []+ message.audio = Nonemessage.parsed = Nonechoice = Mock()@@ -354,6 +355,12 @@ class TestOpenAIChatModel(IsolatedAsyncioTestCase):delta = Mock()delta.content = chunk_data.get("content")delta.reasoning_content = chunk_data.get("reasoning_content")++ audio_mock = Mock()+ audio_mock.__contains__ = lambda self, key: False+ delta.audio = audio_mock+ if "audio" in chunk_data:+ delta.audio = chunk_data["audio"]if "tool_calls" in chunk_data:tool_call_mocks = []for tc_data in chunk_data["tool_calls"]: