Skip to content

Build and run agents you can see, understand and trust.

Support DashScope omni and GPT audio models in ReActAgent class (#759)

qbce361db06d562
diff --git a/README.md b/README.md
index 50d8f06b1..d089cf60b 100644
--- a/README.md
+++ b/README.md
@@ -103,6 +103,7 @@ Quick overview of important features in **AgentScope 1.0**:
| ... | | |
## 📢 News
+- **[2025-09]** **Voice agent** is online! `ReActAgent` supports Qwen-Omni and GPT-Audio natively now, check our [new example](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/voice_agent) and [roadmap](https://github.com/agentscope-ai/agentscope/issues/773).
- **[2025-09]** A new powerful 📋**Plan** module is online now! Check out the [tutorial](https://doc.agentscope.io/tutorial/task_plan.html) for more details.
- **[2025-09]** **AgentScope Runtime** is open-sourced now! Enabling effective agent deployment with sandboxed tool execution for production-ready AI applications. Check out the [GitHub repo](https://github.com/agentscope-ai/agentscope-runtime).
- **[2025-09]** **AgentScope Studio** is open-sourced now! Check out the [GitHub repo](https://github.com/agentscope-ai/agentscope-studio).
@@ -335,6 +336,7 @@ as_studio
- [Long-Term Memory](https://doc.agentscope.io/tutorial/task_long_term_memory.html)
- [Agent](https://doc.agentscope.io/tutorial/task_agent.html)
- [Pipeline](https://doc.agentscope.io/tutorial/task_pipeline.html)
+ - [Plan](https://doc.agentscope.io/tutorial/task_plan.html)
- [State/Session Management](https://doc.agentscope.io/tutorial/task_state.html)
- [Agent Hooks](https://doc.agentscope.io/tutorial/task_hook.html)
- [MCP](https://doc.agentscope.io/tutorial/task_mcp.html)
@@ -361,6 +363,7 @@ as_studio
- [Structured Output](https://github.com/agentscope-ai/agentscope/tree/main/examples/functionality/structured_output)
- [Long-Term Memory](https://github.com/agentscope-ai/agentscope/tree/main/examples/functionality/long_term_memory)
- [Session with SQLite](https://github.com/agentscope-ai/agentscope/tree/main/examples/functionality/session_with_sqlite)
+ - [Voice Agent](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/voice_agent)
## ⚖️ License
diff --git a/README_zh.md b/README_zh.md
index ca67dacf2..769d9b032 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -104,6 +104,7 @@ AgentScope v1.0 新功能概览:
| ... | | |
## 📢 新闻
+- **[2025-09]** **Voice agent** 已上线!`ReActAgent` 已原生支持 Qwen-Omni 和 GPT-Audio 系列模型,欢迎查看 [样例](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/voice_agent) 和 [roadmap](https://github.com/agentscope-ai/agentscope/issues/773)。
- **[2025-09]** 一个全新功能强大的 📋**Plan** 模块已经上线 AgentScope!查看[文档](https://doc.agentscope.io/zh_CN/tutorial/task_plan.html)了解更多详情。
- **[2025-09]** **AgentScope Runtime** 现已开源!支持沙盒化工具执行的高效智能体部署,助力打造生产级AI应用。查看 [GitHub 仓库](https://github.com/agentscope-ai/agentscope-runtime)。
- **[2025-09]** **AgentScope Studio** 现已开源!查看 [GitHub 仓库](https://github.com/agentscope-ai/agentscope-studio)。
@@ -337,6 +338,7 @@ as_studio
- [长期记忆](https://doc.agentscope.io/zh_CN/tutorial/task_long_term_memory.html)
- [智能体](https://doc.agentscope.io/zh_CN/tutorial/task_agent.html)
- [管道(Pipeline)](https://doc.agentscope.io/zh_CN/tutorial/task_pipeline.html)
+ - [计划](https://doc.agentscope.io/zh_CN/tutorial/task_plan.html)
- [状态/会话管理](https://doc.agentscope.io/zh_CN/tutorial/task_state.html)
- [智能体钩子函数](https://doc.agentscope.io/zh_CN/tutorial/task_hook.html)
- [MCP](https://doc.agentscope.io/zh_CN/tutorial/task_mcp.html)
@@ -363,6 +365,7 @@ as_studio
- [结构化输出](https://github.com/agentscope-ai/agentscope/tree/main/examples/functionality/structured_output)
- [长期记忆](https://github.com/agentscope-ai/agentscope/tree/main/examples/functionality/long_term_memory)
- [基于DB的会话管理](https://github.com/agentscope-ai/agentscope/tree/main/examples/functionality/session_with_sqlite)
+ - [语音智能体](https://github.com/agentscope-ai/agentscope/tree/main/examples/agent/voice_agent)
## ⚖️ 许可
diff --git a/examples/agent/voice_agent/README.md b/examples/agent/voice_agent/README.md
new file mode 100644
index 000000000..2b92da2be
--- /dev/null
+++ b/examples/agent/voice_agent/README.md
@@ -0,0 +1,20 @@
+# Voice Agent
+
+> This is experimental functionality in AgentScope.
+
+This example demonstrates how to create a voice agent using AgentScope with Qwen-Omni model, featuring both text and audio output capabilities.
+
+> **Note**:
+> - Qwen-Omni may not generate tool calls when the audio output is enabled.
+> - This example supports DashScope `Qwen-Omni` and OpenAI `GPT-4o Audio` models. You can change model by modifying the `model` parameter in `main.py`.
+> - We haven't tested vLLM yet. Contributions are welcome!
+
+## Quick Start
+
+Ensure you have installed agentscope and set ``DASHSCOPE_API_KEY`` in your environment variables.
+
+Run the following commands to set up and run the example:
+
+```bash
+python main.py
+```
diff --git a/examples/agent/voice_agent/main.py b/examples/agent/voice_agent/main.py
new file mode 100644
index 000000000..085bb0851
--- /dev/null
+++ b/examples/agent/voice_agent/main.py
@@ -0,0 +1,50 @@
+# -*- coding: utf-8 -*-
+"""
+A ReAct agent example that demonstrates audio output capability.
+Note: When audio output is enabled, tool calling functionality may be disabled.
+"""
+import asyncio
+import os
+from agentscope.agent import ReActAgent, UserAgent
+from agentscope.formatter import OpenAIChatFormatter
+
+from agentscope.memory import InMemoryMemory
+from agentscope.model import OpenAIChatModel
+
+
+async def main() -> None:
+ """The main entry point for the ReAct audio agent example."""
+
+ agent = ReActAgent(
+ name="Friday",
+ sys_prompt="You are a helpful assistant",
+ model=OpenAIChatModel(
+ model_name="qwen-omni-turbo",
+ client_args={
+ "base_url": "https://dashscope.aliyuncs.com/"
+ "compatible-mode/v1",
+ },
+ api_key=os.getenv("DASHSCOPE_API_KEY"),
+ stream=True,
+ # More options can be found in the DashScope API docs:
+ # https://help.aliyun.com/zh/model-studio/qwen-omni
+ generate_kwargs={
+ "modalities": ["text", "audio"],
+ "audio": {"voice": "Cherry", "format": "wav"},
+ },
+ ),
+ formatter=OpenAIChatFormatter(),
+ memory=InMemoryMemory(),
+ )
+
+ user = UserAgent("Bob")
+
+ msg = None
+ while True:
+ msg = await user(msg)
+ if msg.get_text_content() == "exit":
+ break
+ msg = await agent(msg)
+
+
+asyncio.run(main())
diff --git a/setup.py b/setup.py
index f30e1ba71..a24492229 100644
--- a/setup.py
+++ b/setup.py
@@ -31,6 +31,7 @@ minimal_requires = [
"python-socketio",
"shortuuid",
"tiktoken",
+ "sounddevice",
]
extra_requires = [
diff --git a/src/agentscope/agent/_agent_base.py b/src/agentscope/agent/_agent_base.py
index 291e4c4fa..dac803bd2 100644
--- a/src/agentscope/agent/_agent_base.py
+++ b/src/agentscope/agent/_agent_base.py
@@ -5,13 +5,21 @@ import json
from asyncio import Task
from collections import OrderedDict
from typing import Callable, Any
-
+import base64
import shortuuid
+import numpy as np
from ._agent_meta import _AgentMeta
from .._logging import logger
from ..module import StateModule
-from ..message import Msg
+from ..message import (
+ Msg,
+ AudioBlock,
+ ToolUseBlock,
+ ToolResultBlock,
+ ImageBlock,
+ VideoBlock,
+)
from ..types import AgentHookTypes
@@ -145,7 +153,9 @@ class AgentBase(StateModule, metaclass=_AgentMeta):
self._instance_pre_observe_hooks = OrderedDict()
self._instance_post_observe_hooks = OrderedDict()
- # The prefix used in streaming printing
+ # The prefix used in streaming printing, which will save the
+ # accumulated text and audio streaming data for each message id.
+ # e.g. {"text": "xxx", "audio": (stream_obj, "{base64_data}")}
self._stream_prefix = {}
# The subscribers that will receive the reply message by their
@@ -190,41 +200,182 @@ class AgentBase(StateModule, metaclass=_AgentMeta):
if self._disable_console_output:
return
+ # The accumulated textual content to print, including the text blocks
+ # and the thinking blocks
thinking_and_text_to_print = []
- for block in msg.get_content_blocks():
- prefix = self._stream_prefix.get(msg.id, "")
- if block["type"] in ["text", "thinking"]:
- block_type = block["type"]
- format_prefix = "" if block_type == "text" else "(thinking)"
- thinking_and_text_to_print.append(
- f"{msg.name}{format_prefix}: {block[block_type]}",
+ for block in msg.get_content_blocks():
+ if block["type"] == "audio":
+ self._process_audio_block(msg.id, block)
+
+ elif block["type"] == "text":
+ self._print_text_block(
+ msg.id,
+ name_prefix=msg.name,
+ text_content=block["text"],
+ thinking_and_text_to_print=thinking_and_text_to_print,
)
- to_print = "\n".join(thinking_and_text_to_print)
- if len(to_print) > len(prefix):
- print(to_print[len(prefix) :], end="")
- self._stream_prefix[msg.id] = to_print
+ elif block["type"] == "thinking":
+ self._print_text_block(
+ msg.id,
+ name_prefix=f"{msg.name}(thinking)",
+ text_content=block["thinking"],
+ thinking_and_text_to_print=thinking_and_text_to_print,
+ )
elif last:
- if prefix:
- if not prefix.endswith("\n"):
- print(
- "\n"
- + json.dumps(block, indent=4, ensure_ascii=False),
- )
- else:
- print(json.dumps(block, indent=4, ensure_ascii=False))
- else:
- print(
- f"{msg.name}: "
- f"{json.dumps(block, indent=4, ensure_ascii=False)}",
- )
+ self._print_last_block(block, msg)
+
+ # Clean up resources if this is the last message in streaming
if last and msg.id in self._stream_prefix:
- last_prefix = self._stream_prefix.pop(msg.id)
- if not last_prefix.endswith("\n"):
+ if "audio" in self._stream_prefix[msg.id]:
+ player, _ = self._stream_prefix[msg.id]["audio"]
+ # Close the miniaudio player
+ player.close()
+ stream_prefix = self._stream_prefix.pop(msg.id)
+ if "text" in stream_prefix and not stream_prefix["text"].endswith(
+ "\n",
+ ):
print()
+ def _process_audio_block(
+ self,
+ msg_id: str,
+ audio_block: AudioBlock,
+ ) -> None:
+ """Process audio block content.
+
+ Args:
+ msg_id (`str`):
+ The unique identifier of the message
+ audio_block (`AudioBlock`):
+ The audio content block
+ """
+ if "source" not in audio_block:
+ raise ValueError(
+ "The audio block must contain the 'source' field.",
+ )
+
+ if audio_block["source"]["type"] == "url":
+ # TODO: maybe download and play the audio from the URL?
+ print(json.dumps(audio_block, indent=4, ensure_ascii=False))
+
+ elif audio_block["source"]["type"] == "base64":
+ data = audio_block["source"]["data"]
+
+ if msg_id not in self._stream_prefix:
+ self._stream_prefix[msg_id] = {}
+
+ audio_prefix = self._stream_prefix[msg_id].get("audio", None)
+
+ import sounddevice as sd
+
+ # The player and the prefix data is cached for streaming audio
+ if audio_prefix:
+ player, audio_prefix_data = audio_prefix
+ else:
+ player = sd.OutputStream(
+ samplerate=24000,
+ channels=1,
+ dtype=np.float32,
+ blocksize=1024,
+ latency="low",
+ )
+ player.start()
+ audio_prefix_data = ""
+
+ # play the audio data
+ new_audio_data = data[len(audio_prefix_data) :]
+ if new_audio_data:
+ audio_bytes = base64.b64decode(new_audio_data)
+ audio_np = np.frombuffer(audio_bytes, dtype=np.int16)
+ audio_float = audio_np.astype(np.float32) / 32768.0
+
+ # Write to the audio output stream
+ player.write(audio_float)
+
+ # save the player and the prefix data
+ self._stream_prefix[msg_id]["audio"] = (
+ player,
+ data,
+ )
+
+ else:
+ raise ValueError(
+ "Unsupported audio source type: "
+ f"{audio_block['source']['type']}",
+ )
+
+ def _print_text_block(
+ self,
+ msg_id: str,
+ name_prefix: str,
+ text_content: str,
+ thinking_and_text_to_print: list[str],
+ ) -> None:
+ """Print the text block and thinking block content.
+
+ Args:
+ msg_id (`str`):
+ The unique identifier of the message
+ name_prefix (`str`):
+ The prefix for the message, e.g. "{name}: " for text block and
+ "{name}(thinking): " for thinking block.
+ text_content (`str`):
+ The textual content to be printed.
+ thinking_and_text_to_print (`list[str]`):
+ A list of textual content to be printed together. Here we
+ gather the text and thinking blocks to print them together.
+ """
+ thinking_and_text_to_print.append(
+ f"{name_prefix}: {text_content}",
+ )
+ # The accumulated text and thinking blocks to print
+ to_print = "\n".join(thinking_and_text_to_print)
+
+ # The text prefix that has been printed
+ if msg_id not in self._stream_prefix:
+ self._stream_prefix[msg_id] = {}
+
+ text_prefix = self._stream_prefix[msg_id].get("text", "")
+
+ # Only print when there is new text content
+ if len(to_print) > len(text_prefix):
+ print(to_print[len(text_prefix) :], end="")
+
+ # Save the printed text prefix
+ self._stream_prefix[msg_id]["text"] = to_print
+
+ def _print_last_block(
+ self,
+ block: ToolUseBlock | ToolResultBlock | ImageBlock | VideoBlock,
+ msg: Msg,
+ ) -> None:
+ """Process and print the last content block, and the block type
+ is not audio, text, or thinking.
+
+ Args:
+ block (`ToolUseBlock | ToolResultBlock | ImageBlock | VideoBlock`):
+ The content block to be printed
+ msg (`Msg`):
+ The message object
+ """
+ text_prefix = self._stream_prefix.get(msg.id, {}).get("text", "")
+
+ if text_prefix:
+ # Add a newline to separate from previous text content
+ print_newline = "" if text_prefix.endswith("\n") else "\n"
+ print(
+ f"{print_newline}"
+ f"{json.dumps(block, indent=4, ensure_ascii=False)}",
+ )
+ else:
+ print(
+ f"{msg.name}:"
+ f" {json.dumps(block, indent=4, ensure_ascii=False)}",
+ )
+
async def __call__(self, *args: Any, **kwargs: Any) -> Msg:
"""Call the reply function with the given arguments."""
self._reply_id = shortuuid.uuid()
diff --git a/src/agentscope/model/_model_response.py b/src/agentscope/model/_model_response.py
index 67d96fb06..4726328ac 100644
--- a/src/agentscope/model/_model_response.py
+++ b/src/agentscope/model/_model_response.py
@@ -7,7 +7,12 @@ from typing import Literal, Sequence
from ._model_usage import ChatUsage
from .._utils._common import _get_timestamp
from .._utils._mixin import DictMixin
-from ..message import TextBlock, ToolUseBlock, ThinkingBlock
+from ..message import (
+ TextBlock,
+ ToolUseBlock,
+ ThinkingBlock,
+ AudioBlock,
+)
from ..types import JSONSerializableObject
@@ -15,7 +20,7 @@ from ..types import JSONSerializableObject
class ChatResponse(DictMixin):
"""The response of chat models."""
- content: Sequence[TextBlock | ToolUseBlock | ThinkingBlock]
+ content: Sequence[TextBlock | ToolUseBlock | ThinkingBlock | AudioBlock]
"""The content of the chat response, which can include text blocks,
tool use blocks, or thinking blocks."""
diff --git a/src/agentscope/model/_openai_model.py b/src/agentscope/model/_openai_model.py
index f767dd47f..777f2a1ba 100644
--- a/src/agentscope/model/_openai_model.py
+++ b/src/agentscope/model/_openai_model.py
@@ -19,7 +19,13 @@ from ._model_base import ChatModelBase
from ._model_usage import ChatUsage
from .._logging import logger
from .._utils._common import _json_loads_with_repair
-from ..message import ToolUseBlock, TextBlock, ThinkingBlock
+from ..message import (
+ ToolUseBlock,
+ TextBlock,
+ ThinkingBlock,
+ AudioBlock,
+ Base64Source,
+)
from ..tracing import trace_llm
from ..types import JSONSerializableObject
@@ -241,9 +247,12 @@ class OpenAIChatModel(ChatModelBase):
usage, res = None, None
text = ""
thinking = ""
+ audio = ""
tool_calls = OrderedDict()
metadata = None
- contents: List[TextBlock | ToolUseBlock | ThinkingBlock] = []
+ contents: List[
+ TextBlock | ToolUseBlock | ThinkingBlock | AudioBlock
+ ] = []
async with response as stream:
async for item in stream:
@@ -278,6 +287,17 @@ class OpenAIChatModel(ChatModelBase):
)
text += choice.delta.content or ""
+ if (
+ hasattr(choice.delta, "audio")
+ and "data" in choice.delta.audio
+ ):
+ audio += choice.delta.audio["data"]
+ if (
+ hasattr(choice.delta, "audio")
+ and "transcript" in choice.delta.audio
+ ):
+ text += choice.delta.audio["transcript"]
+
for tool_call in choice.delta.tool_calls or []:
if tool_call.index in tool_calls:
if tool_call.function.arguments is not None:
@@ -303,6 +323,22 @@ class OpenAIChatModel(ChatModelBase):
),
)
+ if audio:
+ media_type = self.generate_kwargs.get("audio", {}).get(
+ "format",
+ "wav",
+ )
+ contents.append(
+ AudioBlock(
+ type="audio",
+ source=Base64Source(
+ data=audio,
+ media_type=f"audio/{media_type}",
+ type="base64",
+ ),
+ ),
+ )
+
if text:
contents.append(
TextBlock(
@@ -362,7 +398,9 @@ class OpenAIChatModel(ChatModelBase):
If `structured_model` is not `None`, the expected structured output
will be stored in the metadata of the `ChatResponse`.
"""
- content_blocks: List[TextBlock | ToolUseBlock | ThinkingBlock] = []
+ content_blocks: List[
+ TextBlock | ToolUseBlock | ThinkingBlock | AudioBlock
+ ] = []
metadata = None
if response.choices:
@@ -385,6 +423,29 @@ class OpenAIChatModel(ChatModelBase):
text=response.choices[0].message.content,
),
)
+ if choice.message.audio:
+ media_type = self.generate_kwargs.get("audio", {}).get(
+ "format",
+ "mp3",
+ )
+ content_blocks.append(
+ AudioBlock(
+ type="audio",
+ source=Base64Source(
+ data=choice.message.audio.data,
+ media_type=f"audio/{media_type}",
+ type="base64",
+ ),
+ ),
+ )
+
+ if choice.message.audio.transcript:
+ content_blocks.append(
+ TextBlock(
+ type="text",
+ text=choice.message.audio.transcript,
+ ),
+ )
for tool_call in choice.message.tool_calls or []:
content_blocks.append(
diff --git a/tests/model_openai_test.py b/tests/model_openai_test.py
index 8f7d1f5b6..668e3577d 100644
--- a/tests/model_openai_test.py
+++ b/tests/model_openai_test.py
@@ -261,6 +261,7 @@ class TestOpenAIChatModel(IsolatedAsyncioTestCase):
message.content = content
message.reasoning_content = None
message.tool_calls = []
+ message.audio = None
message.parsed = None
choice = Mock()
@@ -354,6 +355,12 @@ class TestOpenAIChatModel(IsolatedAsyncioTestCase):
delta = Mock()
delta.content = chunk_data.get("content")
delta.reasoning_content = chunk_data.get("reasoning_content")
+
+ audio_mock = Mock()
+ audio_mock.__contains__ = lambda self, key: False
+ delta.audio = audio_mock
+ if "audio" in chunk_data:
+ delta.audio = chunk_data["audio"]
if "tool_calls" in chunk_data:
tool_call_mocks = []
for tc_data in chunk_data["tool_calls"]: