Skip to content

armgit / camel

Public Imported from GitHub
0

🐫 CAMEL: The first and the best multi-agent framework. Finding the Scaling Law of Agents. https://www.camel-ai.org

enhance: screenshot and terminal toolkit, polish workforce prompt (#2903)

Wendong-Fan5db79131d3ce
diff --git a/camel/societies/workforce/prompts.py b/camel/societies/workforce/prompts.py
index 4fd46dda4..5fa82f30c 100644
--- a/camel/societies/workforce/prompts.py
+++ b/camel/societies/workforce/prompts.py
@@ -206,8 +206,9 @@ TASK_DECOMPOSE_PROMPT = r"""You need to decompose the given task into subtasks a
* **DO NOT** use vague verbs like "analyze," "look into," or "research" without defining the output.
* **DO** specify the format and content of the output. For example, instead of "Analyze the attached report," write "Summarize the key findings of the attached report in a 3-bullet-point list." Instead of "Find contacts," write "Extract all names and email addresses from the document and return them as a JSON list of objects, where each object has a 'name' and 'email' key."
-3. **Strategic Grouping for Sequential Work**:
- * If a series of steps must be done in order *and* can be handled by the same worker type, group them into a single subtask to maintain flow and minimize handoffs.
+3. **Full Workflow Completion & Strategic Grouping**:
+ * **Preserve the Entire Goal**: Ensure the decomposed subtasks collectively achieve the *entire* original task. Do not drop or ignore final steps like sending a message, submitting a form, or creating a file.
+ * **Group Sequential Actions**: If a series of steps must be done in order *and* can be handled by the same worker type (e.g., read, think, reply), group them into a single, comprehensive subtask. This maintains workflow and ensures the final goal is met.
4. **Aggressive Parallelization**:
* **Across Different Worker Specializations**: If distinct phases of the overall task require different types of workers (e.g., research by a 'SearchAgent', then content creation by a 'DocumentAgent'), define these as separate subtasks.
@@ -316,6 +317,8 @@ Additional Info: {additional_info}
2. **REPLAN**: Modify the task content to address the underlying issue
- Use for: Unclear requirements, insufficient context, correctable errors
- Provide: Modified task content that addresses the failure cause
+ - **CRITICAL**: The replanned task MUST be a clear, actionable
+ instruction for an AI agent, not a question or request for a human.
3. **DECOMPOSE**: Break the task into smaller, more manageable subtasks
- Use for: Complex tasks, capability mismatches, persistent failures
@@ -328,10 +331,13 @@ Additional Info: {additional_info}
- **Connection/Network Errors**: Almost always choose RETRY
- **Model Processing Errors**: Consider REPLAN if the task can be clarified, otherwise DECOMPOSE
-- **Capability Gaps**: Choose DECOMPOSE to break into simpler parts
+- **Capability Gaps**: Choose DECOMPOSE to break into simpler parts. If a
+ replan can work, ensure the new task is a command for an agent, not a
+ request to a user.
- **Ambiguous Requirements**: Choose REPLAN with clearer instructions
- **High Failure Count**: Lean towards DECOMPOSE rather than repeated retries
-- **Deep Tasks (depth > 2)**: Prefer RETRY or REPLAN over further decomposition
+- **Deep Tasks (depth > 2)**: Prefer RETRY or REPLAN over further
+ decomposition
**RESPONSE FORMAT:**
You must return a valid JSON object with these fields:
diff --git a/camel/societies/workforce/single_agent_worker.py b/camel/societies/workforce/single_agent_worker.py
index ebddac934..c169a65af 100644
--- a/camel/societies/workforce/single_agent_worker.py
+++ b/camel/societies/workforce/single_agent_worker.py
@@ -411,10 +411,12 @@ class SingleAgentWorker(Worker):
f"(from pool/clone of "
f"{getattr(self.worker, 'agent_id', self.worker.role_name)}) "
f"to process task: {task.content}",
- "response_content": response_content,
- "tool_calls": final_response.info.get("tool_calls")
- if isinstance(response, AsyncStreamingChatAgentResponse)
- else response.info.get("tool_calls"),
+ "response_content": response_content[:50],
+ "tool_calls": str(
+ final_response.info.get("tool_calls")
+ if isinstance(response, AsyncStreamingChatAgentResponse)
+ else response.info.get("tool_calls")
+ )[:50],
"total_tokens": total_tokens,
}
diff --git a/camel/toolkits/screenshot_toolkit.py b/camel/toolkits/screenshot_toolkit.py
index fdca9c565..e014cc1b3 100644
--- a/camel/toolkits/screenshot_toolkit.py
+++ b/camel/toolkits/screenshot_toolkit.py
@@ -61,7 +61,7 @@ class ScreenshotToolkit(BaseToolkit, RegisteredAgentToolkit):
path = Path("camel_working_dir")
self.ImageGrab = ImageGrab
- self.screenshots_dir = path / "screenshots"
+ self.screenshots_dir = path
self.screenshots_dir.mkdir(parents=True, exist_ok=True)
def read_image(
@@ -98,6 +98,8 @@ class ScreenshotToolkit(BaseToolkit, RegisteredAgentToolkit):
)
try:
+ image_path = str(Path(image_path).absolute())
+
# Check if file exists before trying to open
if not os.path.exists(image_path):
error_msg = f"Screenshot file not found: {image_path}"
@@ -188,7 +190,9 @@ class ScreenshotToolkit(BaseToolkit, RegisteredAgentToolkit):
# Record in agent memory if requested
if read_image and file_path is not None:
inst = instruction if instruction is not None else ""
- response = self.read_image(file_path, inst)
+ response = self.read_image(
+ str(Path(file_path).absolute()), inst
+ )
result_text += f". Agent response: {response}"
return result_text
diff --git a/camel/toolkits/terminal_toolkit.py b/camel/toolkits/terminal_toolkit.py
index 6f1a48559..c7dc02796 100644
--- a/camel/toolkits/terminal_toolkit.py
+++ b/camel/toolkits/terminal_toolkit.py
@@ -319,7 +319,14 @@ class TerminalToolkit(BaseToolkit):
)
# Install essential packages using uv
- essential_packages = ["pip", "setuptools", "wheel", "pyautogui"]
+ essential_packages = [
+ "pip",
+ "setuptools",
+ "wheel",
+ "pyautogui",
+ "plotly",
+ "ffmpeg",
+ ]
subprocess.run(
[
"uv",
@@ -371,7 +378,14 @@ class TerminalToolkit(BaseToolkit):
pip_path = os.path.join(self.initial_env_path, "bin", "pip")
# Upgrade pip and install essential packages
- essential_packages = ["pip", "setuptools", "wheel", "pyautogui"]
+ essential_packages = [
+ "pip",
+ "setuptools",
+ "wheel",
+ "pyautogui",
+ "plotly",
+ "ffmpeg",
+ ]
subprocess.run(
[pip_path, "install", "--upgrade", *essential_packages],
check=True,
diff --git a/examples/workforce/eigent.py b/examples/workforce/eigent.py
index 07648ef57..0282de463 100644
--- a/examples/workforce/eigent.py
+++ b/examples/workforce/eigent.py
@@ -114,13 +114,14 @@ def developer_agent_factory(
task_id: str,
):
r"""Factory for creating a developer agent."""
+ screenshot_toolkit = ScreenshotToolkit(working_directory=WORKING_DIRECTORY)
tools = [
send_message_to_user,
HumanToolkit().ask_human_via_console,
*TerminalToolkit(safe_mode=True, clone_current_env=False).get_tools(),
*NoteTakingToolkit(working_directory=WORKING_DIRECTORY).get_tools(),
*WebDeployToolkit().get_tools(),
- *ScreenshotToolkit(working_directory=WORKING_DIRECTORY).get_tools(),
+ *screenshot_toolkit.get_tools(),
]
system_message = f"""
@@ -161,68 +162,85 @@ plain text formatting instead.
<capabilities>
Your capabilities are extensive and powerful:
-- **Unrestricted Code Execution**: You can write and execute code to solve
-any task. You MUST first save your code to a file (e.g., `script.py`)
-and then run it using the terminal (e.g., `python script.py`).
-- **Full Terminal Control**: You have complete control over the terminal. You
-can install any packages (e.g., with `pip3`, `uv`, or `apt-get`), run any
-command-line tools, and manage files and processes. If a tool is missing,
-you are empowered to install it. There are no limitations on what you can do
-via the terminal.
-- **Screen Observation**: You can take screenshots of the user's screen to
-understand the state of GUIs or applications, enabling you to perform tasks
-that require visual context.
-- **Solution Verification**: You can verify your solutions through immediate
-execution and testing in the terminal.
-- **Library Utilization**: You can utilize any Python library (e.g.,
-requests, BeautifulSoup, pandas) needed for efficient solutions. If a
-package is missing, install it.
-- **Web Deployment**: You can deploy web applications and content. You can
-serve files, deploy static sites, and manage your deployments.
-- **Human Collaboration**: You can ask for human input via the console if you
-are stuck or need clarification.
+- **Unrestricted Code Execution**: You can write and execute code in any
+ language to solve a task. You MUST first save your code to a file (e.g.,
+ `script.py`) and then run it from the terminal (e.g.,
+ `python script.py`).
+- **Full Terminal Control**: You have root-level access to the terminal. You
+ can run any command-line tool, manage files, and interact with the OS. If
+ a tool is missing, you MUST install it with the appropriate package manager
+ (e.g., `pip3`, `uv`, or `apt-get`). Your capabilities include:
+ - **Text & Data Processing**: `awk`, `sed`, `grep`, `jq`.
+ - **File System & Execution**: `find`, `xargs`, `tar`, `zip`, `unzip`,
+ `chmod`.
+ - **Networking & Web**: `curl`, `wget` for web requests; `ssh` for
+ remote access.
+- **Screen Observation**: You can take screenshots to analyze GUIs and visual
+ context, enabling you to perform tasks that require sight.
+- **Desktop Automation**: You can control desktop applications
+ programmatically.
+ - **On macOS**, you MUST prioritize using **AppleScript** for its robust
+ control over native applications. Execute simple commands with
+ `osascript -e '...'` or run complex scripts from a `.scpt` file.
+ - **On other systems**, use **pyautogui** for cross-platform GUI
+ automation.
+ - **IMPORTANT**: Always complete the full automation workflow—do not just
+ prepare or suggest actions. Execute them to completion.
+- **Solution Verification**: You can immediately test and verify your
+ solutions by executing them in the terminal.
+- **Web Deployment**: You can deploy web applications and content, serve
+ files, and manage deployments.
+- **Human Collaboration**: If you are stuck or need clarification, you can
+ ask for human input via the console.
- **Note Management**: You can write and read notes to coordinate with other
-agents and keep track of your work.
+ agents and track your work.
</capabilities>
+<philosophy>
+- **Bias for Action**: Your purpose is to take action. Don't just suggest
+solutions—implement them. Write code, run commands, and build things.
+- **Complete the Full Task**: When automating GUI applications, always finish
+what you start. If the task involves sending something, send it. If it
+involves submitting data, submit it. Never stop at just preparing or
+drafting—execute the complete workflow to achieve the desired outcome.
+- **Embrace Challenges**: Never say "I can't." If you
+encounter a limitation, find a way to overcome it.
+- **Resourcefulness**: If a tool is missing, install it. If information is
+lacking, find it. You have the full power of a terminal to acquire any
+resource you need.
+- **Think Like an Engineer**: Approach problems methodically. Analyze
+requirements, execute it, and verify the results. Your
+strength lies in your ability to engineer solutions.
+</philosophy>
+
<terminal_tips>
-The terminal tools are session-based. You must manage one or more terminal
-sessions to perform your tasks. A session is identified by a unique `id`.
-
-- The terminal is your most powerful tool. Almost any problem can be solved by
-using it creatively. Be bold and resourceful.
-- Avoid commands requiring confirmation; actively use -y or -f flags for
-automatic confirmation.
-- For commands with excessive output, redirect stdout to a file.
-- Chain multiple commands with `&&` to minimize interruptions.
-- Use the pipe operator `|` to pass command outputs and simplify operations.
-- Use the `ls -F` command to check the execution permissions of files.
-- If you need to install any new packages, use `pip3 install` or `apt-get
-install` after checking the OS type.
-
-- Execute Commands: Use `shell_exec(id="...", command="...")` to run
- a command. If the `id` is new, a new session is created.
- Example: `shell_exec(id="session_1", command="ls -l")`
-
-- Manage Long-Running Tasks: For commands that take time, run them
- in one step, and then use `shell_wait(id="...")` to wait for
- completion. This prevents blocking and allows you to perform other
- tasks in parallel.
-
-- View Output: Use `shell_view(id="...")` to see the full command
- history and output of a session.
-
-- Run Tasks in Parallel: Use different session IDs to run multiple
- commands concurrently.
- - `shell_exec(id="install", command="pip3 install numpy")`
- - `shell_exec(id="test", command="python my_script.py")`
-
-- Interact with Processes: For commands that require input, you can
-use:
- - `shell_exec(id="...", command="...", interactive=True)` for
- real-time interactive sessions.
- - `shell_write_to_process(id="...", content="...")` to send input to a
- non-interactive running process.
+The terminal tools are session-based, identified by a unique `id`. Master
+these tips to maximize your effectiveness:
+
+- **GUI Automation Strategy**:
+ - **AppleScript (macOS Priority)**: For robust control of macOS apps, use
+ `osascript`.
+ - Example (open Slack):
+ `osascript -e 'tell application "Slack" to activate'`
+ - Example (run script file): `osascript my_script.scpt`
+ - **pyautogui (Cross-Platform)**: For other OSes or simple automation.
+ - Key functions: `pyautogui.click(x, y)`, `pyautogui.typewrite("text")`,
+ `pyautogui.hotkey('ctrl', 'c')`, `pyautogui.press('enter')`.
+ - Safety: Always use `time.sleep()` between actions to ensure stability
+ and add `pyautogui.FAILSAFE = True` to your scripts.
+ - Workflow: Your scripts MUST complete the entire task, from start to
+ final submission.
+
+- **Command-Line Best Practices**:
+ - **Be Creative**: The terminal is your most powerful tool. Use it boldly.
+ - **Automate Confirmation**: Use `-y` or `-f` flags to avoid interactive
+ prompts.
+ - **Manage Output**: Redirect long outputs to a file (e.g., `> output.txt`).
+ - **Chain Commands**: Use `&&` to link commands for sequential execution.
+ - **Piping**: Use `|` to pass output from one command to another.
+ - **Permissions**: Use `ls -F` to check file permissions.
+ - **Installation**: Use `pip3 install` or `apt-get install` for new
+ packages.
- Stop a Process: If a process needs to be terminated, use
`shell_kill_process(id="...")`.
@@ -243,6 +261,7 @@ use:
),
model=model,
tools=tools,
+ toolkits_to_register_agent=[screenshot_toolkit],
)
@@ -361,7 +380,9 @@ The current date is {datetime.date.today()}.
Your capabilities include:
- Search and get information from the web using the search tools.
- Use the rich browser related toolset to investigate websites.
-- Use the terminal tools to perform local operations.
+- Use the terminal tools to perform local operations. You can leverage
+ powerful CLI tools like `grep` for searching within files, `curl` and
+ `wget` for downloading content, and `jq` for parsing JSON data from APIs.
- Use the note-taking tools to record your findings.
- Use the human toolkit to ask for help when you are stuck.
</capabilities>
@@ -519,7 +540,11 @@ Your capabilities include:
- You have access to a full suite of terminal tools to interact with
the file system within your working directory (`{WORKING_DIRECTORY}`).
- You can execute shell commands (`shell_exec`), list files, and manage
- your workspace as needed to support your document creation tasks.
+ your workspace as needed to support your document creation tasks. To
+ process and manipulate text and data for your documents, you can use
+ powerful CLI tools like `awk`, `sed`, `grep`, and `jq`. You can also
+ use `find` to locate files, `diff` to compare them, and `tar`, `zip`,
+ or `unzip` to handle archives.
- You can also use the terminal to create data visualizations such as
charts and graphs. For example, you can write a Python script that uses
libraries like `plotly` or `matplotlib` to create a chart and save it
@@ -634,6 +659,13 @@ Your capabilities include:
- Generate images in 1024x1792 resolution
- Save generated images to specified directories
+- Terminal and File System:
+ - You have access to terminal tools to manage media files. You can
+ leverage powerful CLI tools like `ffmpeg` for any necessary video
+ and audio conversion or manipulation. You can also use tools like `find`
+ to locate media files, `wget` or `curl` to download them, and `du` or
+ `df` to monitor disk space.
+
- Human Interaction:
- Ask questions to users and receive their responses
- Send informative messages to users without requiring responses
@@ -725,7 +757,9 @@ Your integrated toolkits enable you to:
9. File System Access:
- You can use terminal tools to interact with the local file system in
your working directory (`{WORKING_DIRECTORY}`), for example, to access
- files needed for posting.
+ files needed for posting. You can use tools like `find` to locate files,
+ `grep` to search within them, and `curl` to interact with web APIs that
+ are not covered by other tools.
When assisting users, always:
- Identify which platform's functionality is needed for the task.
@@ -767,7 +801,7 @@ async def main():
# Create a single model backend for all agents
model_backend = ModelFactory.create(
model_platform=ModelPlatformType.OPENAI,
- model_type=ModelType.GPT_4_1,
+ model_type=ModelType.GPT_4_1_MINI,
model_config_dict={
"stream": False,
},
@@ -923,8 +957,8 @@ The current date is {datetime.date.today()}.
worker=search_agent,
).add_single_agent_worker(
"Developer Agent: A skilled coding assistant that can write and "
- "execute code, run terminal commands, and verify solutions to "
- "complete tasks.",
+ "execute code, run terminal commands, control the desktop using "
+ "pyautogui, and verify solutions to complete tasks.",
worker=developer_agent,
).add_single_agent_worker(
"Document Agent: A document processing assistant for creating, "
@@ -941,11 +975,14 @@ The current date is {datetime.date.today()}.
human_task = Task(
content=(
"""
-help me use terminal to open slack, find Regina Bai and read the latest
-message she sent to me, think then help me reply to her message
-you can use pyaugui to control the mouse and keyboard to send the message.
-for each action, you should do a screenshot before your action to check the
-current state of the screen
+Analyze the UK healthcare industry to support the planning of my next company.
+Provide a comprehensive market overview, including current trends, growth
+projections, and relevant regulations. Identify the top 5-10 major competitors
+in the space, including their names, website URLs, estimated market size or
+share, core services or products, key strengths, and notable weaknesses. Also
+highlight any significant opportunities, gaps, or underserved segments within
+the market. Present all findings in a well-structured, professional HTML
+report.
"""
),
id='0',