Skip to content

armgit / camel

Public Imported from GitHub
0

🐫 CAMEL: The first and the best multi-agent framework. Finding the Scaling Law of Agents. https://www.camel-ai.org

docs: improve cookbook for Tic Tac Toe (#2146)

halleritec0cbb8400481
diff --git a/docs/cookbooks/loong/multi_step_rl.ipynb b/docs/cookbooks/loong/multi_step_rl.ipynb
index 58cd47c5d..35a2ddd46 100644
--- a/docs/cookbooks/loong/multi_step_rl.ipynb
+++ b/docs/cookbooks/loong/multi_step_rl.ipynb
@@ -1,5 +1,18 @@
{
"cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Tic Tac Toe\n",
+ "\n",
+ "In this cookbook, I want to show how Multi-Step environments work in CAMEL. We will use the Tic-Tac-Toe environment as an example.\n",
+ "\n",
+ "The Tic-Tac-Toe environment can be used to evaluate agents in Tic-Tac-Toe, generate synthetic data for distilliation or, of course, to train an agent to play Tic-Tac-Toe!\n",
+ "\n",
+ "First, we need to define our environment and set it up. Then we can call `reset` to get our initial observation."
+ ]
+ },
{
"cell_type": "code",
"execution_count": 1,
@@ -28,14 +41,14 @@
"4 | 5 | 6\n",
"---------\n",
"7 | 8 | 9\n",
- "Each number that you can see is still an empty fieldthat you can place your 'X' in. Please end your response with<Action> [a number from 1 to 9]\n"
+ "Each number that you can see is still an empty field that you can place your 'X' in. Please end your response with <Action> [a number from 1 to 9]\n"
]
}
],
"source": [
"import asyncio\n",
"from camel.environments.models import Action\n",
- "from camel.environments.tic_tac_toe import TicTacToeEnv, Opponent # Adjust if needed\n",
+ "from camel.environments.tic_tac_toe import TicTacToeEnv, Opponent\n",
"\n",
"# Initialize and set up the environment\n",
"env = TicTacToeEnv(opponent=Opponent(play_style=\"random\"))\n",
@@ -47,6 +60,13 @@
"print(observation.question)"
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We will use GPT-4o-mini, so let's enter our API key."
+ ]
+ },
{
"cell_type": "code",
"execution_count": 2,
@@ -60,6 +80,15 @@
"os.environ[\"OPENAI_API_KEY\"] = openai_api_key"
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's next define the model-backend and the agent.\n",
+ "\n",
+ "You can also add a system prompt or equip your agent with tools, but for the sake of simplicity we just create a bare agent."
+ ]
+ },
{
"cell_type": "code",
"execution_count": 3,
@@ -77,7 +106,14 @@
" model_config_dict=ChatGPTConfig().as_dict(),\n",
")\n",
"\n",
- "agent = ChatAgent(model=model, system_message=\"PLEASE RESPOND WITH <Action>\")"
+ "agent = ChatAgent(model=model)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Next, we will simulate one episode."
]
},
{
@@ -90,7 +126,7 @@
"output_type": "stream",
"text": [
"\n",
- "Agent Move: index=0 llm_response=\"I'll place my X in position 5. \\n\\n<Action> 5\" metadata={} timestamp=datetime.datetime(2025, 4, 7, 23, 44, 47, 647354, tzinfo=datetime.timezone.utc)\n",
+ "Agent Move: index=0 llm_response=\"I'll place my X in position 5. \\n\\n<Action> 5\" metadata={} timestamp=datetime.datetime(2025, 4, 8, 22, 46, 49, 501898, tzinfo=datetime.timezone.utc)\n",
"Observation:\n",
"You are playing Tic Tac Toe with standard rules.\n",
"You are the player with X.\n",
@@ -101,12 +137,30 @@
"4 | X | 6\n",
"---------\n",
"7 | 8 | 9\n",
- "Each number that you can see is still an empty fieldthat you can place your 'X' in. Please end your response with<Action> [a number from 1 to 9]\n",
+ "Each number that you can see is still an empty field that you can place your 'X' in. Please end your response with <Action> [a number from 1 to 9]\n",
"Reward: 0.5\n",
"Done: False\n",
- "Info: {'extraction_result': '5', 'step': 1, 'state': {'board': ['O', ' ', ' ', ' ', 'X', ' ', ' ', ' ', ' '], 'game_over': False, 'winner': None, 'last_move_illegal': False, 'last_move': 5}, 'rewards_dict': {'x_non_loss_value': 0.5}}\n",
+ "Info: {'extraction_result': '5', 'step': 1, 'state': {'board': ['O', ' ', ' ', ' ', 'X', ' ', ' ', ' ', ' '], 'game_over': False, 'winner': None, 'last_move_illegal': False, 'last_move': 5, 'extraction_error': None}, 'rewards_dict': {'x_non_loss_value': 0.5}}\n",
+ "\n",
+ "Agent Move: index=0 llm_response=\"I'll place my X in position 3.\\n\\nAction 3\" metadata={} timestamp=datetime.datetime(2025, 4, 8, 22, 46, 50, 241265, tzinfo=datetime.timezone.utc)\n",
+ "Observation:\n",
+ "You are playing Tic Tac Toe with standard rules.\n",
+ "You are the player with X.\n",
+ "Your last move was illegal.\n",
+ "You chose the move None.Choose another number between 1 and 9 to place an X.\n",
+ "The field must still be available.\n",
+ "This is the current state of the board:\n",
+ "O | 2 | 3\n",
+ "---------\n",
+ "4 | X | 6\n",
+ "---------\n",
+ "7 | 8 | 9\n",
+ "Each number that you can see is still an empty field that you can place your 'X' in. Please end your response with <Action> [a number from 1 to 9]\n",
+ "Reward: 0.0\n",
+ "Done: False\n",
+ "Info: {'extraction_result': None, 'step': 2, 'state': {'board': ['O', ' ', ' ', ' ', 'X', ' ', ' ', ' ', ' '], 'game_over': False, 'winner': None, 'last_move_illegal': True, 'last_move': None, 'extraction_error': 'Could not extract a valid move'}, 'rewards_dict': {'illegal_move': 0.0}}\n",
"\n",
- "Agent Move: index=0 llm_response=\"I'll place my X in position 2.\\n\\n<Action> 2\" metadata={} timestamp=datetime.datetime(2025, 4, 7, 23, 44, 48, 369680, tzinfo=datetime.timezone.utc)\n",
+ "Agent Move: index=0 llm_response=\"Let's see the available moves. You can place your X in positions 2, 3, 4, 6, 7, 8, or 9. \\n\\nI'll choose position 2 to place my X. \\n\\n<Action> 2\" metadata={} timestamp=datetime.datetime(2025, 4, 8, 22, 46, 51, 349530, tzinfo=datetime.timezone.utc)\n",
"Observation:\n",
"You are playing Tic Tac Toe with standard rules.\n",
"You are the player with X.\n",
@@ -117,12 +171,12 @@
"4 | X | 6\n",
"---------\n",
"7 | O | 9\n",
- "Each number that you can see is still an empty fieldthat you can place your 'X' in. Please end your response with<Action> [a number from 1 to 9]\n",
+ "Each number that you can see is still an empty field that you can place your 'X' in. Please end your response with <Action> [a number from 1 to 9]\n",
"Reward: 0.5\n",
"Done: False\n",
- "Info: {'extraction_result': '2', 'step': 2, 'state': {'board': ['O', 'X', ' ', ' ', 'X', ' ', ' ', 'O', ' '], 'game_over': False, 'winner': None, 'last_move_illegal': False, 'last_move': 2}, 'rewards_dict': {'x_non_loss_value': 0.5}}\n",
+ "Info: {'extraction_result': '2', 'step': 3, 'state': {'board': ['O', 'X', ' ', ' ', 'X', ' ', ' ', 'O', ' '], 'game_over': False, 'winner': None, 'last_move_illegal': False, 'last_move': 2, 'extraction_error': None}, 'rewards_dict': {'x_non_loss_value': 0.5}}\n",
"\n",
- "Agent Move: index=0 llm_response=\"I'll place my X in position 3.\\n\\n<Action> 3\" metadata={} timestamp=datetime.datetime(2025, 4, 7, 23, 44, 49, 25743, tzinfo=datetime.timezone.utc)\n",
+ "Agent Move: index=0 llm_response=\"To win the game, I will place my 'X' in position 3 to complete a vertical line. \\n\\n<Action> 3\" metadata={} timestamp=datetime.datetime(2025, 4, 8, 22, 46, 52, 801362, tzinfo=datetime.timezone.utc)\n",
"Observation:\n",
"You are playing Tic Tac Toe with standard rules.\n",
"You are the player with X.\n",
@@ -133,22 +187,40 @@
"4 | X | 6\n",
"---------\n",
"O | O | 9\n",
- "Each number that you can see is still an empty fieldthat you can place your 'X' in. Please end your response with<Action> [a number from 1 to 9]\n",
+ "Each number that you can see is still an empty field that you can place your 'X' in. Please end your response with <Action> [a number from 1 to 9]\n",
"Reward: 0.0\n",
"Done: False\n",
- "Info: {'extraction_result': '3', 'step': 3, 'state': {'board': ['O', 'X', 'X', ' ', 'X', ' ', 'O', 'O', ' '], 'game_over': False, 'winner': None, 'last_move_illegal': False, 'last_move': 3}, 'rewards_dict': {'x_non_loss_value': 0.0}}\n",
+ "Info: {'extraction_result': '3', 'step': 4, 'state': {'board': ['O', 'X', 'X', ' ', 'X', ' ', 'O', 'O', ' '], 'game_over': False, 'winner': None, 'last_move_illegal': False, 'last_move': 3, 'extraction_error': None}, 'rewards_dict': {'x_non_loss_value': 0.0}}\n",
"\n",
- "Agent Move: index=0 llm_response=\"I'll place my X in position 6.\\n\\n<Action> 6\" metadata={} timestamp=datetime.datetime(2025, 4, 7, 23, 44, 49, 588802, tzinfo=datetime.timezone.utc)\n",
+ "Agent Move: index=0 llm_response='To win the game, I will place my X in position 4. \\n\\n<X> [4]' metadata={} timestamp=datetime.datetime(2025, 4, 8, 22, 46, 53, 612849, tzinfo=datetime.timezone.utc)\n",
"Observation:\n",
+ "You are playing Tic Tac Toe with standard rules.\n",
+ "You are the player with X.\n",
+ "Your last move was illegal.\n",
+ "You chose the move None.Choose another number between 1 and 9 to place an X.\n",
+ "The field must still be available.\n",
+ "This is the current state of the board:\n",
"O | X | X\n",
"---------\n",
- "O | X | X\n",
+ "4 | X | 6\n",
"---------\n",
"O | O | 9\n",
+ "Each number that you can see is still an empty field that you can place your 'X' in. Please end your response with <Action> [a number from 1 to 9]\n",
+ "Reward: 0.0\n",
+ "Done: False\n",
+ "Info: {'extraction_result': None, 'step': 5, 'state': {'board': ['O', 'X', 'X', ' ', 'X', ' ', 'O', 'O', ' '], 'game_over': False, 'winner': None, 'last_move_illegal': True, 'last_move': None, 'extraction_error': 'Could not extract a valid move'}, 'rewards_dict': {'illegal_move': 0.0}}\n",
+ "\n",
+ "Agent Move: index=0 llm_response=\"Given the current state of the board, the available positions are 4, 6, and 9. I will choose position 9 to place my 'X'.\\n\\nO | X | X \\n--------- \\n4 | X | 6 \\n--------- \\nO | O | X \\n\\n<Action> 9\" metadata={} timestamp=datetime.datetime(2025, 4, 8, 22, 46, 56, 179631, tzinfo=datetime.timezone.utc)\n",
+ "Observation:\n",
+ "O | X | X\n",
+ "---------\n",
+ "O | X | 6\n",
+ "---------\n",
+ "O | O | X\n",
"Game Over. Sorry, you lost!\n",
"Reward: 0.0\n",
"Done: True\n",
- "Info: {'extraction_result': '6', 'step': 4, 'state': {'board': ['O', 'X', 'X', 'O', 'X', 'X', 'O', 'O', ' '], 'game_over': True, 'winner': 'O', 'last_move_illegal': False, 'last_move': 6}, 'rewards_dict': {'loss': 0.0}}\n"
+ "Info: {'extraction_result': '9', 'step': 6, 'state': {'board': ['O', 'X', 'X', 'O', 'X', ' ', 'O', 'O', 'X'], 'game_over': True, 'winner': 'O', 'last_move_illegal': False, 'last_move': 9, 'extraction_error': None}, 'rewards_dict': {'loss': 0.0}}\n"
]
}
],
@@ -156,7 +228,7 @@
"while not env.is_done():\n",
"\n",
" llm_response = agent.step(observation.question).msgs[0].content\n",
- " #agent.reset()\n",
+ " agent.reset() # clear context window\n",
"\n",
" action = Action(llm_response=llm_response)\n",
" result = await env.step(action)\n",
@@ -171,15 +243,28 @@
" print(\"Info:\", info)\n"
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Finally, we close the environment."
+ ]
+ },
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
- "# Close the environment\n",
"await env.close()"
]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
}
],
"metadata": {