🐫 CAMEL: The first and the best multi-agent framework. Finding the Scaling Law of Agents. https://www.camel-ai.org
Python agentai-societiesartificial-intelligencecommunicative-aicooperative-aideep-learninglarge-language-modelsmulti-agent-systemsnatural-language-processing
docs: improve cookbook for Tic Tac Toe (#2146)
halleritec0cbb8400481
diff --git a/docs/cookbooks/loong/multi_step_rl.ipynb b/docs/cookbooks/loong/multi_step_rl.ipynbindex 58cd47c5d..35a2ddd46 100644--- a/docs/cookbooks/loong/multi_step_rl.ipynb+++ b/docs/cookbooks/loong/multi_step_rl.ipynb@@ -1,5 +1,18 @@{"cells": [+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "# Tic Tac Toe\n",+ "\n",+ "In this cookbook, I want to show how Multi-Step environments work in CAMEL. We will use the Tic-Tac-Toe environment as an example.\n",+ "\n",+ "The Tic-Tac-Toe environment can be used to evaluate agents in Tic-Tac-Toe, generate synthetic data for distilliation or, of course, to train an agent to play Tic-Tac-Toe!\n",+ "\n",+ "First, we need to define our environment and set it up. Then we can call `reset` to get our initial observation."+ ]+ },{"cell_type": "code","execution_count": 1,@@ -28,14 +41,14 @@"4 | 5 | 6\n","---------\n","7 | 8 | 9\n",- "Each number that you can see is still an empty fieldthat you can place your 'X' in. Please end your response with<Action> [a number from 1 to 9]\n"+ "Each number that you can see is still an empty field that you can place your 'X' in. Please end your response with <Action> [a number from 1 to 9]\n"]}],"source": ["import asyncio\n","from camel.environments.models import Action\n",- "from camel.environments.tic_tac_toe import TicTacToeEnv, Opponent # Adjust if needed\n",+ "from camel.environments.tic_tac_toe import TicTacToeEnv, Opponent\n","\n","# Initialize and set up the environment\n","env = TicTacToeEnv(opponent=Opponent(play_style=\"random\"))\n",@@ -47,6 +60,13 @@"print(observation.question)"]},+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "We will use GPT-4o-mini, so let's enter our API key."+ ]+ },{"cell_type": "code","execution_count": 2,@@ -60,6 +80,15 @@"os.environ[\"OPENAI_API_KEY\"] = openai_api_key"]},+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Let's next define the model-backend and the agent.\n",+ "\n",+ "You can also add a system prompt or equip your agent with tools, but for the sake of simplicity we just create a bare agent."+ ]+ },{"cell_type": "code","execution_count": 3,@@ -77,7 +106,14 @@" model_config_dict=ChatGPTConfig().as_dict(),\n",")\n","\n",- "agent = ChatAgent(model=model, system_message=\"PLEASE RESPOND WITH <Action>\")"+ "agent = ChatAgent(model=model)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Next, we will simulate one episode."]},{@@ -90,7 +126,7 @@"output_type": "stream","text": ["\n",- "Agent Move: index=0 llm_response=\"I'll place my X in position 5. \\n\\n<Action> 5\" metadata={} timestamp=datetime.datetime(2025, 4, 7, 23, 44, 47, 647354, tzinfo=datetime.timezone.utc)\n",+ "Agent Move: index=0 llm_response=\"I'll place my X in position 5. \\n\\n<Action> 5\" metadata={} timestamp=datetime.datetime(2025, 4, 8, 22, 46, 49, 501898, tzinfo=datetime.timezone.utc)\n","Observation:\n","You are playing Tic Tac Toe with standard rules.\n","You are the player with X.\n",@@ -101,12 +137,30 @@"4 | X | 6\n","---------\n","7 | 8 | 9\n",- "Each number that you can see is still an empty fieldthat you can place your 'X' in. Please end your response with<Action> [a number from 1 to 9]\n",+ "Each number that you can see is still an empty field that you can place your 'X' in. Please end your response with <Action> [a number from 1 to 9]\n","Reward: 0.5\n","Done: False\n",- "Info: {'extraction_result': '5', 'step': 1, 'state': {'board': ['O', ' ', ' ', ' ', 'X', ' ', ' ', ' ', ' '], 'game_over': False, 'winner': None, 'last_move_illegal': False, 'last_move': 5}, 'rewards_dict': {'x_non_loss_value': 0.5}}\n",+ "Info: {'extraction_result': '5', 'step': 1, 'state': {'board': ['O', ' ', ' ', ' ', 'X', ' ', ' ', ' ', ' '], 'game_over': False, 'winner': None, 'last_move_illegal': False, 'last_move': 5, 'extraction_error': None}, 'rewards_dict': {'x_non_loss_value': 0.5}}\n",+ "\n",+ "Agent Move: index=0 llm_response=\"I'll place my X in position 3.\\n\\nAction 3\" metadata={} timestamp=datetime.datetime(2025, 4, 8, 22, 46, 50, 241265, tzinfo=datetime.timezone.utc)\n",+ "Observation:\n",+ "You are playing Tic Tac Toe with standard rules.\n",+ "You are the player with X.\n",+ "Your last move was illegal.\n",+ "You chose the move None.Choose another number between 1 and 9 to place an X.\n",+ "The field must still be available.\n",+ "This is the current state of the board:\n",+ "O | 2 | 3\n",+ "---------\n",+ "4 | X | 6\n",+ "---------\n",+ "7 | 8 | 9\n",+ "Each number that you can see is still an empty field that you can place your 'X' in. Please end your response with <Action> [a number from 1 to 9]\n",+ "Reward: 0.0\n",+ "Done: False\n",+ "Info: {'extraction_result': None, 'step': 2, 'state': {'board': ['O', ' ', ' ', ' ', 'X', ' ', ' ', ' ', ' '], 'game_over': False, 'winner': None, 'last_move_illegal': True, 'last_move': None, 'extraction_error': 'Could not extract a valid move'}, 'rewards_dict': {'illegal_move': 0.0}}\n","\n",- "Agent Move: index=0 llm_response=\"I'll place my X in position 2.\\n\\n<Action> 2\" metadata={} timestamp=datetime.datetime(2025, 4, 7, 23, 44, 48, 369680, tzinfo=datetime.timezone.utc)\n",+ "Agent Move: index=0 llm_response=\"Let's see the available moves. You can place your X in positions 2, 3, 4, 6, 7, 8, or 9. \\n\\nI'll choose position 2 to place my X. \\n\\n<Action> 2\" metadata={} timestamp=datetime.datetime(2025, 4, 8, 22, 46, 51, 349530, tzinfo=datetime.timezone.utc)\n","Observation:\n","You are playing Tic Tac Toe with standard rules.\n","You are the player with X.\n",@@ -117,12 +171,12 @@"4 | X | 6\n","---------\n","7 | O | 9\n",- "Each number that you can see is still an empty fieldthat you can place your 'X' in. Please end your response with<Action> [a number from 1 to 9]\n",+ "Each number that you can see is still an empty field that you can place your 'X' in. Please end your response with <Action> [a number from 1 to 9]\n","Reward: 0.5\n","Done: False\n",- "Info: {'extraction_result': '2', 'step': 2, 'state': {'board': ['O', 'X', ' ', ' ', 'X', ' ', ' ', 'O', ' '], 'game_over': False, 'winner': None, 'last_move_illegal': False, 'last_move': 2}, 'rewards_dict': {'x_non_loss_value': 0.5}}\n",+ "Info: {'extraction_result': '2', 'step': 3, 'state': {'board': ['O', 'X', ' ', ' ', 'X', ' ', ' ', 'O', ' '], 'game_over': False, 'winner': None, 'last_move_illegal': False, 'last_move': 2, 'extraction_error': None}, 'rewards_dict': {'x_non_loss_value': 0.5}}\n","\n",- "Agent Move: index=0 llm_response=\"I'll place my X in position 3.\\n\\n<Action> 3\" metadata={} timestamp=datetime.datetime(2025, 4, 7, 23, 44, 49, 25743, tzinfo=datetime.timezone.utc)\n",+ "Agent Move: index=0 llm_response=\"To win the game, I will place my 'X' in position 3 to complete a vertical line. \\n\\n<Action> 3\" metadata={} timestamp=datetime.datetime(2025, 4, 8, 22, 46, 52, 801362, tzinfo=datetime.timezone.utc)\n","Observation:\n","You are playing Tic Tac Toe with standard rules.\n","You are the player with X.\n",@@ -133,22 +187,40 @@"4 | X | 6\n","---------\n","O | O | 9\n",- "Each number that you can see is still an empty fieldthat you can place your 'X' in. Please end your response with<Action> [a number from 1 to 9]\n",+ "Each number that you can see is still an empty field that you can place your 'X' in. Please end your response with <Action> [a number from 1 to 9]\n","Reward: 0.0\n","Done: False\n",- "Info: {'extraction_result': '3', 'step': 3, 'state': {'board': ['O', 'X', 'X', ' ', 'X', ' ', 'O', 'O', ' '], 'game_over': False, 'winner': None, 'last_move_illegal': False, 'last_move': 3}, 'rewards_dict': {'x_non_loss_value': 0.0}}\n",+ "Info: {'extraction_result': '3', 'step': 4, 'state': {'board': ['O', 'X', 'X', ' ', 'X', ' ', 'O', 'O', ' '], 'game_over': False, 'winner': None, 'last_move_illegal': False, 'last_move': 3, 'extraction_error': None}, 'rewards_dict': {'x_non_loss_value': 0.0}}\n","\n",- "Agent Move: index=0 llm_response=\"I'll place my X in position 6.\\n\\n<Action> 6\" metadata={} timestamp=datetime.datetime(2025, 4, 7, 23, 44, 49, 588802, tzinfo=datetime.timezone.utc)\n",+ "Agent Move: index=0 llm_response='To win the game, I will place my X in position 4. \\n\\n<X> [4]' metadata={} timestamp=datetime.datetime(2025, 4, 8, 22, 46, 53, 612849, tzinfo=datetime.timezone.utc)\n","Observation:\n",+ "You are playing Tic Tac Toe with standard rules.\n",+ "You are the player with X.\n",+ "Your last move was illegal.\n",+ "You chose the move None.Choose another number between 1 and 9 to place an X.\n",+ "The field must still be available.\n",+ "This is the current state of the board:\n","O | X | X\n","---------\n",- "O | X | X\n",+ "4 | X | 6\n","---------\n","O | O | 9\n",+ "Each number that you can see is still an empty field that you can place your 'X' in. Please end your response with <Action> [a number from 1 to 9]\n",+ "Reward: 0.0\n",+ "Done: False\n",+ "Info: {'extraction_result': None, 'step': 5, 'state': {'board': ['O', 'X', 'X', ' ', 'X', ' ', 'O', 'O', ' '], 'game_over': False, 'winner': None, 'last_move_illegal': True, 'last_move': None, 'extraction_error': 'Could not extract a valid move'}, 'rewards_dict': {'illegal_move': 0.0}}\n",+ "\n",+ "Agent Move: index=0 llm_response=\"Given the current state of the board, the available positions are 4, 6, and 9. I will choose position 9 to place my 'X'.\\n\\nO | X | X \\n--------- \\n4 | X | 6 \\n--------- \\nO | O | X \\n\\n<Action> 9\" metadata={} timestamp=datetime.datetime(2025, 4, 8, 22, 46, 56, 179631, tzinfo=datetime.timezone.utc)\n",+ "Observation:\n",+ "O | X | X\n",+ "---------\n",+ "O | X | 6\n",+ "---------\n",+ "O | O | X\n","Game Over. Sorry, you lost!\n","Reward: 0.0\n","Done: True\n",- "Info: {'extraction_result': '6', 'step': 4, 'state': {'board': ['O', 'X', 'X', 'O', 'X', 'X', 'O', 'O', ' '], 'game_over': True, 'winner': 'O', 'last_move_illegal': False, 'last_move': 6}, 'rewards_dict': {'loss': 0.0}}\n"+ "Info: {'extraction_result': '9', 'step': 6, 'state': {'board': ['O', 'X', 'X', 'O', 'X', ' ', 'O', 'O', 'X'], 'game_over': True, 'winner': 'O', 'last_move_illegal': False, 'last_move': 9, 'extraction_error': None}, 'rewards_dict': {'loss': 0.0}}\n"]}],@@ -156,7 +228,7 @@"while not env.is_done():\n","\n"," llm_response = agent.step(observation.question).msgs[0].content\n",- " #agent.reset()\n",+ " agent.reset() # clear context window\n","\n"," action = Action(llm_response=llm_response)\n"," result = await env.step(action)\n",@@ -171,15 +243,28 @@" print(\"Info:\", info)\n"]},+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Finally, we close the environment."+ ]+ },{"cell_type": "code","execution_count": 5,"metadata": {},"outputs": [],"source": [- "# Close the environment\n","await env.close()"]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": []}],"metadata": {