🐫 CAMEL: The first and the best multi-agent framework. Finding the Scaling Law of Agents. https://www.camel-ai.org
Python agentai-societiesartificial-intelligencecommunicative-aicooperative-aideep-learninglarge-language-modelsmulti-agent-systemsnatural-language-processing
docs:add o1datagene_cookbook (#1359)
正经人王同学8d7aaa4b5db4
diff --git a/README.md b/README.mdindex 20c492172..04a783040 100644--- a/README.md+++ b/README.md@@ -325,6 +325,7 @@ Practical guides and tutorials for implementing specific functionalities in CAME| **[Customer Service Discord Bot with Agentic RAG](https://docs.camel-ai.org/cookbooks/customer_service_Discord_bot_with_agentic_RAG.html)** | Learn how to build a robust customer service bot for Discord using Agentic RAG. || **[Create AI Agents that work with your PDFs using Chunkr & Mistral AI](https://docs.camel-ai.org/cookbooks/agent_with_chunkr_for_pdf_parsing.html)** | Learn how to create AI agents that work with your PDFs using Chunkr and Mistral AI. || **[Data Gen with Real Function Calls and Hermes Format](https://docs.camel-ai.org/cookbooks/data_gen_with_real_function_calls_and_hermes_format.html)** | Explore how to generate data with real function calls and the Hermes format. |+| **[O1 Data Generation with CAMEL and Upload Data to Huggingface](https://docs.camel-ai.org/cookbooks/O1datagen_with_CAMEL_And_Upload_Data_To_Huggingface.html)** | Uncover how to generate O1 data with CAMEL and seamlessly upload it to Huggingface. |## Utilize Various LLMs as Backendsdiff --git a/docs/cookbooks/O1datagen_with_CAMEL_And_Upload_Data_To_Huggingface.ipynb b/docs/cookbooks/O1datagen_with_CAMEL_And_Upload_Data_To_Huggingface.ipynbnew file mode 100644index 000000000..b95042227--- /dev/null+++ b/docs/cookbooks/O1datagen_with_CAMEL_And_Upload_Data_To_Huggingface.ipynb@@ -0,0 +1,1057 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "id": "d80bea45",+ "metadata": {+ "id": "d80bea45"+ },+ "source": [+ "# O1 Data Generation with CAMEL and Upload Data to Huggingface"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "MgAnJZTCCNDl",+ "metadata": {+ "id": "MgAnJZTCCNDl"+ },+ "source": [+ "You can also check this cookbook in colab [here](https://drive.google.com/file/d/1oGFFuhaiQgGPieT4BwbfAWEdV-L5cJQz/view?usp=sharing) (Use the colab share link)"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "RiZXE5RDB8tu",+ "metadata": {+ "id": "RiZXE5RDB8tu"+ },+ "source": [+ "\n",+ "\n",+ "This notebook demonstrates how to set up and leverage CAMEL's **O1DataGenerator** for generating high-quality question-answer pairs like o1 thinking data, uploading the data to Hugging Face.\n",+ "\n",+ "In this notebook, you'll explore:\n",+ "\n",+ "- **CAMEL**: A powerful multi-agent framework that enables SFT data generation and multi-agent role-playing scenarios, allowing for sophisticated AI-driven tasks.\n",+ "- **O1DataGenerator**: A tool for generating like o1 thinking data.\n",+ "\n",+ "- **Hugging Face Integration**: Uploading datasets to the Hugging Face platform for sharing\n"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "EnupE_558wFh",+ "metadata": {+ "id": "EnupE_558wFh"+ },+ "source": [+ "⭐ **Star the Repo**\n",+ "\n",+ "If you find CAMEL useful or interesting, please consider giving it a star on our [CAMEL GitHub Repo](https://github.com/camel-ai/camel)! Your stars help others find this project and motivate us to continue improving it."+ ]+ },+ {+ "cell_type": "markdown",+ "id": "0J0_iW-YVcq2",+ "metadata": {+ "id": "0J0_iW-YVcq2"+ },+ "source": [+ "## 📦 Installation"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "87b853af",+ "metadata": {+ "id": "87b853af"+ },+ "outputs": [],+ "source": [+ "%%capture\n",+ "!pip install camel-ai==0.2.15a0"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "b77fd525",+ "metadata": {+ "id": "b77fd525",+ "lines_to_next_cell": 2+ },+ "outputs": [],+ "source": [+ "import os\n",+ "from datetime import datetime\n",+ "import json\n",+ "from camel.datagen.o1datagen import O1DataGenerator"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "lfNvFbhD6o8B",+ "metadata": {+ "id": "lfNvFbhD6o8B"+ },+ "source": [+ "## 🔑 Setting Up API Keys"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "a382a3a8",+ "metadata": {+ "id": "a382a3a8"+ },+ "source": [+ "First we will set the OPENAI_API_KEY that will be used to generate the data.\n",+ "\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "5e59f61c",+ "metadata": {+ "id": "5e59f61c",+ "lines_to_next_cell": 2+ },+ "outputs": [],+ "source": [+ "from getpass import getpass"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "c90eae14",+ "metadata": {+ "id": "c90eae14",+ "lines_to_next_cell": 2,+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "outputId": "2d75236b-5361-42d8-aa8b-b6e3bed3d0c4"+ },+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Enter your OpenAI API key: ··········\n"+ ]+ }+ ],+ "source": [+ "openai_api_key = getpass('Enter your OpenAI API key: ')\n",+ "os.environ[\"OPENAI_API_KEY\"] = openai_api_key"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "aecd3b42",+ "metadata": {+ "id": "aecd3b42"+ },+ "source": [+ "## Set ChatAgent"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "bd726b2a",+ "metadata": {+ "id": "bd726b2a"+ },+ "source": [+ "Create a system message to define agent's default role and behaviors."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "a768c2af",+ "metadata": {+ "id": "a768c2af",+ "lines_to_next_cell": 2+ },+ "outputs": [],+ "source": [+ "sys_msg = 'You are a genius at slow-thinking data and code'"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "f0f39ad6",+ "metadata": {+ "id": "f0f39ad6",+ "lines_to_next_cell": 2+ },+ "source": [+ "Use ModelFactory to set up the backend model for agent\n",+ "\n",+ "CAMEL supports many other models. See [here](https://docs.camel-ai.org/key_modules/models.html) for a list."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "662e2f69",+ "metadata": {+ "id": "662e2f69"+ },+ "outputs": [],+ "source": [+ "from camel.models import ModelFactory\n",+ "from camel.types import ModelPlatformType, ModelType\n",+ "from camel.configs import ChatGPTConfig"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "108da101",+ "metadata": {+ "id": "108da101"+ },+ "outputs": [],+ "source": [+ "# Define the model, here in this case we use gpt-4o-mini\n",+ "model = ModelFactory.create(\n",+ " model_platform=ModelPlatformType.OPENAI,\n",+ " model_type=ModelType.GPT_4O_MINI,\n",+ " model_config_dict=ChatGPTConfig().as_dict(), # [Optional] the config for model\n",+ ")"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "946b410d",+ "metadata": {+ "id": "946b410d",+ "lines_to_next_cell": 2+ },+ "outputs": [],+ "source": [+ "from camel.agents import ChatAgent\n",+ "chat_agent = ChatAgent(\n",+ " system_message=sys_msg,\n",+ " model=model,\n",+ " message_window_size=10,\n",+ ")"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "ed4208d6",+ "metadata": {+ "id": "ed4208d6",+ "lines_to_next_cell": 2+ },+ "source": [+ "## Load Q&A data from a JSON file"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "64e2b2cd",+ "metadata": {+ "id": "64e2b2cd"+ },+ "source": [+ "### please prepare the qa data like below in json file:\n",+ "\n",+ "'''\n",+ "{\n",+ " \"question1\": \"answer1\",\n",+ " \"question2\": \"answer2\",\n",+ " ...\n",+ "}\n",+ "'''"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "0XuYttZgwupa",+ "metadata": {+ "id": "0XuYttZgwupa",+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "outputId": "da8f7655-1af4-4499-afd9-9e59758fc17e"+ },+ "outputs": [+ {+ "output_type": "stream",+ "name": "stdout",+ "text": [+ "/content\n"+ ]+ }+ ],+ "source": [+ "!pwd"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "RERFki4MpiHH",+ "metadata": {+ "id": "RERFki4MpiHH"+ },+ "source": [+ "The script fetches a example JSON file containing question-answer pairs from a GitHub repository and saves it locally. The JSON file is then loaded into the qa_data variable."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "qACKggkeDAVQ",+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "id": "qACKggkeDAVQ",+ "outputId": "736f06a6-c613-4497-a954-51d551324266"+ },+ "outputs": [+ {+ "output_type": "stream",+ "name": "stdout",+ "text": [+ "JSON data successfully saved to qa_data.json\n"+ ]+ }+ ],+ "source": [+ "#get example json data\n",+ "\n",+ "import requests\n",+ "import json\n",+ "\n",+ "# URL of the JSON file\n",+ "url = 'https://raw.githubusercontent.com/zjrwtx/alldata/refs/heads/main/qa_data.json'\n",+ "\n",+ "# Send a GET request to fetch the JSON file\n",+ "response = requests.get(url)\n",+ "\n",+ "# Check if the request was successful\n",+ "if response.status_code == 200:\n",+ " # Parse the response content as JSON\n",+ " json_data = response.json()\n",+ "\n",+ " # Specify the file path to save the JSON data\n",+ " file_path = 'qa_data.json'\n",+ "\n",+ " # Write the JSON data to the file\n",+ " with open(file_path, 'w', encoding='utf-8') as json_file:\n",+ " json.dump(json_data, json_file, ensure_ascii=False, indent=4)\n",+ "\n",+ " print(f\"JSON data successfully saved to {file_path}\")\n",+ "else:\n",+ " print(f\"Failed to retrieve JSON file. Status code: {response.status_code}\")"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "orMVK60weVn7",+ "metadata": {+ "id": "orMVK60weVn7"+ },+ "outputs": [],+ "source": [+ "with open(file_path, 'r', encoding='utf-8') as f:\n",+ " qa_data = json.load(f)"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "12631bd0",+ "metadata": {+ "id": "12631bd0",+ "lines_to_next_cell": 2+ },+ "source": [+ "## Create an instance of O1DataGenerator"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "4fdb7792",+ "metadata": {+ "id": "4fdb7792"+ },+ "outputs": [],+ "source": [+ "# Create an instance of O1DataGenerator\n",+ "testo1 = O1DataGenerator(chat_agent, golden_answers=qa_data)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "23e2d5a8",+ "metadata": {+ "id": "23e2d5a8",+ "lines_to_next_cell": 2+ },+ "outputs": [],+ "source": [+ "# Record generated answers\n",+ "generated_answers = {}"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "6d92f64e",+ "metadata": {+ "id": "6d92f64e"+ },+ "source": [+ "## Test Q&A"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "v4Vi60CQrZTZ",+ "metadata": {+ "id": "v4Vi60CQrZTZ"+ },+ "source": [+ "The script iterates through the questions, generates answers, and verifies their correctness. The generated answers are stored in a dictionary"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "7fd7fe57",+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "id": "7fd7fe57",+ "lines_to_next_cell": 2,+ "outputId": "03c903a7-6f24-4187-bbb3-7eff5644e7b8"+ },+ "outputs": [+ {+ "output_type": "stream",+ "name": "stdout",+ "text": [+ "Question: What is the coefficient of $x^2y^6$ in the expansion of $\\left(\\frac{3}{5}x-\\frac{y}{2}\\right)^8$? Express your answer as a common fraction\n",+ "AI's thought process and answer:\n",+ "To find the coefficient of \\(x^2y^6\\) in the expansion of \\(\\left(\\frac{3}{5}x - \\frac{y}{2}\\right)^8\\), we will follow a systematic approach. \n",+ "\n",+ "### Step 1: Analyze the Problem Requirements\n",+ "We need to expand the expression \\(\\left(\\frac{3}{5}x - \\frac{y}{2}\\right)^8\\) and identify the specific term that contains \\(x^2y^6\\). This involves using the binomial theorem, which states that:\n",+ "\n",+ "\\[\n",+ "(a + b)^n = \\sum_{k=0}^{n} \\binom{n}{k} a^{n-k} b^k\n",+ "\\]\n",+ "\n",+ "In our case, \\(a = \\frac{3}{5}x\\) and \\(b = -\\frac{y}{2}\\), and \\(n = 8\\).\n",+ "\n",+ "### Step 2: List the Steps to Solve the Problem\n",+ "1. Identify the general term in the binomial expansion.\n",+ "2. Set up the equation to find the specific term that corresponds to \\(x^2y^6\\).\n",+ "3. Solve for the coefficients and powers.\n",+ "4. Calculate the coefficient for the term \\(x^2y^6\\).\n",+ "\n",+ "### Step 3: Execute the Solution Process\n",+ "1. **Identify the General Term**: The general term in the expansion can be expressed as:\n",+ "\n",+ "\\[\n",+ "T_k = \\binom{8}{k} \\left(\\frac{3}{5}x\\right)^{8-k} \\left(-\\frac{y}{2}\\right)^k\n",+ "\\]\n",+ "\n",+ "2. **Set Up the Equation**: We need to find \\(k\\) such that the term \\(T_k\\) contains \\(x^2y^6\\). This means we need:\n",+ "\n",+ "- The power of \\(x\\) to be 2: \\(8 - k = 2\\) implies \\(k = 6\\).\n",+ "- The power of \\(y\\) to be 6: \\(k = 6\\).\n",+ "\n",+ "Since both conditions are satisfied with \\(k = 6\\), we will use this value to find the corresponding term.\n",+ "\n",+ "3. **Calculate the Coefficient**: Substitute \\(k = 6\\) into the general term:\n",+ "\n",+ "\\[\n",+ "T_6 = \\binom{8}{6} \\left(\\frac{3}{5}x\\right)^{2} \\left(-\\frac{y}{2}\\right)^{6}\n",+ "\\]\n",+ "\n",+ "Now calculate each component:\n",+ "\n",+ "- \\(\\binom{8}{6} = \\binom{8}{2} = \\frac{8 \\times 7}{2 \\times 1} = 28\\)\n",+ "- \\(\\left(\\frac{3}{5}\\right)^{2} = \\frac{9}{25}\\)\n",+ "- \\(\\left(-\\frac{1}{2}\\right)^{6} = \\frac{1}{64}\\)\n",+ "\n",+ "Now, combine these to find the coefficient:\n",+ "\n",+ "\\[\n",+ "T_6 = 28 \\cdot \\frac{9}{25} \\cdot \\frac{1}{64} \\cdot x^2 \\cdot y^6\n",+ "\\]\n",+ "\n",+ "Calculating the coefficient:\n",+ "\n",+ "\\[\n",+ "\\text{Coefficient} = 28 \\cdot \\frac{9}{25} \\cdot \\frac{1}{64} = \\frac{28 \\cdot 9}{25 \\cdot 64}\n",+ "\\]\n",+ "\n",+ "Calculating \\(28 \\cdot 9 = 252\\):\n",+ "\n",+ "\\[\n",+ "\\text{Coefficient} = \\frac{252}{1600}\n",+ "\\]\n",+ "\n",+ "4. **Simplify the Fraction**: We can simplify \\(\\frac{252}{1600}\\) by finding the greatest common divisor (GCD) of 252 and 1600. The GCD is 4.\n",+ "\n",+ "\\[\n",+ "\\frac{252 \\div 4}{1600 \\div 4} = \\frac{63}{400}\n",+ "\\]\n",+ "\n",+ "### Step 4: Provide the Final Answer\n",+ "The coefficient of \\(x^2y^6\\) in the expansion of \\(\\left(\\frac{3}{5}x - \\frac{y}{2}\\right)^8\\) is:\n",+ "\n",+ "\\[\n",+ "\\boxed{\\frac{63}{400}}\n",+ "\\]\n",+ "Answer verification result: Correct\n",+ "--------------------------------------------------\n",+ "\n",+ "Question: how many r in strawberry?\n",+ "AI's thought process and answer:\n",+ "To solve the problem of how many times the letter \"r\" appears in the word \"strawberry,\" we will follow the outlined requirements step by step.\n",+ "\n",+ "### Step 1: Analyze the Problem Requirements\n",+ "The problem requires us to determine the frequency of the letter \"r\" in the word \"strawberry.\" This involves:\n",+ "- Identifying the specific letter we are counting, which is \"r.\"\n",+ "- Understanding that we need to count occurrences of this letter in the given word.\n",+ "\n",+ "### Step 2: List the Steps to Solve the Problem\n",+ "To solve the problem, we can break it down into the following steps:\n",+ "1. Write down the word \"strawberry.\"\n",+ "2. Identify and isolate the letter \"r\" within the word.\n",+ "3. Count how many times the letter \"r\" appears in the word.\n",+ "\n",+ "### Step 3: Execute the Solution Process\n",+ "Now, let's execute the steps we have outlined:\n",+ "\n",+ "1. **Write down the word**: The word is \"strawberry.\"\n",+ "2. **Identify the letter \"r\"**: We will look at each letter in the word:\n",+ " - s\n",+ " - t\n",+ " - r (1st occurrence)\n",+ " - a\n",+ " - w\n",+ " - b\n",+ " - e\n",+ " - r (2nd occurrence)\n",+ " - r (3rd occurrence)\n",+ " - y\n",+ "\n",+ "3. **Count the occurrences**: As we go through the letters, we find:\n",+ " - The first \"r\" is the 3rd letter.\n",+ " - The second \"r\" is the 8th letter.\n",+ " - The third \"r\" is the 9th letter.\n",+ "\n",+ "Thus, we have identified that the letter \"r\" appears **three times** in the word \"strawberry.\"\n",+ "\n",+ "### Step 4: Provide the Final Answer\n",+ "After counting the occurrences, we conclude that the letter \"r\" appears **3 times** in the word \"strawberry.\"\n",+ "\n",+ "### Summary of Thought Process\n",+ "- We carefully analyzed the problem to understand what was being asked.\n",+ "- We broke down the solution into manageable steps to ensure clarity and accuracy.\n",+ "- We executed the counting process methodically, ensuring that we did not miss any occurrences of the letter \"r.\"\n",+ "- Finally, we provided a clear and concise answer based on our findings. \n",+ "\n",+ "The final answer is: **3**.\n",+ "Answer verification result: Correct\n",+ "--------------------------------------------------\n",+ "\n"+ ]+ }+ ],+ "source": [+ "# Test Q&A\n",+ "for question in qa_data.keys():\n",+ " print(f\"Question: {question}\")\n",+ "\n",+ " # Get AI's thought process and answer\n",+ " answer = testo1.get_answer(question)\n",+ " generated_answers[question] = answer\n",+ " print(f\"AI's thought process and answer:\\n{answer}\")\n",+ "\n",+ " # Verify the answer\n",+ " is_correct = testo1.verify_answer(question, answer)\n",+ " print(f\"Answer verification result: {'Correct' if is_correct else 'Incorrect'}\")\n",+ " print(\"-\" * 50)\n",+ " print() # Add a new line at the end of each iteration"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "8b35c0e9",+ "metadata": {+ "id": "8b35c0e9",+ "lines_to_next_cell": 2+ },+ "source": [+ "### Export the generated answers to a JSON file and transform these to Alpaca traing data format\n",+ "\n",+ "\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "d47d8529",+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "id": "d47d8529",+ "lines_to_next_cell": 2,+ "outputId": "60a4a6f6-5614-480d-8d5c-d4116ce5457c"+ },+ "outputs": [+ {+ "output_type": "stream",+ "name": "stdout",+ "text": [+ "The generated answers have been exported to: generated_answers_20241227_111615.json\n"+ ]+ }+ ],+ "source": [+ "simplified_output = {\n",+ " 'timestamp': datetime.now().isoformat(),\n",+ " 'qa_pairs': generated_answers\n",+ "}\n",+ "simplified_file = f'generated_answers_{datetime.now().strftime(\"%Y%m%d_%H%M%S\")}.json'\n",+ "with open(simplified_file, 'w', encoding='utf-8') as f:\n",+ " json.dump(simplified_output, f, ensure_ascii=False, indent=2)\n",+ "print(f\"The generated answers have been exported to: {simplified_file}\")"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "riC_gYznYFN6",+ "metadata": {+ "id": "riC_gYznYFN6"+ },+ "source": [+ "The script transforms the Q&A data into the Alpaca training data format, which is suitable for supervised fine-tuning (SFT). The transformed data is saved to a new JSON file."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "2ef8dfbc",+ "metadata": {+ "id": "2ef8dfbc"+ },+ "outputs": [],+ "source": [+ "import json\n",+ "from datetime import datetime\n",+ "\n",+ "def transform_qa_format(input_file):\n",+ " # Read the input JSON file\n",+ " with open(input_file, 'r', encoding='utf-8') as f:\n",+ " data = json.load(f)\n",+ "\n",+ " # Transform the data\n",+ " transformed_data = []\n",+ " for question, answer in data['qa_pairs'].items():\n",+ " transformed_pair = {\n",+ " \"instruction\": question,\n",+ " \"input\": \"\",\n",+ " \"output\": answer\n",+ " }\n",+ " transformed_data.append(transformed_pair)\n",+ "\n",+ " # Generate output filename with timestamp\n",+ " timestamp = datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n",+ " output_file = f'transformed_qa_{timestamp}.json'\n",+ "\n",+ " # Write the transformed data\n",+ " with open(output_file, 'w', encoding='utf-8') as f:\n",+ " json.dump(transformed_data, f, ensure_ascii=False, indent=2)\n",+ "\n",+ " return output_file, transformed_data"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "a83515f3",+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "id": "a83515f3",+ "outputId": "47e2fffd-6989-452c-a305-e4fcdf3998f6"+ },+ "outputs": [+ {+ "output_type": "stream",+ "name": "stdout",+ "text": [+ "Transformation complete. Output saved to: transformed_qa_20241227_111622.json\n"+ ]+ }+ ],+ "source": [+ "output_file, transformed_data = transform_qa_format(simplified_file)\n",+ "print(f\"Transformation complete. Output saved to: {output_file}\")"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "lK8qEmdGYUeX",+ "metadata": {+ "id": "lK8qEmdGYUeX"+ },+ "source": [+ "## Upload the Data to Huggingface"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "1696368d",+ "metadata": {+ "id": "1696368d"+ },+ "source": [+ "This defines a function upload_to_huggingface that uploads a dataset to Hugging Face. The script is modular, with helper functions handling specific tasks such as dataset name generation, dataset creation, metadata card creation, and record addition"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "5TYOIiF3yQZC",+ "metadata": {+ "id": "5TYOIiF3yQZC"+ },+ "outputs": [],+ "source": [+ "# Import necessary modules and classes\n",+ "from camel.datahubs.huggingface import HuggingFaceDatasetManager # Manages interactions with Hugging Face datasets\n",+ "from camel.datahubs.models import Record # Represents a single record in the dataset\n",+ "from datetime import datetime # Handles date and time operations\n",+ "\n",+ "# Main function: Upload dataset to Hugging Face\n",+ "def upload_to_huggingface(transformed_data, username, dataset_name=None):\n",+ " r\"\"\"Uploads transformed data to the Hugging Face dataset platform.\n",+ "\n",+ " Args:\n",+ " transformed_data (list): Transformed data, typically a list of dictionaries.\n",+ " username (str): Hugging Face username.\n",+ " dataset_name (str, optional): Custom dataset name.\n",+ "\n",+ " Returns:\n",+ " str: URL of the uploaded dataset.\n",+ " \"\"\"\n",+ " # Initialize HuggingFaceDatasetManager to interact with Hugging Face datasets\n",+ " manager = HuggingFaceDatasetManager()\n",+ "\n",+ " # Generate or validate the dataset name\n",+ " dataset_name = generate_or_validate_dataset_name(username, dataset_name)\n",+ "\n",+ " # Create the dataset on Hugging Face and get the dataset URL\n",+ " dataset_url = create_dataset(manager, dataset_name)\n",+ "\n",+ " # Create a dataset card to add metadata\n",+ " create_dataset_card(manager, dataset_name, username)\n",+ "\n",+ " # Convert the transformed data into a list of Record objects\n",+ " records = create_records(transformed_data)\n",+ "\n",+ " # Add the Record objects to the dataset\n",+ " add_records_to_dataset(manager, dataset_name, records)\n",+ "\n",+ " # Return the dataset URL\n",+ " return dataset_url\n",+ "\n",+ "# Generate or validate the dataset name\n",+ "def generate_or_validate_dataset_name(username, dataset_name):\n",+ " r\"\"\"Generates a default dataset name or validates and formats a user-provided name.\n",+ "\n",+ " Args:\n",+ " username (str): Hugging Face username.\n",+ " dataset_name (str, optional): User-provided custom dataset name.\n",+ "\n",+ " Returns:\n",+ " str: Formatted dataset name.\n",+ " \"\"\"\n",+ " if dataset_name is None:\n",+ " # If no dataset name is provided, generate a default name with the username and current date\n",+ " dataset_name = f\"{username}/qa-dataset-{datetime.now().strftime('%Y%m%d')}\"\n",+ " else:\n",+ " # If a dataset name is provided, format it to include the username\n",+ " dataset_name = f\"{username}/{dataset_name}\"\n",+ " return dataset_name\n",+ "\n",+ "# Create a dataset on Hugging Face\n",+ "def create_dataset(manager, dataset_name):\n",+ " r\"\"\"Creates a new dataset on Hugging Face and returns the dataset URL.\n",+ "\n",+ " Args:\n",+ " manager (HuggingFaceDatasetManager): Instance of HuggingFaceDatasetManager.\n",+ " dataset_name (str): Name of the dataset.\n",+ "\n",+ " Returns:\n",+ " str: URL of the created dataset.\n",+ " \"\"\"\n",+ " print(f\"Creating dataset: {dataset_name}\")\n",+ " # Use HuggingFaceDatasetManager to create the dataset\n",+ " dataset_url = manager.create_dataset(name=dataset_name)\n",+ " print(f\"Dataset created: {dataset_url}\")\n",+ " return dataset_url\n",+ "\n",+ "# Create a dataset card with metadata\n",+ "def create_dataset_card(manager, dataset_name, username):\n",+ " r\"\"\"Creates a dataset card to add metadata\n",+ "\n",+ " Args:\n",+ " manager (HuggingFaceDatasetManager): Instance of HuggingFaceDatasetManager.\n",+ " dataset_name (str): Name of the dataset.\n",+ " username (str): Hugging Face username.\n",+ " \"\"\"\n",+ " print(\"Creating dataset card...\")\n",+ " # Use HuggingFaceDatasetManager to create the dataset card\n",+ " manager.create_dataset_card(\n",+ " dataset_name=dataset_name,\n",+ " description=\"Question-Answer dataset generated by CAMEL O1DataGene\", # Dataset description\n",+ " license=\"mit\", # Dataset license\n",+ " language=[\"en\"], # Dataset language\n",+ " size_category=\"<1MB\", # Dataset size category\n",+ " version=\"0.1.0\", # Dataset version\n",+ " tags=[\"camel\", \"question-answering\"], # Dataset tags\n",+ " task_categories=[\"question-answering\"], # Dataset task categories\n",+ " authors=[username] # Dataset authors\n",+ " )\n",+ " print(\"Dataset card created successfully.\")\n",+ "\n",+ "# Convert transformed data into Record objects\n",+ "def create_records(transformed_data):\n",+ " r\"\"\"Converts transformed data into a list of Record objects.\n",+ "\n",+ " Args:\n",+ " transformed_data (list): Transformed data, typically a list of dictionaries.\n",+ "\n",+ " Returns:\n",+ " list: List of Record objects.\n",+ " \"\"\"\n",+ " records = []\n",+ " # Iterate through the transformed data and convert each dictionary into a Record object\n",+ " for item in transformed_data:\n",+ " record = Record(**item) # Use the dictionary key-value pairs to create a Record object\n",+ " records.append(record)\n",+ " return records\n",+ "\n",+ "# Add Record objects to the dataset\n",+ "def add_records_to_dataset(manager, dataset_name, records):\n",+ " r\"\"\"Adds a list of Record objects to the dataset.\n",+ "\n",+ " Args:\n",+ " manager (HuggingFaceDatasetManager): Instance of HuggingFaceDatasetManager.\n",+ " dataset_name (str): Name of the dataset.\n",+ " records (list): List of Record objects.\n",+ " \"\"\"\n",+ " print(\"Adding records to the dataset...\")\n",+ " # Use HuggingFaceDatasetManager to add the records to the dataset\n",+ " manager.add_records(dataset_name=dataset_name, records=records)\n",+ " print(\"Records added successfully.\")"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "TrxbW2_bY0im",+ "metadata": {+ "id": "TrxbW2_bY0im"+ },+ "source": [+ "# Config Access Token of Huggingface"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "L3dwOEvrLoIV",+ "metadata": {+ "id": "L3dwOEvrLoIV"+ },+ "source": [+ "You can go to [here](https://huggingface.co/settings/tokens) to get API Key from Huggingface"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "_QHHkipbY7kJ",+ "metadata": {+ "id": "_QHHkipbY7kJ",+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "outputId": "f88df1b1-268e-4a42-821c-486ac4dfa872"+ },+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Enter your HUGGING_FACE_TOKEN: ··········\n"+ ]+ }+ ],+ "source": [+ "HUGGING_FACE_TOKEN = getpass('Enter your HUGGING_FACE_TOKEN: ')\n",+ "os.environ[\"HUGGING_FACE_TOKEN\"] = HUGGING_FACE_TOKEN"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "e84fb857",+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "id": "e84fb857",+ "outputId": "527a49bc-63e4-4e7d-b730-7c830f8a412e"+ },+ "outputs": [+ {+ "output_type": "stream",+ "name": "stdout",+ "text": [+ "Enter your HuggingFace username: zjrwtxtechstudio\n",+ "Enter dataset name (press Enter to use default): o1data88\n",+ "Creating dataset: zjrwtxtechstudio/o1data88\n",+ "Dataset created: https://huggingface.co/datasets/zjrwtxtechstudio/o1data88\n",+ "Creating dataset card...\n",+ "Dataset card created successfully.\n",+ "Adding records to the dataset...\n",+ "Records added successfully.\n",+ "\n",+ "Data successfully uploaded to HuggingFace!\n",+ "Dataset URL: https://huggingface.co/datasets/zjrwtxtechstudio/o1data88\n"+ ]+ }+ ],+ "source": [+ "# Set your personal huggingface config, then upload to HuggingFace\n",+ "username = input(\"Enter your HuggingFace username: \")\n",+ "dataset_name = input(\"Enter dataset name (press Enter to use default): \").strip()\n",+ "if not dataset_name:\n",+ " dataset_name = None\n",+ "\n",+ "try:\n",+ " dataset_url = upload_to_huggingface(transformed_data, username, dataset_name)\n",+ " print(f\"\\nData successfully uploaded to HuggingFace!\")\n",+ " print(f\"Dataset URL: {dataset_url}\")\n",+ "except Exception as e:\n",+ " print(f\"Error uploading to HuggingFace: {str(e)}\")\n"+ ]+ },+ {+ "cell_type": "markdown",+ "source": [+ "### final example preview"+ ],+ "metadata": {+ "id": "2nLsMgSnzp6l"+ },+ "id": "2nLsMgSnzp6l"+ },+ {+ "cell_type": "markdown",+ "source": [+ ""+ ],+ "metadata": {+ "id": "d1nFtlFFzldR"+ },+ "id": "d1nFtlFFzldR"+ },+ {+ "cell_type": "markdown",+ "id": "flYNal6-R4yR",+ "metadata": {+ "id": "flYNal6-R4yR"+ },+ "source": [+ "## 🌟 Highlights"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "-qsPHvwW6oW3",+ "metadata": {+ "id": "-qsPHvwW6oW3"+ },+ "source": [+ "\n",+ "\n",+ "This cookbook demonstrates the process of using **CAMEL's O1DataGenerator** to create high-quality question-answer pairs, similar to o1 thinking data. The notebook covers the following steps:\n",+ "\n",+ "1. **Setup**: Installation of the `camel-ai` library and configuration of the OpenAI API key.\n",+ "2. **Data Generation**: Utilization of the `O1DataGenerator` to generate answers for predefined questions using llm model.\n",+ "3. **Data Transformation**: Conversion of the generated Q&A data into a format compliant with the Alpaca training data schema.\n",+ "4. **Upload to Hugging Face**: Integration with Hugging Face to upload the transformed dataset, including the creation of a dataset card and metadata.\n",+ "\n",+ "The cookbook also includes detailed instructions for setting up the environment, handling API keys, and configuring the Hugging Face dataset upload process. The final output is a dataset uploaded to Hugging Face, ready for sharing and further use in AI training tasks."+ ]+ },+ {+ "cell_type": "markdown",+ "id": "cGoeNB1X7Qv2",+ "metadata": {+ "id": "cGoeNB1X7Qv2"+ },+ "source": [+ "⭐ **Star the Repo**\n",+ "\n",+ "If you find CAMEL useful or interesting, please consider giving it a star on [GitHub](https://github.com/camel-ai/camel)! Your stars help others find this project and motivate us to continue improving it."+ ]+ }+ ],+ "metadata": {+ "accelerator": "GPU",+ "colab": {+ "gpuType": "T4",+ "provenance": []+ },+ "jupytext": {+ "cell_metadata_filter": "-all",+ "main_language": "python",+ "notebook_metadata_filter": "-all"+ },+ "kernelspec": {+ "display_name": "o1data",+ "language": "python",+ "name": "python3"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.11.11"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 5+}\ No newline at end of filediff --git a/docs/index.rst b/docs/index.rstindex 7b2f8e55d..47cfb3b39 100644--- a/docs/index.rst+++ b/docs/index.rst@@ -77,6 +77,7 @@ Main Documentationcookbooks/customer_service_Discord_bot_using_SambaNova_with_agentic_RAG.ipynbcookbooks/data_gen_with_real_function_calls_and_hermes_format.ipynbcookbooks/agent_with_chunkr_for_pdf_parsing.ipynb+ cookbooks/O1datagen_with_CAMEL_And_Upload_Data_To_Huggingface.ipynb.. toctree:::maxdepth: 1