Skip to content

A programming framework for agentic AI

Improving website navigation and help click-through analytics collection (#2205)

Eric Zhu6ca478502dd7
diff --git a/README.md b/README.md
index a5f4ea36f..98aebbf06 100644
--- a/README.md
+++ b/README.md
@@ -2,7 +2,7 @@
[![Build](https://github.com/microsoft/autogen/actions/workflows/python-package.yml/badge.svg)](https://github.com/microsoft/autogen/actions/workflows/python-package.yml)
![Python Version](https://img.shields.io/badge/3.8%20%7C%203.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue)
[![Downloads](https://static.pepy.tech/badge/pyautogen/week)](https://pepy.tech/project/pyautogen)
-[![Discord](https://img.shields.io/discord/1153072414184452236?logo=discord&style=flat)](https://discord.gg/pAbnFJrkgZ)
+[![Discord](https://img.shields.io/discord/1153072414184452236?logo=discord&style=flat)](https://aka.ms/autogen-dc)
[![Twitter](https://img.shields.io/twitter/url/https/twitter.com/cloudposse.svg?style=social&label=Follow%20%40pyautogen)](https://twitter.com/pyautogen)
@@ -62,7 +62,7 @@ AutoGen is powered by collaborative [research studies](https://microsoft.github.
## Roadmaps
To see what we are working on and what we plan to work on, please check our
-[Roadmap Issues](https://github.com/microsoft/autogen/issues?q=is%3Aopen+is%3Aissue+label%3Aroadmap).
+[Roadmap Issues](https://aka.ms/autogen-roadmap).
## Quickstart
The easiest way to start playing is
@@ -172,7 +172,7 @@ In addition, you can find:
- [Research](https://microsoft.github.io/autogen/docs/Research), [blogposts](https://microsoft.github.io/autogen/blog) around AutoGen, and [Transparency FAQs](https://github.com/microsoft/autogen/blob/main/TRANSPARENCY_FAQS.md)
-- [Discord](https://discord.gg/pAbnFJrkgZ)
+- [Discord](https://aka.ms/autogen-dc)
- [Contributing guide](https://microsoft.github.io/autogen/docs/Contribute)
diff --git a/website/blog/2023-04-21-LLM-tuning-math/index.md b/website/blog/2023-04-21-LLM-tuning-math/index.md
index a5378af6c..a8eeb2538 100644
--- a/website/blog/2023-04-21-LLM-tuning-math/index.md
+++ b/website/blog/2023-04-21-LLM-tuning-math/index.md
@@ -71,4 +71,4 @@ The need for model selection, parameter tuning and cost saving is not specific t
* [Research paper about the tuning technique](https://arxiv.org/abs/2303.04673)
* [Documentation about inference tuning](/docs/Use-Cases/enhanced_inference)
-*Do you have any experience to share about LLM applications? Do you like to see more support or research of LLM optimization or automation? Please join our [Discord](https://discord.gg/pAbnFJrkgZ) server for discussion.*
+*Do you have any experience to share about LLM applications? Do you like to see more support or research of LLM optimization or automation? Please join our [Discord](https://aka.ms/autogen-dc) server for discussion.*
diff --git a/website/blog/2023-05-18-GPT-adaptive-humaneval/index.mdx b/website/blog/2023-05-18-GPT-adaptive-humaneval/index.mdx
index 924ca4eb3..3a4acf511 100644
--- a/website/blog/2023-05-18-GPT-adaptive-humaneval/index.mdx
+++ b/website/blog/2023-05-18-GPT-adaptive-humaneval/index.mdx
@@ -7,8 +7,8 @@ tags: [LLM, GPT, research]
![An adaptive way of using GPT-3.5 and GPT-4 outperforms GPT-4 in both coding success rate and inference cost](img/humaneval.png)
**TL;DR:**
-* **A case study using the HumanEval benchmark shows that an adaptive way of using multiple GPT models can achieve both much higher accuracy (from 68% to 90%) and lower inference cost (by 18%) than using GPT-4 for coding.**
+- **A case study using the HumanEval benchmark shows that an adaptive way of using multiple GPT models can achieve both much higher accuracy (from 68% to 90%) and lower inference cost (by 18%) than using GPT-4 for coding.**
GPT-4 is a big upgrade of foundation model capability, e.g., in code and math, accompanied by a much higher (more than 10x) price per token to use over GPT-3.5-Turbo. On a code completion benchmark, [HumanEval](https://huggingface.co/datasets/openai_humaneval), developed by OpenAI, GPT-4 can successfully solve 68% tasks while GPT-3.5-Turbo does 46%. It is possible to increase the success rate of GPT-4 further by generating multiple responses or making multiple calls. However, that will further increase the cost, which is already nearly 20 times of using GPT-3.5-Turbo and with more restricted API call rate limit. Can we achieve more with less?
@@ -16,10 +16,10 @@ In this blog post, we will explore a creative, adaptive way of using GPT models
## Observations
-* GPT-3.5-Turbo can already solve 40%-50% tasks. For these tasks if we never use GPT-4, we can save nearly 40-50% cost.
-* If we use the saved cost to generate more responses with GPT-4 for the remaining unsolved tasks, it is possible to solve some more of them while keeping the amortized cost down.
+- GPT-3.5-Turbo can already solve 40%-50% tasks. For these tasks if we never use GPT-4, we can save nearly 40-50% cost.
+- If we use the saved cost to generate more responses with GPT-4 for the remaining unsolved tasks, it is possible to solve some more of them while keeping the amortized cost down.
-The obstacle of leveraging these observations is that we do not know *a priori* which tasks can be solved by the cheaper model, which tasks can be solved by the expensive model, and which tasks can be solved by paying even more to the expensive model.
+The obstacle of leveraging these observations is that we do not know _a priori_ which tasks can be solved by the cheaper model, which tasks can be solved by the expensive model, and which tasks can be solved by paying even more to the expensive model.
To overcome that obstacle, one may want to predict which task requires what model to solve and how many responses are required for each task. Let's look at one example code completion task:
@@ -49,8 +49,8 @@ Some simple example test cases are provided in the docstr. If we already have a
Combining these observations, we can design a solution with two intuitive ideas:
-* Make use of auto-generated feedback, i.e., code execution results, to filter responses.
-* Try inference configurations one by one, until one response can pass the filter.
+- Make use of auto-generated feedback, i.e., code execution results, to filter responses.
+- Try inference configurations one by one, until one response can pass the filter.
![Design](img/design.png)
@@ -72,6 +72,7 @@ The inference cost includes the cost for generating the assertions in our soluti
Here are a few examples of function definitions which are solved by different configurations in the portfolio.
1. Solved by GPT-3.5-Turbo, n=1, temperature=0
+
```python
def compare(game,guess):
"""I think we all remember that feeling when the result of some long-awaited
@@ -89,8 +90,10 @@ def compare(game,guess):
compare([0,5,0,0,0,4],[4,1,1,0,0,-2]) -> [4,4,1,0,0,6]
"""
```
+
2. Solved by GPT-3.5-Turbo, n=7, temperature=1, stop=["\nclass", "\ndef", "\nif", "\nprint"]: the `vowels_count` function presented earlier.
3. Solved by GPT-4, n=1, temperature=0:
+
```python
def string_xor(a: str, b: str) -> str:
""" Input are two strings a and b consisting only of 1s and 0s.
@@ -99,7 +102,9 @@ def string_xor(a: str, b: str) -> str:
'100'
"""
```
+
4. Solved by GPT-4, n=2, temperature=1, stop=["\nclass", "\ndef", "\nif", "\nprint"]:
+
```python
def is_palindrome(string: str) -> bool:
""" Test if given string is a palindrome """
@@ -119,7 +124,9 @@ def make_palindrome(string: str) -> str:
'catac'
"""
```
+
5. Solved by GPT-4, n=1, temperature=1, stop=["\nclass", "\ndef", "\nif", "\nprint"]:
+
```python
def sort_array(arr):
"""
@@ -135,8 +142,9 @@ def sort_array(arr):
```
The last problem is an example with wrong example test cases in the original definition. It misleads the adaptive solution because a correct implementation is regarded as wrong and more trials are made. The last configuration in the sequence returns the right implementation, even though it does not pass the auto-generated assertions. This example demonstrates that:
-* Our adaptive solution has a certain degree of fault tolerance.
-* The success rate and inference cost for the adaptive solution can be further improved if correct example test cases are used.
+
+- Our adaptive solution has a certain degree of fault tolerance.
+- The success rate and inference cost for the adaptive solution can be further improved if correct example test cases are used.
It is worth noting that the reduced inference cost is the amortized cost over all the tasks. For each individual task, the cost can be either larger or smaller than directly using GPT-4. This is the nature of the adaptive solution: The cost is in general larger for difficult tasks than that for easy tasks.
@@ -147,8 +155,9 @@ An example notebook to run this experiment can be found at: https://github.com/m
Our solution is quite simple to implement using a generic interface offered in [`autogen`](/docs/Use-Cases/enhanced_inference#logic-error), yet the result is quite encouraging.
While the specific way of generating assertions is application-specific, the main ideas are general in LLM operations:
-* Generate multiple responses to select - especially useful when selecting a good response is relatively easier than generating a good response at one shot.
-* Consider multiple configurations to generate responses - especially useful when:
+
+- Generate multiple responses to select - especially useful when selecting a good response is relatively easier than generating a good response at one shot.
+- Consider multiple configurations to generate responses - especially useful when:
- Model and other inference parameter choice affect the utility-cost tradeoff; or
- Different configurations have complementary effect.
@@ -156,13 +165,14 @@ A [previous blog post](/blog/2023/04/21/LLM-tuning-math) provides evidence that
`autogen` uses a technique [EcoOptiGen](https://arxiv.org/abs/2303.04673) to support inference parameter tuning and model selection.
There are many directions of extensions in research and development:
-* Generalize the way to provide feedback.
-* Automate the process of optimizing the configurations.
-* Build adaptive agents for different applications.
-*Do you find this approach applicable to your use case? Do you have any other challenge to share about LLM applications? Do you like to see more support or research of LLM optimization or automation? Please join our [Discord](https://discord.gg/pAbnFJrkgZ) server for discussion.*
+- Generalize the way to provide feedback.
+- Automate the process of optimizing the configurations.
+- Build adaptive agents for different applications.
+
+_Do you find this approach applicable to your use case? Do you have any other challenge to share about LLM applications? Do you like to see more support or research of LLM optimization or automation? Please join our [Discord](https://aka.ms/autogen-dc) server for discussion._
## For Further Reading
-* [Documentation](/docs/Getting-Started) about `autogen` and [Research paper](https://arxiv.org/abs/2303.04673).
-* [Blog post](/blog/2023/04/21/LLM-tuning-math) about a related study for math.
+- [Documentation](/docs/Getting-Started) about `autogen` and [Research paper](https://arxiv.org/abs/2303.04673).
+- [Blog post](/blog/2023/04/21/LLM-tuning-math) about a related study for math.
diff --git a/website/blog/2023-06-28-MathChat/index.mdx b/website/blog/2023-06-28-MathChat/index.mdx
index 62b024818..4c1007c61 100644
--- a/website/blog/2023-06-28-MathChat/index.mdx
+++ b/website/blog/2023-06-28-MathChat/index.mdx
@@ -6,9 +6,10 @@ tags: [LLM, GPT, research]
![MathChat WorkFlow](img/mathchatflow.png)
**TL;DR:**
-* **We introduce MathChat, a conversational framework leveraging Large Language Models (LLMs), specifically GPT-4, to solve advanced mathematical problems.**
-* **MathChat improves LLM's performance on challenging math problem-solving, outperforming basic prompting and other strategies by about 6%. The improvement was especially notable in the Algebra category, with a 15% increase in accuracy.**
-* **Despite the advancement, GPT-4 still struggles to solve very challenging math problems, even with effective prompting strategies. Further improvements are needed, such as the development of more specific assistant models or the integration of new tools and prompts.**
+
+- **We introduce MathChat, a conversational framework leveraging Large Language Models (LLMs), specifically GPT-4, to solve advanced mathematical problems.**
+- **MathChat improves LLM's performance on challenging math problem-solving, outperforming basic prompting and other strategies by about 6%. The improvement was especially notable in the Algebra category, with a 15% increase in accuracy.**
+- **Despite the advancement, GPT-4 still struggles to solve very challenging math problems, even with effective prompting strategies. Further improvements are needed, such as the development of more specific assistant models or the integration of new tools and prompts.**
Recent Large Language Models (LLMs) like GTP-3.5 and GPT-4 have demonstrated astonishing abilities over previous models on various tasks, such as text generation, question answering, and code generation. Moreover, these models can communicate with humans through conversations and remember previous contexts, making it easier for humans to interact with them. These models play an increasingly important role in our daily lives assisting people with different tasks, such as writing emails, summarizing documents, and writing code.
@@ -25,27 +26,28 @@ The proxy agent first presents a math problem to the LLM assistant, framed by an
- **Tool-using Prompt:** This guides the LLM assistant to use Python code in the correct format to address the problem, allowing the user proxy agent to parse the code and provide the appropriate results.
- **Problem-Solving Strategy Selection Prompt:** The assistant is instructed to choose one of three potential problem-solving strategies, including:
+
1. Writing a Python program to solve the problem directly.
2. Solving the problem directly without using Python, which exercises LLMs' inherent reasoning capabilities.
3. Solving the problem step by step with Python. This should be used when the first two strategies aren't suitable.
-- **Final Answer Encapsulation Prompt:** This part instructs the assistant to put the final answer in *\boxed*{}.
+- **Final Answer Encapsulation Prompt:** This part instructs the assistant to put the final answer in _\boxed_{}.
The prompt design in MathChat is flexible and can easily be refined to include other tools, such as Wolfram Alpha.
Let's take a look at an example between the **User Proxy Agent** and the **LLM Assistant** (GPT-4). The conversation focuses on how to solve inequality using Python.
(The conversation is modified for readability.)
-| Participant | Dialogue |
-|-------------|----------|
-| User Proxy Agent | (Initial Prompt: Let's solve a math problem with Python...) Problem: Find all x that satisfy the inequality (2x+10)(x+3) < (3x+9)(x+8). Express your answer in interval notation. |
-| LLM assistant | The key idea to solve this problem is to find the critical points, which are the points where the inequality is equal. Then, test the intervals between the critical points to find the interval for which the inequality holds. |
-| LLM assistant (continued) | Here's Python code to find the critical points: `from sympy import symbols, Eq, solve; x = symbols("x"); lhs = (2*x + 10)*(x + 3); rhs = (3*x + 9)*(x + 8); equation = Eq(lhs, rhs); critical_points = solve(equation, x); critical_points` |
-| User Proxy Agent | The critical points are: [-14, -3] |
-| LLM assistant | We found the critical points to be x = -14 and x = -3. Now we'll test the intervals between these critical points as well as points before and after the critical points. The intervals are: x < -14, -14 < x < -3, x > -3. |
+| Participant | Dialogue |
+| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| User Proxy Agent | (Initial Prompt: Let's solve a math problem with Python...) Problem: Find all x that satisfy the inequality (2x+10)(x+3) < (3x+9)(x+8). Express your answer in interval notation. |
+| LLM assistant | The key idea to solve this problem is to find the critical points, which are the points where the inequality is equal. Then, test the intervals between the critical points to find the interval for which the inequality holds. |
+| LLM assistant (continued) | Here's Python code to find the critical points: `from sympy import symbols, Eq, solve; x = symbols("x"); lhs = (2*x + 10)*(x + 3); rhs = (3*x + 9)*(x + 8); equation = Eq(lhs, rhs); critical_points = solve(equation, x); critical_points` |
+| User Proxy Agent | The critical points are: [-14, -3] |
+| LLM assistant | We found the critical points to be x = -14 and x = -3. Now we'll test the intervals between these critical points as well as points before and after the critical points. The intervals are: x < -14, -14 < x < -3, x > -3. |
| LLM assistant (continued) | Here's Python code to test the inequality for each interval: `from sympy import symbols; x = symbols("x"); inequality = (2*x + 10)*(x + 3) < (3*x + 9)*(x + 8); test_points = [-15, -5, 0]; intervals = []; for point in test_points: if inequality.subs(x, point): intervals.append(point); intervals` |
-| User Proxy Agent | The intervals where the inequality holds true are for test points: [-15, 0] |
-| LLM assistant | We have tested the intervals and found that the inequality holds for: 1. x < -14 (test point: -15) 2. x > -3 (test point: 0). Therefore, the solution to the inequality (2x+10)(x+3) < (3x+9)(x+8) is x ∈ (-∞, -14) ∪ (-3, ∞) |
+| User Proxy Agent | The intervals where the inequality holds true are for test points: [-15, 0] |
+| LLM assistant | We have tested the intervals and found that the inequality holds for: 1. x < -14 (test point: -15) 2. x > -3 (test point: 0). Therefore, the solution to the inequality (2x+10)(x+3) < (3x+9)(x+8) is x ∈ (-∞, -14) ∪ (-3, ∞) |
## Experiment Setup
@@ -53,16 +55,15 @@ We evaluate the improvement brought by MathChat.
For the experiment, we focus on the level-5 problems from the MATH dataset, which are composed of high school competition problems. These problems include the application of theorems and complex equation derivation and are challenging even for undergraduate students. We evaluate 6 of 7 categories from the dataset (excluding Geometry): Prealgebra, Algebra, Number Theory, Counting and Probability, Intermediate Algebra, and Precalculus.
-We evaluate GPT-4 and use the default configuration of the OpenAI API. To access the final performance, we manually compare the final answer with the correct answer. For the vanilla prompt, Program Synthesis, and MathChat, we have GPT-4 enclose the final answer in *\boxed*{}, and we take the return of the function in PoT as the final answer.
-
+We evaluate GPT-4 and use the default configuration of the OpenAI API. To access the final performance, we manually compare the final answer with the correct answer. For the vanilla prompt, Program Synthesis, and MathChat, we have GPT-4 enclose the final answer in _\boxed_{}, and we take the return of the function in PoT as the final answer.
We also evaluate the following methods for comparison:
-1. **Vanilla prompting:** Evaluates GPT-4's direct problem-solving capability. The prompt used is: *" Solve the problem carefully. Put the final answer in \boxed{}"*.
+1. **Vanilla prompting:** Evaluates GPT-4's direct problem-solving capability. The prompt used is: _" Solve the problem carefully. Put the final answer in \boxed{}"_.
-2. **Program of Thoughts (PoT):** Uses a zero-shot PoT prompt that requests the model to create a *Solver* function to solve the problem and return the final answer.
+2. **Program of Thoughts (PoT):** Uses a zero-shot PoT prompt that requests the model to create a _Solver_ function to solve the problem and return the final answer.
-3. **Program Synthesis (PS) prompting:** Like PoT, it prompts the model to write a program to solve the problem. The prompt used is: *"Write a program that answers the following question: \{Problem\}"*.
+3. **Program Synthesis (PS) prompting:** Like PoT, it prompts the model to write a program to solve the problem. The prompt used is: _"Write a program that answers the following question: \{Problem\}"_.
## Experiment Results
@@ -82,13 +83,14 @@ We now provide an implementation of MathChat using the interactive agents in Aut
Despite MathChat's improvements over previous methods, the results show that complex math problem is still challenging for recent powerful LLMs, like GPT-4, even with help from external tools.
Further work can be done to enhance this framework or math problem-solving in general:
+
- Although enabling the model to use tools like Python can reduce calculation errors, LLMs are still prone to logic errors. Methods like self-consistency (Sample several solutions and take a major vote on the final answer), or self-verification (use another LLM instance to check whether an answer is correct) might improve the performance.
- Sometimes, whether the LLM can solve the problem depends on the plan it uses. Some plans require less computation and logical reasoning, leaving less room for mistakes.
- MathChat has the potential to be adapted into a copilot system, which could assist users with math problems. This system could allow users to be more involved in the problem-solving process, potentially enhancing learning.
## For Further Reading
-* [Research paper of MathChat](https://arxiv.org/abs/2306.01337)
-* [Documentation about `autogen`](/docs/Getting-Started)
+- [Research paper of MathChat](https://arxiv.org/abs/2306.01337)
+- [Documentation about `autogen`](/docs/Getting-Started)
-*Are you working on applications that involve math problem-solving? Would you appreciate additional research or support on the application of LLM-based agents for math problem-solving? Please join our [Discord](https://discord.gg/pAbnFJrkgZ) server for discussion.*
+_Are you working on applications that involve math problem-solving? Would you appreciate additional research or support on the application of LLM-based agents for math problem-solving? Please join our [Discord](https://aka.ms/autogen-dc) server for discussion._
diff --git a/website/blog/2023-10-26-TeachableAgent/index.mdx b/website/blog/2023-10-26-TeachableAgent/index.mdx
index 6d687fd4b..ca3992489 100644
--- a/website/blog/2023-10-26-TeachableAgent/index.mdx
+++ b/website/blog/2023-10-26-TeachableAgent/index.mdx
@@ -7,11 +7,12 @@ tags: [LLM, teach]
![Teachable Agent Architecture](img/teachable-arch.png)
**TL;DR:**
-* We introduce **Teachable Agents** so that users can teach their LLM-based assistants new facts, preferences, and skills.
-* We showcase examples of teachable agents learning and later recalling facts, preferences, and skills in subsequent chats.
+- We introduce **Teachable Agents** so that users can teach their LLM-based assistants new facts, preferences, and skills.
+- We showcase examples of teachable agents learning and later recalling facts, preferences, and skills in subsequent chats.
## Introduction
+
Conversational assistants based on LLMs can remember the current chat with the user, and can also demonstrate in-context learning of user teachings during the conversation. But the assistant's memories and learnings are lost once the chat is over, or when a single chat grows too long for the LLM to handle effectively. Then in subsequent chats the user is forced to repeat any necessary instructions over and over.
`Teachability` addresses these limitations by persisting user teachings across chat boundaries in long-term memory implemented as a vector database. Instead of copying all of memory into the context window, which would eat up valuable space, individual memories (called memos) are retrieved into context as needed. This allows the user to teach frequently used facts and skills to the teachable agent just once, and have it recall them in later chats.
@@ -19,7 +20,6 @@ Conversational assistants based on LLMs can remember the current chat with the u
Any instantiated `agent` that inherits from `ConversableAgent` can be made teachable by instantiating a `Teachability` object and calling its `add_to_agent(agent)` method.
In order to make effective decisions about memo storage and retrieval, the `Teachability` object calls an instance of `TextAnalyzerAgent` (another AutoGen agent) to identify and reformulate text as needed for remembering facts, preferences, and skills. Note that this adds extra LLM calls involving a relatively small number of tokens, which can add a few seconds to the time a user waits for each response.
-
## Run It Yourself
AutoGen contains four code examples that use `Teachability`.
@@ -33,14 +33,17 @@ AutoGen contains four code examples that use `Teachability`.
4. Use the Jupyter notebook [agentchat_teachable_oai_assistants.ipynb](https://github.com/microsoft/autogen/blob/main/notebook/agentchat_teachable_oai_assistants.ipynb) to make arbitrary OpenAI Assistants teachable through `GPTAssistantAgent`.
## Basic Usage of Teachability
+
1. Install dependencies
Please install pyautogen with the [teachable] option before using `Teachability`.
+
```bash
pip install "pyautogen[teachable]"
```
2. Import agents
+
```python
from autogen import UserProxyAgent, config_list_from_json
from autogen.agentchat.contrib.capabilities.teachability import Teachability
@@ -48,6 +51,7 @@ from autogen import ConversableAgent # As an example
```
3. Create llm_config
+
```python
# Load LLM inference endpoints from an env variable or a file
# See https://microsoft.github.io/autogen/docs/FAQ#set-your-api-endpoints
@@ -58,6 +62,7 @@ llm_config={"config_list": config_list, "timeout": 120}
```
4. Create the agents
+
```python
# Start by instantiating any agent that inherits from ConversableAgent, which we use directly here for simplicity.
@@ -80,6 +85,7 @@ user = UserProxyAgent("user", human_input_mode="ALWAYS")
```
5. Chat with the teachable agent
+
```python
# This function will return once the user types 'exit'.
teachable_agent.initiate_chat(user, message="Hi, I'm a teachable user assistant! What's on your mind?")
@@ -89,6 +95,7 @@ teachable_agent.initiate_chat(user, message="Hi, I'm a teachable user assistant!
A user can teach the agent facts about themselves.
(Note that due to their finetuning, LLMs can be reluctant to admit that they know personal information.)
+
```
Loading previous memory (if any) from disk.
teachable_agent (to user):
@@ -113,6 +120,7 @@ Provide feedback to teachable_agent. Press enter to skip and use auto-reply, or
```
In a later conversation, the user can check whether the teachable agent remembers their name. (For readability, the user prompts and some logged notices are not repeated below.)
+
```
--------------------------------------------------------------------------------
user (to teachable_agent):
@@ -127,10 +135,10 @@ Hi Ricky! How can I help you today?
--------------------------------------------------------------------------------
```
-
## Example 2 - Learning new facts
A user can teach the agent more complex, related facts.
+
```
--------------------------------------------------------------------------------
user (to teachable_agent):
@@ -179,6 +187,7 @@ Orca's advanced capabilities allow it to excel in tasks such as text completion,
Then in a later chat the teachable agent can answer questions about the facts it has been taught.
(Remember to first close the previous chat by typing 'exit'.)
+
```
--------------------------------------------------------------------------------
user (to teachable_agent):
@@ -193,13 +202,13 @@ The Vicuna and Orca models are both 13B-parameter language models released by Me
--------------------------------------------------------------------------------
```
-
## Example 3 - Learning user preferences
A user can teach the agent how they prefer to have things done.
Be aware that a message like the next one cannot be entered as a single message through a command line because it contains a newline character.
Such messages can be entered in a Jupyter notebook, or through some UI layer like that of ChatGPT.
+
```
--------------------------------------------------------------------------------
user (to teachable_agent):
@@ -233,6 +242,7 @@ teachable_agent (to user):
```
Then in later chats the teacher doesn't need to reiterate their detailed preferences.
+
```
--------------------------------------------------------------------------------
user (to teachable_agent):
@@ -253,12 +263,12 @@ teachable_agent (to user):
--------------------------------------------------------------------------------
```
-
## Example 4 - Learning new skills
Users can extend the teachable agent's capabilities by teaching it new skills for accomplishing challenging tasks. It usually works best to first describe the task, then (in the same turn) provide a hint or advice for approaching the task.
The [Sparks of AGI](https://arxiv.org/abs/2303.12712) paper evaluated GPT-4 on math problems like the following, which it could only solve 32% of the time. We first show a failure case, then teach the agent a strategy which lifts GPT-4's success rate above 95%.
+
```
--------------------------------------------------------------------------------
user (to teachable_agent):
@@ -346,6 +356,7 @@ The solution is: 9 * 7 + 6 * 6
```
In a later chat the user doesn't need to repeat the detailed advice.
+
```
--------------------------------------------------------------------------------
user (to teachable_agent):
@@ -375,13 +386,12 @@ The solution is: 8 * 3 + 7 * 5
--------------------------------------------------------------------------------
```
-
## Planned improvements
+
- Understanding user instructions distributed over multiple turns.
- Learning from the agent's own experience, to reduce dependence on explicit user teachings.
- Learning skills built on top of previously learned skills.
-
## Conclusion
-`Teachability` is still under active research and development. For any problems you find or improvements you have in mind, please join our discussions in this repo and on our [Discord channel](https://discord.gg/pAbnFJrkgZ). We look forward to seeing how you and the rest of the community can use and improve teachable agents in AutoGen!
+`Teachability` is still under active research and development. For any problems you find or improvements you have in mind, please join our discussions in this repo and on our [Discord channel](https://aka.ms/autogen-dc). We look forward to seeing how you and the rest of the community can use and improve teachable agents in AutoGen!
diff --git a/website/blog/2023-12-01-AutoGenStudio/index.mdx b/website/blog/2023-12-01-AutoGenStudio/index.mdx
index ba862dad5..014aa870e 100644
--- a/website/blog/2023-12-01-AutoGenStudio/index.mdx
+++ b/website/blog/2023-12-01-AutoGenStudio/index.mdx
@@ -7,11 +7,14 @@ authors:
tags: [AutoGen, UI, web, UX]
---
-
-
-
![AutoGen Studio Playground View: Solving a task with multiple agents that generate a pdf document with images.](img/autogenstudio_home.png)
-<p align="center"><em>AutoGen Studio: Solving a task with multiple agents that generate a pdf document with images.</em></p>
+
+<p align="center">
+ <em>
+ AutoGen Studio: Solving a task with multiple agents that generate a pdf
+ document with images.
+ </em>
+</p>
## TL;DR
@@ -22,13 +25,12 @@ To help you rapidly prototype multi-agent solutions for your tasks, we are intro
- Explicitly add skills to your agents and accomplish more tasks.
- Publish your sessions to a local gallery.
-AutoGen Studio is open source [code here](https://github.com/microsoft/autogen/tree/main/samples/apps/autogen-studio), and can be installed via pip. Give it a try!
+AutoGen Studio is open source [code here](https://github.com/microsoft/autogen/tree/main/samples/apps/autogen-studio), and can be installed via pip. Give it a try!
```bash
pip install autogenstudio
```
-
## Introduction
The accelerating pace of technology has ushered us into an era where digital assistants (or agents) are becoming integral to our lives. [AutoGen](https://github.com/microsoft/autogen/tree/main/autogen) has emerged as a leading framework for orchestrating the power of agents. In the spirit of expanding this frontier and democratizing this capability, we are thrilled to introduce a new user-friendly interface: **AutoGen Studio**.
@@ -45,7 +47,6 @@ The following guide will help you get AutoGen Studio up and running on your syst
To get started, you need access to a language model. You can get this set up by following the steps in the AutoGen documentation [here](https://microsoft.github.io/autogen/docs/FAQ#set-your-api-endpoints). Configure your environment with either `OPENAI_API_KEY` or `AZURE_OPENAI_API_KEY`.
-
For example, in your terminal, you would set the API key like this:
```bash
@@ -67,7 +68,6 @@ llm_config = LLMConfig(
)
```
-
### Installation
There are two ways to install AutoGen Studio - from PyPi or from source. We **recommend installing from PyPi** unless you plan to modify the source code.
@@ -103,7 +103,6 @@ There are two ways to install AutoGen Studio - from PyPi or from source. We **re
For Windows users, to build the frontend, you may need alternative commands provided in the [autogen studio readme](https://github.com/microsoft/autogen/tree/main/samples/apps/autogen-studio).
-
### Running the Application
Once installed, run the web UI by entering the following in your terminal:
@@ -117,6 +116,7 @@ This will start the application on the specified port. Open your web browser and
Now that you have AutoGen Studio installed and running, you are ready to explore its capabilities, including defining and modifying agent workflows, interacting with agents and sessions, and expanding agent skills.
## What Can You Do with AutoGen Studio?
+
The AutoGen Studio UI is organized into 3 high level sections - **Build**, **Playground**, and **Gallery**.
### Build
@@ -128,20 +128,28 @@ This section focuses on defining the properties of agents and agent workflows. I
**Skills**: Skills are functions (e.g., Python functions) that describe how to solve a task. In general, a good skill has a descriptive name (e.g. `generate_images`), extensive docstrings and good defaults (e.g., writing out files to disk for persistence and reuse). You can add new skills to AutoGen Studio via the provided UI. At inference time, these skills are made available to the assistant agent as they address your tasks.
![View and add skills.](img/autogenstudio_skills.png)
-<p align="center"><em>AutoGen Studio Build View: View, add or edit skills that an agent can leverage in addressing tasks.</em></p>
+<p align="center">
+ <em>
+ AutoGen Studio Build View: View, add or edit skills that an agent can
+ leverage in addressing tasks.
+ </em>
+</p>
**Agents**: This provides an interface to declaratively specify properties for an AutoGen agent (mirrors most of the members of a base [AutoGen conversable agent](https://github.com/microsoft/autogen/blob/main/autogen/agentchat/conversable_agent.py) class).
**Agent Workflows**: An agent workflow is a specification of a set of agents that can work together to accomplish a task. The simplest version of this is a setup with two agents – a user proxy agent (that represents a user i.e. it compiles code and prints result) and an assistant that can address task requests (e.g., generating plans, writing code, evaluating responses, proposing error recovery steps, etc.). A more complex flow could be a group chat where even more agents work towards a solution.
-
-
### Playground
![AutoGen Studio Playground View: Solving a task with multiple agents that generate a pdf document with images.](img/autogenstudio_home.png)
-<p align="center"><em>AutoGen Studio Playground View: Agents collaborate, use available skills (ability to generate images) to address a user task (generate pdf's).</em></p>
+<p align="center">
+ <em>
+ AutoGen Studio Playground View: Agents collaborate, use available skills
+ (ability to generate images) to address a user task (generate pdf's).
+ </em>
+</p>
The playground section is focused on interacting with agent workflows defined in the previous build section. It includes the following concepts:
@@ -153,16 +161,11 @@ The playground section is focused on interacting with agent workflows defined in
This section is focused on sharing and reusing artifacts (e.g., workflow configurations, sessions, etc.).
-
-
-
AutoGen Studio comes with 3 example skills: `fetch_profile`, `find_papers`, `generate_images`. Please feel free to review the repo to learn more about how they work.
-
-
## The AutoGen Studio API
-While AutoGen Studio is a web interface, it is powered by an underlying python API that is reusable and modular. Importantly, we have implemented an API where agent workflows can be declaratively specified (in JSON), loaded and run. An example of the current API is shown below. Please consult the [AutoGen Studio repo](https://github.com/microsoft/autogen/tree/main/samples/apps/autogen-studio) for more details.
+While AutoGen Studio is a web interface, it is powered by an underlying python API that is reusable and modular. Importantly, we have implemented an API where agent workflows can be declaratively specified (in JSON), loaded and run. An example of the current API is shown below. Please consult the [AutoGen Studio repo](https://github.com/microsoft/autogen/tree/main/samples/apps/autogen-studio) for more details.
```python
import json
@@ -191,8 +194,6 @@ As we continue to develop and refine AutoGen Studio, the road map below outlines
- **Expansion of Agent Skills**: We will work towards improving the workflow for authoring, composing and reusing agent skills.
- **Community Features**: Facilitation of sharing and collaboration within AutoGen Studio user community is a key goal. We're exploring options for sharing sessions and results more easily among users and contributing to a shared repository of skills, agents, and agent workflows.
-
-
## Contribution Guide
We welcome contributions to AutoGen Studio. We recommend the following general steps to contribute to the project:
@@ -205,7 +206,6 @@ We welcome contributions to AutoGen Studio. We recommend the following general s
- If you are modifying AutoGen Studio in vscode, it has its own devcontainer to simplify dev work. See instructions in `.devcontainer/README.md` on how to use it.
- Please use the tag `studio` for any issues, questions, and PRs related to Studio.
-
### FAQ
**Q: Where can I adjust the default skills, agent and workflow configurations?**
@@ -218,19 +218,17 @@ A: To reset your conversation history, you can delete the `database.sqlite` file
A: Yes, you can view the generated messages in the debug console of the web UI, providing insights into the agent interactions. Alternatively, you can inspect the `database.sqlite` file for a comprehensive record of messages.
**Q: Where can I find documentation and support for AutoGen Studio?**
-A: We are constantly working to improve AutoGen Studio. For the latest updates, please refer to the [AutoGen Studio Readme](https://github.com/microsoft/autogen/tree/main/samples/apps/autogen-studio). For additional support, please open an issue on [GitHub](https://github.com/microsoft/autogen) or ask questions on [Discord](https://discord.gg/pAbnFJrkgZ).
+A: We are constantly working to improve AutoGen Studio. For the latest updates, please refer to the [AutoGen Studio Readme](https://github.com/microsoft/autogen/tree/main/samples/apps/autogen-studio). For additional support, please open an issue on [GitHub](https://github.com/microsoft/autogen) or ask questions on [Discord](https://aka.ms/autogen-dc).
**Q: Can I use Other Models with AutoGen Studio?**
Yes. AutoGen standardizes on the openai model api format, and you can use any api server that offers an openai compliant endpoint. In the AutoGen Studio UI, each agent has an `llm_config` field where you can input your model endpoint details including `model name`, `api key`, `base url`, `model type` and `api version`. For Azure OpenAI models, you can find these details in the Azure portal. Note that for Azure OpenAI, the `model name` is the deployment id or engine, and the `model type` is "azure".
For other OSS models, we recommend using a server such as vllm to instantiate an openai compliant endpoint.
**Q: The Server Starts But I Can't Access the UI**
-A: If you are running the server on a remote machine (or a local machine that fails to resolve localhost correstly), you may need to specify the host address. By default, the host address is set to `localhost`. You can specify the host address using the `--host <host>` argument. For example, to start the server on port 8081 and local address such that it is accessible from other machines on the network, you can run the following command:
+A: If you are running the server on a remote machine (or a local machine that fails to resolve localhost correstly), you may need to specify the host address. By default, the host address is set to `localhost`. You can specify the host address using the `--host <host>` argument. For example, to start the server on port 8081 and local address such that it is accessible from other machines on the network, you can run the following command:
```bash
autogenstudio ui --port 8081 --host 0.0.0.0
```
-
-
<br />
diff --git a/website/blog/2024-01-25-AutoGenBench/index.mdx b/website/blog/2024-01-25-AutoGenBench/index.mdx
index 517fecc8a..28cdcd8e6 100644
--- a/website/blog/2024-01-25-AutoGenBench/index.mdx
+++ b/website/blog/2024-01-25-AutoGenBench/index.mdx
@@ -7,22 +7,29 @@ tags: [AutoGen]
---
![AutoGenBench](img/teaser.jpg)
-<p align="center"><em>AutoGenBench is a standalone tool for evaluating AutoGen agents and workflows on common benchmarks.</em></p>
+<p align="center">
+ <em>
+ AutoGenBench is a standalone tool for evaluating AutoGen agents and
+ workflows on common benchmarks.
+ </em>
+</p>
## TL;DR
+
Today we are releasing AutoGenBench - a tool for evaluating AutoGen agents and workflows on established LLM and agentic benchmarks.
AutoGenBench is a standalone command line tool, installable from PyPI, which handles downloading, configuring, running, and reporting supported benchmarks. AutoGenBench works best when run alongside Docker, since it uses Docker to isolate tests from one another.
-* See the [AutoGenBench README](https://github.com/microsoft/autogen/blob/main/samples/tools/autogenbench/README.md) for information on installation and running benchmarks.
-* See the [AutoGenBench CONTRIBUTING guide](https://github.com/microsoft/autogen/blob/main/samples/tools/autogenbench/CONTRIBUTING.md) for information on developing or contributing benchmark datasets.
-
+- See the [AutoGenBench README](https://github.com/microsoft/autogen/blob/main/samples/tools/autogenbench/README.md) for information on installation and running benchmarks.
+- See the [AutoGenBench CONTRIBUTING guide](https://github.com/microsoft/autogen/blob/main/samples/tools/autogenbench/CONTRIBUTING.md) for information on developing or contributing benchmark datasets.
### Quick Start
+
Get started quickly by running the following commands in a bash terminal.
-*Note:* You may need to adjust the path to the `OAI_CONFIG_LIST`, as appropriate.
+_Note:_ You may need to adjust the path to the `OAI_CONFIG_LIST`, as appropriate.
+
```sh
export OAI_CONFIG_LIST=$(cat ./OAI_CONFIG_LIST)
pip install autogenbench
@@ -34,23 +41,28 @@ autogenbench tabulate Results/human_eval_two_agents
```
## Introduction
+
Measurement and evaluation are core components of every major AI or ML research project. The same is true for AutoGen. To this end, today we are releasing AutoGenBench, a standalone command line tool that we have been using to guide development of AutoGen. Conveniently, AutoGenBench handles: downloading, configuring, running, and reporting results of agents on various public benchmark datasets. In addition to reporting top-line numbers, each AutoGenBench run produces a comprehensive set of logs and telemetry that can be used for debugging, profiling, computing custom metrics, and as input to [AgentEval](https://microsoft.github.io/autogen/blog/2023/11/20/AgentEval). In the remainder of this blog post, we outline core design principles for AutoGenBench (key to understanding its operation); present a guide to installing and running AutoGenBench; outline a roadmap for evaluation; and conclude with an open call for contributions.
## Design Principles
+
AutoGenBench is designed around three core design principles. Knowing these principles will help you understand the tool, its operation and its output. These three principles are:
-- **Repetition:** LLMs are stochastic, and in many cases, so too is the code they write to solve problems. For example, a Python script might call an external search engine, and the results may vary run-to-run. This can lead to variance in agent performance. Repetition is key to measuring and understanding this variance. To this end, AutoGenBench is built from the ground up with an understanding that tasks may be run multiple times, and that variance is a metric we often want to measure.
- - **Isolation:** Agents interact with their worlds in both subtle and overt ways. For example an agent may install a python library or write a file to disk. This can lead to ordering effects that can impact future measurements. Consider, for example, comparing two agents on a common benchmark. One agent may appear more efficient than the other simply because it ran second, and benefitted from the hard work the first agent did in installing and debugging necessary Python libraries. To address this, AutoGenBench isolates each task in its own Docker container. This ensures that all runs start with the same initial conditions. (Docker is also a *much safer way to run agent-produced code*, in general.)
+- **Repetition:** LLMs are stochastic, and in many cases, so too is the code they write to solve problems. For example, a Python script might call an external search engine, and the results may vary run-to-run. This can lead to variance in agent performance. Repetition is key to measuring and understanding this variance. To this end, AutoGenBench is built from the ground up with an understanding that tasks may be run multiple times, and that variance is a metric we often want to measure.
-- **Instrumentation:** While top-line metrics are great for comparing agents or models, we often want much more information about how the agents are performing, where they are getting stuck, and how they can be improved. We may also later think of new research questions that require computing a different set of metrics. To this end, AutoGenBench is designed to log everything, and to compute metrics from those logs. This ensures that one can always go back to the logs to answer questions about what happened, run profiling software, or feed the logs into tools like [AgentEval](https://microsoft.github.io/autogen/blog/2023/11/20/AgentEval).
+- **Isolation:** Agents interact with their worlds in both subtle and overt ways. For example an agent may install a python library or write a file to disk. This can lead to ordering effects that can impact future measurements. Consider, for example, comparing two agents on a common benchmark. One agent may appear more efficient than the other simply because it ran second, and benefitted from the hard work the first agent did in installing and debugging necessary Python libraries. To address this, AutoGenBench isolates each task in its own Docker container. This ensures that all runs start with the same initial conditions. (Docker is also a _much safer way to run agent-produced code_, in general.)
+
+- **Instrumentation:** While top-line metrics are great for comparing agents or models, we often want much more information about how the agents are performing, where they are getting stuck, and how they can be improved. We may also later think of new research questions that require computing a different set of metrics. To this end, AutoGenBench is designed to log everything, and to compute metrics from those logs. This ensures that one can always go back to the logs to answer questions about what happened, run profiling software, or feed the logs into tools like [AgentEval](https://microsoft.github.io/autogen/blog/2023/11/20/AgentEval).
## Installing and Running AutoGenBench
+
As noted above, isolation is a key design principle, and so AutoGenBench must be run in an environment where Docker is available (desktop or Engine). **It will not run in GitHub codespaces**, unless you opt for native execution (which is strongly discouraged). To install Docker Desktop see [https://www.docker.com/products/docker-desktop/](https://www.docker.com/products/docker-desktop/).
Once Docker is installed, AutoGenBench can then be installed as a standalone tool from PyPI. With `pip`, installation can be achieved as follows:
```sh
pip install autogenbench
```
+
After installation, you must configure your API keys. As with other AutoGen applications, AutoGenBench will look for the OpenAI keys in the OAI_CONFIG_LIST file in the current working directory, or the OAI_CONFIG_LIST environment variable. This behavior can be overridden using a command-line parameter.
If you will be running multiple benchmarks, it is often most convenient to leverage the environment variable option. You can load your keys into the environment variable by executing:
@@ -58,7 +70,9 @@ If you will be running multiple benchmarks, it is often most convenient to lever
```sh
export OAI_CONFIG_LIST=$(cat ./OAI_CONFIG_LIST)
```
+
## A Typical Session
+
Once AutoGenBench and necessary keys are installed, a typical session will look as follows:
```
@@ -70,10 +84,11 @@ autogenbench tabulate results/human_eval_two_agents
```
Where:
+
- `autogenbench clone HumanEval` downloads and expands the HumanEval benchmark scenario.
- `cd HumanEval; cat README.md` navigates to the benchmark directory, and prints the README (which you should always read!)
- `autogenbench run --subsample 0.1 --repeat 3 Tasks/human_eval_two_agents.jsonl`
- runs a 10% subsample of the tasks defined in `Tasks/human_eval_two_agents.jsonl`. Each task is run 3 times.
+ runs a 10% subsample of the tasks defined in `Tasks/human_eval_two_agents.jsonl`. Each task is run 3 times.
- `autogenbench tabulate results/human_eval_two_agents` tabulates the results of the run.
After running the above `tabulate` command, you should see output similar to the following:
@@ -108,7 +123,7 @@ CAUTION: 'autogenbench tabulate' is in early preview.
Please do not cite these values in academic work without first inspecting and verifying the results in the logs yourself.
```
-From this output we can see the results of the three separate repetitions of each task, and final summary statistics of each run. In this case, the results were generated via GPT-4 (as defined in the OAI_CONFIG_LIST that was provided), and used the `TwoAgents` template. **It is important to remember that AutoGenBench evaluates *specific* end-to-end configurations of agents (as opposed to evaluating a model or cognitive framework more generally).**
+From this output we can see the results of the three separate repetitions of each task, and final summary statistics of each run. In this case, the results were generated via GPT-4 (as defined in the OAI_CONFIG_LIST that was provided), and used the `TwoAgents` template. **It is important to remember that AutoGenBench evaluates _specific_ end-to-end configurations of agents (as opposed to evaluating a model or cognitive framework more generally).**
Finally, complete execution traces and logs can be found in the `Results` folder. See the [AutoGenBench README](https://github.com/microsoft/autogen/blob/main/samples/tools/autogenbench/README.md) for more details about command-line options and output formats. Each of these commands also offers extensive in-line help via:
@@ -117,15 +132,17 @@ Finally, complete execution traces and logs can be found in the `Results` folder
- `autogenbench run --help`
- `autogenbench tabulate --help`
-
## Roadmap
+
While we are announcing AutoGenBench, we note that it is very much an evolving project in its own right. Over the next few weeks and months we hope to:
+
- Onboard many additional benchmarks beyond those shipping today
- Greatly improve logging and telemetry
- Introduce new core metrics including total costs, task completion time, conversation turns, etc.
- Provide tighter integration with AgentEval and AutoGen Studio
-For an up to date tracking of our work items on this project, please see [AutoGenBench Work Items]( https://github.com/microsoft/autogen/issues/973)
+For an up to date tracking of our work items on this project, please see [AutoGenBench Work Items](https://github.com/microsoft/autogen/issues/973)
## Call for Participation
-Finally, we want to end this blog post with an open call for contributions. AutoGenBench is still nascent, and has much opportunity for improvement. New benchmarks are constantly being published, and will need to be added. Everyone may have their own distinct set of metrics that they care most about optimizing, and these metrics should be onboarded. To this end, we welcome any and all contributions to this corner of the AutoGen project. If contributing is something that interests you, please see the [contributor’s guide](https://github.com/microsoft/autogen/blob/main/samples/tools/autogenbench/CONTRIBUTING.md) and join our [Discord](https://discord.gg/pAbnFJrkgZ) discussion in the [#autogenbench](https://discord.com/channels/1153072414184452236/1199851779328847902) channel!
+
+Finally, we want to end this blog post with an open call for contributions. AutoGenBench is still nascent, and has much opportunity for improvement. New benchmarks are constantly being published, and will need to be added. Everyone may have their own distinct set of metrics that they care most about optimizing, and these metrics should be onboarded. To this end, we welcome any and all contributions to this corner of the AutoGen project. If contributing is something that interests you, please see the [contributor’s guide](https://github.com/microsoft/autogen/blob/main/samples/tools/autogenbench/CONTRIBUTING.md) and join our [Discord](https://aka.ms/autogen-dc) discussion in the [#autogenbench](https://discord.com/channels/1153072414184452236/1199851779328847902) channel!
diff --git a/website/blog/2024-03-03-AutoGen-Update/index.mdx b/website/blog/2024-03-03-AutoGen-Update/index.mdx
index 155715625..da036094c 100644
--- a/website/blog/2024-03-03-AutoGen-Update/index.mdx
+++ b/website/blog/2024-03-03-AutoGen-Update/index.mdx
@@ -7,6 +7,7 @@ tags: [news, summary, roadmap]
![autogen is loved](img/love.png)
**TL;DR**
+
- **AutoGen has received tremendous interest and recognition.**
- **AutoGen has many exciting new features and ongoing research.**
@@ -15,6 +16,7 @@ Five months have passed since the initial spinoff of AutoGen from [FLAML](https:
## Background
AutoGen was motivated by two big questions:
+
- What are future AI applications like?
- How do we empower every developer to build them?
@@ -28,14 +30,14 @@ In October, we moved AutoGen from FLAML to a standalone repo on GitHub, and publ
Since then, we've got new feedback every day, everywhere. Users have shown really high recognition of the new levels of capability enabled by AutoGen. For example, there are many comments like the following on X (Twitter) or YouTube.
> Autogen gave me the same a-ha moment that I haven't felt since trying out GPT-3
-for the first time.
+> for the first time.
> I have never been this surprised since ChatGPT.
-
Many users have deep understanding of the value in different dimensions, such as the modularity, flexibility and simplicity.
> The same reason autogen is significant is the same reason OOP is a good idea. Autogen packages up all that complexity into an agent I can create in one line, or modify with another.
+
<!--
I had lots of ideas I wanted to implement, but it needed a framework like this
and I am just not the guy to make such a robust and intelligent framework.
@@ -44,42 +46,47 @@ and I am just not the guy to make such a robust and intelligent framework.
Over time, more and more users share their experiences in using or contributing to autogen.
> In our Data Science department Autogen is helping us develop a production ready
-multi-agents framework.
->> Sam Khalil, VP Data Insights & FounData, Novo Nordisk
+> multi-agents framework.
+>
+> > Sam Khalil, VP Data Insights & FounData, Novo Nordisk
> When I built an interactive learning tool for students, I looked for a tool that
-could streamline the logistics but also give enough flexibility so I could use
-customized tools. AutoGen has both. It simplified the work. Thanks to Chi and his
-team for sharing such a wonderful tool with the community.
->> Yongsheng Lian, Professor at the University of Louisville, Mechanical Engineering
+> could streamline the logistics but also give enough flexibility so I could use
+> customized tools. AutoGen has both. It simplified the work. Thanks to Chi and his
+> team for sharing such a wonderful tool with the community.
+>
+> > Yongsheng Lian, Professor at the University of Louisville, Mechanical Engineering
> Exciting news: the latest AutoGen release now features my contribution…
-This experience has been a wonderful blend of learning and contributing,
-demonstrating the dynamic and collaborative spirit of the tech community.
->> Davor Runje, Cofounder @ airt / President of the board @ CISEx
+> This experience has been a wonderful blend of learning and contributing,
+> demonstrating the dynamic and collaborative spirit of the tech community.
+>
+> > Davor Runje, Cofounder @ airt / President of the board @ CISEx
> With the support of a grant through the Data Intensive Studies Center at Tufts
-University, our group is hoping to solve some of the challenges students face when
-transitioning from undergraduate to graduate-level courses, particularly in Tufts'
-Doctor of Physical Therapy program in the School of Medicine. We're experimenting
-with Autogen to create tailored assessments, individualized study guides, and focused
-tutoring. This approach has led to significantly better results than those we
-achieved using standard chatbots. With the help of Chi and his group at Microsoft,
-our current experiments include using multiple agents in sequential chat, teachable
-agents, and round-robin style debate formats. These methods have proven more
-effective in generating assessments and feedback compared to other large language
-models (LLMs) we've explored. I've also used OpenAI Assistant agents through Autogen
-in my Primary Care class to facilitate student engagement in patient interviews
-through digital simulations. The agent retrieved information from a real patient
-featured in a published case study, allowing students to practice their interview
-skills with realistic information.
->> Benjamin D Stern, MS, DPT, Assistant Professor, Doctor of Physical Therapy Program,
-Tufts University School of Medicine
+> University, our group is hoping to solve some of the challenges students face when
+> transitioning from undergraduate to graduate-level courses, particularly in Tufts'
+> Doctor of Physical Therapy program in the School of Medicine. We're experimenting
+> with Autogen to create tailored assessments, individualized study guides, and focused
+> tutoring. This approach has led to significantly better results than those we
+> achieved using standard chatbots. With the help of Chi and his group at Microsoft,
+> our current experiments include using multiple agents in sequential chat, teachable
+> agents, and round-robin style debate formats. These methods have proven more
+> effective in generating assessments and feedback compared to other large language
+> models (LLMs) we've explored. I've also used OpenAI Assistant agents through Autogen
+> in my Primary Care class to facilitate student engagement in patient interviews
+> through digital simulations. The agent retrieved information from a real patient
+> featured in a published case study, allowing students to practice their interview
+> skills with realistic information.
+>
+> > Benjamin D Stern, MS, DPT, Assistant Professor, Doctor of Physical Therapy Program,
+> > Tufts University School of Medicine
> Autogen has been a game changer for how we analyze companies and products! Through
-collaborative discourse between AI Agents we are able to shave days off our research
-and analysis process.
->> Justin Trugman, Cofounder & Head of Technology at BetterFutureLabs
+> collaborative discourse between AI Agents we are able to shave days off our research
+> and analysis process.
+>
+> > Justin Trugman, Cofounder & Head of Technology at BetterFutureLabs
These are just a small fraction of examples. We have seen big enterprise customers’ interest from pretty much every vertical industry: Accounting, Airlines, Biotech, Consulting, Consumer Packaged Goods, Electronics, Entertainment, Finance, Fintech, Government, Healthcare, Manufacturer, Metals, Pharmacy, Research, Retailer, Social Media, Software, Supply Chain, Technology, Telecom…
@@ -88,6 +95,7 @@ AutoGen is used or contributed by companies, organizations, universities from A
## Milestones
AutoGen has a large and active community of developers, researchers and AI practitioners.
+
- 22K+ stars on [GitHub](https://aka.ms/autogen-gh), 3K+ forks
- 14K+ members on [Discord](https://aka.ms/autogen-dc)
- 100K+ downloads per months
@@ -96,6 +104,7 @@ AutoGen has a large and active community of developers, researchers and AI pract
I am so amazed by their creativity and passion.
I also appreciate the recognition and awards AutoGen has received, such as:
+
- Selected by [TheSequence: My Five Favorite AI Papers of 2023](https://thesequence.substack.com/p/my-five-favorite-ai-papers-of-2023)
- Top trending repo on GitHub in Oct'23
- Selected into [Open100: Top 100 Open Source achievements](https://www.benchcouncil.org/evaluation/opencs/annual.html) only 35 days after spinoff
@@ -140,7 +149,7 @@ We are making rapid progress in further improving the interface to make it even
- [AutoBuild](/blog/2023/11/26/Agent-AutoBuild). AutoBuild is an ongoing area of research to automatically create or select a group of agents for a given task and objective. If successful, it will greatly reduce the effort from users or developers when using the multi-agent technology. It also paves the way for agentic decomposition to handle complex tasks. It is available as an experimental feature and demonstrated in two modes: free-form [creation](https://github.com/microsoft/autogen/blob/main/notebook/autobuild_basic.ipynb) and [selection](https://github.com/microsoft/autogen/blob/main/notebook/autobuild_agent_library.ipynb) from a library.
- [AutoGen Studio](/blog/2023/12/01/AutoGenStudio). AutoGen Studio is a no-code UI for fast experimentation with the multi-agent conversations. It lowers the barrier of entrance to the AutoGen technology. Models, agents, and workflows can all be configured without writing code. And chatting with multiple agents in a playground is immediately available after the configuration. Although only a subset of `pyautogen` features are available in this sample app, it demonstrates a promising experience. It has generated tremendous excitement in the community.
-- Conversation Programming+. The [AutoGen paper](https://arxiv.org/abs/2308.08155) introduced a key concept of *Conversation Programming*, which can be used to program diverse conversation patterns such as 1-1 chat, group chat, hierarchical chat, nested chat etc. While we offered dynamic group chat as an example of high-level orchestration, it made other patterns relatively less discoverable. Therefore, we have added more convenient conversation programming features which enables easier definition of other types of complex workflow, such as [finite state machine based group chat](/blog/2024/02/11/FSM-GroupChat), [sequential chats](/docs/notebooks/agentchats_sequential_chats), and [nested chats](/docs/notebooks/agentchat_nestedchat). Many users have found them useful in implementing specific patterns, which have been always possible but more obvious with the added features. I will write another blog post for a deep dive.
+- Conversation Programming+. The [AutoGen paper](https://arxiv.org/abs/2308.08155) introduced a key concept of _Conversation Programming_, which can be used to program diverse conversation patterns such as 1-1 chat, group chat, hierarchical chat, nested chat etc. While we offered dynamic group chat as an example of high-level orchestration, it made other patterns relatively less discoverable. Therefore, we have added more convenient conversation programming features which enables easier definition of other types of complex workflow, such as [finite state machine based group chat](/blog/2024/02/11/FSM-GroupChat), [sequential chats](/docs/notebooks/agentchats_sequential_chats), and [nested chats](/docs/notebooks/agentchat_nestedchat). Many users have found them useful in implementing specific patterns, which have been always possible but more obvious with the added features. I will write another blog post for a deep dive.
### Learning/Optimization/Teaching
@@ -149,12 +158,13 @@ The features in this category allow agents to remember teachings from users or o
- [AgentOptimizer](/blog/2023/12/23/AgentOptimizer). This [research](https://arxiv.org/abs/2402.11359) finds an approach of training LLM agents without modifying the model. As a case study, this technique optimizes a set of Python functions for agents to use in solving a set of training tasks. It is planned to be available as an experimental feature.
- [EcoAssistant](/blog/2023/11/09/EcoAssistant). This [research](https://arxiv.org/abs/2310.03046) finds a multi-agent teaching approach when using agents with different capacities powered by different LLMs. For example, a GPT-4 agent can teach a GPT-3.5 agent by demonstration. With this approach, one only needs 1/3 or 1/2 of GPT-4's cost, while getting 10-20\% higher success rate than GPT-4 on coding-based QA. No finetuning is needed. All you need is a GPT-4 endpoint and a GPT-3.5-turbo endpoint. Help is appreciated to offer this technique as a feature in the AutoGen library.
- [Teachability](/blog/2023/10/26/TeachableAgent). Every LLM agent in AutoGen can be made teachable, i.e., remember facts, preferences, skills etc. from interacting with other agents. For example, a user behind a user proxy agent can teach an assistant agent instructions in solving a difficult math problem. After teaching once, the problem solving rate for the assistant agent can have a dramatic improvement (e.g., 37% -> 95% for gpt-4-0613).
-![teach](img/teach.png)
-This feature works for GPTAssistantAgent (using OpenAI's assistant API) and group chat as well. One interesting use case of teachability + FSM group chat: [teaching resilience](https://www.linkedin.com/pulse/combatting-ai-naivete-teaching-resilience-emotional-leah-bonser-jdhrc).
+ ![teach](img/teach.png)
+ This feature works for GPTAssistantAgent (using OpenAI's assistant API) and group chat as well. One interesting use case of teachability + FSM group chat: [teaching resilience](https://www.linkedin.com/pulse/combatting-ai-naivete-teaching-resilience-emotional-leah-bonser-jdhrc).
### Integration
The extensible design of AutoGen makes it easy to integrate with new technologies. For example:
+
- [Custom models and clients](/blog/2024/01/26/Custom-Models) can be used as backends of an agent, such as Huggingface models and inference APIs.
- [OpenAI assistant](/blog/2023/11/13/OAI-assistants) can be used as the backend of an agent (GPTAssistantAgent). It will be nice to reimplement it as a custom client to increase the compatibility with ConversableAgent.
- [Multimodality](/blog/2023/11/06/LMM-Agent). LMM models like GPT-4V can be used to provide vision to an agent, and accomplish interesting multimodal tasks by conversing with other agents, including advanced image analysis, figure generation, and automatic iterative improvement in image generation.
@@ -162,6 +172,7 @@ The extensible design of AutoGen makes it easy to integrate with new technologie
![multimodal](img/dalle_gpt4v.png)
The above only covers a subset of new features and roadmap. There are many other interesting new features, integration examples or sample apps:
+
- new features like stateful code execution, [tool decorators](/docs/Use-Cases/agent_chat#tool-calling), [long context handling](https://github.com/microsoft/autogen/blob/main/notebook/agentchat_capability_long_context_handling.ipynb), [web agents](https://github.com/microsoft/autogen/blob/main/notebook/agentchat_surfer.ipynb).
- integration examples like using [guidance](https://github.com/microsoft/autogen/blob/main/notebook/agentchat_guidance.ipynb) to generate structured response.
- sample apps like [AutoAnny](/blog/2024/02/02/AutoAnny).
@@ -173,6 +184,6 @@ Despite all the exciting progress, there are tons of open problems, issues and f
We need more help to tackle the challenging problems and accelerate the development.
You're all welcome to join our community and define the future of AI agents together.
-*Do you find this update helpful? Would you like to join force? Please join our [Discord](https://discord.gg/pAbnFJrkgZ) server for discussion.*
+_Do you find this update helpful? Would you like to join force? Please join our [Discord](https://aka.ms/autogen-dc) server for discussion._
![contributors](img/contributors.png)
diff --git a/website/docs/Contribute.md b/website/docs/Contribute.md
index c25d50b11..92bea4897 100644
--- a/website/docs/Contribute.md
+++ b/website/docs/Contribute.md
@@ -6,7 +6,7 @@ This project welcomes and encourages all forms of contributions, including but n
- Code review of pull requests.
- Documentation, examples and test cases.
- Readability improvement, e.g., improvement on docstr and comments.
-- Community participation in [issues](https://github.com/microsoft/autogen/issues), [discussions](https://github.com/microsoft/autogen/discussions), [discord](https://discord.gg/pAbnFJrkgZ), and [twitter](https://twitter.com/pyautogen).
+- Community participation in [issues](https://github.com/microsoft/autogen/issues), [discussions](https://github.com/microsoft/autogen/discussions), [discord](https://aka.ms/autogen-dc), and [twitter](https://twitter.com/pyautogen).
- Tutorials, blog posts, talks that promote the project.
- Sharing application scenarios and/or related research.
@@ -27,7 +27,7 @@ contact [[email protected]](mailto:[email protected]) with any additio
## Roadmaps
To see what we are working on and what we plan to work on, please check our
-[Roadmap Issues](https://github.com/microsoft/autogen/issues?q=is%3Aopen+is%3Aissue+label%3Aroadmap).
+[Roadmap Issues](https://aka.ms/autogen-roadmap).
## How to make a good bug report
diff --git a/website/docs/Getting-Started.mdx b/website/docs/Getting-Started.mdx
index a933ae3cc..d1ed2a56a 100644
--- a/website/docs/Getting-Started.mdx
+++ b/website/docs/Getting-Started.mdx
@@ -56,6 +56,7 @@ user_proxy.initiate_chat(
</TabItem>
<TabItem value="local" label="Local execution" default>
+
:::warning
When asked, be sure to check the generated code before continuing to ensure it is safe to run.
:::
@@ -103,6 +104,7 @@ with autogen.coding.DockerCommandLineCodeExecutor(work_dir="coding") as code_exe
Open `coding/plot.png` to see the generated plot.
</TabItem>
+
</Tabs>
:::tip
@@ -125,9 +127,9 @@ The figure below shows an example conversation flow with AutoGen.
- Understand the use cases for [multi-agent conversation](/docs/Use-Cases/agent_chat) and [enhanced LLM inference](/docs/Use-Cases/enhanced_inference)
- Read the [API](/docs/reference/agentchat/conversable_agent/) docs
- Learn about [research](/docs/Research) around AutoGen
-- Chat on [Discord](https://discord.gg/pAbnFJrkgZ)
+- Chat on [Discord](https://aka.ms/autogen-dc)
- Follow on [Twitter](https://twitter.com/pyautogen)
-- See our [roadmaps](https://github.com/microsoft/autogen/issues?q=is%3Aopen+is%3Aissue+label%3Aroadmap)
+- See our [roadmaps](https://aka.ms/autogen-roadmap)
If you like our project, please give it a [star](https://github.com/microsoft/autogen/stargazers) on GitHub. If you are interested in contributing, please read [Contributor's Guide](/docs/Contribute).
diff --git a/website/docs/tutorial/what-next.md b/website/docs/tutorial/what-next.md
index 74a28364a..dbbc19c0b 100644
--- a/website/docs/tutorial/what-next.md
+++ b/website/docs/tutorial/what-next.md
@@ -33,13 +33,13 @@ topics:
If you have any questions, you can ask in our [GitHub
Discussions](https://github.com/microsoft/autogen/discussions), or join
-our [Discord Server](https://discord.gg/pAbnFJrkgZ).
+our [Discord Server](https://aka.ms/autogen-dc).
-[![](https://img.shields.io/discord/1153072414184452236?logo=discord&style=flat.png)](https://discord.gg/pAbnFJrkgZ)
+[![](https://img.shields.io/discord/1153072414184452236?logo=discord&style=flat.png)](https://aka.ms/autogen-dc)
## Get Involved
-- Check out [Roadmap Issues](https://github.com/microsoft/autogen/issues?q=is%3Aopen+is%3Aissue+label%3Aroadmap) to see what we are working on.
+- Check out [Roadmap Issues](https://aka.ms/autogen-roadmap) to see what we are working on.
- Contribute your work to our [gallery](../Gallery)
- Follow our [contribution guide](../Contribute) to make a pull request to AutoGen
- You can also share your work with the community on the Discord server.
diff --git a/website/docusaurus.config.js b/website/docusaurus.config.js
index 0f2da9d6d..1d60bb603 100644
--- a/website/docusaurus.config.js
+++ b/website/docusaurus.config.js
@@ -41,60 +41,111 @@ module.exports = {
},
items: [
{
- type: "doc",
- docId: "Getting-Started",
+ type: "dropdown",
position: "left",
label: "Docs",
+ items: [
+ {
+ type: "doc",
+ label: "Getting Started",
+ docId: "Getting-Started",
+ },
+ {
+ type: "doc",
+ label: "Installation",
+ docId: "installation/Installation",
+ },
+ {
+ type: "doc",
+ label: "Tutorial",
+ docId: "tutorial/introduction",
+ },
+ {
+ type: "doc",
+ label: "User Guide",
+ docId: "topics",
+ },
+ {
+ type: "doc",
+ docId: "reference/agentchat/conversable_agent",
+ label: "API Reference",
+ },
+ {
+ type: "doc",
+ docId: "FAQ",
+ label: "FAQs",
+ },
+ {
+ type: "doc",
+ docId: "ecosystem",
+ label: "Ecosystem",
+ },
+ {
+ type: "doc",
+ label: "Contribute",
+ docId: "Contribute",
+ },
+ {
+ type: "doc",
+ label: "Research",
+ docId: "Research",
+ },
+ ],
},
{
- type: "doc",
- docId: "reference/agentchat/conversable_agent",
- position: "left",
- label: "API",
- },
- { to: "blog", label: "Blog", position: "left" },
- {
- type: "doc",
- docId: "FAQ",
- position: "left",
- label: "FAQ",
- },
- // {
- // to: 'examples',
- // label: 'Examples',
- // },
- {
- type: "doc",
- docId: "Examples",
+ type: "dropdown",
position: "left",
label: "Examples",
- },
- {
- to: "docs/notebooks",
- position: "left",
- label: "Notebooks",
- },
- {
- type: "doc",
- position: "left",
- docId: "Gallery",
+ items: [
+ {
+ type: "doc",
+ label: "Examples by Category",
+ docId: "Examples",
+ },
+ {
+ type: "doc",
+ label: "Examples by Notebook",
+ docId: "notebooks",
+ },
+ {
+ type: "doc",
+ label: "Application Gallery",
+ docId: "Gallery",
+ },
+ ],
},
{
label: "Other Languages",
type: "dropdown",
- position: "right",
+ position: "left",
items: [
{
label: "Dotnet",
href: "https://microsoft.github.io/autogen-for-net/",
}
],
+
+ },
+ {
+ to: "blog",
+ label: "Blog",
+ position: "left",
},
{
href: "https://github.com/microsoft/autogen",
label: "GitHub",
position: "right",
- }
+ },
+ {
+ href: "https://aka.ms/autogen-dc",
+ label: "Discord",
+ position: "right",
+ },
+ {
+ href: "https://twitter.com/pyautogen",
+ label: "Twitter",
+ position: "right",
+ },
],
},
footer: {
@@ -118,7 +169,8 @@ module.exports = {
// // },
{
label: "Discord",
- href: "https://discord.gg/pAbnFJrkgZ",
+ href: "https://aka.ms/autogen-dc",
+
},
{
label: "Twitter",
diff --git a/website/sidebars.js b/website/sidebars.js
index 44829e607..cac530617 100644
--- a/website/sidebars.js
+++ b/website/sidebars.js
@@ -25,6 +25,13 @@
{
type: 'category',
label: 'Tutorial',
+ collapsed: false,
+ link: {
+ type: 'generated-index',
+ title: 'Tutorial',
+ description: 'Tutorial on the basic concepts of AutoGen',
+ slug: 'tutorial',
+ },
items: [
{
type: 'doc',
@@ -66,18 +73,44 @@
{'Use Cases': [{type: 'autogenerated', dirName: 'Use-Cases'}]},
{
type: 'category',
- label: 'Topics',
+ label: 'User Guide',
+ collapsed: false,
link: {
type: 'generated-index',
- title: 'Topics',
- description: 'Learn about various topics in AutoGen',
- slug: 'topics',
+ title: 'User Guide',
+ slug: 'topics',
},
items: [{type: 'autogenerated', dirName: 'topics'}]
},
+ {
+ 'type': 'category',
+ 'label': 'API Reference',
+ 'link': {
+ type: 'generated-index',
+ title: 'API Reference',
+ description: 'API Reference of AutoGen modules and classes',
+ slug: 'reference',
+ },
+ 'items': [{type: 'autogenerated', dirName: 'reference'}]
+ },
+ {
+ type: 'doc',
+ label: 'FAQs',
+ id: 'FAQ',
+ },
+ {
+ 'type': 'category',
+ 'label': 'Ecosystem',
+ 'link': {
+ type: 'generated-index',
+ title: 'Ecosystem',
+ description: 'Learn about the ecosystem of AutoGen',
+ slug: 'ecosystem',
+ },
+ 'items': [{type: 'autogenerated', dirName: 'ecosystem'}]
+ },
'Contribute',
'Research',
- {'Ecosystem': [{type: 'autogenerated', dirName: 'ecosystem'}]},
'Migration-Guide'
],
// pydoc-markdown auto-generated markdowns from docstrings