Skip to content

A programming framework for agentic AI

Improve docs for model clients (#5952)

Eric Zhu9bde5ef911e2
diff --git a/python/packages/autogen-core/docs/src/user-guide/agentchat-user-guide/tutorial/models.ipynb b/python/packages/autogen-core/docs/src/user-guide/agentchat-user-guide/tutorial/models.ipynb
index 9b359fe58..2f39ef26b 100644
--- a/python/packages/autogen-core/docs/src/user-guide/agentchat-user-guide/tutorial/models.ipynb
+++ b/python/packages/autogen-core/docs/src/user-guide/agentchat-user-guide/tutorial/models.ipynb
@@ -16,6 +16,32 @@
"```"
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Log Model Calls\n",
+ "\n",
+ "AutoGen uses standard Python logging module to log events like model calls and responses.\n",
+ "The logger name is {py:attr}`autogen_core.EVENT_LOGGER_NAME`, and the event type is `LLMCall`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "\n",
+ "from autogen_core import EVENT_LOGGER_NAME\n",
+ "\n",
+ "logging.basicConfig(level=logging.WARNING)\n",
+ "logger = logging.getLogger(EVENT_LOGGER_NAME)\n",
+ "logger.addHandler(logging.StreamHandler())\n",
+ "logger.setLevel(logging.INFO)"
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {},
@@ -135,11 +161,15 @@
"metadata": {},
"outputs": [],
"source": [
+ "from autogen_ext.auth.azure import AzureTokenProvider\n",
"from autogen_ext.models.openai import AzureOpenAIChatCompletionClient\n",
- "from azure.identity import DefaultAzureCredential, get_bearer_token_provider\n",
+ "from azure.identity import DefaultAzureCredential\n",
"\n",
"# Create the token provider\n",
- "token_provider = get_bearer_token_provider(DefaultAzureCredential(), \"https://cognitiveservices.azure.com/.default\")\n",
+ "token_provider = AzureTokenProvider(\n",
+ " DefaultAzureCredential(),\n",
+ " \"https://cognitiveservices.azure.com/.default\",\n",
+ ")\n",
"\n",
"az_model_client = AzureOpenAIChatCompletionClient(\n",
" azure_deployment=\"{your-azure-deployment}\",\n",
@@ -475,7 +505,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.11.9"
+ "version": "3.12.7"
}
},
"nbformat": 4,
diff --git a/python/packages/autogen-core/docs/src/user-guide/core-user-guide/components/model-clients.ipynb b/python/packages/autogen-core/docs/src/user-guide/core-user-guide/components/model-clients.ipynb
index 20a033f76..94f601038 100644
--- a/python/packages/autogen-core/docs/src/user-guide/core-user-guide/components/model-clients.ipynb
+++ b/python/packages/autogen-core/docs/src/user-guide/core-user-guide/components/model-clients.ipynb
@@ -14,301 +14,51 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "Currently there are three built-in model clients:\n",
- "* {py:class}`~autogen_ext.models.openai.OpenAIChatCompletionClient`\n",
- "* {py:class}`~autogen_ext.models.openai.AzureOpenAIChatCompletionClient`\n",
- "* {py:class}`~autogen_ext.models.azure.AzureAIChatCompletionClient`"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## OpenAI\n",
+ "Currently we support the following built-in model clients:\n",
+ "* {py:class}`~autogen_ext.models.openai.OpenAIChatCompletionClient`: for OpenAI models and models with OpenAI API compatibility (e.g., Gemini).\n",
+ "* {py:class}`~autogen_ext.models.openai.AzureOpenAIChatCompletionClient`: for Azure OpenAI models.\n",
+ "* {py:class}`~autogen_ext.models.azure.AzureAIChatCompletionClient`: for GitHub models and models hosted on Azure.\n",
+ "* {py:class}`~autogen_ext.models.ollama.OllamaChatCompletionClient` (Experimental): for local models hosted on Ollama.\n",
+ "* {py:class}`~autogen_ext.models.anthropic.AnthropicChatCompletionClient` (Experimental): for models hosted on Anthropic.\n",
+ "* {py:class}`~autogen_ext.models.semantic_kernel.SKChatCompletionAdapter`: adapter for Semantic Kernel AI connectors.\n",
"\n",
- "To use the {py:class}`~autogen_ext.models.openai.OpenAIChatCompletionClient`, you need to install the `openai` extra."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "vscode": {
- "languageId": "shellscript"
- }
- },
- "outputs": [],
- "source": [
- "# pip install \"autogen-ext[openai]\""
+ "For more information on how to use these model clients, please refer to the documentation of each client."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "You also need to provide the API key\n",
- "either through the environment variable `OPENAI_API_KEY` or through the `api_key` argument."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "from autogen_core.models import UserMessage\n",
- "from autogen_ext.models.openai import OpenAIChatCompletionClient\n",
+ "## Log Model Calls\n",
"\n",
- "# Create an OpenAI model client.\n",
- "model_client = OpenAIChatCompletionClient(\n",
- " model=\"gpt-4o\",\n",
- " # api_key=\"sk-...\", # Optional if you have an API key set in the environment.\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "You can call the {py:meth}`~autogen_ext.models.openai.BaseOpenAIChatCompletionClient.create` method to create a\n",
- "chat completion request, and await for an {py:class}`~autogen_core.models.CreateResult` object in return."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "The capital of France is Paris.\n"
- ]
- }
- ],
- "source": [
- "# Send a message list to the model and await the response.\n",
- "messages = [\n",
- " UserMessage(content=\"What is the capital of France?\", source=\"user\"),\n",
- "]\n",
- "response = await model_client.create(messages=messages)\n",
- "\n",
- "# Print the response\n",
- "print(response.content)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "RequestUsage(prompt_tokens=15, completion_tokens=7)\n"
- ]
- }
- ],
- "source": [
- "# Print the response token usage\n",
- "print(response.usage)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Azure OpenAI\n",
- "\n",
- "To use the {py:class}`~autogen_ext.models.openai.AzureOpenAIChatCompletionClient`, you need to provide\n",
- "the deployment id, Azure Cognitive Services endpoint, api version, and model capabilities.\n",
- "For authentication, you can either provide an API key or an Azure Active Directory (AAD) token credential."
+ "AutoGen uses standard Python logging module to log events like model calls and responses.\n",
+ "The logger name is {py:attr}`autogen_core.EVENT_LOGGER_NAME`, and the event type is `LLMCall`."
]
},
{
"cell_type": "code",
"execution_count": null,
- "metadata": {
- "vscode": {
- "languageId": "shellscript"
- }
- },
- "outputs": [],
- "source": [
- "# pip install \"autogen-ext[openai,azure]\""
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The following code snippet shows how to use AAD authentication.\n",
- "The identity used must be assigned the [**Cognitive Services OpenAI User**](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/role-based-access-control#cognitive-services-openai-user) role."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from autogen_ext.models.openai import AzureOpenAIChatCompletionClient\n",
- "from azure.identity import DefaultAzureCredential, get_bearer_token_provider\n",
- "\n",
- "# Create the token provider\n",
- "token_provider = get_bearer_token_provider(DefaultAzureCredential(), \"https://cognitiveservices.azure.com/.default\")\n",
- "\n",
- "az_model_client = AzureOpenAIChatCompletionClient(\n",
- " azure_deployment=\"{your-azure-deployment}\",\n",
- " model=\"{model-name, such as gpt-4o}\",\n",
- " api_version=\"2024-06-01\",\n",
- " azure_endpoint=\"https://{your-custom-endpoint}.openai.azure.com/\",\n",
- " azure_ad_token_provider=token_provider, # Optional if you choose key-based authentication.\n",
- " # api_key=\"sk-...\", # For key-based authentication.\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
"metadata": {},
- "source": [
- "```{note}\n",
- "See [here](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/managed-identity#chat-completions) for how to use the Azure client directly or for more info.\n",
- "```"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Azure AI Foundry\n",
- "\n",
- "[Azure AI Foundry](https://learn.microsoft.com/en-us/azure/ai-studio/) (previously known as Azure AI Studio) offers models hosted on Azure.\n",
- "To use those models, you use the {py:class}`~autogen_ext.models.azure.AzureAIChatCompletionClient`.\n",
- "\n",
- "You need to install the `azure` extra to use this client."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "vscode": {
- "languageId": "shellscript"
- }
- },
"outputs": [],
"source": [
- "# pip install \"autogen-ext[openai,azure]\""
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Below is an example of using this client with the Phi-4 model from [GitHub Marketplace](https://github.com/marketplace/models)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "finish_reason='stop' content='The capital of France is Paris.' usage=RequestUsage(prompt_tokens=14, completion_tokens=8) cached=False logprobs=None\n"
- ]
- }
- ],
- "source": [
- "import os\n",
+ "import logging\n",
"\n",
- "from autogen_core.models import UserMessage\n",
- "from autogen_ext.models.azure import AzureAIChatCompletionClient\n",
- "from azure.core.credentials import AzureKeyCredential\n",
- "\n",
- "client = AzureAIChatCompletionClient(\n",
- " model=\"Phi-4\",\n",
- " endpoint=\"https://models.inference.ai.azure.com\",\n",
- " # To authenticate with the model you will need to generate a personal access token (PAT) in your GitHub settings.\n",
- " # Create your PAT token by following instructions here: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens\n",
- " credential=AzureKeyCredential(os.environ[\"GITHUB_TOKEN\"]),\n",
- " model_info={\n",
- " \"json_output\": False,\n",
- " \"function_calling\": False,\n",
- " \"vision\": False,\n",
- " \"family\": \"unknown\",\n",
- " },\n",
- ")\n",
+ "from autogen_core import EVENT_LOGGER_NAME\n",
"\n",
- "result = await client.create([UserMessage(content=\"What is the capital of France?\", source=\"user\")])\n",
- "print(result)"
+ "logging.basicConfig(level=logging.WARNING)\n",
+ "logger = logging.getLogger(EVENT_LOGGER_NAME)\n",
+ "logger.addHandler(logging.StreamHandler())\n",
+ "logger.setLevel(logging.INFO)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Ollama (experimental)\n",
- "\n",
- "[Ollama](https://ollama.com/) is a local model server that can run models locally on your machine.\n",
+ "## Call Model Client\n",
"\n",
- "```{note}\n",
- "Small local models are typically not as capable as larger models on the cloud.\n",
- "For some tasks they may not perform as well and the output may be suprising.\n",
- "```\n",
- "\n",
- "To use Ollama, install the `ollama` extension and use the {py:class}`~autogen_ext.models.ollama.OllamaChatCompletionClient`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "vscode": {
- "languageId": "shellscript"
- }
- },
- "outputs": [],
- "source": [
- "pip install -U \"autogen-ext[ollama]\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "finish_reason='unknown' content='The capital of France is Paris.' usage=RequestUsage(prompt_tokens=32, completion_tokens=8) cached=False logprobs=None thought=None\n"
- ]
- }
- ],
- "source": [
- "from autogen_core.models import UserMessage\n",
- "from autogen_ext.models.ollama import OllamaChatCompletionClient\n",
- "\n",
- "# Assuming your Ollama server is running locally on port 11434.\n",
- "ollama_model_client = OllamaChatCompletionClient(model=\"llama3.2\")\n",
- "\n",
- "response = await ollama_model_client.create([UserMessage(content=\"What is the capital of France?\", source=\"user\")])\n",
- "print(response)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Gemini (beta)\n",
- "\n",
- "The below example shows how to use the Gemini model via the {py:class}`~autogen_ext.models.openai.OpenAIChatCompletionClient`."
+ "To call a model client, you can use the {py:meth}`~autogen_core.models.ChatCompletionClient.create` method.\n",
+ "This example uses the {py:class}`~autogen_ext.models.openai.OpenAIChatCompletionClient` to call an OpenAI model."
]
},
{
@@ -320,7 +70,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
- "finish_reason='stop' content='Paris\\n' usage=RequestUsage(prompt_tokens=7, completion_tokens=2) cached=False logprobs=None thought=None\n"
+ "finish_reason='stop' content='The capital of France is Paris.' usage=RequestUsage(prompt_tokens=15, completion_tokens=8) cached=False logprobs=None thought=None\n"
]
}
],
@@ -329,123 +79,27 @@
"from autogen_ext.models.openai import OpenAIChatCompletionClient\n",
"\n",
"model_client = OpenAIChatCompletionClient(\n",
- " model=\"gemini-1.5-flash-8b\",\n",
- " # api_key=\"GEMINI_API_KEY\",\n",
- ")\n",
- "\n",
- "response = await model_client.create([UserMessage(content=\"What is the capital of France?\", source=\"user\")])\n",
- "print(response)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Semantic Kernel Adapter\n",
- "\n",
- "The {py:class}`~autogen_ext.models.semantic_kernel.SKChatCompletionAdapter`\n",
- "allows you to use Semantic kernel model clients as a\n",
- "{py:class}`~autogen_core.models.ChatCompletionClient` by adapting them to the required interface.\n",
+ " model=\"gpt-4\", temperature=0.3\n",
+ ") # assuming OPENAI_API_KEY is set in the environment.\n",
"\n",
- "You need to install the relevant provider extras to use this adapter. \n",
- "\n",
- "The list of extras that can be installed:\n",
- "\n",
- "- `semantic-kernel-anthropic`: Install this extra to use Anthropic models.\n",
- "- `semantic-kernel-google`: Install this extra to use Google Gemini models.\n",
- "- `semantic-kernel-ollama`: Install this extra to use Ollama models.\n",
- "- `semantic-kernel-mistralai`: Install this extra to use MistralAI models.\n",
- "- `semantic-kernel-aws`: Install this extra to use AWS models.\n",
- "- `semantic-kernel-hugging-face`: Install this extra to use Hugging Face models.\n",
- "\n",
- "For example, to use Anthropic models, you need to install `semantic-kernel-anthropic`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "vscode": {
- "languageId": "shellscript"
- }
- },
- "outputs": [],
- "source": [
- "# pip install \"autogen-ext[semantic-kernel-anthropic]\""
+ "result = await model_client.create([UserMessage(content=\"What is the capital of France?\", source=\"user\")])\n",
+ "print(result)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "To use this adapter, you need create a Semantic Kernel model client and pass it to the adapter.\n",
+ "## Streaming Tokens\n",
"\n",
- "For example, to use the Anthropic model:"
+ "You can use the {py:meth}`~autogen_core.models.ChatCompletionClient.create_stream` method to create a\n",
+ "chat completion request with streaming token chunks."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "finish_reason='stop' content='The capital of France is Paris. It is also the largest city in France and one of the most populous metropolitan areas in Europe.' usage=RequestUsage(prompt_tokens=0, completion_tokens=0) cached=False logprobs=None\n"
- ]
- }
- ],
- "source": [
- "import os\n",
- "\n",
- "from autogen_core.models import UserMessage\n",
- "from autogen_ext.models.semantic_kernel import SKChatCompletionAdapter\n",
- "from semantic_kernel import Kernel\n",
- "from semantic_kernel.connectors.ai.anthropic import AnthropicChatCompletion, AnthropicChatPromptExecutionSettings\n",
- "from semantic_kernel.memory.null_memory import NullMemory\n",
- "\n",
- "sk_client = AnthropicChatCompletion(\n",
- " ai_model_id=\"claude-3-5-sonnet-20241022\",\n",
- " api_key=os.environ[\"ANTHROPIC_API_KEY\"],\n",
- " service_id=\"my-service-id\", # Optional; for targeting specific services within Semantic Kernel\n",
- ")\n",
- "settings = AnthropicChatPromptExecutionSettings(\n",
- " temperature=0.2,\n",
- ")\n",
- "\n",
- "anthropic_model_client = SKChatCompletionAdapter(\n",
- " sk_client, kernel=Kernel(memory=NullMemory()), prompt_settings=settings\n",
- ")\n",
- "\n",
- "# Call the model directly.\n",
- "model_result = await anthropic_model_client.create(\n",
- " messages=[UserMessage(content=\"What is the capital of France?\", source=\"User\")]\n",
- ")\n",
- "print(model_result)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Read more about the [Semantic Kernel Adapter](../../../reference/python/autogen_ext.models.semantic_kernel.rst)."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Streaming Response\n",
- "\n",
- "You can use the {py:meth}`~autogen_ext.models.openai.BaseOpenAIChatCompletionClient.create_stream` method to create a\n",
- "chat completion request with streaming response."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
"outputs": [
{
"name": "stdout",
@@ -476,6 +130,11 @@
}
],
"source": [
+ "from autogen_core.models import CreateResult, UserMessage\n",
+ "from autogen_ext.models.openai import OpenAIChatCompletionClient\n",
+ "\n",
+ "model_client = OpenAIChatCompletionClient(model=\"gpt-4o\") # assuming OPENAI_API_KEY is set in the environment.\n",
+ "\n",
"messages = [\n",
" UserMessage(content=\"Write a very short story about a dragon.\", source=\"user\"),\n",
"]\n",
@@ -485,18 +144,17 @@
"\n",
"# Iterate over the stream and print the responses.\n",
"print(\"Streamed responses:\")\n",
- "async for response in stream: # type: ignore\n",
- " if isinstance(response, str):\n",
- " # A partial response is a string.\n",
- " print(response, flush=True, end=\"\")\n",
+ "async for chunk in stream: # type: ignore\n",
+ " if isinstance(chunk, str):\n",
+ " # The chunk is a string.\n",
+ " print(chunk, flush=True, end=\"\")\n",
" else:\n",
+ " # The final chunk is a CreateResult object.\n",
+ " assert isinstance(chunk, CreateResult) and isinstance(chunk.content, str)\n",
" # The last response is a CreateResult object with the complete message.\n",
" print(\"\\n\\n------------\\n\")\n",
" print(\"The complete response:\", flush=True)\n",
- " print(response.content, flush=True)\n",
- " print(\"\\n\\n------------\\n\")\n",
- " print(\"The token usage was:\", flush=True)\n",
- " print(response.usage, flush=True)"
+ " print(chunk.content, flush=True)"
]
},
{
@@ -509,87 +167,12 @@
"```\n",
"\n",
"```{note}\n",
- "The default usage response is to return zero values\n",
+ "The default usage response is to return zero values. To enable usage, \n",
+ "see {py:meth}`~autogen_ext.models.openai.BaseOpenAIChatCompletionClient.create_stream`\n",
+ "for more details.\n",
"```"
]
},
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Comparing usage returns in the above Non Streaming `model_client.create(messages=messages)` vs streaming `model_client.create_stream(messages=messages)` we see differences.\n",
- "The non streaming response by default returns valid prompt and completion token usage counts. \n",
- "The streamed response by default returns zero values.\n",
- "\n",
- "as documented in the OPENAI API Reference an additional parameter `stream_options` can be specified to return valid usage counts. see [stream_options](https://platform.openai.com/docs/api-reference/chat/create#chat-create-stream_options)\n",
- "\n",
- "Only set this when you using streaming ie , using `create_stream` \n",
- "\n",
- "to enable this in `create_stream` set `extra_create_args={\"stream_options\": {\"include_usage\": True}},`\n",
- "\n",
- "```{note}\n",
- "Note whilst other API's like LiteLLM also support this, it is not always guarenteed that it is fully supported or correct.\n",
- "\n",
- "See the example below for how to use the `stream_options` parameter to return usage."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Streamed responses:\n",
- "In a lush, emerald valley hidden by towering peaks, there lived a dragon named Ember. Unlike others of her kind, Ember cherished solitude over treasure, and the songs of the stream over the roar of flames. One misty dawn, a young shepherd stumbled into her sanctuary, lost and frightened. \n",
- "\n",
- "Instead of fury, he was met with kindness as Ember extended a wing, guiding him back to safety. In gratitude, the shepherd visited yearly, bringing tales of his world beyond the mountains. Over time, a friendship blossomed, binding man and dragon in shared stories and laughter.\n",
- "\n",
- "As the years passed, the legend of Ember the gentle-hearted spread far and wide, forever changing the way dragons were seen in the hearts of many.\n",
- "\n",
- "------------\n",
- "\n",
- "The complete response:\n",
- "In a lush, emerald valley hidden by towering peaks, there lived a dragon named Ember. Unlike others of her kind, Ember cherished solitude over treasure, and the songs of the stream over the roar of flames. One misty dawn, a young shepherd stumbled into her sanctuary, lost and frightened. \n",
- "\n",
- "Instead of fury, he was met with kindness as Ember extended a wing, guiding him back to safety. In gratitude, the shepherd visited yearly, bringing tales of his world beyond the mountains. Over time, a friendship blossomed, binding man and dragon in shared stories and laughter.\n",
- "\n",
- "As the years passed, the legend of Ember the gentle-hearted spread far and wide, forever changing the way dragons were seen in the hearts of many.\n",
- "\n",
- "\n",
- "------------\n",
- "\n",
- "The token usage was:\n",
- "RequestUsage(prompt_tokens=17, completion_tokens=146)\n"
- ]
- }
- ],
- "source": [
- "messages = [\n",
- " UserMessage(content=\"Write a very short story about a dragon.\", source=\"user\"),\n",
- "]\n",
- "\n",
- "# Create a stream.\n",
- "stream = model_client.create_stream(messages=messages, extra_create_args={\"stream_options\": {\"include_usage\": True}})\n",
- "\n",
- "# Iterate over the stream and print the responses.\n",
- "print(\"Streamed responses:\")\n",
- "async for response in stream: # type: ignore\n",
- " if isinstance(response, str):\n",
- " # A partial response is a string.\n",
- " print(response, flush=True, end=\"\")\n",
- " else:\n",
- " # The last response is a CreateResult object with the complete message.\n",
- " print(\"\\n\\n------------\\n\")\n",
- " print(\"The complete response:\", flush=True)\n",
- " print(response.content, flush=True)\n",
- " print(\"\\n\\n------------\\n\")\n",
- " print(\"The token usage was:\", flush=True)\n",
- " print(response.usage, flush=True)"
- ]
- },
{
"cell_type": "markdown",
"metadata": {},
diff --git a/python/packages/autogen-ext/src/autogen_ext/models/azure/_azure_ai_client.py b/python/packages/autogen-ext/src/autogen_ext/models/azure/_azure_ai_client.py
index 93a8c2c50..17dadcc52 100644
--- a/python/packages/autogen-ext/src/autogen_ext/models/azure/_azure_ai_client.py
+++ b/python/packages/autogen-ext/src/autogen_ext/models/azure/_azure_ai_client.py
@@ -197,40 +197,88 @@ class AzureAIChatCompletionClient(ChatCompletionClient):
seed: (optional,int)
model_extras: (optional,Dict[str, Any])
- To use this client, you must install the `azure-ai-inference` extension:
+ To use this client, you must install the `azure` extra:
- .. code-block:: bash
+ .. code-block:: bash
- pip install "autogen-ext[azure]"
+ pip install "autogen-ext[azure]"
- The following code snippet shows how to use the client:
+ The following code snippet shows how to use the client with GitHub Models:
- .. code-block:: python
+ .. code-block:: python
- import asyncio
- from azure.core.credentials import AzureKeyCredential
- from autogen_ext.models.azure import AzureAIChatCompletionClient
- from autogen_core.models import UserMessage
+ import asyncio
+ import os
+ from azure.core.credentials import AzureKeyCredential
+ from autogen_ext.models.azure import AzureAIChatCompletionClient
+ from autogen_core.models import UserMessage
- async def main():
- client = AzureAIChatCompletionClient(
- endpoint="endpoint",
- credential=AzureKeyCredential("api_key"),
- model_info={
- "json_output": False,
- "function_calling": False,
- "vision": False,
- "family": "unknown",
- },
- )
+ async def main():
+ client = AzureAIChatCompletionClient(
+ model="Phi-4",
+ endpoint="https://models.inference.ai.azure.com",
+ # To authenticate with the model you will need to generate a personal access token (PAT) in your GitHub settings.
+ # Create your PAT token by following instructions here: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens
+ credential=AzureKeyCredential(os.environ["GITHUB_TOKEN"]),
+ model_info={
+ "json_output": False,
+ "function_calling": False,
+ "vision": False,
+ "family": "unknown",
+ },
+ )
+
+ result = await client.create([UserMessage(content="What is the capital of France?", source="user")])
+ print(result)
+
+ # Close the client.
+ await client.close()
+
+
+ if __name__ == "__main__":
+ asyncio.run(main())
+
+ To use streaming, you can use the `create_stream` method:
+
+ .. code-block:: python
+
+ import asyncio
+ import os
+
+ from autogen_core.models import UserMessage
+ from autogen_ext.models.azure import AzureAIChatCompletionClient
+ from azure.core.credentials import AzureKeyCredential
+
+
+ async def main():
+ client = AzureAIChatCompletionClient(
+ model="Phi-4",
+ endpoint="https://models.inference.ai.azure.com",
+ # To authenticate with the model you will need to generate a personal access token (PAT) in your GitHub settings.
+ # Create your PAT token by following instructions here: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens
+ credential=AzureKeyCredential(os.environ["GITHUB_TOKEN"]),
+ model_info={
+ "json_output": False,
+ "function_calling": False,
+ "vision": False,
+ "family": "unknown",
+ },
+ )
+
+ # Create a stream.
+ stream = client.create_stream([UserMessage(content="Write a poem about the ocean", source="user")])
+ async for chunk in stream:
+ print(chunk, end="", flush=True)
+ print()
+
+ # Close the client.
+ await client.close()
- result = await client.create([UserMessage(content="What is the capital of France?", source="user")])
- print(result)
+ if __name__ == "__main__":
+ asyncio.run(main())
- if __name__ == "__main__":
- asyncio.run(main())
"""
diff --git a/python/packages/autogen-ext/src/autogen_ext/models/openai/_openai_client.py b/python/packages/autogen-ext/src/autogen_ext/models/openai/_openai_client.py
index 968a23337..267dea0f0 100644
--- a/python/packages/autogen-ext/src/autogen_ext/models/openai/_openai_client.py
+++ b/python/packages/autogen-ext/src/autogen_ext/models/openai/_openai_client.py
@@ -641,17 +641,22 @@ class BaseOpenAIChatCompletionClient(ChatCompletionClient):
json_output (Optional[bool], optional): If True, the output will be in JSON format. Defaults to None.
extra_create_args (Mapping[str, Any], optional): Additional arguments for the creation process. Default to `{}`.
cancellation_token (Optional[CancellationToken], optional): A token to cancel the operation. Defaults to None.
- max_consecutive_empty_chunk_tolerance (int): [Deprecated] The maximum number of consecutive empty chunks to tolerate before raising a ValueError. This seems to only be needed to set when using `AzureOpenAIChatCompletionClient`. Defaults to 0. This parameter is deprecated, empty chunks will be skipped.
+ max_consecutive_empty_chunk_tolerance (int): [Deprecated] This parameter is deprecated, empty chunks will be skipped.
Yields:
AsyncGenerator[Union[str, CreateResult], None]: A generator yielding the completion results as they are produced.
- In streaming, the default behaviour is not return token usage counts. See: [OpenAI API reference for possible args](https://platform.openai.com/docs/api-reference/chat/create).
- However `extra_create_args={"stream_options": {"include_usage": True}}` will (if supported by the accessed API)
- return a final chunk with usage set to a RequestUsage object having prompt and completion token counts,
- all preceding chunks will have usage as None. See: [stream_options](https://platform.openai.com/docs/api-reference/chat/create#chat-create-stream_options).
+ In streaming, the default behaviour is not return token usage counts.
+ See: `OpenAI API reference for possible args <https://platform.openai.com/docs/api-reference/chat/create>`_.
- Other examples of OPENAI supported arguments that can be included in `extra_create_args`:
+ You can set `extra_create_args={"stream_options": {"include_usage": True}}`
+ (if supported by the accessed API) to
+ return a final chunk with usage set to a :class:`~autogen_core.models.RequestUsage` object
+ with prompt and completion token counts,
+ all preceding chunks will have usage as `None`.
+ See: `OpenAI API reference for stream options <https://platform.openai.com/docs/api-reference/chat/create#chat-create-stream_options>`_.
+
+ Other examples of supported arguments that can be included in `extra_create_args`:
- `temperature` (float): Controls the randomness of the output. Higher values (e.g., 0.8) make the output more random, while lower values (e.g., 0.2) make it more focused and deterministic.
- `max_tokens` (int): The maximum number of tokens to generate in the completion.
- `top_p` (float): An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass.
@@ -1126,6 +1131,8 @@ class OpenAIChatCompletionClient(BaseOpenAIChatCompletionClient, Component[OpenA
result = await openai_client.create([UserMessage(content="What is the capital of France?", source="user")]) # type: ignore
print(result)
+ # Close the client when done.
+ # await openai_client.close()
To use the client with a non-OpenAI model, you need to provide the base URL of the model and the model info.
For example, to use Ollama, you can use the following code snippet:
@@ -1147,6 +1154,45 @@ class OpenAIChatCompletionClient(BaseOpenAIChatCompletionClient, Component[OpenA
},
)
+ # Close the client when done.
+ # await custom_model_client.close()
+
+ To use streaming mode, you can use the following code snippet:
+
+ .. code-block:: python
+
+ import asyncio
+ from autogen_core.models import UserMessage
+ from autogen_ext.models.openai import OpenAIChatCompletionClient
+
+
+ async def main() -> None:
+ # Similar for AzureOpenAIChatCompletionClient.
+ model_client = OpenAIChatCompletionClient(model="gpt-4o") # assuming OPENAI_API_KEY is set in the environment.
+
+ messages = [UserMessage(content="Write a very short story about a dragon.", source="user")]
+
+ # Create a stream.
+ stream = model_client.create_stream(messages=messages)
+
+ # Iterate over the stream and print the responses.
+ print("Streamed responses:")
+ async for response in stream:
+ if isinstance(response, str):
+ # A partial response is a string.
+ print(response, flush=True, end="")
+ else:
+ # The last response is a CreateResult object with the complete message.
+ print("\\n\\n------------\\n")
+ print("The complete response:", flush=True)
+ print(response.content, flush=True)
+
+ # Close the client when done.
+ await model_client.close()
+
+
+ asyncio.run(main())
+
To use structured output as well as function calling, you can use the following code snippet:
.. code-block:: python
@@ -1182,14 +1228,14 @@ class OpenAIChatCompletionClient(BaseOpenAIChatCompletionClient, Component[OpenA
# which is required for structured output mode.
tool = FunctionTool(sentiment_analysis, description="Sentiment Analysis", strict=True)
- # Create an OpenAIChatCompletionClient instance.
- model_client = OpenAIChatCompletionClient(
- model="gpt-4o-mini",
- response_format=AgentResponse, # type: ignore
- )
-
async def main() -> None:
+ # Create an OpenAIChatCompletionClient instance.
+ model_client = OpenAIChatCompletionClient(
+ model="gpt-4o-mini",
+ response_format=AgentResponse, # type: ignore
+ )
+
# Generate a response using the tool.
response1 = await model_client.create(
messages=[
@@ -1217,6 +1263,8 @@ class OpenAIChatCompletionClient(BaseOpenAIChatCompletionClient, Component[OpenA
# Should be a structured output.
# {"thoughts": "The user is happy.", "response": "happy"}
+ # Close the client when done.
+ await model_client.close()
asyncio.run(main())
@@ -1311,6 +1359,12 @@ class AzureOpenAIChatCompletionClient(
):
"""Chat completion client for Azure OpenAI hosted models.
+ To use this client, you must install the `azure` and `openai` extensions:
+
+ .. code-block:: bash
+
+ pip install "autogen-ext[openai,azure]"
+
Args:
model (str): Which OpenAI model to use.
@@ -1337,36 +1391,34 @@ class AzureOpenAIChatCompletionClient(
default_headers (optional, dict[str, str]): Custom headers; useful for authentication or other custom requirements.
-
- To use this client, you must install the `azure` and `openai` extensions:
-
- .. code-block:: bash
-
- pip install "autogen-ext[openai,azure]"
-
- To use the client, you need to provide your deployment id, Azure Cognitive Services endpoint,
- api version, and model capabilities.
+ To use the client, you need to provide your deployment name, Azure Cognitive Services endpoint, and api version.
For authentication, you can either provide an API key or an Azure Active Directory (AAD) token credential.
The following code snippet shows how to use AAD authentication.
The identity used must be assigned the `Cognitive Services OpenAI User <https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/role-based-access-control#cognitive-services-openai-user>`_ role.
- .. code-block:: python
+ .. code-block:: python
- from autogen_ext.models.openai import AzureOpenAIChatCompletionClient
- from azure.identity import DefaultAzureCredential, get_bearer_token_provider
+ from autogen_ext.auth.azure import AzureTokenProvider
+ from autogen_ext.models.openai import AzureOpenAIChatCompletionClient
+ from azure.identity import DefaultAzureCredential
- # Create the token provider
- token_provider = get_bearer_token_provider(DefaultAzureCredential(), "https://cognitiveservices.azure.com/.default")
+ # Create the token provider
+ token_provider = AzureTokenProvider(
+ DefaultAzureCredential(),
+ "https://cognitiveservices.azure.com/.default",
+ )
- az_model_client = AzureOpenAIChatCompletionClient(
- azure_deployment="{your-azure-deployment}",
- model="{deployed-model, such as 'gpt-4o'}",
- api_version="2024-06-01",
- azure_endpoint="https://{your-custom-endpoint}.openai.azure.com/",
- azure_ad_token_provider=token_provider, # Optional if you choose key-based authentication.
- # api_key="sk-...", # For key-based authentication. `AZURE_OPENAI_API_KEY` environment variable can also be used instead.
- )
+ az_model_client = AzureOpenAIChatCompletionClient(
+ azure_deployment="{your-azure-deployment}",
+ model="{model-name, such as gpt-4o}",
+ api_version="2024-06-01",
+ azure_endpoint="https://{your-custom-endpoint}.openai.azure.com/",
+ azure_ad_token_provider=token_provider, # Optional if you choose key-based authentication.
+ # api_key="sk-...", # For key-based authentication.
+ )
+
+ See other usage examples in the :class:`OpenAIChatCompletionClient` class.
To load the client that uses identity based aith from a configuration, you can use the `load_component` method:
@@ -1396,7 +1448,6 @@ class AzureOpenAIChatCompletionClient(
To view the full list of available configuration options, see the :py:class:`AzureOpenAIClientConfigurationConfigModel` class.
-
.. note::
Right now only `DefaultAzureCredential` is supported with no additional args passed to it.