The agent engineering platform.
Python agentsaiai-agentsanthropicchatgptdeepagentsenterpriseframeworkgeminigenerative-ailangchainlanggraphllmmultiagentopen-sourceopenaipydanticpythonragtypescript
docs: document built-in tools for ChatVertexAI (#31564)
ccurme446a9d564791
diff --git a/docs/docs/integrations/chat/google_vertex_ai_palm.ipynb b/docs/docs/integrations/chat/google_vertex_ai_palm.ipynbindex f626ee7712..11f0fcac65 100644--- a/docs/docs/integrations/chat/google_vertex_ai_palm.ipynb+++ b/docs/docs/integrations/chat/google_vertex_ai_palm.ipynb@@ -1,269 +1,327 @@{- "cells": [- {- "cell_type": "raw",- "id": "afaf8039",- "metadata": {},- "source": [- "---\n",- "sidebar_label: Google Cloud Vertex AI\n",- "---"- ]- },- {- "cell_type": "markdown",- "id": "e49f1e0d",- "metadata": {},- "source": [- "# ChatVertexAI\n",- "\n",- "This page provides a quick overview for getting started with VertexAI [chat models](/docs/concepts/chat_models). For detailed documentation of all ChatVertexAI features and configurations head to the [API reference](https://python.langchain.com/api_reference/google_vertexai/chat_models/langchain_google_vertexai.chat_models.ChatVertexAI.html).\n",- "\n",- "ChatVertexAI exposes all foundational models available in Google Cloud, like `gemini-1.5-pro`, `gemini-1.5-flash`, etc. For a full and updated list of available models visit [VertexAI documentation](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/overview).\n",- "\n",- ":::info Google Cloud VertexAI vs Google PaLM\n",- "\n",- "The Google Cloud VertexAI integration is separate from the [Google PaLM integration](/docs/integrations/chat/google_generative_ai/). Google has chosen to offer an enterprise version of PaLM through GCP, and this supports the models made available through there.\n",- "\n",- ":::\n",- "\n",- "## Overview\n",- "### Integration details\n",- "\n",- "| Class | Package | Local | Serializable | [JS support](https://js.langchain.com/docs/integrations/chat/google_vertex_ai) | Package downloads | Package latest |\n",- "| :--- | :--- | :---: | :---: | :---: | :---: | :---: |\n",- "| [ChatVertexAI](https://python.langchain.com/api_reference/google_vertexai/chat_models/langchain_google_vertexai.chat_models.ChatVertexAI.html) | [langchain-google-vertexai](https://python.langchain.com/api_reference/google_vertexai/index.html) | ❌ | beta | ✅ |  |  |\n",- "\n",- "### Model features\n",- "| [Tool calling](/docs/how_to/tool_calling) | [Structured output](/docs/how_to/structured_output/) | JSON mode | [Image input](/docs/how_to/multimodal_inputs/) | Audio input | Video input | [Token-level streaming](/docs/how_to/chat_streaming/) | Native async | [Token usage](/docs/how_to/chat_token_usage_tracking/) | [Logprobs](/docs/how_to/logprobs/) |\n",- "| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n",- "| ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |\n",- "\n",- "## Setup\n",- "\n",- "To access VertexAI models you'll need to create a Google Cloud Platform account, set up credentials, and install the `langchain-google-vertexai` integration package.\n",- "\n",- "### Credentials\n",- "\n",- "To use the integration you must:\n",- "- Have credentials configured for your environment (gcloud, workload identity, etc...)\n",- "- Store the path to a service account JSON file as the GOOGLE_APPLICATION_CREDENTIALS environment variable\n",- "\n",- "This codebase uses the `google.auth` library which first looks for the application credentials variable mentioned above, and then looks for system-level auth.\n",- "\n",- "For more information, see:\n",- "- https://cloud.google.com/docs/authentication/application-default-credentials#GAC\n",- "- https://googleapis.dev/python/google-auth/latest/reference/google.auth.html#module-google.auth\n",- "\n",- "To enable automated tracing of your model calls, set your [LangSmith](https://docs.smith.langchain.com/) API key:"- ]- },- {- "cell_type": "code",- "execution_count": 1,- "id": "a15d341e-3e26-4ca3-830b-5aab30ed66de",- "metadata": {},- "outputs": [],- "source": [- "# os.environ[\"LANGSMITH_API_KEY\"] = getpass.getpass(\"Enter your LangSmith API key: \")\n",- "# os.environ[\"LANGSMITH_TRACING\"] = \"true\""- ]- },- {- "cell_type": "markdown",- "id": "0730d6a1-c893-4840-9817-5e5251676d5d",- "metadata": {},- "source": [- "### Installation\n",- "\n",- "The LangChain VertexAI integration lives in the `langchain-google-vertexai` package:"- ]- },- {- "cell_type": "code",- "execution_count": 2,- "id": "652d6238-1f87-422a-b135-f5abbb8652fc",- "metadata": {},- "outputs": [- {- "name": "stdout",- "output_type": "stream",- "text": [- "Note: you may need to restart the kernel to use updated packages.\n"- ]- }- ],- "source": [- "%pip install -qU langchain-google-vertexai"- ]- },- {- "cell_type": "markdown",- "id": "a38cde65-254d-4219-a441-068766c0d4b5",- "metadata": {},- "source": [- "## Instantiation\n",- "\n",- "Now we can instantiate our model object and generate chat completions:"- ]- },- {- "cell_type": "code",- "execution_count": 3,- "id": "cb09c344-1836-4e0c-acf8-11d13ac1dbae",- "metadata": {},- "outputs": [],- "source": [- "from langchain_google_vertexai import ChatVertexAI\n",- "\n",- "llm = ChatVertexAI(\n",- " model=\"gemini-1.5-flash-001\",\n",- " temperature=0,\n",- " max_tokens=None,\n",- " max_retries=6,\n",- " stop=None,\n",- " # other params...\n",- ")"- ]- },- {- "cell_type": "markdown",- "id": "2b4f3e15",- "metadata": {},- "source": [- "## Invocation"- ]- },- {- "cell_type": "code",- "execution_count": 4,- "id": "62e0dbc3",- "metadata": {- "tags": []- },- "outputs": [- {- "data": {- "text/plain": [- "AIMessage(content=\"J'adore programmer. \\n\", response_metadata={'is_blocked': False, 'safety_ratings': [{'category': 'HARM_CATEGORY_HATE_SPEECH', 'probability_label': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_DANGEROUS_CONTENT', 'probability_label': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_HARASSMENT', 'probability_label': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_SEXUALLY_EXPLICIT', 'probability_label': 'NEGLIGIBLE', 'blocked': False}], 'usage_metadata': {'prompt_token_count': 20, 'candidates_token_count': 7, 'total_token_count': 27}}, id='run-7032733c-d05c-4f0c-a17a-6c575fdd1ae0-0', usage_metadata={'input_tokens': 20, 'output_tokens': 7, 'total_tokens': 27})"- ]- },- "execution_count": 4,- "metadata": {},- "output_type": "execute_result"- }- ],- "source": [- "messages = [\n",- " (\n",- " \"system\",\n",- " \"You are a helpful assistant that translates English to French. Translate the user sentence.\",\n",- " ),\n",- " (\"human\", \"I love programming.\"),\n",- "]\n",- "ai_msg = llm.invoke(messages)\n",- "ai_msg"- ]- },+ "cells": [+ {+ "cell_type": "raw",+ "id": "afaf8039",+ "metadata": {},+ "source": [+ "---\n",+ "sidebar_label: Google Cloud Vertex AI\n",+ "---"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "e49f1e0d",+ "metadata": {},+ "source": [+ "# ChatVertexAI\n",+ "\n",+ "This page provides a quick overview for getting started with VertexAI [chat models](/docs/concepts/chat_models). For detailed documentation of all ChatVertexAI features and configurations head to the [API reference](https://python.langchain.com/api_reference/google_vertexai/chat_models/langchain_google_vertexai.chat_models.ChatVertexAI.html).\n",+ "\n",+ "ChatVertexAI exposes all foundational models available in Google Cloud, like `gemini-1.5-pro`, `gemini-1.5-flash`, etc. For a full and updated list of available models visit [VertexAI documentation](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/overview).\n",+ "\n",+ ":::info Google Cloud VertexAI vs Google PaLM\n",+ "\n",+ "The Google Cloud VertexAI integration is separate from the [Google PaLM integration](/docs/integrations/chat/google_generative_ai/). Google has chosen to offer an enterprise version of PaLM through GCP, and this supports the models made available through there.\n",+ "\n",+ ":::\n",+ "\n",+ "## Overview\n",+ "### Integration details\n",+ "\n",+ "| Class | Package | Local | Serializable | [JS support](https://js.langchain.com/docs/integrations/chat/google_vertex_ai) | Package downloads | Package latest |\n",+ "| :--- | :--- | :---: | :---: | :---: | :---: | :---: |\n",+ "| [ChatVertexAI](https://python.langchain.com/api_reference/google_vertexai/chat_models/langchain_google_vertexai.chat_models.ChatVertexAI.html) | [langchain-google-vertexai](https://python.langchain.com/api_reference/google_vertexai/index.html) | ❌ | beta | ✅ |  |  |\n",+ "\n",+ "### Model features\n",+ "| [Tool calling](/docs/how_to/tool_calling) | [Structured output](/docs/how_to/structured_output/) | JSON mode | [Image input](/docs/how_to/multimodal_inputs/) | Audio input | Video input | [Token-level streaming](/docs/how_to/chat_streaming/) | Native async | [Token usage](/docs/how_to/chat_token_usage_tracking/) | [Logprobs](/docs/how_to/logprobs/) |\n",+ "| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n",+ "| ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |\n",+ "\n",+ "## Setup\n",+ "\n",+ "To access VertexAI models you'll need to create a Google Cloud Platform account, set up credentials, and install the `langchain-google-vertexai` integration package.\n",+ "\n",+ "### Credentials\n",+ "\n",+ "To use the integration you must:\n",+ "- Have credentials configured for your environment (gcloud, workload identity, etc...)\n",+ "- Store the path to a service account JSON file as the GOOGLE_APPLICATION_CREDENTIALS environment variable\n",+ "\n",+ "This codebase uses the `google.auth` library which first looks for the application credentials variable mentioned above, and then looks for system-level auth.\n",+ "\n",+ "For more information, see:\n",+ "- https://cloud.google.com/docs/authentication/application-default-credentials#GAC\n",+ "- https://googleapis.dev/python/google-auth/latest/reference/google.auth.html#module-google.auth\n",+ "\n",+ "To enable automated tracing of your model calls, set your [LangSmith](https://docs.smith.langchain.com/) API key:"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 1,+ "id": "a15d341e-3e26-4ca3-830b-5aab30ed66de",+ "metadata": {},+ "outputs": [],+ "source": [+ "# os.environ[\"LANGSMITH_API_KEY\"] = getpass.getpass(\"Enter your LangSmith API key: \")\n",+ "# os.environ[\"LANGSMITH_TRACING\"] = \"true\""+ ]+ },+ {+ "cell_type": "markdown",+ "id": "0730d6a1-c893-4840-9817-5e5251676d5d",+ "metadata": {},+ "source": [+ "### Installation\n",+ "\n",+ "The LangChain VertexAI integration lives in the `langchain-google-vertexai` package:"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 2,+ "id": "652d6238-1f87-422a-b135-f5abbb8652fc",+ "metadata": {},+ "outputs": [{- "cell_type": "code",- "execution_count": 5,- "id": "d86145b3-bfef-46e8-b227-4dda5c9c2705",- "metadata": {},- "outputs": [- {- "name": "stdout",- "output_type": "stream",- "text": [- "J'adore programmer. \n",- "\n"- ]- }- ],- "source": [- "print(ai_msg.content)"- ]- },+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Note: you may need to restart the kernel to use updated packages.\n"+ ]+ }+ ],+ "source": [+ "%pip install -qU langchain-google-vertexai"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "a38cde65-254d-4219-a441-068766c0d4b5",+ "metadata": {},+ "source": [+ "## Instantiation\n",+ "\n",+ "Now we can instantiate our model object and generate chat completions:"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 3,+ "id": "cb09c344-1836-4e0c-acf8-11d13ac1dbae",+ "metadata": {},+ "outputs": [],+ "source": [+ "from langchain_google_vertexai import ChatVertexAI\n",+ "\n",+ "llm = ChatVertexAI(\n",+ " model=\"gemini-1.5-flash-001\",\n",+ " temperature=0,\n",+ " max_tokens=None,\n",+ " max_retries=6,\n",+ " stop=None,\n",+ " # other params...\n",+ ")"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "2b4f3e15",+ "metadata": {},+ "source": [+ "## Invocation"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 4,+ "id": "62e0dbc3",+ "metadata": {+ "tags": []+ },+ "outputs": [{- "cell_type": "markdown",- "id": "18e2bfc0-7e78-4528-a73f-499ac150dca8",- "metadata": {},- "source": [- "## Chaining\n",- "\n",- "We can [chain](/docs/how_to/sequence/) our model with a prompt template like so:"+ "data": {+ "text/plain": [+ "AIMessage(content=\"J'adore programmer. \\n\", response_metadata={'is_blocked': False, 'safety_ratings': [{'category': 'HARM_CATEGORY_HATE_SPEECH', 'probability_label': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_DANGEROUS_CONTENT', 'probability_label': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_HARASSMENT', 'probability_label': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_SEXUALLY_EXPLICIT', 'probability_label': 'NEGLIGIBLE', 'blocked': False}], 'usage_metadata': {'prompt_token_count': 20, 'candidates_token_count': 7, 'total_token_count': 27}}, id='run-7032733c-d05c-4f0c-a17a-6c575fdd1ae0-0', usage_metadata={'input_tokens': 20, 'output_tokens': 7, 'total_tokens': 27})"]- },+ },+ "execution_count": 4,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "messages = [\n",+ " (\n",+ " \"system\",\n",+ " \"You are a helpful assistant that translates English to French. Translate the user sentence.\",\n",+ " ),\n",+ " (\"human\", \"I love programming.\"),\n",+ "]\n",+ "ai_msg = llm.invoke(messages)\n",+ "ai_msg"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 5,+ "id": "d86145b3-bfef-46e8-b227-4dda5c9c2705",+ "metadata": {},+ "outputs": [{- "cell_type": "code",- "execution_count": 6,- "id": "e197d1d7-a070-4c96-9f8a-a0e86d046e0b",- "metadata": {},- "outputs": [- {- "data": {- "text/plain": [- "AIMessage(content='Ich liebe Programmieren. \\n', response_metadata={'is_blocked': False, 'safety_ratings': [{'category': 'HARM_CATEGORY_HATE_SPEECH', 'probability_label': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_DANGEROUS_CONTENT', 'probability_label': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_HARASSMENT', 'probability_label': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_SEXUALLY_EXPLICIT', 'probability_label': 'NEGLIGIBLE', 'blocked': False}], 'usage_metadata': {'prompt_token_count': 15, 'candidates_token_count': 8, 'total_token_count': 23}}, id='run-c71955fd-8dc1-422b-88a7-853accf4811b-0', usage_metadata={'input_tokens': 15, 'output_tokens': 8, 'total_tokens': 23})"- ]- },- "execution_count": 6,- "metadata": {},- "output_type": "execute_result"- }- ],- "source": [- "from langchain_core.prompts import ChatPromptTemplate\n",- "\n",- "prompt = ChatPromptTemplate.from_messages(\n",- " [\n",- " (\n",- " \"system\",\n",- " \"You are a helpful assistant that translates {input_language} to {output_language}.\",\n",- " ),\n",- " (\"human\", \"{input}\"),\n",- " ]\n",- ")\n",- "\n",- "chain = prompt | llm\n",- "chain.invoke(\n",- " {\n",- " \"input_language\": \"English\",\n",- " \"output_language\": \"German\",\n",- " \"input\": \"I love programming.\",\n",- " }\n",- ")"- ]- },+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "J'adore programmer. \n",+ "\n"+ ]+ }+ ],+ "source": [+ "print(ai_msg.content)"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "28ccabbb-a450-403c-8de1-fb077e0b5d3d",+ "metadata": {},+ "source": [+ "## Built-in tools\n",+ "\n",+ "Gemini supports a range of tools that are executed server-side.\n",+ "\n",+ "### Google search\n",+ "\n",+ ":::info Requires ``langchain-google-vertexai>=2.0.11``\n",+ ":::\n",+ "\n",+ "Gemini can execute a Google search and use the results to [ground its responses](https://ai.google.dev/gemini-api/docs/grounding):"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "ffdbec37-85f8-4755-bd72-47efaecfe944",+ "metadata": {},+ "outputs": [],+ "source": [+ "from langchain_google_vertexai import ChatVertexAI\n",+ "\n",+ "llm = ChatVertexAI(model=\"gemini-2.0-flash-001\").bind_tools([{\"google_search\": {}}])\n",+ "\n",+ "response = llm.invoke(\"What is today's news?\")"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "f63824f5-7d6a-4ad7-aa17-1f5c44119a21",+ "metadata": {},+ "source": [+ "### Code execution\n",+ "\n",+ ":::info Requires ``langchain-google-vertexai>=2.0.25``\n",+ ":::\n",+ "\n",+ "Gemini can [generate and execute Python code](https://ai.google.dev/gemini-api/docs/code-execution):"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "aa079529-ef1c-463d-9d25-6390423a328d",+ "metadata": {},+ "outputs": [],+ "source": [+ "from langchain_google_vertexai import ChatVertexAI\n",+ "\n",+ "llm = ChatVertexAI(model=\"gemini-2.0-flash-001\").bind_tools([{\"code_execution\": {}}])\n",+ "\n",+ "response = llm.invoke(\"What is 3^3?\")"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "18e2bfc0-7e78-4528-a73f-499ac150dca8",+ "metadata": {},+ "source": [+ "## Chaining\n",+ "\n",+ "We can [chain](/docs/how_to/sequence/) our model with a prompt template like so:"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 6,+ "id": "e197d1d7-a070-4c96-9f8a-a0e86d046e0b",+ "metadata": {},+ "outputs": [{- "cell_type": "markdown",- "id": "3a5bb5ca-c3ae-4a58-be67-2cd18574b9a3",- "metadata": {},- "source": [- "## API reference\n",- "\n",- "For detailed documentation of all ChatVertexAI features and configurations, like how to send multimodal inputs and configure safety settings, head to the API reference: https://python.langchain.com/api_reference/google_vertexai/chat_models/langchain_google_vertexai.chat_models.ChatVertexAI.html"+ "data": {+ "text/plain": [+ "AIMessage(content='Ich liebe Programmieren. \\n', response_metadata={'is_blocked': False, 'safety_ratings': [{'category': 'HARM_CATEGORY_HATE_SPEECH', 'probability_label': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_DANGEROUS_CONTENT', 'probability_label': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_HARASSMENT', 'probability_label': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_SEXUALLY_EXPLICIT', 'probability_label': 'NEGLIGIBLE', 'blocked': False}], 'usage_metadata': {'prompt_token_count': 15, 'candidates_token_count': 8, 'total_token_count': 23}}, id='run-c71955fd-8dc1-422b-88a7-853accf4811b-0', usage_metadata={'input_tokens': 15, 'output_tokens': 8, 'total_tokens': 23})"]+ },+ "execution_count": 6,+ "metadata": {},+ "output_type": "execute_result"}- ],- "metadata": {- "kernelspec": {- "display_name": "poetry-venv-2",- "language": "python",- "name": "poetry-venv-2"- },- "language_info": {- "codemirror_mode": {- "name": "ipython",- "version": 3- },- "file_extension": ".py",- "mimetype": "text/x-python",- "name": "python",- "nbconvert_exporter": "python",- "pygments_lexer": "ipython3",- "version": "3.9.1"- }+ ],+ "source": [+ "from langchain_core.prompts import ChatPromptTemplate\n",+ "\n",+ "prompt = ChatPromptTemplate.from_messages(\n",+ " [\n",+ " (\n",+ " \"system\",\n",+ " \"You are a helpful assistant that translates {input_language} to {output_language}.\",\n",+ " ),\n",+ " (\"human\", \"{input}\"),\n",+ " ]\n",+ ")\n",+ "\n",+ "chain = prompt | llm\n",+ "chain.invoke(\n",+ " {\n",+ " \"input_language\": \"English\",\n",+ " \"output_language\": \"German\",\n",+ " \"input\": \"I love programming.\",\n",+ " }\n",+ ")"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "3a5bb5ca-c3ae-4a58-be67-2cd18574b9a3",+ "metadata": {},+ "source": [+ "## API reference\n",+ "\n",+ "For detailed documentation of all ChatVertexAI features and configurations, like how to send multimodal inputs and configure safety settings, head to the API reference: https://python.langchain.com/api_reference/google_vertexai/chat_models/langchain_google_vertexai.chat_models.ChatVertexAI.html"+ ]+ }+ ],+ "metadata": {+ "kernelspec": {+ "display_name": "Python 3 (ipykernel)",+ "language": "python",+ "name": "python3"},- "nbformat": 4,- "nbformat_minor": 5+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.10.4"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 5}