The agent engineering platform.
Python agentsaiai-agentsanthropicchatgptdeepagentsenterpriseframeworkgeminigenerative-ailangchainlanggraphllmmultiagentopen-sourceopenaipydanticpythonragtypescript
Add OCI Generative AI new model support (#22880)
Rave Harpazf5ff7f178b3e
diff --git a/docs/docs/integrations/chat/oci_generative_ai.ipynb b/docs/docs/integrations/chat/oci_generative_ai.ipynbnew file mode 100644index 0000000000..4ce58a13fb--- /dev/null+++ b/docs/docs/integrations/chat/oci_generative_ai.ipynb@@ -0,0 +1,190 @@+{+ "cells": [+ {+ "cell_type": "raw",+ "id": "afaf8039",+ "metadata": {},+ "source": [+ "---\n",+ "sidebar_label: OCIGenAI\n",+ "---"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "e49f1e0d",+ "metadata": {},+ "source": [+ "# ChatOCIGenAI\n",+ "\n",+ "This notebook provides a quick overview for getting started with OCIGenAI [chat models](/docs/concepts/#chat-models). For detailed documentation of all ChatOCIGenAI features and configurations head to the [API reference](https://api.python.langchain.com/en/latest/chat_models/langchain_community.chat_models.oci_generative_ai.ChatOCIGenAI.html).\n",+ "\n",+ "Oracle Cloud Infrastructure (OCI) Generative AI is a fully managed service that provides a set of state-of-the-art, customizable large language models (LLMs) that cover a wide range of use cases, and which is available through a single API.\n",+ "Using the OCI Generative AI service you can access ready-to-use pretrained models, or create and host your own fine-tuned custom models based on your own data on dedicated AI clusters. Detailed documentation of the service and API is available __[here](https://docs.oracle.com/en-us/iaas/Content/generative-ai/home.htm)__ and __[here](https://docs.oracle.com/en-us/iaas/api/#/en/generative-ai/20231130/)__.\n",+ "\n",+ "\n",+ "## Overview\n",+ "### Integration details\n",+ "\n",+ "| Class | Package | Local | Serializable | [JS support](https://js.langchain.com/v0.2/docs/integrations/chat/oci_generative_ai) | Package downloads | Package latest |\n",+ "| :--- | :--- | :---: | :---: | :---: | :---: | :---: |\n",+ "| [ChatOCIGenAI](https://api.python.langchain.com/en/latest/chat_models/langchain_community.chat_models.oci_generative_ai.ChatOCIGenAI.html) | [langchain-community](https://api.python.langchain.com/en/latest/community_api_reference.html) | ❌ | ❌ | ❌ |  |  |\n",+ "\n",+ "### Model features\n",+ "| [Tool calling](/docs/how_to/tool_calling/) | [Structured output](/docs/how_to/structured_output/) | JSON mode | [Image input](/docs/how_to/multimodal_inputs/) | Audio input | Video input | [Token-level streaming](/docs/how_to/chat_streaming/) | Native async | [Token usage](/docs/how_to/chat_token_usage_tracking/) | [Logprobs](/docs/how_to/logprobs/) |\n",+ "| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n",+ "| ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | \n",+ "\n",+ "## Setup\n",+ "\n",+ "To access OCIGenAI models you'll need to install the `oci` and `langchain-community` packages.\n",+ "\n",+ "### Credentials\n",+ "\n",+ "The credentials and authentication methods supported for this integration are equivalent to those used with other OCI services and follow the __[standard SDK authentication](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdk_authentication_methods.htm)__ methods, specifically API Key, session token, instance principal, and resource principal.\n",+ "\n",+ "API key is the default authentication method used in the examples above. The following example demonstrates how to use a different authentication method (session token)"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "0730d6a1-c893-4840-9817-5e5251676d5d",+ "metadata": {},+ "source": [+ "### Installation\n",+ "\n",+ "The LangChain OCIGenAI integration lives in the `langchain-community` package and you will also need to install the `oci` package:"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "652d6238-1f87-422a-b135-f5abbb8652fc",+ "metadata": {},+ "outputs": [],+ "source": [+ "%pip install -qU langchain-community oci"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "a38cde65-254d-4219-a441-068766c0d4b5",+ "metadata": {},+ "source": [+ "## Instantiation\n",+ "\n",+ "Now we can instantiate our model object and generate chat completions:\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "cb09c344-1836-4e0c-acf8-11d13ac1dbae",+ "metadata": {},+ "outputs": [],+ "source": [+ "from langchain_community.chat_models.oci_generative_ai import ChatOCIGenAI\n",+ "from langchain_core.messages import AIMessage, HumanMessage, SystemMessage\n",+ "\n",+ "chat = ChatOCIGenAI(\n",+ " model_id=\"cohere.command-r-16k\",\n",+ " service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",+ " compartment_id=\"MY_OCID\",\n",+ " model_kwargs={\"temperature\": 0.7, \"max_tokens\": 500},\n",+ ")"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "2b4f3e15",+ "metadata": {},+ "source": [+ "## Invocation"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "62e0dbc3",+ "metadata": {+ "tags": []+ },+ "outputs": [],+ "source": [+ "messages = [\n",+ " SystemMessage(content=\"your are an AI assistant.\"),\n",+ " AIMessage(content=\"Hi there human!\"),\n",+ " HumanMessage(content=\"tell me a joke.\"),\n",+ "]\n",+ "response = chat.invoke(messages)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "d86145b3-bfef-46e8-b227-4dda5c9c2705",+ "metadata": {},+ "outputs": [],+ "source": [+ "print(response.content)"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "18e2bfc0-7e78-4528-a73f-499ac150dca8",+ "metadata": {},+ "source": [+ "## Chaining\n",+ "\n",+ "We can [chain](/docs/how_to/sequence/) our model with a prompt template like so:\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "e197d1d7-a070-4c96-9f8a-a0e86d046e0b",+ "metadata": {},+ "outputs": [],+ "source": [+ "from langchain_core.prompts import ChatPromptTemplate\n",+ "\n",+ "prompt = ChatPromptTemplate.from_template(\"Tell me a joke about {topic}\")\n",+ "chain = prompt | chat\n",+ "\n",+ "response = chain.invoke({\"topic\": \"dogs\"})\n",+ "print(response.content)"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "3a5bb5ca-c3ae-4a58-be67-2cd18574b9a3",+ "metadata": {},+ "source": [+ "## API reference\n",+ "\n",+ "For detailed documentation of all ChatOCIGenAI features and configurations head to the API reference: https://api.python.langchain.com/en/latest/chat_models/langchain_community.chat_models.oci_generative_ai.ChatOCIGenAI.html"+ ]+ }+ ],+ "metadata": {+ "kernelspec": {+ "display_name": "Python 3 (ipykernel)",+ "language": "python",+ "name": "python3"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.9.1"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 5+}diff --git a/docs/docs/integrations/llms/oci_generative_ai.ipynb b/docs/docs/integrations/llms/oci_generative_ai.ipynbindex 0c2368efdc..3da80aef0e 100644--- a/docs/docs/integrations/llms/oci_generative_ai.ipynb+++ b/docs/docs/integrations/llms/oci_generative_ai.ipynb@@ -14,15 +14,15 @@"Oracle Cloud Infrastructure (OCI) Generative AI is a fully managed service that provides a set of state-of-the-art, customizable large language models (LLMs) that cover a wide range of use cases, and which is available through a single API.\n","Using the OCI Generative AI service you can access ready-to-use pretrained models, or create and host your own fine-tuned custom models based on your own data on dedicated AI clusters. Detailed documentation of the service and API is available __[here](https://docs.oracle.com/en-us/iaas/Content/generative-ai/home.htm)__ and __[here](https://docs.oracle.com/en-us/iaas/api/#/en/generative-ai/20231130/)__.\n","\n",- "This notebook explains how to use OCI's Genrative AI models with LangChain."+ "This notebook explains how to use OCI's Generative AI complete models with LangChain."]},{"cell_type": "markdown","metadata": {},"source": [- "### Prerequisite\n",- "We will need to install the oci sdk"+ "## Setup\n",+ "Ensure that the oci sdk and the langchain-community package are installed"]},{@@ -31,38 +31,40 @@"metadata": {},"outputs": [],"source": [- "!pip install -U oci"+ "!pip install -U oci langchain-community"]},{"cell_type": "markdown","metadata": {},"source": [- "### OCI Generative AI API endpoint \n",- "https://inference.generativeai.us-chicago-1.oci.oraclecloud.com"+ "## Usage"]},{- "cell_type": "markdown",+ "cell_type": "code",+ "execution_count": null,"metadata": {},+ "outputs": [],"source": [- "## Authentication\n",- "The authentication methods supported for this langchain integration are:\n",+ "from langchain_community.llms.oci_generative_ai import OCIGenAI\n","\n",- "1. API Key\n",- "2. Session token\n",- "3. Instance principal\n",- "4. Resource principal \n",+ "llm = OCIGenAI(\n",+ " model_id=\"cohere.command\",\n",+ " service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",+ " compartment_id=\"MY_OCID\",\n",+ " model_kwargs={\"temperature\": 0, \"max_tokens\": 500},\n",+ ")\n","\n",- "These follows the standard SDK authentication methods detailed __[here](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdk_authentication_methods.htm)__.\n",- " "+ "response = llm.invoke(\"Tell me one fact about earth\", temperature=0.7)\n",+ "print(response)"]},{"cell_type": "markdown","metadata": {},"source": [- "## Usage"+ "#### Chaining with prompt templates"]},{@@ -71,44 +73,54 @@"metadata": {},"outputs": [],"source": [- "from langchain_community.llms import OCIGenAI\n",+ "from langchain_core.prompts import PromptTemplate\n","\n",- "# use default authN method API-key\n","llm = OCIGenAI(\n",- " model_id=\"MY_MODEL\",\n",+ " model_id=\"cohere.command\",\n"," service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n"," compartment_id=\"MY_OCID\",\n",+ " model_kwargs={\"temperature\": 0, \"max_tokens\": 500},\n",")\n","\n",- "response = llm.invoke(\"Tell me one fact about earth\", temperature=0.7)\n",+ "prompt = PromptTemplate(input_variables=[\"query\"], template=\"{query}\")\n",+ "llm_chain = prompt | llm\n",+ "\n",+ "response = llm_chain.invoke(\"what is the capital of france?\")\n","print(response)"]},+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "#### Streaming"+ ]+ },{"cell_type": "code","execution_count": null,"metadata": {},"outputs": [],"source": [- "from langchain.chains import LLMChain\n",- "from langchain_core.prompts import PromptTemplate\n",- "\n",- "# Use Session Token to authN\n","llm = OCIGenAI(\n",- " model_id=\"MY_MODEL\",\n",+ " model_id=\"cohere.command\",\n"," service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n"," compartment_id=\"MY_OCID\",\n",- " auth_type=\"SECURITY_TOKEN\",\n",- " auth_profile=\"MY_PROFILE\", # replace with your profile name\n",- " model_kwargs={\"temperature\": 0.7, \"top_p\": 0.75, \"max_tokens\": 200},\n",+ " model_kwargs={\"temperature\": 0, \"max_tokens\": 500},\n",")\n","\n",- "prompt = PromptTemplate(input_variables=[\"query\"], template=\"{query}\")\n",- "\n",- "llm_chain = LLMChain(llm=llm, prompt=prompt)\n",+ "for chunk in llm.stream(\"Write me a song about sparkling water.\"):\n",+ " print(chunk, end=\"\", flush=True)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## Authentication\n",+ "The authentication methods supported for LlamaIndex are equivalent to those used with other OCI services and follow the __[standard SDK authentication](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdk_authentication_methods.htm)__ methods, specifically API Key, session token, instance principal, and resource principal.\n","\n",- "response = llm_chain.invoke(\"what is the capital of france?\")\n",- "print(response)"+ "API key is the default authentication method used in the examples above. The following example demonstrates how to use a different authentication method (session token)"]},{@@ -117,49 +129,39 @@"metadata": {},"outputs": [],"source": [- "from langchain_community.embeddings import OCIGenAIEmbeddings\n",- "from langchain_community.vectorstores import FAISS\n",- "from langchain_core.output_parsers import StrOutputParser\n",- "from langchain_core.runnables import RunnablePassthrough\n",- "\n",- "embeddings = OCIGenAIEmbeddings(\n",- " model_id=\"MY_EMBEDDING_MODEL\",\n",+ "llm = OCIGenAI(\n",+ " model_id=\"cohere.command\",\n"," service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n"," compartment_id=\"MY_OCID\",\n",- ")\n",- "\n",- "vectorstore = FAISS.from_texts(\n",- " [\n",- " \"Larry Ellison co-founded Oracle Corporation in 1977 with Bob Miner and Ed Oates.\",\n",- " \"Oracle Corporation is an American multinational computer technology company headquartered in Austin, Texas, United States.\",\n",- " ],\n",- " embedding=embeddings,\n",- ")\n",- "\n",- "retriever = vectorstore.as_retriever()\n",- "\n",- "template = \"\"\"Answer the question based only on the following context:\n",- "{context}\n",- " \n",- "Question: {question}\n",- "\"\"\"\n",- "prompt = PromptTemplate.from_template(template)\n",+ " auth_type=\"SECURITY_TOKEN\",\n",+ " auth_profile=\"MY_PROFILE\", # replace with your profile name\n",+ ")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## Dedicated AI Cluster\n",+ "To access models hosted in a dedicated AI cluster __[create an endpoint](https://docs.oracle.com/en-us/iaas/api/#/en/generative-ai-inference/20231130/)__ whose assigned OCID (currently prefixed by ‘ocid1.generativeaiendpoint.oc1.us-chicago-1’) is used as your model ID.\n","\n",+ "When accessing models hosted in a dedicated AI cluster you will need to initialize the OCIGenAI interface with two extra required params (\"provider\" and \"context_size\")."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": ["llm = OCIGenAI(\n",- " model_id=\"MY_MODEL\",\n",+ " model_id=\"ocid1.generativeaiendpoint.oc1.us-chicago-1....\",\n"," service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",- " compartment_id=\"MY_OCID\",\n",- ")\n",- "\n",- "chain = (\n",- " {\"context\": retriever, \"question\": RunnablePassthrough()}\n",- " | prompt\n",- " | llm\n",- " | StrOutputParser()\n",- ")\n",- "\n",- "print(chain.invoke(\"when was oracle founded?\"))\n",- "print(chain.invoke(\"where is oracle headquartered?\"))"+ " compartment_id=\"DEDICATED_COMPARTMENT_OCID\",\n",+ " auth_profile=\"MY_PROFILE\", # replace with your profile name,\n",+ " provider=\"MODEL_PROVIDER\", # e.g., \"cohere\" or \"meta\"\n",+ " context_size=\"MODEL_CONTEXT_SIZE\", # e.g., 128000\n",+ ")"]}],diff --git a/docs/docs/integrations/providers/oci.mdx b/docs/docs/integrations/providers/oci.mdxindex e0b3570028..5037fb86f1 100644--- a/docs/docs/integrations/providers/oci.mdx+++ b/docs/docs/integrations/providers/oci.mdx@@ -2,27 +2,29 @@The `LangChain` integrations related to [Oracle Cloud Infrastructure](https://www.oracle.com/artificial-intelligence/).-## LLMs--### OCI Generative AI+## OCI Generative AI> Oracle Cloud Infrastructure (OCI) [Generative AI](https://docs.oracle.com/en-us/iaas/Content/generative-ai/home.htm) is a fully managed service that provides a set of state-of-the-art,> customizable large language models (LLMs) that cover a wide range of use cases, and which are available through a single API.> Using the OCI Generative AI service you can access ready-to-use pretrained models, or create and host your own fine-tuned> custom models based on your own data on dedicated AI clusters.-To use, you should have the latest `oci` python SDK installed.+To use, you should have the latest `oci` python SDK and the langchain_community package installed.```bash-pip install -U oci+pip install -U oci langchain-community```-See [usage examples](/docs/integrations/llms/oci_generative_ai).+See [chat](/docs/integrations/llms/oci_generative_ai), [complete](/docs/integrations/chat/oci_generative_ai), and [embedding](/docs/integrations/text_embedding/oci_generative_ai) usage examples.```python+from langchain_community.chat_models import ChatOCIGenAI+from langchain_community.llms import OCIGenAI++from langchain_community.embeddings import OCIGenAIEmbeddings```-### OCI Data Science Model Deployment Endpoint+## OCI Data Science Model Deployment Endpoint> [OCI Data Science](https://docs.oracle.com/en-us/iaas/data-science/using/home.htm) is a> fully managed and serverless platform for data science teams. Using the OCI Data Science@@ -47,12 +49,3 @@ from langchain_community.llms import OCIModelDeploymentVLLMfrom langchain_community.llms import OCIModelDeploymentTGI```-## Text Embedding Models--### OCI Generative AI--See [usage examples](/docs/integrations/text_embedding/oci_generative_ai).--```python-from langchain_community.embeddings import OCIGenAIEmbeddings-```\ No newline at end of filediff --git a/libs/community/extended_testing_deps.txt b/libs/community/extended_testing_deps.txtindex 9f5e8284af..db8d9cdfd0 100644--- a/libs/community/extended_testing_deps.txt+++ b/libs/community/extended_testing_deps.txt@@ -46,7 +46,7 @@ mwxml>=0.3.3,<0.4newspaper3k>=0.2.8,<0.3numexpr>=2.8.6,<3nvidia-riva-client>=2.14.0,<3-oci>=2.119.1,<3+oci>=2.128.0,<3openai<2openapi-pydantic>=0.3.2,<0.4oracle-ads>=2.9.1,<3diff --git a/libs/community/langchain_community/chat_models/__init__.py b/libs/community/langchain_community/chat_models/__init__.pyindex 7b942a26ca..af25b60184 100644--- a/libs/community/langchain_community/chat_models/__init__.py+++ b/libs/community/langchain_community/chat_models/__init__.py@@ -121,6 +121,9 @@ if TYPE_CHECKING:from langchain_community.chat_models.mlx import (ChatMLX,)+ from langchain_community.chat_models.oci_generative_ai import (+ ChatOCIGenAI, # noqa: F401+ )from langchain_community.chat_models.octoai import ChatOctoAIfrom langchain_community.chat_models.ollama import (ChatOllama,@@ -194,6 +197,7 @@ __all__ = ["ChatMLflowAIGateway","ChatMaritalk","ChatMlflow",+ "ChatOCIGenAI","ChatOllama","ChatOpenAI","ChatPerplexity",@@ -248,6 +252,7 @@ _module_lookup = {"ChatMaritalk": "langchain_community.chat_models.maritalk","ChatMlflow": "langchain_community.chat_models.mlflow","ChatOctoAI": "langchain_community.chat_models.octoai",+ "ChatOCIGenAI": "langchain_community.chat_models.oci_generative_ai","ChatOllama": "langchain_community.chat_models.ollama","ChatOpenAI": "langchain_community.chat_models.openai","ChatPerplexity": "langchain_community.chat_models.perplexity",diff --git a/libs/community/langchain_community/chat_models/oci_generative_ai.py b/libs/community/langchain_community/chat_models/oci_generative_ai.pynew file mode 100644index 0000000000..9409b1a2fb--- /dev/null+++ b/libs/community/langchain_community/chat_models/oci_generative_ai.py@@ -0,0 +1,363 @@+import json+from abc import ABC, abstractmethod+from typing import Any, Dict, Iterator, List, Mapping, Optional, Sequence++from langchain_core.callbacks import CallbackManagerForLLMRun+from langchain_core.language_models.chat_models import (+ BaseChatModel,+ generate_from_stream,+)+from langchain_core.messages import (+ AIMessage,+ AIMessageChunk,+ BaseMessage,+ ChatMessage,+ HumanMessage,+ SystemMessage,+)+from langchain_core.outputs import ChatGeneration, ChatGenerationChunk, ChatResult+from langchain_core.pydantic_v1 import Extra++from langchain_community.llms.oci_generative_ai import OCIGenAIBase+from langchain_community.llms.utils import enforce_stop_tokens++CUSTOM_ENDPOINT_PREFIX = "ocid1.generativeaiendpoint"+++class Provider(ABC):+ @property+ @abstractmethod+ def stop_sequence_key(self) -> str:+ ...++ @abstractmethod+ def chat_response_to_text(self, response: Any) -> str:+ ...++ @abstractmethod+ def chat_stream_to_text(self, event_data: Dict) -> str:+ ...++ @abstractmethod+ def chat_generation_info(self, response: Any) -> Dict[str, Any]:+ ...++ @abstractmethod+ def get_role(self, message: BaseMessage) -> str:+ ...++ @abstractmethod+ def messages_to_oci_params(self, messages: Any) -> Dict[str, Any]:+ ...+++class CohereProvider(Provider):+ stop_sequence_key = "stop_sequences"++ def __init__(self) -> None:+ from oci.generative_ai_inference import models++ self.oci_chat_request = models.CohereChatRequest+ self.oci_chat_message = {+ "USER": models.CohereUserMessage,+ "CHATBOT": models.CohereChatBotMessage,+ "SYSTEM": models.CohereSystemMessage,+ }+ self.chat_api_format = models.BaseChatRequest.API_FORMAT_COHERE++ def chat_response_to_text(self, response: Any) -> str:+ return response.data.chat_response.text++ def chat_stream_to_text(self, event_data: Dict) -> str:+ if "text" in event_data and "finishReason" not in event_data:+ return event_data["text"]+ else:+ return ""++ def chat_generation_info(self, response: Any) -> Dict[str, Any]:+ return {+ "finish_reason": response.data.chat_response.finish_reason,+ }++ def get_role(self, message: BaseMessage) -> str:+ if isinstance(message, HumanMessage):+ return "USER"+ elif isinstance(message, AIMessage):+ return "CHATBOT"+ elif isinstance(message, SystemMessage):+ return "SYSTEM"+ else:+ raise ValueError(f"Got unknown type {message}")++ def messages_to_oci_params(self, messages: Sequence[ChatMessage]) -> Dict[str, Any]:+ oci_chat_history = [+ self.oci_chat_message[self.get_role(msg)](message=msg.content)+ for msg in messages[:-1]+ ]+ oci_params = {+ "message": messages[-1].content,+ "chat_history": oci_chat_history,+ "api_format": self.chat_api_format,+ }++ return oci_params+++class MetaProvider(Provider):+ stop_sequence_key = "stop"++ def __init__(self) -> None:+ from oci.generative_ai_inference import models++ self.oci_chat_request = models.GenericChatRequest+ self.oci_chat_message = {+ "USER": models.UserMessage,+ "SYSTEM": models.SystemMessage,+ "ASSISTANT": models.AssistantMessage,+ }+ self.oci_chat_message_content = models.TextContent+ self.chat_api_format = models.BaseChatRequest.API_FORMAT_GENERIC++ def chat_response_to_text(self, response: Any) -> str:+ return response.data.chat_response.choices[0].message.content[0].text++ def chat_stream_to_text(self, event_data: Dict) -> str:+ if "message" in event_data:+ return event_data["message"]["content"][0]["text"]+ else:+ return ""++ def chat_generation_info(self, response: Any) -> Dict[str, Any]:+ return {+ "finish_reason": response.data.chat_response.choices[0].finish_reason,+ "time_created": str(response.data.chat_response.time_created),+ }++ def get_role(self, message: BaseMessage) -> str:+ # meta only supports alternating user/assistant roles+ if isinstance(message, HumanMessage):+ return "USER"+ elif isinstance(message, AIMessage):+ return "ASSISTANT"+ elif isinstance(message, SystemMessage):+ return "SYSTEM"+ else:+ raise ValueError(f"Got unknown type {message}")++ def messages_to_oci_params(self, messages: List[BaseMessage]) -> Dict[str, Any]:+ oci_messages = [+ self.oci_chat_message[self.get_role(msg)](+ content=[self.oci_chat_message_content(text=msg.content)]+ )+ for msg in messages+ ]+ oci_params = {+ "messages": oci_messages,+ "api_format": self.chat_api_format,+ "top_k": -1,+ }++ return oci_params+++class ChatOCIGenAI(BaseChatModel, OCIGenAIBase):+ """ChatOCIGenAI chat model integration.++ Setup:+ Install ``langchain-community`` and the ``oci`` sdk.++ .. code-block:: bash++ pip install -U langchain-community oci++ Key init args — completion params:+ model_id: str+ Id of the OCIGenAI chat model to use, e.g., cohere.command-r-16k.+ is_stream: bool+ Whether to stream back partial progress+ model_kwargs: Optional[Dict]+ Keyword arguments to pass to the specific model used, e.g., temperature, max_tokens.++ Key init args — client params:+ service_endpoint: str+ The endpoint URL for the OCIGenAI service, e.g., https://inference.generativeai.us-chicago-1.oci.oraclecloud.com.+ compartment_id: str+ The compartment OCID.+ auth_type: str+ The authentication type to use, e.g., API_KEY (default), SECURITY_TOKEN, INSTANCE_PRINCIPAL, RESOURCE_PRINCIPAL.+ auth_profile: Optional[str]+ The name of the profile in ~/.oci/config, if not specified , DEFAULT will be used.+ provider: str+ Provider name of the model. Default to None, will try to be derived from the model_id otherwise, requires user input.+ See full list of supported init args and their descriptions in the params section.++ Instantiate:+ .. code-block:: python++ from langchain_community.chat_models import ChatOCIGenAI++ chat = ChatOCIGenAI(+ model_id="cohere.command-r-16k",+ service_endpoint="https://inference.generativeai.us-chicago-1.oci.oraclecloud.com",+ compartment_id="MY_OCID",+ model_kwargs={"temperature": 0.7, "max_tokens": 500},+ )++ Invoke:+ .. code-block:: python+ messages = [+ SystemMessage(content="your are an AI assistant."),+ AIMessage(content="Hi there human!"),+ HumanMessage(content="tell me a joke."),+ ]+ response = chat.invoke(messages)++ Stream:+ .. code-block:: python++ for r in chat.stream(messages):+ print(r.content, end="", flush=True)++ Response metadata+ .. code-block:: python++ response = chat.invoke(messages)+ print(response.response_metadata)++ """ # noqa: E501++ class Config:+ """Configuration for this pydantic object."""++ extra = Extra.forbid++ @property+ def _llm_type(self) -> str:+ """Return type of llm."""+ return "oci_generative_ai_chat"++ @property+ def _provider_map(self) -> Mapping[str, Any]:+ """Get the provider map"""+ return {+ "cohere": CohereProvider(),+ "meta": MetaProvider(),+ }++ @property+ def _provider(self) -> Any:+ """Get the internal provider object"""+ return self._get_provider(provider_map=self._provider_map)++ def _prepare_request(+ self,+ messages: List[BaseMessage],+ stop: Optional[List[str]],+ kwargs: Dict[str, Any],+ stream: bool,+ ) -> Dict[str, Any]:+ try:+ from oci.generative_ai_inference import models++ except ImportError as ex:+ raise ModuleNotFoundError(+ "Could not import oci python package. "+ "Please make sure you have the oci package installed."+ ) from ex+ oci_params = self._provider.messages_to_oci_params(messages)+ oci_params["is_stream"] = stream # self.is_stream+ _model_kwargs = self.model_kwargs or {}++ if stop is not None:+ _model_kwargs[self._provider.stop_sequence_key] = stop++ chat_params = {**_model_kwargs, **kwargs, **oci_params}++ if self.model_id.startswith(CUSTOM_ENDPOINT_PREFIX):+ serving_mode = models.DedicatedServingMode(endpoint_id=self.model_id)+ else:+ serving_mode = models.OnDemandServingMode(model_id=self.model_id)++ request = models.ChatDetails(+ compartment_id=self.compartment_id,+ serving_mode=serving_mode,+ chat_request=self._provider.oci_chat_request(**chat_params),+ )++ return request++ def _generate(+ self,+ messages: List[BaseMessage],+ stop: Optional[List[str]] = None,+ run_manager: Optional[CallbackManagerForLLMRun] = None,+ **kwargs: Any,+ ) -> ChatResult:+ """Call out to a OCIGenAI chat model.++ Args:+ messages: list of LangChain messages+ stop: Optional list of stop words to use.++ Returns:+ LangChain ChatResult++ Example:+ .. code-block:: python++ messages = [+ HumanMessage(content="hello!"),+ AIMessage(content="Hi there human!"),+ HumanMessage(content="Meow!")+ ]++ response = llm.invoke(messages)+ """+ if self.is_stream:+ stream_iter = self._stream(+ messages, stop=stop, run_manager=run_manager, **kwargs+ )+ return generate_from_stream(stream_iter)++ request = self._prepare_request(messages, stop, kwargs, stream=False)+ response = self.client.chat(request)++ content = self._provider.chat_response_to_text(response)++ if stop is not None:+ content = enforce_stop_tokens(content, stop)++ generation_info = self._provider.chat_generation_info(response)++ llm_output = {+ "model_id": response.data.model_id,+ "model_version": response.data.model_version,+ "request_id": response.request_id,+ "content-length": response.headers["content-length"],+ }++ return ChatResult(+ generations=[+ ChatGeneration(+ message=AIMessage(content=content), generation_info=generation_info+ )+ ],+ llm_output=llm_output,+ )++ def _stream(+ self,+ messages: List[BaseMessage],+ stop: Optional[List[str]] = None,+ run_manager: Optional[CallbackManagerForLLMRun] = None,+ **kwargs: Any,+ ) -> Iterator[ChatGenerationChunk]:+ request = self._prepare_request(messages, stop, kwargs, stream=True)+ response = self.client.chat(request)++ for event in response.data.events():+ delta = self._provider.chat_stream_to_text(json.loads(event.data))+ chunk = ChatGenerationChunk(message=AIMessageChunk(content=delta))+ if run_manager:+ run_manager.on_llm_new_token(delta, chunk=chunk)+ yield chunkdiff --git a/libs/community/langchain_community/llms/oci_generative_ai.py b/libs/community/langchain_community/llms/oci_generative_ai.pyindex 2c2935cc76..178694656e 100644--- a/libs/community/langchain_community/llms/oci_generative_ai.py+++ b/libs/community/langchain_community/llms/oci_generative_ai.py@@ -1,17 +1,53 @@from __future__ import annotations-from abc import ABC+import json+from abc import ABC, abstractmethodfrom enum import Enum-from typing import Any, Dict, List, Mapping, Optional+from typing import Any, Dict, Iterator, List, Mapping, Optionalfrom langchain_core.callbacks import CallbackManagerForLLMRunfrom langchain_core.language_models.llms import LLM+from langchain_core.outputs import GenerationChunkfrom langchain_core.pydantic_v1 import BaseModel, Extra, root_validatorfrom langchain_community.llms.utils import enforce_stop_tokensCUSTOM_ENDPOINT_PREFIX = "ocid1.generativeaiendpoint"-VALID_PROVIDERS = ("cohere", "meta")+++class Provider(ABC):+ @property+ @abstractmethod+ def stop_sequence_key(self) -> str:+ ...++ @abstractmethod+ def completion_response_to_text(self, response: Any) -> str:+ ...+++class CohereProvider(Provider):+ stop_sequence_key = "stop_sequences"++ def __init__(self) -> None:+ from oci.generative_ai_inference import models++ self.llm_inference_request = models.CohereLlmInferenceRequest++ def completion_response_to_text(self, response: Any) -> str:+ return response.data.inference_response.generated_texts[0].text+++class MetaProvider(Provider):+ stop_sequence_key = "stop"++ def __init__(self) -> None:+ from oci.generative_ai_inference import models++ self.llm_inference_request = models.LlamaLlmInferenceRequest++ def completion_response_to_text(self, response: Any) -> str:+ return response.data.inference_response.choices[0].textclass OCIAuthType(Enum):@@ -33,8 +69,8 @@ class OCIGenAIBase(BaseModel, ABC):API_KEY,SECURITY_TOKEN,- INSTANCE_PRINCIPLE,- RESOURCE_PRINCIPLE+ INSTANCE_PRINCIPAL,+ RESOURCE_PRINCIPALIf not specified, API_KEY will be used"""@@ -65,11 +101,6 @@ class OCIGenAIBase(BaseModel, ABC):is_stream: bool = False"""Whether to stream back partial progress"""- llm_stop_sequence_mapping: Mapping[str, str] = {- "cohere": "stop_sequences",- "meta": "stop",- }-@root_validator()def validate_environment(cls, values: Dict) -> Dict:"""Validate that OCI config and python package exists in environment."""@@ -121,24 +152,28 @@ class OCIGenAIBase(BaseModel, ABC):"signer"] = oci.auth.signers.get_resource_principals_signer()else:- raise ValueError("Please provide valid value to auth_type")+ raise ValueError(+ "Please provide valid value to auth_type, "+ f"{values['auth_type']} is not valid."+ )values["client"] = oci.generative_ai_inference.GenerativeAiInferenceClient(**client_kwargs)except ImportError as ex:- raise ImportError(+ raise ModuleNotFoundError("Could not import oci python package. ""Please make sure you have the oci package installed.") from exexcept Exception as e:raise ValueError(- "Could not authenticate with OCI client. "- "Please check if ~/.oci/config exists. "- "If INSTANCE_PRINCIPLE or RESOURCE_PRINCIPLE is used, "- "Please check the specified "- "auth_profile and auth_type are valid."+ """Could not authenticate with OCI client.+ Please check if ~/.oci/config exists.+ If INSTANCE_PRINCIPAL or RESOURCE_PRINCIPAL is used,+ please check the specified+ auth_profile and auth_type are valid.""",+ e,) from ereturn values@@ -151,19 +186,19 @@ class OCIGenAIBase(BaseModel, ABC):**{"model_kwargs": _model_kwargs},}- def _get_provider(self) -> str:+ def _get_provider(self, provider_map: Mapping[str, Any]) -> Any:if self.provider is not None:provider = self.providerelse:provider = self.model_id.split(".")[0].lower()- if provider not in VALID_PROVIDERS:+ if provider not in provider_map:raise ValueError(f"Invalid provider derived from model_id: {self.model_id} ""Please explicitly pass in the supported provider ""when using custom endpoint")- return provider+ return provider_map[provider]class OCIGenAI(LLM, OCIGenAIBase):@@ -173,7 +208,7 @@ class OCIGenAI(LLM, OCIGenAIBase):https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdk_authentication_methods.htmThe authentifcation method is passed through auth_type and should be one of:- API_KEY (default), SECURITY_TOKEN, INSTANCE_PRINCIPLE, RESOURCE_PRINCIPLE+ API_KEY (default), SECURITY_TOKEN, INSTANCE_PRINCIPAL, RESOURCE_PRINCIPALMake sure you have the required policies (profile/roles) toaccess the OCI Generative AI service.@@ -204,21 +239,29 @@ class OCIGenAI(LLM, OCIGenAIBase):@propertydef _llm_type(self) -> str:"""Return type of llm."""- return "oci"+ return "oci_generative_ai_completion"++ @property+ def _provider_map(self) -> Mapping[str, Any]:+ """Get the provider map"""+ return {+ "cohere": CohereProvider(),+ "meta": MetaProvider(),+ }++ @property+ def _provider(self) -> Any:+ """Get the internal provider object"""+ return self._get_provider(provider_map=self._provider_map)def _prepare_invocation_object(self, prompt: str, stop: Optional[List[str]], kwargs: Dict[str, Any]) -> Dict[str, Any]:from oci.generative_ai_inference import models- oci_llm_request_mapping = {- "cohere": models.CohereLlmInferenceRequest,- "meta": models.LlamaLlmInferenceRequest,- }- provider = self._get_provider()_model_kwargs = self.model_kwargs or {}if stop is not None:- _model_kwargs[self.llm_stop_sequence_mapping[provider]] = stop+ _model_kwargs[self._provider.stop_sequence_key] = stopif self.model_id.startswith(CUSTOM_ENDPOINT_PREFIX):serving_mode = models.DedicatedServingMode(endpoint_id=self.model_id)@@ -232,19 +275,13 @@ class OCIGenAI(LLM, OCIGenAIBase):invocation_obj = models.GenerateTextDetails(compartment_id=self.compartment_id,serving_mode=serving_mode,- inference_request=oci_llm_request_mapping[provider](**inference_params),+ inference_request=self._provider.llm_inference_request(**inference_params),)return invocation_objdef _process_response(self, response: Any, stop: Optional[List[str]]) -> str:- provider = self._get_provider()- if provider == "cohere":- text = response.data.inference_response.generated_texts[0].text- elif provider == "meta":- text = response.data.inference_response.choices[0].text- else:- raise ValueError(f"Invalid provider: {provider}")+ text = self._provider.completion_response_to_text(response)if stop is not None:text = enforce_stop_tokens(text, stop)@@ -272,7 +309,51 @@ class OCIGenAI(LLM, OCIGenAIBase):response = llm.invoke("Tell me a joke.")"""+ if self.is_stream:+ text = ""+ for chunk in self._stream(prompt, stop, run_manager, **kwargs):+ text += chunk.text+ if stop is not None:+ text = enforce_stop_tokens(text, stop)+ return textinvocation_obj = self._prepare_invocation_object(prompt, stop, kwargs)response = self.client.generate_text(invocation_obj)return self._process_response(response, stop)++ def _stream(+ self,+ prompt: str,+ stop: Optional[List[str]] = None,+ run_manager: Optional[CallbackManagerForLLMRun] = None,+ **kwargs: Any,+ ) -> Iterator[GenerationChunk]:+ """Stream OCIGenAI LLM on given prompt.++ Args:+ prompt: The prompt to pass into the model.+ stop: Optional list of stop words to use when generating.++ Returns:+ An iterator of GenerationChunks.++ Example:+ .. code-block:: python++ response = llm.stream("Tell me a joke.")+ """++ self.is_stream = True+ invocation_obj = self._prepare_invocation_object(prompt, stop, kwargs)+ response = self.client.generate_text(invocation_obj)++ for event in response.data.events():+ json_load = json.loads(event.data)+ if "text" in json_load:+ event_data_text = json_load["text"]+ else:+ event_data_text = ""+ chunk = GenerationChunk(text=event_data_text)+ if run_manager:+ run_manager.on_llm_new_token(chunk.text, chunk=chunk)+ yield chunkdiff --git a/libs/community/tests/unit_tests/chat_models/test_imports.py b/libs/community/tests/unit_tests/chat_models/test_imports.pyindex a0e573068c..3c9b5e2254 100644--- a/libs/community/tests/unit_tests/chat_models/test_imports.py+++ b/libs/community/tests/unit_tests/chat_models/test_imports.py@@ -27,6 +27,7 @@ EXPECTED_ALL = ["ChatMlflow","ChatMLflowAIGateway","ChatMLX",+ "ChatOCIGenAI","ChatOllama","ChatOpenAI","ChatPerplexity",diff --git a/libs/community/tests/unit_tests/chat_models/test_oci_generative_ai.py b/libs/community/tests/unit_tests/chat_models/test_oci_generative_ai.pynew file mode 100644index 0000000000..b7d80d19c4--- /dev/null+++ b/libs/community/tests/unit_tests/chat_models/test_oci_generative_ai.py@@ -0,0 +1,105 @@+"""Test OCI Generative AI LLM service"""+from unittest.mock import MagicMock++import pytest+from langchain_core.messages import HumanMessage+from pytest import MonkeyPatch++from langchain_community.chat_models.oci_generative_ai import ChatOCIGenAI+++class MockResponseDict(dict):+ def __getattr__(self, val): # type: ignore[no-untyped-def]+ return self[val]++[email protected]("oci")+ "test_model_id", ["cohere.command-r-16k", "meta.llama-3-70b-instruct"]+)+def test_llm_chat(monkeypatch: MonkeyPatch, test_model_id: str) -> None:+ """Test valid chat call to OCI Generative AI LLM service."""+ oci_gen_ai_client = MagicMock()+ llm = ChatOCIGenAI(model_id=test_model_id, client=oci_gen_ai_client)++ provider = llm.model_id.split(".")[0].lower()++ def mocked_response(*args): # type: ignore[no-untyped-def]+ response_text = "Assistant chat reply."+ response = None+ if provider == "cohere":+ response = MockResponseDict(+ {+ "status": 200,+ "data": MockResponseDict(+ {+ "chat_response": MockResponseDict(+ {+ "text": response_text,+ "finish_reason": "completed",+ }+ ),+ "model_id": "cohere.command-r-16k",+ "model_version": "1.0.0",+ }+ ),+ "request_id": "1234567890",+ "headers": MockResponseDict(+ {+ "content-length": "123",+ }+ ),+ }+ )+ elif provider == "meta":+ response = MockResponseDict(+ {+ "status": 200,+ "data": MockResponseDict(+ {+ "chat_response": MockResponseDict(+ {+ "choices": [+ MockResponseDict(+ {+ "message": MockResponseDict(+ {+ "content": [+ MockResponseDict(+ {+ "text": response_text, # noqa: E501+ }+ )+ ]+ }+ ),+ "finish_reason": "completed",+ }+ )+ ],+ "time_created": "2024-09-01T00:00:00Z",+ }+ ),+ "model_id": "cohere.command-r-16k",+ "model_version": "1.0.0",+ }+ ),+ "request_id": "1234567890",+ "headers": MockResponseDict(+ {+ "content-length": "123",+ }+ ),+ }+ )+ return response++ monkeypatch.setattr(llm.client, "chat", mocked_response)++ messages = [+ HumanMessage(content="User message"),+ ]++ expected = "Assistant chat reply."+ actual = llm.invoke(messages, temperature=0.2)+ assert actual.content == expecteddiff --git a/libs/community/tests/unit_tests/llms/test_oci_generative_ai.py b/libs/community/tests/unit_tests/llms/test_oci_generative_ai.pyindex b1c36ec7a5..cc3599abe6 100644--- a/libs/community/tests/unit_tests/llms/test_oci_generative_ai.py+++ b/libs/community/tests/unit_tests/llms/test_oci_generative_ai.py@@ -4,7 +4,7 @@ from unittest.mock import MagicMockimport pytestfrom pytest import MonkeyPatch-from langchain_community.llms import OCIGenAI+from langchain_community.llms.oci_generative_ai import OCIGenAIclass MockResponseDict(dict):@@ -16,12 +16,12 @@ class MockResponseDict(dict):@pytest.mark.parametrize("test_model_id", ["cohere.command", "cohere.command-light", "meta.llama-2-70b-chat"])-def test_llm_call(monkeypatch: MonkeyPatch, test_model_id: str) -> None:- """Test valid call to OCI Generative AI LLM service."""+def test_llm_complete(monkeypatch: MonkeyPatch, test_model_id: str) -> None:+ """Test valid completion call to OCI Generative AI LLM service."""oci_gen_ai_client = MagicMock()llm = OCIGenAI(model_id=test_model_id, client=oci_gen_ai_client)- provider = llm._get_provider()+ provider = llm.model_id.split(".")[0].lower()def mocked_response(*args): # type: ignore[no-untyped-def]response_text = "This is the completion."@@ -71,6 +71,5 @@ def test_llm_call(monkeypatch: MonkeyPatch, test_model_id: str) -> None:)monkeypatch.setattr(llm.client, "generate_text", mocked_response)-output = llm.invoke("This is a prompt.", temperature=0.2)assert output == "This is the completion."