Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI
Python agentagentopsagents-sdkaianthropicautogencost-estimationcrewaievalsevaluation-metricsgroqlangchainllmmistralollamaopenaiopenai-agents
updated notebook
reibs457bb5babdb5
diff --git a/examples/langchain_examples.ipynb b/examples/langchain_examples.ipynbindex 077ea331..c573c580 100644--- a/examples/langchain_examples.ipynb+++ b/examples/langchain_examples.ipynb@@ -2,198 +2,272 @@"cells": [{"cell_type": "markdown",+ "id": "e0deea1ab1db2a19",+ "metadata": {},"source": ["# AgentOps Langchain Agent Implementation\n","\n","Using AgentOps monitoring with Langchain is simple. We've created a LangchainCallbackHandler that will do all of the heavy lifting!\n","\n","First we'll import the typical Langchain packages:"- ],- "metadata": {- "collapsed": false- },- "id": "e0deea1ab1db2a19"+ ]},{"cell_type": "code","execution_count": 1,"id": "initial_id","metadata": {- "collapsed": true,"ExecuteTime": {"end_time": "2023-12-15T20:21:11.477270Z","start_time": "2023-12-15T20:21:10.289895Z"}},- "outputs": [- {- "name": "stderr",- "output_type": "stream",- "text": [- "/Users/braelynboynton/Developer/agentops/venv/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020\n",- " warnings.warn(\n"- ]- }- ],+ "outputs": [],"source": ["import os\n","from langchain.chat_models import ChatOpenAI\n","from langchain.agents import initialize_agent, AgentType\n",- "from dotenv import load_dotenv"+ "from dotenv import load_dotenv\n",+ "from langchain.agents import tool"]},{"cell_type": "markdown",+ "id": "57ddb8eca4e8a3cb",+ "metadata": {},"source": ["The only difference with using AgentOps is that we'll also import this special Callback Handler"- ],- "metadata": {- "collapsed": false- },- "id": "57ddb8eca4e8a3cb"+ ]},{"cell_type": "code","execution_count": 2,- "outputs": [],- "source": [- "from agentops.langchain_callback_handler import LangchainCallbackHandler"- ],+ "id": "585f00bb186711a7","metadata": {- "collapsed": false,"ExecuteTime": {"end_time": "2023-12-15T20:21:11.478111Z","start_time": "2023-12-15T20:21:11.471462Z"}},- "id": "585f00bb186711a7"+ "outputs": [],+ "source": [+ "from agentops.langchain_callback_handler import LangchainCallbackHandler"+ ]},{"cell_type": "markdown",+ "id": "523be945b85dc5d5",+ "metadata": {},"source": ["Next, we'll grab our two API keys. You can use dotenv like below or however else you like to load environment variables"- ],- "metadata": {- "collapsed": false- },- "id": "523be945b85dc5d5"+ ]},{"cell_type": "code","execution_count": 3,- "outputs": [],- "source": [- "load_dotenv()\n",- "OPENAI_API_KEY = os.getenv('OPENAI_API_KEY')\n",- "AGENTOPS_API_KEY = os.getenv('AGENTOPS_API_KEY')"- ],+ "id": "1490411415d7317c","metadata": {- "collapsed": false,"ExecuteTime": {"end_time": "2023-12-15T20:21:11.494019Z","start_time": "2023-12-15T20:21:11.479154Z"}},- "id": "1490411415d7317c"+ "outputs": [+ {+ "data": {+ "text/plain": [+ "False"+ ]+ },+ "execution_count": 3,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "load_dotenv()"+ ]},{"cell_type": "markdown",+ "id": "8371ec020e634dd0",+ "metadata": {},"source": [- "This is where AgentOps comes into play. Before creating our LLM instance via Langchain, first we'll create an instance of the AO LangchainCallbackHandler.\n",+ "This is where AgentOps comes into play. Before creating our LLM instance via Langchain, first we'll create an instance of the AO LangchainCallbackHandler. After the handler is initialiezd, a session will be recorded automatically.\n","\n","Pass in your API key, and optionally any tags to describe this session for easier lookup in the AO dashboard. You can also retrieve the `session_id` of the newly created session."- ],- "metadata": {- "collapsed": false- },- "id": "8371ec020e634dd0"+ ]},{"cell_type": "code","execution_count": 4,+ "id": "432921383f39c9d5",+ "metadata": {+ "ExecuteTime": {+ "end_time": "2023-12-15T20:21:12.346995Z",+ "start_time": "2023-12-15T20:21:11.483591Z"+ }+ },"outputs": [{"name": "stdout","output_type": "stream","text": [- "Agent Ops session ID: 0feeb023-d8eb-4eae-9c77-29c2a3c88b38\n"+ "Agent Ops session ID: 80c6593c-ae90-4b48-98e1-d17fcd801c70\n"]}],"source": [+ "AGENTOPS_API_KEY = os.environ.get('AGENTOPS_API_KEY')\n",+ "OPENAI_API_KEY = os.environ.get('OPENAI_API_KEY')\n",+ "\n","handler = LangchainCallbackHandler(api_key=AGENTOPS_API_KEY, tags=['Langchain Example'])\n",+ "\n","print(\"Agent Ops session ID: \" + handler.session_id)\n",- "llm = ChatOpenAI(openai_api_key=OPENAI_API_KEY, callbacks=[handler])"- ],- "metadata": {- "collapsed": false,- "ExecuteTime": {- "end_time": "2023-12-15T20:21:12.346995Z",- "start_time": "2023-12-15T20:21:11.483591Z"- }- },- "id": "432921383f39c9d5"+ "llm = ChatOpenAI(openai_api_key=OPENAI_API_KEY,\n",+ " model='gpt-3.5-turbo')"+ ]},{"cell_type": "markdown",+ "id": "93aa09ec",+ "metadata": {},"source": [- "Finally, let's use our agent! All the actions will be recorded in the AO Dashboard"- ],- "metadata": {- "collapsed": false- },- "id": "58bbca0b49302b2b"+ "Agents generally use tools. Let's define a simple tool here. Tool usage is also recorded."+ ]},{"cell_type": "code","execution_count": 5,+ "id": "6abf26f9",+ "metadata": {},"outputs": [],"source": [- "agent = initialize_agent([],\n",- " llm,\n",- " agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION,\n",- " verbose=True,\n",- " handle_parsing_errors=True)"- ],+ "@tool\n",+ "def find_movie(term) -> str:\n",+ " \"\"\"Find available movies\"\"\"\n",+ " return 'Citizen Kane'\n",+ "\n",+ "\n",+ "tools = [find_movie]"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "58bbca0b49302b2b",+ "metadata": {},+ "source": [+ "Finally, let's use our agent! Pass in the callback handler to the agent, and all the actions will be recorded in the AO Dashboard"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 6,+ "id": "d538b20aa954ee80","metadata": {- "collapsed": false,"ExecuteTime": {"end_time": "2023-12-15T20:21:12.352862Z","start_time": "2023-12-15T20:21:12.351126Z"}},- "id": "d538b20aa954ee80"+ "outputs": [],+ "source": [+ "agent = initialize_agent(tools,\n",+ " llm,\n",+ " agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION,\n",+ " verbose=True,\n",+ " callbacks=[handler], # You must pass in a callback handler to record your agent\n",+ " handle_parsing_errors=True)"+ ]},{"cell_type": "code",- "execution_count": null,- "outputs": [],- "source": [- "agent.run(\"Please write me a short poem about secret agents\")"+ "execution_count": 7,+ "id": "6dfb127553751384",+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "\n",+ "\n",+ "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",+ "\u001b[32;1m\u001b[1;3mCould not parse LLM output: I can use the `find_movie` tool to find the available movies.\u001b[0m\n",+ "Observation: Invalid or incomplete response\n",+ "Thought:\u001b[32;1m\u001b[1;3mCould not parse LLM output: I will use the `find_movie` tool to find the available movies.\u001b[0m\n",+ "Observation: Invalid or incomplete response\n",+ "Thought:\u001b[32;1m\u001b[1;3mI will use the `find_movie` tool to find the available movies.\n",+ "Action:\n",+ "```\n",+ "{\n",+ " \"action\": \"find_movie\",\n",+ " \"action_input\": \"\"\n",+ "}\n",+ "```\u001b[0m\n",+ "Observation: \u001b[36;1m\u001b[1;3mCitizen Kane\u001b[0m\n",+ "Thought:\u001b[32;1m\u001b[1;3mCould not parse LLM output: The movie \"Citizen Kane\" is currently playing.\u001b[0m\n",+ "Observation: Invalid or incomplete response\n",+ "Thought:\u001b[32;1m\u001b[1;3mI will use the `find_movie` tool to find the available movies.\n",+ "Action:\n",+ "```\n",+ "{\n",+ " \"action\": \"find_movie\",\n",+ " \"action_input\": \"\"\n",+ "}\n",+ "```\u001b[0m\n",+ "Observation: \u001b[36;1m\u001b[1;3mCitizen Kane\u001b[0m\n",+ "Thought:\u001b[32;1m\u001b[1;3mI will use the `find_movie` tool to find the available movies.\n",+ "Action:\n",+ "```\n",+ "{\n",+ " \"action\": \"find_movie\",\n",+ " \"action_input\": \"\"\n",+ "}\n",+ "```\n",+ "\u001b[0m\n",+ "Observation: \u001b[36;1m\u001b[1;3mCitizen Kane\u001b[0m\n",+ "Thought:\u001b[32;1m\u001b[1;3mCould not parse LLM output: I now know the final answer.\u001b[0m\n",+ "Observation: Invalid or incomplete response\n",+ "Thought:\u001b[32;1m\u001b[1;3mI now know the final answer.\n",+ "Final Answer: The movie \"Citizen Kane\" is currently playing.\u001b[0m\n",+ "\n",+ "\u001b[1m> Finished chain.\u001b[0m\n"+ ]+ },+ {+ "data": {+ "text/plain": [+ "'The movie \"Citizen Kane\" is currently playing.'"+ ]+ },+ "execution_count": 7,+ "metadata": {},+ "output_type": "execute_result"+ }],- "metadata": {- "collapsed": false- },- "id": "6dfb127553751384"+ "source": [+ "agent.run(\"What movies are playing?\", callbacks=[handler])"+ ]}],"metadata": {"kernelspec": {- "display_name": "Python 3",+ "display_name": "Python 3 (ipykernel)","language": "python","name": "python3"},"language_info": {"codemirror_mode": {"name": "ipython",- "version": 2+ "version": 3},"file_extension": ".py","mimetype": "text/x-python","name": "python","nbconvert_exporter": "python",- "pygments_lexer": "ipython2",- "version": "2.7.6"+ "pygments_lexer": "ipython3",+ "version": "3.11.3"}},"nbformat": 4,