Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI
Python agentagentopsagents-sdkaianthropicautogencost-estimationcrewaievalsevaluation-metricsgroqlangchainllmmistralollamaopenaiopenai-agents
Removed broken images
Howard Gila6e9dedb4afb
diff --git a/examples/langchain/langchain_examples.ipynb b/examples/langchain/langchain_examples.ipynbindex 5e198cd7..3b083046 100644--- a/examples/langchain/langchain_examples.ipynb+++ b/examples/langchain/langchain_examples.ipynb@@ -13,96 +13,98 @@]},{+ "cell_type": "code",+ "execution_count": 1,+ "id": "8e3d38adc7861277","metadata": {"ExecuteTime": {"end_time": "2024-08-08T00:29:29.288644Z","start_time": "2024-08-08T00:29:29.026314Z"}},- "cell_type": "code",+ "outputs": [],"source": ["%pip install langchain==0.2.9\n","%pip install -U agentops\n","%pip install -U python-dotenv"- ],- "id": "8e3d38adc7861277",- "outputs": [],- "execution_count": 1+ ]},{- "metadata": {},"cell_type": "markdown",- "source": "Then import them",- "id": "effc8ee7453a6c3"+ "id": "effc8ee7453a6c3",+ "metadata": {},+ "source": [+ "Then import them"+ ]},{"cell_type": "code",- "source": [- "import os\n",- "from langchain_openai import ChatOpenAI\n",- "from langchain.agents import tool, AgentExecutor, create_openai_tools_agent\n",- "from dotenv import load_dotenv"- ],+ "execution_count": 2,+ "id": "d344305baa26d651","metadata": {- "collapsed": false,"ExecuteTime": {"end_time": "2024-08-08T00:29:30.052348Z","start_time": "2024-08-08T00:29:29.289841Z"- }+ },+ "collapsed": false},- "id": "d344305baa26d651","outputs": [],- "execution_count": 2+ "source": [+ "import os\n",+ "from langchain_openai import ChatOpenAI\n",+ "from langchain.agents import tool, AgentExecutor, create_openai_tools_agent\n",+ "from dotenv import load_dotenv"+ ]},{"cell_type": "markdown",- "source": [- "The only difference with using AgentOps is that we'll also import this special Callback Handler"- ],+ "id": "3584d84a441169b0","metadata": {"collapsed": false},- "id": "3584d84a441169b0"+ "source": [+ "The only difference with using AgentOps is that we'll also import this special Callback Handler"+ ]},{"cell_type": "code",- "source": [- "from agentops.partners.langchain_callback_handler import (\n",- " LangchainCallbackHandler as AgentOpsLangchainCallbackHandler,\n",- ")"- ],+ "execution_count": 3,+ "id": "256f01cac66d7d90","metadata": {- "collapsed": false,"ExecuteTime": {"end_time": "2024-08-08T00:29:30.058535Z","start_time": "2024-08-08T00:29:30.054497Z"- }+ },+ "collapsed": false},- "id": "256f01cac66d7d90","outputs": [],- "execution_count": 3+ "source": [+ "from agentops.partners.langchain_callback_handler import (\n",+ " LangchainCallbackHandler as AgentOpsLangchainCallbackHandler,\n",+ ")"+ ]},{"cell_type": "markdown",- "source": [- "Next, we'll grab our two API keys. You can use dotenv like below or however else you like to load environment variables"- ],+ "id": "afab0f7bbd847916","metadata": {"collapsed": false},- "id": "afab0f7bbd847916"+ "source": [+ "Next, we'll grab our two API keys. You can use dotenv like below or however else you like to load environment variables"+ ]},{"cell_type": "code",- "source": "load_dotenv()",+ "execution_count": 4,+ "id": "3f9189f4d13ebd07","metadata": {- "collapsed": false,"ExecuteTime": {"end_time": "2024-08-08T00:29:30.072366Z","start_time": "2024-08-08T00:29:30.063997Z"- }+ },+ "collapsed": false},- "id": "3f9189f4d13ebd07","outputs": [{"data": {@@ -115,22 +117,42 @@"output_type": "execute_result"}],- "execution_count": 4+ "source": [+ "load_dotenv()"+ ]},{"cell_type": "markdown",+ "id": "6f5a9ab030c636c6",+ "metadata": {+ "collapsed": false+ },"source": ["This is where AgentOps comes into play. Before creating our LLM instance via Langchain, first we'll create an instance of the AO LangchainCallbackHandler. After the handler is initialized, a session will be recorded automatically.\n","\n","Pass in your API key, and optionally any tags to describe this session for easier lookup in the AO dashboard."- ],- "metadata": {- "collapsed": false- },- "id": "6f5a9ab030c636c6"+ ]},{"cell_type": "code",+ "execution_count": 5,+ "id": "7d4accd2f68404fb",+ "metadata": {+ "ExecuteTime": {+ "end_time": "2024-08-08T00:29:31.018329Z",+ "start_time": "2024-08-08T00:29:30.074528Z"+ },+ "collapsed": false+ },+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "🖇 AgentOps: \u001b[34m\u001b[34mSession Replay: https://app.agentops.ai/drilldown?session_id=418f60fc-a709-4a61-952c-6b9670406198\u001b[0m\u001b[0m\n"+ ]+ }+ ],"source": ["from langchain_core.prompts import ChatPromptTemplate\n","\n",@@ -157,49 +179,29 @@" # (\"tool_names\", \"find_movie\")\n"," ]\n",")"- ],- "metadata": {- "collapsed": false,- "ExecuteTime": {- "end_time": "2024-08-08T00:29:31.018329Z",- "start_time": "2024-08-08T00:29:30.074528Z"- }- },- "id": "7d4accd2f68404fb",- "outputs": [- {- "name": "stderr",- "output_type": "stream",- "text": [- "🖇 AgentOps: \u001B[34m\u001B[34mSession Replay: https://app.agentops.ai/drilldown?session_id=418f60fc-a709-4a61-952c-6b9670406198\u001B[0m\u001B[0m\n"- ]- }- ],- "execution_count": 5+ ]},{"cell_type": "markdown",- "source": [- "You can also retrieve the `session_id` of the newly created session."- ],+ "id": "821b2367b1082673","metadata": {"collapsed": false},- "id": "821b2367b1082673"+ "source": [+ "You can also retrieve the `session_id` of the newly created session."+ ]},{"cell_type": "code",- "source": [- "print(\"Agent Ops session ID: \" + str(agentops_handler.current_session_ids))"- ],+ "execution_count": 6,+ "id": "6bb30c31eaa5ba02","metadata": {- "collapsed": false,"ExecuteTime": {"end_time": "2024-08-08T00:29:31.021351Z","start_time": "2024-08-08T00:29:31.019165Z"- }+ },+ "collapsed": false},- "id": "6bb30c31eaa5ba02","outputs": [{"name": "stdout",@@ -209,20 +211,32 @@]}],- "execution_count": 6+ "source": [+ "print(\"Agent Ops session ID: \" + str(agentops_handler.current_session_ids))"+ ]},{"cell_type": "markdown",- "source": [- "Agents generally use tools. Let's define a simple tool here. Tool usage is also recorded."- ],+ "id": "f7362d89b6a7af4c","metadata": {"collapsed": false},- "id": "f7362d89b6a7af4c"+ "source": [+ "Agents generally use tools. Let's define a simple tool here. Tool usage is also recorded."+ ]},{"cell_type": "code",+ "execution_count": 7,+ "id": "4a28f85842129016",+ "metadata": {+ "ExecuteTime": {+ "end_time": "2024-08-08T00:29:31.071133Z",+ "start_time": "2024-08-08T00:29:31.022304Z"+ },+ "collapsed": false+ },+ "outputs": [],"source": ["@tool\n","def find_movie(genre: str) -> str:\n",@@ -234,97 +248,89 @@"\n","\n","tools = [find_movie]"- ],- "metadata": {- "collapsed": false,- "ExecuteTime": {- "end_time": "2024-08-08T00:29:31.071133Z",- "start_time": "2024-08-08T00:29:31.022304Z"- }- },- "id": "4a28f85842129016",- "outputs": [],- "execution_count": 7+ ]},{"cell_type": "markdown",- "source": [- "For each tool, you need to also add the callback handler"- ],+ "id": "186984add993f839","metadata": {"collapsed": false},- "id": "186984add993f839"+ "source": [+ "For each tool, you need to also add the callback handler"+ ]},{"cell_type": "code",- "source": [- "for t in tools:\n",- " t.callbacks = [agentops_handler]"- ],+ "execution_count": 8,+ "id": "afe9113b61c67e80","metadata": {- "collapsed": false,"ExecuteTime": {"end_time": "2024-08-08T00:29:31.073655Z","start_time": "2024-08-08T00:29:31.071945Z"- }+ },+ "collapsed": false},- "id": "afe9113b61c67e80","outputs": [],- "execution_count": 8+ "source": [+ "for t in tools:\n",+ " t.callbacks = [agentops_handler]"+ ]},{- "metadata": {},"cell_type": "markdown",- "source": "Add the tools to our LLM",- "id": "6dd5f322d26f692"+ "id": "6dd5f322d26f692",+ "metadata": {},+ "source": [+ "Add the tools to our LLM"+ ]},{- "metadata": {},"cell_type": "code",- "outputs": [],"execution_count": 9,- "source": "llm_with_tools = llm.bind_tools([find_movie])",- "id": "e469ab337023053e"+ "id": "e469ab337023053e",+ "metadata": {},+ "outputs": [],+ "source": [+ "llm_with_tools = llm.bind_tools([find_movie])"+ ]},{"cell_type": "markdown",- "source": "Finally, let's create our agent! Pass in the callback handler to the agent, and all the actions will be recorded in the AO Dashboard",+ "id": "88b96e62db542900","metadata": {"collapsed": false},- "id": "88b96e62db542900"+ "source": [+ "Finally, let's create our agent! Pass in the callback handler to the agent, and all the actions will be recorded in the AO Dashboard"+ ]},{+ "cell_type": "code",+ "execution_count": 10,+ "id": "d45ff1b7a3843191","metadata": {"ExecuteTime": {"end_time": "2024-08-08T00:29:31.127744Z","start_time": "2024-08-08T00:29:31.124945Z"}},- "cell_type": "code",+ "outputs": [],"source": ["agent = create_openai_tools_agent(llm, tools, prompt)\n","agent_executor = AgentExecutor(agent=agent, tools=tools)"- ],- "id": "d45ff1b7a3843191",- "outputs": [],- "execution_count": 10+ ]},{+ "cell_type": "code",+ "execution_count": 11,+ "id": "bb90bf54b5a42a63","metadata": {"ExecuteTime": {"end_time": "2024-08-08T00:29:32.666768Z","start_time": "2024-08-08T00:29:31.128749Z"}},- "cell_type": "code",- "source": [- "agent_executor.invoke(\n",- " {\"input\": \"What comedies are playing?\"}, config={\"callback\": [agentops_handler]}\n",- ")"- ],- "id": "bb90bf54b5a42a63","outputs": [{"data": {@@ -338,48 +344,59 @@"output_type": "execute_result"}],- "execution_count": 11+ "source": [+ "agent_executor.invoke(\n",+ " {\"input\": \"What comedies are playing?\"}, config={\"callback\": [agentops_handler]}\n",+ ")"+ ]},{- "metadata": {},"cell_type": "markdown",- "source": "Now if we look in the AgentOps dashboard, you will see a session recorded with the LLM calls and tool usage.",- "id": "c466cb71c9dc68db"+ "id": "c466cb71c9dc68db",+ "metadata": {},+ "source": [+ "Now if we look in the AgentOps dashboard, you will see a session recorded with the LLM calls and tool usage."+ ]},{- "metadata": {},"cell_type": "code",- "outputs": [],"execution_count": null,- "source": "assert False",- "id": "8571b7ae931b3cb2"+ "id": "8571b7ae931b3cb2",+ "metadata": {},+ "outputs": [],+ "source": [+ "assert False"+ ]},{"cell_type": "markdown",- "source": [- "## Langchain V0.1 Example\n",- "This example is out of date."- ],+ "id": "97a1a264a71876","metadata": {"collapsed": false},- "id": "97a1a264a71876"+ "source": [+ "## Langchain V0.1 Example\n",+ "This example is out of date."+ ]},{+ "cell_type": "code",+ "execution_count": null,+ "id": "decac159bb492462","metadata": {"ExecuteTime": {"end_time": "2024-08-08T00:29:32.741186Z","start_time": "2024-08-08T00:29:32.741133Z"}},- "cell_type": "code",- "source": "%pip install langchain==0.1.6",- "id": "decac159bb492462","outputs": [],- "execution_count": null+ "source": [+ "%pip install langchain==0.1.6"+ ]},{"cell_type": "code",+ "execution_count": null,"id": "initial_id","metadata": {"ExecuteTime": {@@ -387,14 +404,13 @@"start_time": "2024-08-08T00:29:32.741599Z"}},+ "outputs": [],"source": ["import os\n","from langchain_openai import ChatOpenAI\n","from langchain.agents import initialize_agent, AgentType\n","from langchain.agents import tool"- ],- "outputs": [],- "execution_count": null+ ]},{"cell_type": "markdown",@@ -406,56 +422,62 @@},{"cell_type": "code",- "source": [- "from agentops.partners.langchain_callback_handler import (\n",- " LangchainCallbackHandler as AgentOpsLangchainCallbackHandler,\n",- ")"- ],+ "execution_count": null,+ "id": "7e8f8cd098ad5b57","metadata": {"collapsed": false},- "id": "7e8f8cd098ad5b57","outputs": [],- "execution_count": null+ "source": [+ "from agentops.partners.langchain_callback_handler import (\n",+ " LangchainCallbackHandler as AgentOpsLangchainCallbackHandler,\n",+ ")"+ ]},{"cell_type": "markdown",- "source": [- "Next, we'll grab our two API keys. You can use dotenv like below or however else you like to load environment variables"- ],+ "id": "14a1b8e08a2e9eb3","metadata": {"collapsed": false},- "id": "14a1b8e08a2e9eb3"+ "source": [+ "Next, we'll grab our two API keys. You can use dotenv like below or however else you like to load environment variables"+ ]},{"cell_type": "code",- "source": [- "from dotenv import load_dotenv\n",- "\n",- "load_dotenv()"- ],+ "execution_count": null,+ "id": "ff6cfc570599871f","metadata": {"collapsed": false},- "id": "ff6cfc570599871f","outputs": [],- "execution_count": null+ "source": [+ "from dotenv import load_dotenv\n",+ "\n",+ "load_dotenv()"+ ]},{"cell_type": "markdown",+ "id": "51f083697b783fa4",+ "metadata": {+ "collapsed": false+ },"source": ["This is where AgentOps comes into play. Before creating our LLM instance via Langchain, first we'll create an instance of the AO LangchainCallbackHandler. After the handler is initialized, a session will be recorded automatically.\n","\n","Pass in your API key, and optionally any tags to describe this session for easier lookup in the AO dashboard."- ],- "metadata": {- "collapsed": false- },- "id": "51f083697b783fa4"+ ]},{"cell_type": "code",+ "execution_count": null,+ "id": "d432fe915edb6365",+ "metadata": {+ "collapsed": false+ },+ "outputs": [],"source": ["AGENTOPS_API_KEY = os.environ.get(\"AGENTOPS_API_KEY\")\n","OPENAI_API_KEY = os.environ.get(\"OPENAI_API_KEY\")\n",@@ -467,48 +489,48 @@"llm = ChatOpenAI(\n"," openai_api_key=OPENAI_API_KEY, callbacks=[agentops_handler], model=\"gpt-3.5-turbo\"\n",")"- ],- "metadata": {- "collapsed": false- },- "id": "d432fe915edb6365",- "outputs": [],- "execution_count": null+ ]},{"cell_type": "markdown",- "source": [- "You can also retrieve the `session_id` of the newly created session."- ],+ "id": "38d309f07363b58e","metadata": {"collapsed": false},- "id": "38d309f07363b58e"+ "source": [+ "You can also retrieve the `session_id` of the newly created session."+ ]},{"cell_type": "code",- "source": [- "print(\"Agent Ops session ID: \" + str(agentops_handler.current_session_ids))"- ],+ "execution_count": null,+ "id": "f7e3a37cde3f9c22","metadata": {"collapsed": false},- "id": "f7e3a37cde3f9c22","outputs": [],- "execution_count": null+ "source": [+ "print(\"Agent Ops session ID: \" + str(agentops_handler.current_session_ids))"+ ]},{"cell_type": "markdown",- "source": [- "Agents generally use tools. Let's define a simple tool here. Tool usage is also recorded."- ],+ "id": "42f226ace56ef6f5","metadata": {"collapsed": false},- "id": "42f226ace56ef6f5"+ "source": [+ "Agents generally use tools. Let's define a simple tool here. Tool usage is also recorded."+ ]},{"cell_type": "code",+ "execution_count": null,+ "id": "c103a2edbe837abd",+ "metadata": {+ "collapsed": false+ },+ "outputs": [],"source": ["@tool\n","def find_movie(genre) -> str:\n",@@ -520,49 +542,49 @@"\n","\n","tools = [find_movie]"- ],- "metadata": {- "collapsed": false- },- "id": "c103a2edbe837abd",- "outputs": [],- "execution_count": null+ ]},{"cell_type": "markdown",- "source": [- "For each tool, you need to also add the callback handler"- ],+ "id": "4fb7633857b19bf0","metadata": {"collapsed": false},- "id": "4fb7633857b19bf0"+ "source": [+ "For each tool, you need to also add the callback handler"+ ]},{"cell_type": "code",- "source": [- "for t in tools:\n",- " t.callbacks = [agentops_handler]"- ],+ "execution_count": null,+ "id": "a0345f08bf1c5ecd","metadata": {"collapsed": false},- "id": "a0345f08bf1c5ecd","outputs": [],- "execution_count": null+ "source": [+ "for t in tools:\n",+ " t.callbacks = [agentops_handler]"+ ]},{"cell_type": "markdown",- "source": [- "Finally, let's use our agent! Pass in the callback handler to the agent, and all the actions will be recorded in the AO Dashboard"- ],+ "id": "12a02b833716676b","metadata": {"collapsed": false},- "id": "12a02b833716676b"+ "source": [+ "Finally, let's use our agent! Pass in the callback handler to the agent, and all the actions will be recorded in the AO Dashboard"+ ]},{"cell_type": "code",+ "execution_count": null,+ "id": "2d2e83fa69b30add",+ "metadata": {+ "collapsed": false+ },+ "outputs": [],"source": ["agent = initialize_agent(\n"," tools,\n",@@ -574,81 +596,80 @@" ], # You must pass in a callback handler to record your agent\n"," handle_parsing_errors=True,\n",")"- ],- "metadata": {- "collapsed": false- },- "id": "2d2e83fa69b30add",- "outputs": [],- "execution_count": null+ ]},{"cell_type": "code",- "source": [- "agent.invoke(\"What comedies are playing?\", callbacks=[agentops_handler])"- ],+ "execution_count": null,+ "id": "df2bc3a384493e1e","metadata": {"collapsed": false},- "id": "df2bc3a384493e1e","outputs": [],- "execution_count": null+ "source": [+ "agent.invoke(\"What comedies are playing?\", callbacks=[agentops_handler])"+ ]},{"cell_type": "markdown",- "source": [- "## Check your session\n",- "Finally, check your run on [AgentOps](https://app.agentops.ai)\n",- ""- ],+ "id": "2230edd919182a55","metadata": {"collapsed": false},- "id": "2230edd919182a55"+ "source": [+ "## Check your session\n",+ "Finally, check your run on [AgentOps](https://app.agentops.ai)"+ ]},{"cell_type": "markdown",+ "id": "fbf4a3ec5fa60d74",+ "metadata": {+ "collapsed": false+ },"source": ["# Async Agents\n","\n","Several langchain agents require async callback handlers. AgentOps also supports this."- ],- "metadata": {- "collapsed": false- },- "id": "fbf4a3ec5fa60d74"+ ]},{"cell_type": "code",+ "execution_count": null,+ "id": "ed63a166b343e1a2",+ "metadata": {+ "collapsed": false+ },+ "outputs": [],"source": ["import os\n","from langchain.chat_models import ChatOpenAI\n","from langchain.agents import initialize_agent, AgentType\n","from langchain.agents import tool"- ],+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "aa15223969f97b3d","metadata": {"collapsed": false},- "id": "ed63a166b343e1a2","outputs": [],- "execution_count": null- },- {- "cell_type": "code","source": ["from agentops.partners.langchain_callback_handler import (\n"," AsyncLangchainCallbackHandler as AgentOpsAsyncLangchainCallbackHandler,\n",")"- ],+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "818357483f039b60","metadata": {"collapsed": false},- "id": "aa15223969f97b3d","outputs": [],- "execution_count": null- },- {- "cell_type": "code","source": ["from dotenv import load_dotenv\n","\n",@@ -656,16 +677,16 @@"\n","AGENTOPS_API_KEY = os.environ.get(\"AGENTOPS_API_KEY\")\n","OPENAI_API_KEY = os.environ.get(\"OPENAI_API_KEY\")"- ],+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "ae76cfe058f5e4e4","metadata": {"collapsed": false},- "id": "818357483f039b60","outputs": [],- "execution_count": null- },- {- "cell_type": "code","source": ["agentops_handler = AgentOpsAsyncLangchainCallbackHandler(\n"," api_key=AGENTOPS_API_KEY, tags=[\"Async Example\"]\n",@@ -676,16 +697,16 @@")\n","\n","print(\"Agent Ops session ID: \" + str(await agentops_handler.session_id))"- ],+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "1201049766be84a7","metadata": {"collapsed": false},- "id": "ae76cfe058f5e4e4","outputs": [],- "execution_count": null- },- {- "cell_type": "code","source": ["@tool\n","def find_movie(genre) -> str:\n",@@ -700,16 +721,16 @@"\n","for t in tools:\n"," t.callbacks = [agentops_handler]"- ],+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "8d4f9dd39b79d542","metadata": {"collapsed": false},- "id": "1201049766be84a7","outputs": [],- "execution_count": null- },- {- "cell_type": "code","source": ["agent = initialize_agent(\n"," tools,\n",@@ -721,26 +742,18 @@")\n","\n","await agent.arun(\"What comedies are playing?\")"- ],- "metadata": {- "collapsed": false- },- "id": "8d4f9dd39b79d542",- "outputs": [],- "execution_count": null+ ]},{"cell_type": "markdown",- "source": [- "## Check your session\n",- "Finally, check your run on [AgentOps](https://app.agentops.ai)\n",- "\n",- ""- ],+ "id": "fb276a2e-f1c3-4f0f-8818-b7730e9d3ff7","metadata": {"collapsed": false},- "id": "fb276a2e-f1c3-4f0f-8818-b7730e9d3ff7"+ "source": [+ "## Check your session\n",+ "Finally, check your run on [AgentOps](https://app.agentops.ai)"+ ]}],"metadata": {