Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI
Python agentagentopsagents-sdkaianthropicautogencost-estimationcrewaievalsevaluation-metricsgroqlangchainllmmistralollamaopenaiopenai-agents
Sdk docs (#797)
Alex Reibmanee7d3cb7e686
diff --git a/README.md b/README.mdindex ba660572..a728ba8b 100644--- a/README.md+++ b/README.md@@ -52,6 +52,24 @@ https://github.com/user-attachments/assets/dfb4fa8d-d8c4-4965-9ff6-5b8514c1c22fAgentOps helps developers build, evaluate, and monitor AI agents. From prototype to production.+## Key Integrations 🔌++<div align="center" style="background-color: white; padding: 20px; border-radius: 10px; margin: 0 auto; max-width: 800px;">+ <div style="display: flex; flex-wrap: wrap; justify-content: center; align-items: center; gap: 30px; margin-bottom: 20px;">+ <a href="https://docs.agentops.ai/v1/integrations/openai-agents"><img src="docs/images/external/openai/agents-sdk.svg" height="45" alt="OpenAI Agents SDK"></a>+ <a href="https://docs.agentops.ai/v1/integrations/crewai"><img src="docs/v1/img/docs-icons/crew-banner.png" height="45" alt="CrewAI"></a>+ <a href="https://docs.ag2.ai/docs/ecosystem/agentops"><img src="docs/images/external/autogen/ag2_blue.svg" height="45" alt="AG2 (AutoGen)"></a>+ <a href="https://docs.agentops.ai/v1/integrations/microsoft"><img src="docs/images/external/microsoft/microsoft_logo.svg" height="45" alt="Microsoft"></a>+ </div>++ <div style="display: flex; flex-wrap: wrap; justify-content: center; align-items: center; gap: 30px; margin-bottom: 20px;">+ <a href="https://docs.agentops.ai/v1/integrations/langchain"><img src="docs/images/external/langchain/langchain-logo.png" height="45" alt="LangChain"></a>+ <a href="https://docs.agentops.ai/v1/integrations/camel"><img src="docs/images/external/camel/camel.png" height="45" alt="Camel AI"></a>+ <a href="https://docs.llamaindex.ai/en/stable/module_guides/observability/?h=agentops#agentops"><img src="docs/images/external/ollama/ollama-icon.png" height="45" alt="LlamaIndex"></a>+ <a href="https://docs.agentops.ai/v1/integrations/cohere"><img src="docs/images/external/cohere/cohere-logo.svg" height="45" alt="Cohere"></a>+ </div>+</div>+| | || ------------------------------------- | ------------------------------------------------------------- || 📊 **Replay Analytics and Debugging** | Step-by-step agent execution graphs |@@ -739,7 +757,7 @@ from swarmzero import Agent, Swarm| Platform | Dashboard | Evals || ---------------------------------------------------------------------------- | ------------------------------------------ | -------------------------------------- || ✅ Python SDK | ✅ Multi-session and Cross-session metrics | ✅ Custom eval metrics |-| 🚧 Evaluation builder API | ✅ Custom event tag tracking | 🔜 Agent scorecards |+| 🚧 Evaluation builder API | ✅ Custom event tag tracking | 🔜 Agent scorecards || ✅ [Javascript/Typescript SDK](https://github.com/AgentOps-AI/agentops-node) | ✅ Session replays | 🔜 Evaluation playground + leaderboard |## Debugging Roadmap 🧭diff --git a/docs/images/external/openai/agents-sdk.svg b/docs/images/external/openai/agents-sdk.svgnew file mode 100644index 00000000..df50e724--- /dev/null+++ b/docs/images/external/openai/agents-sdk.svg@@ -0,0 +1,10 @@+<svg width="151" height="50" viewBox="0 0 151 50" fill="none" xmlns="http://www.w3.org/2000/svg">+<path d="M56.9066 8.73828C50.8538 8.73828 45.8887 13.6967 45.8887 19.7698C45.8887 25.843 50.841 30.8014 56.9066 30.8014C62.9721 30.8014 67.9244 25.8686 67.9244 19.7698C67.9244 13.6711 62.9849 8.73828 56.9066 8.73828ZM56.9066 26.8423C53.1444 26.8423 50.1116 23.7545 50.1116 19.7698C50.1116 15.7852 53.1444 12.6845 56.9066 12.6845C60.6688 12.6845 63.7016 15.7723 63.7016 19.757C63.7016 23.7417 60.6688 26.8423 56.9066 26.8423Z" fill="black"/>+<path d="M79.0451 14.8623C77.0616 14.8623 75.1294 15.6567 74.1184 17.002V15.1698H70.1387V36.6051H74.1184V28.8536C75.1294 30.1092 76.9977 30.7882 79.0451 30.7882C83.332 30.7882 86.6975 27.4186 86.6975 22.8189C86.6975 18.2192 83.332 14.8623 79.0451 14.8623ZM78.3797 27.3289C76.1147 27.3289 74.0928 25.5479 74.0928 22.8317C74.0928 20.1154 76.1147 18.3345 78.3797 18.3345C80.6447 18.3345 82.6666 20.1154 82.6666 22.8317C82.6666 25.5479 80.6447 27.3289 78.3797 27.3289Z" fill="black"/>+<path d="M96.0507 14.8623C91.7127 14.8623 88.2832 18.2576 88.2832 22.8317C88.2832 27.4057 91.2776 30.8011 96.1787 30.8011C100.184 30.8011 102.756 28.3795 103.55 25.6504H99.6594C99.1731 26.7779 97.7911 27.5851 96.1403 27.5851C94.0929 27.5851 92.5317 26.1501 92.1606 24.0873H103.729V22.5242C103.729 18.3473 100.824 14.8623 96.0507 14.8623ZM92.199 21.2557C92.6213 19.3211 94.2208 18.0654 96.1403 18.0654C98.1878 18.0654 99.749 19.4107 99.9281 21.2557H92.199Z" fill="black"/>+<path d="M114.311 14.8623C112.532 14.8623 110.677 15.6567 109.82 16.9764V15.1698H105.84V30.4807H109.82V22.2423C109.82 19.8592 111.099 18.2961 113.185 18.2961C115.117 18.2961 116.154 19.7695 116.154 21.8195V30.4807H120.134V21.1661C120.134 17.3736 117.805 14.8623 114.311 14.8623Z" fill="black"/>+<path d="M130.309 9.0459L121.658 30.4812H125.907L127.737 25.7918H137.59L139.42 30.4812H143.732L135.133 9.0459H130.309ZM129.119 22.2428L132.663 13.274L136.182 22.2428H129.119Z" fill="black"/>+<path d="M149.4 9.0459H145.369V30.4812H149.413V9.0459H149.4Z" fill="black"/>+<path d="M15.7653 15.1572V10.8907C15.7653 10.6088 15.8421 10.4038 16.1364 10.2372L24.0447 5.63754C25.1068 5.02254 26.4633 4.74066 27.7813 4.74066C32.7848 4.74066 35.9455 8.59723 35.9455 12.7485C35.9455 13.0816 35.9455 13.53 35.8688 13.9016L27.5894 9.05848C27.1799 8.81504 26.732 8.7766 26.2329 9.05848L15.7653 15.1572ZM34.0516 30.3144V20.8332C34.0516 20.295 33.8469 19.9363 33.3606 19.6416L22.8674 13.53L26.5528 11.3904C26.7576 11.2622 27.0903 11.2622 27.295 11.3904L35.2545 15.99C37.5067 17.3097 39.0679 20.1797 39.0679 22.9216C39.0551 26.1632 37.0844 29.1613 34.0516 30.3144ZM13.5899 22.14L9.94283 19.9619C9.6613 19.7954 9.57173 19.5904 9.57173 19.3085V10.1604C9.57173 5.68879 12.9756 2.3191 17.608 2.3191C19.4123 2.3191 21.1399 2.9341 22.5219 4.04879L14.2937 8.84066C13.8074 9.12254 13.6027 9.4941 13.6027 10.0322V22.14H13.5899ZM20.7304 26.2528L15.7653 23.4597V17.5532L20.7304 14.76L25.6443 17.5532V23.4597L20.7304 26.2528ZM23.8016 38.681C21.9972 38.681 20.2697 38.066 18.8877 36.9513L27.1287 32.1594C27.6149 31.8775 27.8197 31.506 27.8197 30.9678V18.86L31.5051 21.0382C31.7866 21.2047 31.8762 21.4097 31.8762 21.6916V30.8525C31.8762 35.3241 28.4339 38.681 23.8016 38.681ZM14.1273 29.6097L6.16783 25.01C3.91563 23.6903 2.35444 20.8204 2.35444 18.0785C2.35444 14.7985 4.36351 11.8388 7.3963 10.6857V20.2054C7.3963 20.7435 7.60105 21.1022 8.08732 21.3969L18.5421 27.47L14.8567 29.6097C14.6648 29.7379 14.3321 29.7379 14.1273 29.6097ZM13.6283 36.2594C8.9191 36.2594 5.46402 32.7232 5.46402 28.3413C5.46402 27.9313 5.50241 27.5213 5.54079 27.1497L13.7818 31.916C14.2681 32.1979 14.6776 32.1979 15.1766 31.916L25.6315 25.8428V30.1094C25.6315 30.3913 25.5547 30.5963 25.2604 30.7629L17.3649 35.3625C16.3027 35.9775 14.9463 36.2594 13.6283 36.2594ZM23.8016 40.8975C28.7155 40.8975 32.8232 37.3613 33.7189 32.6847C38.3129 31.5316 41.2689 27.265 41.2689 22.9088C41.2689 20.0388 40.0405 17.2841 37.7882 15.2725C37.993 14.4141 38.1594 13.5044 38.1594 12.646C38.1594 6.85473 33.4502 2.51129 28.0372 2.51129C26.9239 2.51129 25.8234 2.71629 24.7613 3.08785C22.9058 1.24285 20.3977 0.102539 17.608 0.102539C12.6941 0.102539 8.58639 3.63879 7.69063 8.31535C3.09664 9.46848 0.140625 13.735 0.140625 18.0913C0.140625 20.9613 1.3691 23.716 3.6213 25.7275C3.41656 26.586 3.2502 27.4957 3.2502 28.3541C3.2502 34.1454 7.95935 38.4888 13.3723 38.4888C14.4856 38.4888 15.5861 38.2838 16.6483 37.9122C18.5166 39.7572 21.0119 40.8975 23.8016 40.8975Z" fill="black"/>+<path d="M82.3047 37.2949L81.0566 41.7715H83.5527L82.3047 37.2949ZM81.5898 36.252H83.0254L85.7031 45H84.4785L83.834 42.7207H80.7695L80.1367 45H78.9121L81.5898 36.252ZM90.9531 41.666C90.9531 40.8574 90.8203 40.2441 90.5547 39.8262C90.293 39.4043 89.9102 39.1934 89.4062 39.1934C88.8789 39.1934 88.4766 39.4043 88.1992 39.8262C87.9219 40.2441 87.7832 40.8574 87.7832 41.666C87.7832 42.4746 87.9219 43.0918 88.1992 43.5176C88.4805 43.9395 88.8867 44.1504 89.418 44.1504C89.9141 44.1504 90.293 43.9375 90.5547 43.5117C90.8203 43.0859 90.9531 42.4707 90.9531 41.666ZM92.0312 44.5781C92.0312 45.5625 91.7988 46.3086 91.334 46.8164C90.8691 47.3242 90.1855 47.5781 89.2832 47.5781C88.9863 47.5781 88.6758 47.5508 88.3516 47.4961C88.0273 47.4414 87.7031 47.3613 87.3789 47.2559V46.1895C87.7617 46.3691 88.1094 46.502 88.4219 46.5879C88.7344 46.6738 89.0215 46.7168 89.2832 46.7168C89.8652 46.7168 90.2891 46.5586 90.5547 46.2422C90.8203 45.9258 90.9531 45.4238 90.9531 44.7363V44.6895V43.957C90.7812 44.3242 90.5469 44.5977 90.25 44.7773C89.9531 44.957 89.5918 45.0469 89.166 45.0469C88.4004 45.0469 87.7891 44.7402 87.332 44.127C86.875 43.5137 86.6465 42.6934 86.6465 41.666C86.6465 40.6348 86.875 39.8125 87.332 39.1992C87.7891 38.5859 88.4004 38.2793 89.166 38.2793C89.5879 38.2793 89.9453 38.3633 90.2383 38.5312C90.5312 38.6992 90.7695 38.959 90.9531 39.3105V38.4609H92.0312V44.5781ZM99.7188 41.1797V41.707H95.0488V41.7422C95.0176 42.6367 95.1875 43.2793 95.5586 43.6699C95.9336 44.0605 96.4609 44.2559 97.1406 44.2559C97.4844 44.2559 97.8438 44.2012 98.2188 44.0918C98.5938 43.9824 98.9941 43.8164 99.4199 43.5938V44.666C99.0098 44.834 98.6133 44.959 98.2305 45.041C97.8516 45.127 97.4844 45.1699 97.1289 45.1699C96.1094 45.1699 95.3125 44.8652 94.7383 44.2559C94.1641 43.6426 93.877 42.7988 93.877 41.7246C93.877 40.6777 94.1582 39.8418 94.7207 39.2168C95.2832 38.5918 96.0332 38.2793 96.9707 38.2793C97.8066 38.2793 98.4648 38.5625 98.9453 39.1289C99.4297 39.6953 99.6875 40.3789 99.7188 41.1797ZM98.6406 40.8633C98.5938 40.4102 98.4277 40.0195 98.1426 39.6914C97.8613 39.3594 97.4551 39.1934 96.9238 39.1934C96.4043 39.1934 95.9766 39.3652 95.6406 39.709C95.3047 40.0527 95.1211 40.4395 95.0898 40.8691L98.6406 40.8633ZM106.545 40.9336V45H105.461V40.9336C105.461 40.3438 105.357 39.9102 105.15 39.6328C104.943 39.3555 104.619 39.2168 104.178 39.2168C103.674 39.2168 103.285 39.3965 103.012 39.7559C102.742 40.1113 102.607 40.623 102.607 41.291V45H101.529V38.4375H102.607V39.4219C102.799 39.0469 103.059 38.7637 103.387 38.5723C103.715 38.377 104.104 38.2793 104.553 38.2793C105.221 38.2793 105.719 38.5 106.047 38.9414C106.379 39.3789 106.545 40.043 106.545 40.9336ZM111.215 36.5742V38.4375H113.664V39.2754H111.215V42.8379C111.215 43.3223 111.307 43.6602 111.49 43.8516C111.674 44.043 111.994 44.1387 112.451 44.1387H113.664V45H112.346C111.537 45 110.967 44.8379 110.635 44.5137C110.303 44.1895 110.137 43.6309 110.137 42.8379V39.2754H108.385V38.4375H110.137V36.5742H111.215ZM120.549 38.666V39.7207C120.24 39.541 119.93 39.4062 119.617 39.3164C119.305 39.2266 118.986 39.1816 118.662 39.1816C118.174 39.1816 117.809 39.2617 117.566 39.4219C117.328 39.5781 117.209 39.8184 117.209 40.1426C117.209 40.4355 117.299 40.6543 117.479 40.7988C117.658 40.9434 118.105 41.084 118.82 41.2207L119.254 41.3027C119.789 41.4043 120.193 41.6074 120.467 41.9121C120.744 42.2168 120.883 42.6133 120.883 43.1016C120.883 43.75 120.652 44.2578 120.191 44.625C119.73 44.9883 119.09 45.1699 118.27 45.1699C117.945 45.1699 117.605 45.1348 117.25 45.0645C116.895 44.998 116.51 44.8965 116.096 44.7598V43.6465C116.498 43.8535 116.883 44.0098 117.25 44.1152C117.617 44.2168 117.965 44.2676 118.293 44.2676C118.77 44.2676 119.139 44.1719 119.4 43.9805C119.662 43.7852 119.793 43.5137 119.793 43.166C119.793 42.666 119.314 42.3203 118.357 42.1289L118.311 42.1172L117.906 42.0352C117.285 41.9141 116.832 41.7109 116.547 41.4258C116.262 41.1367 116.119 40.7441 116.119 40.248C116.119 39.6191 116.332 39.1348 116.758 38.7949C117.184 38.4512 117.791 38.2793 118.58 38.2793C118.932 38.2793 119.27 38.3125 119.594 38.3789C119.918 38.4414 120.236 38.5371 120.549 38.666ZM135.238 36.5508V37.752C134.879 37.5215 134.518 37.3477 134.154 37.2305C133.795 37.1133 133.432 37.0547 133.064 37.0547C132.506 37.0547 132.064 37.1855 131.74 37.4473C131.416 37.7051 131.254 38.0547 131.254 38.4961C131.254 38.8828 131.359 39.1777 131.57 39.3809C131.785 39.584 132.184 39.7539 132.766 39.8906L133.387 40.0312C134.207 40.2227 134.805 40.5234 135.18 40.9336C135.555 41.3438 135.742 41.9023 135.742 42.6094C135.742 43.4414 135.484 44.0762 134.969 44.5137C134.453 44.9512 133.703 45.1699 132.719 45.1699C132.309 45.1699 131.896 45.125 131.482 45.0352C131.068 44.9492 130.652 44.8184 130.234 44.6426V43.3828C130.684 43.668 131.107 43.877 131.506 44.0098C131.908 44.1426 132.312 44.209 132.719 44.209C133.316 44.209 133.781 44.0762 134.113 43.8105C134.445 43.541 134.611 43.166 134.611 42.6855C134.611 42.248 134.496 41.9141 134.266 41.6836C134.039 41.4531 133.643 41.2754 133.076 41.1504L132.443 41.0039C131.631 40.8203 131.041 40.543 130.674 40.1719C130.307 39.8008 130.123 39.3027 130.123 38.6777C130.123 37.8965 130.385 37.2715 130.908 36.8027C131.436 36.3301 132.135 36.0938 133.006 36.0938C133.342 36.0938 133.695 36.1328 134.066 36.2109C134.438 36.2852 134.828 36.3984 135.238 36.5508ZM139.094 44.0273C140.09 44.0273 140.785 43.7832 141.18 43.2949C141.574 42.8027 141.771 41.916 141.771 40.6348C141.771 39.3418 141.574 38.4512 141.18 37.9629C140.789 37.4707 140.094 37.2246 139.094 37.2246H138.531V44.0273H139.094ZM139.117 36.252C140.453 36.252 141.438 36.6074 142.07 37.3184C142.703 38.0293 143.02 39.1348 143.02 40.6348C143.02 42.127 142.703 43.2285 142.07 43.9395C141.438 44.6465 140.453 45 139.117 45H137.342V36.252H139.117ZM144.572 36.252H145.762V40.1426L149.459 36.252H150.848L147.443 39.8262L150.947 45H149.518L146.664 40.6172L145.762 41.5781V45H144.572V36.252Z" fill="black"/>+</svg>diff --git a/docs/v1/img/docs-icons/crew-banner.png b/docs/v1/img/docs-icons/crew-banner.pngnew file mode 100644index 00000000..ebe8a004Binary files /dev/null and b/docs/v1/img/docs-icons/crew-banner.png differdiff --git a/docs/v1/integrations/agentssdk.mdx b/docs/v1/integrations/agentssdk.mdxindex e92ed349..999993b9 100644--- a/docs/v1/integrations/agentssdk.mdx+++ b/docs/v1/integrations/agentssdk.mdx@@ -6,13 +6,13 @@ description: 'AgentOps and OpenAI Agents SDK integration for powerful multi-agenimport CodeTooltip from '/snippets/add-code-tooltip.mdx'import EnvTooltip from '/snippets/add-env-tooltip.mdx'-<Check>[Give us a star](https://github.com/AgentOps-AI/agentops) to bookmark us on GitHub, save for later 🖇️)</Check>+<Check>[Give us a star](https://github.com/AgentOps-AI/agentops) to bookmark on GitHub, save for later 🖇️)</Check>[OpenAI Agents SDK](https://github.com/openai/agentsdk_prototype) is a lightweight yet powerful framework for building multi-agent workflows. The SDK provides a comprehensive set of tools for creating, managing, and monitoring agent-based applications.-<Frame type="glass" caption="OpenAI Agents Tracing UI">+{/* <Frame type="glass" caption="OpenAI Agents Tracing UI"><img height="400" src="/images/openai-agents-tracing-ui.png" />-</Frame>+</Frame> */}## Core Conceptsdiff --git a/docs/v1/introduction.mdx b/docs/v1/introduction.mdxindex 6aad3885..cd0e8b18 100644--- a/docs/v1/introduction.mdx+++ b/docs/v1/introduction.mdx@@ -28,7 +28,7 @@ Observability and monitoring for your AI agents and LLM apps. And we do it all i## The AgentOps Dashboard-<Check>[Give us a star](https://github.com/AgentOps-AI/agentops) to bookmark us on GitHub, save for later 🖇️)</Check>+<Check>[Give us a star](https://github.com/AgentOps-AI/agentops) to bookmark on GitHub, save for later 🖇️)</Check>With just two lines of code, you can free yourself from the chains of the terminal and, instead, visualize your agents' behaviorin your AgentOps Dashboard. After setting up AgentOps, each execution of your program is recorded as a session and the above