Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI
Python agentagentopsagents-sdkaianthropicautogencost-estimationcrewaievalsevaluation-metricsgroqlangchainllmmistralollamaopenaiopenai-agents
docs (#795)
Alex Reibmanbb02634958f6
diff --git a/docs/mint.json b/docs/mint.jsonindex df1b2474..b241b368 100644--- a/docs/mint.json+++ b/docs/mint.json@@ -87,6 +87,7 @@{"group": "Integrations","pages": [+ "v1/integrations/agentssdk","v1/integrations/ai21","v1/integrations/anthropic","v1/integrations/autogen",diff --git a/docs/v1/integrations/agentssdk.mdx b/docs/v1/integrations/agentssdk.mdxnew file mode 100644index 00000000..e92ed349--- /dev/null+++ b/docs/v1/integrations/agentssdk.mdx@@ -0,0 +1,195 @@+---+title: 'OpenAI Agents SDK'+description: 'AgentOps and OpenAI Agents SDK integration for powerful multi-agent workflow monitoring.'+---++import CodeTooltip from '/snippets/add-code-tooltip.mdx'+import EnvTooltip from '/snippets/add-env-tooltip.mdx'++<Check>[Give us a star](https://github.com/AgentOps-AI/agentops) to bookmark us on GitHub, save for later 🖇️)</Check>++[OpenAI Agents SDK](https://github.com/openai/agentsdk_prototype) is a lightweight yet powerful framework for building multi-agent workflows. The SDK provides a comprehensive set of tools for creating, managing, and monitoring agent-based applications.++<Frame type="glass" caption="OpenAI Agents Tracing UI">+ <img height="400" src="/images/openai-agents-tracing-ui.png" />+</Frame>++## Core Concepts++- **Agents**: LLMs configured with instructions, tools, guardrails, and handoffs+- **Handoffs**: Allow agents to transfer control to other agents for specific tasks+- **Guardrails**: Configurable safety checks for input and output validation+- **Tracing**: Built-in tracking of agent runs, allowing you to view, debug and optimize your workflows++<Steps>+ <Step title="Install the AgentOps SDK">+ <CodeGroup>+ ```bash pip+ pip install agentops+ ```+ ```bash poetry+ poetry add agentops+ ```+ </CodeGroup>+ </Step>+ <Step title="Install OpenAI Agents SDK">+ <CodeGroup>+ ```bash pip+ pip install openai-agents+ ```+ ```bash poetry+ poetry add openai-agents+ ```+ </CodeGroup>+ <Tip>+ This will be updated to a PyPI link when the package is officially released.+ </Tip>+ </Step>+ <Step title="Add 2 lines of code">+ <CodeTooltip/>+ <CodeGroup>+ ```python python+ import agentops+ agentops.init(<INSERT YOUR API KEY HERE>)+ ```+ </CodeGroup>+ <EnvTooltip />+ <CodeGroup>+ ```python .env+ AGENTOPS_API_KEY=<YOUR API KEY>+ OPENAI_API_KEY=<YOUR OPENAI API KEY>+ ```+ </CodeGroup>+ Read more about environment variables in [Advanced Configuration](/v1/usage/advanced-configuration)+ </Step>+ <Step title="Run your agents">+ Execute your program and visit [app.agentops.ai/drilldown](https://app.agentops.ai/drilldown) to observe your Agents! 🕵️+ <Tip>+ After your run, AgentOps prints a clickable url to console linking directly to your session in the Dashboard+ </Tip>+ <div/>{/* Intentionally blank div for newline */}+ <Frame type="glass" caption="Clickable link to session">+ <img height="200" src="https://github.com/AgentOps-AI/agentops/blob/main/docs/images/link-to-session.gif?raw=true" />+ </Frame>+ </Step>+</Steps>++## Hello World Example++```python+from agents import Agent, Runner+import agentops++# Initialize AgentOps+agentops.init()++agent = Agent(name="Assistant", instructions="You are a helpful assistant")++result = Runner.run_sync(agent, "Write a haiku about recursion in programming.")+print(result.final_output)++# Output:+# Code within the code,+# Functions calling themselves,+# Infinite loop's dance.+```++## Handoffs Example++```python+from agents import Agent, Runner+import asyncio+import agentops++# Initialize AgentOps+agentops.init()++spanish_agent = Agent(+ name="Spanish agent",+ instructions="You only speak Spanish.",+)++english_agent = Agent(+ name="English agent",+ instructions="You only speak English",+)++triage_agent = Agent(+ name="Triage agent",+ instructions="Handoff to the appropriate agent based on the language of the request.",+ handoffs=[spanish_agent, english_agent],+)+++async def main():+ result = await Runner.run(triage_agent, input="Hola, ¿cómo estás?")+ print(result.final_output)+ # ¡Hola! Estoy bien, gracias por preguntar. ¿Y tú, cómo estás?+++if __name__ == "__main__":+ asyncio.run(main())+```++## Functions Example++```python+import asyncio+from agents import Agent, Runner, function_tool+import agentops++# Initialize AgentOps+agentops.init()++@function_tool+def get_weather(city: str) -> str:+ return f"The weather in {city} is sunny."+++agent = Agent(+ name="Hello world",+ instructions="You are a helpful agent.",+ tools=[get_weather],+)+++async def main():+ result = await Runner.run(agent, input="What's the weather in Tokyo?")+ print(result.final_output)+ # The weather in Tokyo is sunny.+++if __name__ == "__main__":+ asyncio.run(main())+```++## The Agent Loop++When you call `Runner.run()`, the SDK runs a loop until it gets a final output:++1. The LLM is called using the model and settings on the agent, along with the message history.+2. The LLM returns a response, which may include tool calls.+3. If the response has a final output, the loop ends and returns it.+4. If the response has a handoff, the agent is set to the new agent and the loop continues from step 1.+5. Tool calls are processed (if any) and tool response messages are appended. Then the loop continues from step 1.++You can use the `max_turns` parameter to limit the number of loop executions.++## Final Output++Final output is the last thing the agent produces in the loop:++- If you set an `output_type` on the agent, the final output is when the LLM returns something of that type using structured outputs.+- If there's no `output_type` (i.e., plain text responses), then the first LLM response without any tool calls or handoffs is considered the final output.++<CardGroup cols={3}>+ <Card title="Basic Agent" icon="robot" href="https://github.com/openai/agentsdk_prototype/tree/main/examples/basic" />+ <Card title="Multi-Agent" icon="users" href="https://github.com/openai/agentsdk_prototype/tree/main/examples/multi_agent" />+ <Card title="Tool Usage" icon="toolbox" href="https://github.com/openai/agentsdk_prototype/tree/main/examples/tools" />+</CardGroup>++<script type="module" src="/scripts/github_stars.js"></script>+<script type="module" src="/scripts/scroll-img-fadein-animation.js"></script>+<script type="module" src="/scripts/button_heartbeat_animation.js"></script>+<script type="css" src="/styles/styles.css"></script>+<script type="module" src="/scripts/adjust_api_dynamically.js"></script>\ No newline at end of filediff --git a/docs/v1/introduction.mdx b/docs/v1/introduction.mdxindex 3c3d2c72..6aad3885 100644--- a/docs/v1/introduction.mdx+++ b/docs/v1/introduction.mdx@@ -1,27 +1,23 @@---title: "Introduction"-description: "Build your next agent with evals, observability, and replays"+description: "AgentOps is the developer favorite platform for testing, debugging, and deploying AI agents and LLM apps."mode: "wide"----<Check>[Give us a star](https://github.com/AgentOps-AI/agentops) on GitHub if you find AgentOps helpful (you may be our <span id="stars-text">3,000th</span> 😊)</Check>--# AgentOps solves what Terminal can't+## Integrate with developer favorite agent frameworks<CardGroup cols={2}>- <Card title="Terminal CAN'T..." icon="trash" iconType="solid" color="#d6483e">- - • Track agents across executions- - • Parse out LLM completions from output logs- - • Give you insight into what your agents did- </Card>- <Card title="AgentOps does it all and more..." icon="face-smile-halo" iconType="regular" color="#2bd600">- - • Record LLM prompts, completions, & timestamps- - • Log events, calls, and any other agent activity- - • Link agent errors back to their causal event- </Card>+ <Card title="OpenAI Agents SDK" icon={<img src="https://www.github.com/agentops-ai/agentops/blob/main/docs/images/external/openai/openai-logomark.png?raw=true" alt="OpenAI Agents SDK" />} iconType="image" href="/v1/integrations/agentssdk" />+ <Card title="CrewAI" icon={<img src="https://www.github.com/agentops-ai/agentops/blob/main/docs/v1/img/docs-icons/crew.png?raw=true" alt="CrewAI"/>} iconType="image" href="/v1/integrations/crewai" />+ <Card title="AG2" icon={<img src="https://www.github.com/agentops-ai/agentops/blob/main/docs/images/external/autogen/ag2_blue.svg?raw=true" alt="AG2"/>} iconType="image" href="/v1/integrations/autogen" />+ <Card title="AutoGen" icon={<img src="https://www.github.com/agentops-ai/agentops/blob/main/docs/images/external/microsoft/microsoft_logo.svg?raw=true" alt="AutoGen" />} iconType="image" href="/v1/integrations/autogen" />+ <Card title="Anthropic" icon={<img src="https://www.github.com/agentops-ai/agentops/blob/main/docs/images/external/anthropic/anthropic_icon_slate.png?raw=true" alt="Anthropic" />} iconType="image" href="/v1/integrations/anthropic" />+ <Card title="Ollama" icon={<img src="https://www.github.com/agentops-ai/agentops/blob/main/docs/images/external/ollama/ollama-icon.png?raw=true" alt="Ollama" />} iconType="image" href="/v1/integrations/ollama" />+ <Card title="Cohere" icon={<img src="https://www.github.com/agentops-ai/agentops/blob/main/docs/images/external/cohere/cohere-logo.svg?raw=true" alt="Cohere" />} iconType="image" href="/v1/integrations/cohere" />+ <Card title="Groq" icon="bolt" href="https://docs.agentops.ai/v1/integrations/groq" /></CardGroup>-And we do it all in just two lines of code...+Observability and monitoring for your AI agents and LLM apps. And we do it all in just two lines of code...<CodeGroup>```python pythonimport agentops@@ -32,6 +28,8 @@ And we do it all in just two lines of code...## The AgentOps Dashboard+<Check>[Give us a star](https://github.com/AgentOps-AI/agentops) to bookmark us on GitHub, save for later 🖇️)</Check>+With just two lines of code, you can free yourself from the chains of the terminal and, instead, visualize your agents' behaviorin your AgentOps Dashboard. After setting up AgentOps, each execution of your program is recorded as a session and the abovedata is automatically recorded for you.diff --git a/examples/crewai_examples/db/2acd4cdf-cba0-40f5-ae71-ffed7e64c152/data_level0.bin b/examples/crewai_examples/db/2acd4cdf-cba0-40f5-ae71-ffed7e64c152/data_level0.binnew file mode 100644index 00000000..ea3192e8Binary files /dev/null and b/examples/crewai_examples/db/2acd4cdf-cba0-40f5-ae71-ffed7e64c152/data_level0.bin differdiff --git a/examples/crewai_examples/db/2acd4cdf-cba0-40f5-ae71-ffed7e64c152/header.bin b/examples/crewai_examples/db/2acd4cdf-cba0-40f5-ae71-ffed7e64c152/header.binnew file mode 100644index 00000000..3e0932a7Binary files /dev/null and b/examples/crewai_examples/db/2acd4cdf-cba0-40f5-ae71-ffed7e64c152/header.bin differdiff --git a/examples/crewai_examples/db/2acd4cdf-cba0-40f5-ae71-ffed7e64c152/length.bin b/examples/crewai_examples/db/2acd4cdf-cba0-40f5-ae71-ffed7e64c152/length.binnew file mode 100644index 00000000..fd74c49cBinary files /dev/null and b/examples/crewai_examples/db/2acd4cdf-cba0-40f5-ae71-ffed7e64c152/length.bin differdiff --git a/examples/crewai_examples/db/2acd4cdf-cba0-40f5-ae71-ffed7e64c152/link_lists.bin b/examples/crewai_examples/db/2acd4cdf-cba0-40f5-ae71-ffed7e64c152/link_lists.binnew file mode 100644index 00000000..e69de29bdiff --git a/examples/crewai_examples/db/chroma.sqlite3 b/examples/crewai_examples/db/chroma.sqlite3new file mode 100644index 00000000..45d56951Binary files /dev/null and b/examples/crewai_examples/db/chroma.sqlite3 differdiff --git a/examples/crewai_examples/job_posting.md b/examples/crewai_examples/job_posting.mdnew file mode 100644index 00000000..4404867a--- /dev/null+++ b/examples/crewai_examples/job_posting.md@@ -0,0 +1,46 @@+# Job Title: AI Engineer++**Location:** San Francisco, CA (Hybrid Work Model)++## About Us:+At AgentOps, we are pioneers in the realm of AI agent observability and evaluation, dedicated to enhancing the reliability and effectiveness of AI applications. Our mission is to empower engineers and teams to construct reliable AI agents that deliver exceptional performance through robust observability and evaluation tools.++## Introduction:+Join us at AgentOps, where innovation meets opportunity! As an AI Engineer, you'll be at the forefront of developing groundbreaking solutions that reshape how businesses implement AI technologies. Work alongside some of the brightest minds in San Francisco within a dynamic environment that thrives on creativity and initiative. If you're passionate about AI and eager to impact the future of technology, we want to hear from you!++## Role Description:+In this role, you will play a vital part in designing and developing sophisticated AI systems that optimize agent observability. Your responsibilities include collaborating with cross-functional teams to create tools to enhance AI performance, ensure efficiency, and maintain stringent compliance standards. Leverage your expertise to drive innovation, streamline operations, and contribute to our mission of delivering unparalleled customer success.++## Responsibilities:+- Design, implement, and optimize AI solutions focused on agent observability and evaluation.+- Collaborate with engineering teams to develop and integrate SDK tools that enhance agent performance.+- Conduct thorough testing and evaluations of AI models, ensuring reliability and compliance with industry standards.+- Utilize debugging techniques, including time travel methods, to troubleshoot and improve agent systems.+- Support the development of comprehensive documentation and reporting on AI performance metrics.+- Engage with customers to gather feedback and enhance product features, prioritizing customer success.+- Stay abreast of current trends and advancements in AI technology, applying insights to improve our offerings.++## Required Skills and Qualifications:+- Bachelor’s/Master's degree in Computer Science, Engineering, or related fields.+- Proficiency in AI and machine learning algorithms, notably in natural language processing and model evaluation.+- Strong programming skills in Python, C++, or Java with experience in AI frameworks such as TensorFlow or PyTorch.+- Experience in developing or integrating SDKs focused on AI agents.+- Familiarity with cloud platforms (AWS, Azure, or Google Cloud) for deploying AI applications.+- Proven track record in evaluating AI models, including debugging and troubleshooting capabilities.+- Excellent collaboration, communication, and problem-solving skills.++## Preferred Qualifications:+- Experience working with or developing enterprise-grade applications.+- Exposure to regulatory frameworks and compliance standards (SOC-2, HIPAA, etc.).+- Passion for innovative thinking and a strong customer-centric approach.++## What We Offer:+- An opportunity to work with the most talented developers in San Francisco.+- A collaborative work environment that promotes innovation and high agency.+- Flexible work arrangements with a focus on work-life balance.+- Competitive salary and benefits package.+- Ongoing professional development opportunities and mentorship programs.++## Join Us:+If you're ready to take on the challenge of shaping the future of AI observability and evaluation, we invite you to apply for the AI Engineer position at AgentOps. Together, we can build technology that makes a difference!+```\ No newline at end of file