Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI
Python agentagentopsagents-sdkaianthropicautogencost-estimationcrewaievalsevaluation-metricsgroqlangchainllmmistralollamaopenaiopenai-agents
Basic agent ops example (#58)
Braelyn Boynton663223c7c168
diff --git a/examples/openai-gpt.ipynb b/examples/openai-gpt.ipynbnew file mode 100644index 00000000..2a686332--- /dev/null+++ b/examples/openai-gpt.ipynb@@ -0,0 +1,298 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "source": [+ "# Monitoring \n",+ "This is an example of how to use the AgentOps library for basic Agent monitoring with OpenAI's GPT"+ ],+ "metadata": {+ "collapsed": false+ },+ "id": "212799f5255f3513"+ },+ {+ "cell_type": "markdown",+ "source": [+ "_At this time, make sure to install openai==0.28.1_"+ ],+ "metadata": {+ "collapsed": false+ },+ "id": "6d717e4c8bf0de9c"+ },+ {+ "cell_type": "code",+ "execution_count": 3,+ "outputs": [],+ "source": [+ "import openai\n",+ "import agentops\n",+ "from dotenv import load_dotenv\n",+ "import os\n",+ "from openai import ChatCompletion"+ ],+ "metadata": {+ "collapsed": false,+ "ExecuteTime": {+ "end_time": "2023-12-15T21:33:47.472626Z",+ "start_time": "2023-12-15T21:33:47.448868Z"+ }+ },+ "id": "c51dcbda80eb8c53"+ },+ {+ "cell_type": "markdown",+ "source": [+ "Next, we'll grab our two API keys. You can use dotenv like below or however else you like to load environment variables"+ ],+ "metadata": {+ "collapsed": false+ },+ "id": "33bb35b82e9a8f71"+ },+ {+ "cell_type": "code",+ "execution_count": 4,+ "outputs": [],+ "source": [+ "load_dotenv()\n",+ "OPENAI_API_KEY = os.getenv('OPENAI_API_KEY') or \"<your_openai_key>\"\n",+ "AGENTOPS_API_KEY = os.getenv('AGENTOPS_API_KEY') or \"<your_agentops_key>\""+ ],+ "metadata": {+ "collapsed": false,+ "ExecuteTime": {+ "end_time": "2023-12-15T21:33:47.473321Z",+ "start_time": "2023-12-15T21:33:47.460144Z"+ }+ },+ "id": "cbc5d13679e269dc"+ },+ {+ "cell_type": "markdown",+ "source": [+ "The AgentOps library is designed to be a plug-and-play replacement for the OpenAI Client, maximizing use with minimal install effort."+ ],+ "metadata": {+ "collapsed": false+ },+ "id": "82fdff21ff00ef37"+ },+ {+ "cell_type": "code",+ "execution_count": 5,+ "outputs": [],+ "source": [+ "ao_client = agentops.Client(AGENTOPS_API_KEY)\n",+ "openai.api_key = OPENAI_API_KEY"+ ],+ "metadata": {+ "collapsed": false,+ "ExecuteTime": {+ "end_time": "2023-12-15T21:33:48.179508Z",+ "start_time": "2023-12-15T21:33:47.463468Z"+ }+ },+ "id": "fe8116d5969f1d23"+ },+ {+ "cell_type": "markdown",+ "source": [+ "Now just use OpenAI as you would normally!"+ ],+ "metadata": {+ "collapsed": false+ },+ "id": "3c20bbfa91b3419c"+ },+ {+ "cell_type": "markdown",+ "source": [+ "## Single Session with ChatCompletion"+ ],+ "metadata": {+ "collapsed": false+ },+ "id": "b42f5685ac4af5c2"+ },+ {+ "cell_type": "code",+ "execution_count": 6,+ "outputs": [+ {+ "data": {+ "text/plain": "<OpenAIObject chat.completion id=chatcmpl-8WA6WNna1UiKr7dg1F1fGtAuTRZeB at 0x116b48860> JSON: {\n \"id\": \"chatcmpl-8WA6WNna1UiKr7dg1F1fGtAuTRZeB\",\n \"object\": \"chat.completion\",\n \"created\": 1702676028,\n \"model\": \"gpt-3.5-turbo-0613\",\n \"choices\": [\n {\n \"index\": 0,\n \"message\": {\n \"role\": \"assistant\",\n \"content\": \"Hello! How can I assist you today?\"\n },\n \"logprobs\": null,\n \"finish_reason\": \"stop\"\n }\n ],\n \"usage\": {\n \"prompt_tokens\": 8,\n \"completion_tokens\": 9,\n \"total_tokens\": 17\n },\n \"system_fingerprint\": null\n}"+ },+ "execution_count": 6,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "message = {\"role\": \"user\", \"content\": \"Hello\"},\n",+ "ChatCompletion.create(\n",+ " model='gpt-3.5-turbo', messages=message, temperature=0.5)"+ ],+ "metadata": {+ "collapsed": false,+ "ExecuteTime": {+ "end_time": "2023-12-15T21:33:49.022014Z",+ "start_time": "2023-12-15T21:33:48.181090Z"+ }+ },+ "id": "9cd47d3fa1e252e1"+ },+ {+ "cell_type": "markdown",+ "source": [+ "Make sure to end your session with a `Result` (Success|Fail|Indeterminate) for better tracking"+ ],+ "metadata": {+ "collapsed": false+ },+ "id": "bf75276ad9fbb3f4"+ },+ {+ "cell_type": "code",+ "execution_count": 7,+ "outputs": [],+ "source": [+ "ao_client.end_session('Success')"+ ],+ "metadata": {+ "collapsed": false,+ "ExecuteTime": {+ "end_time": "2023-12-15T21:33:50.619934Z",+ "start_time": "2023-12-15T21:33:49.021089Z"+ }+ },+ "id": "f59fe80a7e00e6e8"+ },+ {+ "cell_type": "markdown",+ "source": [+ "Now if you check the AgentOps dashboard, you should see information related to this run!"+ ],+ "metadata": {+ "collapsed": false+ },+ "id": "318a7186c1be2d59"+ },+ {+ "cell_type": "markdown",+ "source": [+ "# Events\n",+ "Additionally, you can track custom events via AgentOps.\n",+ "Lets record an event when an LLM says \"Hello\""+ ],+ "metadata": {+ "collapsed": false+ },+ "id": "ccf998561cb9a834"+ },+ {+ "cell_type": "code",+ "execution_count": 8,+ "outputs": [],+ "source": [+ "# Create new session\n",+ "ao_client.start_session()\n",+ "\n",+ "# Optionally, we can add tags to the session\n",+ "# ao_client.start_session(['Hello Tracker'])"+ ],+ "metadata": {+ "collapsed": false,+ "ExecuteTime": {+ "end_time": "2023-12-15T21:33:50.914863Z",+ "start_time": "2023-12-15T21:33:50.620442Z"+ }+ },+ "id": "f5a1a63ff4ecf127"+ },+ {+ "cell_type": "code",+ "execution_count": 9,+ "outputs": [],+ "source": [+ "message = {\"role\": \"user\", \"content\": \"Hello\"},\n",+ "response = ChatCompletion.create(\n",+ " model='gpt-3.5-turbo', messages=message, temperature=0.5)"+ ],+ "metadata": {+ "collapsed": false,+ "ExecuteTime": {+ "end_time": "2023-12-15T21:33:51.321518Z",+ "start_time": "2023-12-15T21:33:50.915612Z"+ }+ },+ "id": "7daafba014c1a4d8"+ },+ {+ "cell_type": "code",+ "execution_count": 12,+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "📝 Recording Event\n"+ ]+ }+ ],+ "source": [+ "from agentops import Event\n",+ "\n",+ "if \"hello\" in str(response.choices[0].message.content).lower():\n",+ " print('📝 Recording Event')\n",+ " ao_client.record(Event(event_type=\"said_hello\"))"+ ],+ "metadata": {+ "collapsed": false,+ "ExecuteTime": {+ "end_time": "2023-12-15T21:34:24.301681Z",+ "start_time": "2023-12-15T21:34:24.297258Z"+ }+ },+ "id": "b45754a57148eed1"+ },+ {+ "cell_type": "code",+ "execution_count": 13,+ "outputs": [],+ "source": [+ "ao_client.end_session('Success')"+ ],+ "metadata": {+ "collapsed": false,+ "ExecuteTime": {+ "end_time": "2023-12-15T21:34:26.439101Z",+ "start_time": "2023-12-15T21:34:26.436252Z"+ }+ },+ "id": "4ca2b49fc06adddb"+ }+ ],+ "metadata": {+ "kernelspec": {+ "display_name": "Python 3",+ "language": "python",+ "name": "python3"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 2+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython2",+ "version": "2.7.6"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 5+}