Skip to content

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI

Instrumenting Cohere (#202)

Howard Gil9acda5a46d2e
diff --git a/README.md b/README.md
index aeab3368..8884092f 100644
--- a/README.md
+++ b/README.md
@@ -25,37 +25,42 @@
# AgentOps 🖇️
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) ![PyPI - Version](https://img.shields.io/pypi/v/agentops) <a href="https://pepy.tech/project/agentops">
- <img src="https://static.pepy.tech/badge/agentops/month"> <a href="https://twitter.com/agentopsai">
- <img src="https://img.shields.io/badge/follow-%40agentops-1DA1F2?logo=twitter&style=social" alt="AgentOps Twitter" />
- </a>
- <a href="https://discord.gg/mKW3ZhN9p2">
- <img src="https://img.shields.io/badge/chat-on%20Discord-blueviolet" alt="Discord community channel" />
- </a>
- <a href="mailto:[email protected]"><img src="https://img.shields.io/website?color=%23f26522&down_message=Y%20Combinator&label=Not%20Backed%20By&logo=ycombinator&style=flat-square&up_message=Y%20Combinator&url=https%3A%2F%2Fwww.ycombinator.com"/>
- <a href="https://github.com/agentops-ai/agentops/issues">
- <img src="https://img.shields.io/github/commit-activity/m/agentops-ai/agentops" alt="git commit activity" />
- </a>
-
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+![PyPI - Version](https://img.shields.io/pypi/v/agentops)
+<a href="https://pepy.tech/project/agentops">
+ <img src="https://static.pepy.tech/badge/agentops/month">
+</a>
+<a href="https://twitter.com/agentopsai">
+ <img src="https://img.shields.io/badge/follow-%40agentops-1DA1F2?logo=twitter&style=social" alt="AgentOps Twitter"/>
+</a>
+<a href="https://discord.gg/mKW3ZhN9p2">
+ <img src="https://img.shields.io/badge/chat-on%20Discord-blueviolet" alt="Discord community channel"/>
+</a>
+<a href="mailto:[email protected]">
+ <img src="https://img.shields.io/website?color=%23f26522&down_message=Y%20Combinator&label=Not%20Backed%20By&logo=ycombinator&style=flat-square&up_message=Y%20Combinator&url=https%3A%2F%2Fwww.ycombinator.com"/>
+</a>
+<a href="https://github.com/agentops-ai/agentops/issues">
+ <img src="https://img.shields.io/github/commit-activity/m/agentops-ai/agentops" alt="git commit activity"/>
+</a>
AgentOps helps developers build, evaluate, and monitor AI agents. Tools to build agents from prototype to production.
-
-|||
-|------------------------------------------|----------------------------------------------------|
-| 📊 **Replay Analytics and Debugging** | Step-by-step agent execution graphs |
-| 💸 **LLM Cost Management** | Track spend with LLM foundation model providers |
-| 🧪 **Agent Benchmarking** | Test your agents against 1,000+ evals |
-| 🔐 **Compliance and Security** | Detect common prompt injection and data exfiltration exploits |
-| 🤝 **Framework Integrations** | Easily plugs in with frameworks like CrewAI and LangChain |
+| | |
+| ------------------------------------- | ------------------------------------------------------------- |
+| 📊 **Replay Analytics and Debugging** | Step-by-step agent execution graphs |
+| 💸 **LLM Cost Management** | Track spend with LLM foundation model providers |
+| 🧪 **Agent Benchmarking** | Test your agents against 1,000+ evals |
+| 🔐 **Compliance and Security** | Detect common prompt injection and data exfiltration exploits |
+| 🤝 **Framework Integrations** | Easily plugs in with frameworks like CrewAI and LangChain |
## Quick Start ⌨️
-```python
+```bash
pip install agentops
```
### Session replays in 3 lines of code
+
Initialize the AgentOps client and automatically get analytics on every LLM call.
```python
@@ -77,7 +82,6 @@ agentops.end_session('Success')
All your sessions are available on the [AgentOps dashboard](https://app.agentops.ai?ref=gh). Refer to our [API documentation](http://docs.agentops.ai) for detailed instructions.
-
<details open>
<summary>Agent Dashboard</summary>
<a href="https://app.agentops.ai?ref=gh">
@@ -105,14 +109,19 @@ All your sessions are available on the [AgentOps dashboard](https://app.agentops
Build Crew agents with observability with only 2 lines of code. Simply set an `AGENTOPS_API_KEY` in your environment, and your crews will get automatic monitoring on the AgentOps dashboard.
-AgentOps is officially supported on Crew's latest rc branch: `crewai==0.28.9rc1`
+AgentOps is integrated with CrewAI on a pre-release fork. Install crew with
-* [AgentOps integration example](https://docs.agentops.ai/v1/integrations/crewai)
-* [Offical CrewAI documentation](https://docs.crewai.com/how-to/AgentOps-Observability)
+```bash
+pip install git+https://github.com/AgentOps-AI/crewAI.git@main
+```
+- [AgentOps integration example](https://docs.agentops.ai/v1/integrations/crewai)
+- [Official CrewAI documentation](https://docs.crewai.com/how-to/AgentOps-Observability)
### Langchain 🦜🔗
+
AgentOps works seamlessly with applications built using Langchain. To use the handler, install Langchain as an optional dependency:
+
```shell
pip install agentops[langchain]
```
@@ -142,41 +151,89 @@ agent = initialize_agent(tools,
Check out the [Langchain Examples Notebook](./examples/langchain_examples.ipynb) for more details including Async handlers.
-### LlamaIndex 🦙
-(Coming Soon)
+### Cohere
+
+First class support for Cohere(>=5.4.0). This is a living integration, should you need any added functionality please message us on Discord!
+
+- [AgentOps integration example](https://docs.agentops.ai/v1/integrations/cohere)
+- [Official Cohere documentation](https://docs.cohere.com/reference/about)
+
+```bash
+pip install cohere
+```
+
+```python python
+import cohere
+import agentops
+
+# Beginning of program's code (i.e. main.py, __init__.py)
+agentops.init(<INSERT YOUR API KEY HERE>)
+co = cohere.Client()
+chat = co.chat(
+ message="Is it pronounced ceaux-hear or co-hehray?"
+)
+print(chat)
+
+agentops.end_session('Success')
+```
+
+```python python
+import cohere
+import agentops
+
+# Beginning of program's code (i.e. main.py, __init__.py)
+agentops.init(<INSERT YOUR API KEY HERE>)
+
+co = cohere.Client()
+
+stream = co.chat_stream(
+ message="Write me a haiku about the synergies between Cohere and AgentOps"
+)
+
+for event in stream:
+ if event.event_type == "text-generation":
+ print(event.text, end='')
+
+agentops.end_session('Success')
+```
+
+### LlamaIndex 🦙
+
+(Coming Soon)
## Time travel debugging 🔮
+
(coming soon!)
## Agent Arena 🥊
+
(coming soon!)
## Evaluations Roadmap 🧭
-| Platform | Dashboard | Evals |
-|---|---|---|
-|✅ Python SDK | ✅ Multi-session and Cross-session metrics | ✅ Custom eval metrics |
-|🚧 Evaluation builder API | ✅ Custom event tag tracking | 🔜 Agent scorecards |
-|✅ [Javascript/Typescript SDK](https://github.com/AgentOps-AI/agentops-node) | ✅ Session replays| 🔜 Evaluation playground + leaderboard|
-
+| Platform | Dashboard | Evals |
+| ---------------------------------------------------------------------------- | ------------------------------------------ | -------------------------------------- |
+| ✅ Python SDK | ✅ Multi-session and Cross-session metrics | ✅ Custom eval metrics |
+| 🚧 Evaluation builder API | ✅ Custom event tag tracking  | 🔜 Agent scorecards |
+| ✅ [Javascript/Typescript SDK](https://github.com/AgentOps-AI/agentops-node) | ✅ Session replays | 🔜 Evaluation playground + leaderboard |
## Debugging Roadmap 🧭
-| Performance testing | Environments | LLM Testing | Reasoning and execution testing |
-|---|---|---|---|
-|✅ Event latency analysis | 🔜 Non-stationary environment testing | 🔜 LLM non-deterministic function detection | 🚧 Infinite loops and recursive thought detection |
-|✅ Agent workflow execution pricing | 🔜 Multi-modal environments | 🚧 Token limit overflow flags | 🔜 Faulty reasoning detection |
-|🚧 Success validators (external) | 🔜 Execution containers | 🔜 Context limit overflow flags | 🔜 Generative code validators |
-|🔜 Agent controllers/skill tests | ✅ Honeypot and prompt injection detection ([PromptArmor](https://promptarmor.com)) | 🔜 API bill tracking | 🔜 Error breakpoint analysis |
-|🔜 Information context constraint testing | 🔜 Anti-agent roadblocks (i.e. Captchas) | 🔜 CI/CD integration checks | |
-|🔜 Regression testing | 🔜 Multi-agent framework visualization | | |
+| Performance testing | Environments | LLM Testing | Reasoning and execution testing |
+| ----------------------------------------- | ----------------------------------------------------------------------------------- | ------------------------------------------- | ------------------------------------------------- |
+| ✅ Event latency analysis | 🔜 Non-stationary environment testing | 🔜 LLM non-deterministic function detection | 🚧 Infinite loops and recursive thought detection |
+| ✅ Agent workflow execution pricing | 🔜 Multi-modal environments | 🚧 Token limit overflow flags | 🔜 Faulty reasoning detection |
+| 🚧 Success validators (external) | 🔜 Execution containers | 🔜 Context limit overflow flags | 🔜 Generative code validators |
+| 🔜 Agent controllers/skill tests | ✅ Honeypot and prompt injection detection ([PromptArmor](https://promptarmor.com)) | 🔜 API bill tracking | 🔜 Error breakpoint analysis |
+| 🔜 Information context constraint testing | 🔜 Anti-agent roadblocks (i.e. Captchas) | 🔜 CI/CD integration checks | |
+| 🔜 Regression testing | 🔜 Multi-agent framework visualization | | |
### Why AgentOps? 🤔
Our mission is to bring your agent from prototype to production.
-Agent developers often work with little to no visibility into agent testing performance. This means their agents never leave the lab. We're changing that.
+Agent developers often work with little to no visibility into agent testing performance. This means their agents never leave the lab. We're changing that.
AgentOps is the easiest way to evaluate, grade, and test agents. Is there a feature you'd like to see AgentOps cover? Just raise it in the issues tab, and we'll work on adding it to the roadmap.
diff --git a/agentops/client.py b/agentops/client.py
index c082642c..5447f510 100644
--- a/agentops/client.py
+++ b/agentops/client.py
@@ -255,7 +255,7 @@ class Client(metaclass=MetaClient):
self._session = None
return logger.warning("🖇 AgentOps: Cannot start session")
- logger.info('View info on this session at https://app.agentops.ai/drilldown?session_id=%s',
+ logger.info('View info on this session at https://app.agentops.ai/drilldown?session_id=%s',
self._session.session_id)
return self._session.session_id
@@ -277,19 +277,19 @@ class Client(metaclass=MetaClient):
if not any(end_state == state.value for state in EndState):
return logger.warning("🖇 AgentOps: Invalid end_state. Please use one of the EndState enums")
-
+
if self._worker is None or self._worker._session is None:
return logger.warning("🖇 AgentOps: Cannot end session - no current worker or session")
self._session.video = video
self._session.end_session(end_state, end_state_reason)
token_cost = self._worker.end_session(self._session)
-
+
if token_cost == 'unknown':
print('🖇 AgentOps: Could not determine cost of run.')
else:
token_cost_d = Decimal(token_cost)
- print('🖇 AgentOps: This run cost ${}'.format('{:.2f}'.format(
+ print('\n🖇 AgentOps: This run cost ${}'.format('{:.2f}'.format(
token_cost_d) if token_cost_d == 0 else '{:.6f}'.format(token_cost_d)))
self._session = None
self._worker = None
diff --git a/agentops/llm_tracker.py b/agentops/llm_tracker.py
index 2b823fe9..250aa41c 100644
--- a/agentops/llm_tracker.py
+++ b/agentops/llm_tracker.py
@@ -13,6 +13,7 @@ import pprint
original_create = None
original_create_async = None
+
class LlmTracker:
SUPPORTED_APIS = {
'litellm': {'1.3.1': ("openai_chat_completions.completion",)},
@@ -25,6 +26,12 @@ class LlmTracker:
"ChatCompletion.create",
"ChatCompletion.acreate",
),
+ },
+ 'cohere': {
+ '5.4.0': (
+ "chat",
+ "chat_stream"
+ ),
}
}
@@ -95,13 +102,9 @@ class LlmTracker:
yield chunk
return generator()
- self.llm_event = LLMEvent(
- init_timestamp=init_timestamp,
- params=kwargs,
- returns=response
- )
# v0.0.0 responses are dicts
try:
+ self.llm_event.returns = response
self.llm_event.agent_id = check_call_stack_for_agent_id()
self.llm_event.prompt = kwargs["messages"]
self.llm_event.prompt_tokens = response['usage']['prompt_tokens']
@@ -201,10 +204,6 @@ class LlmTracker:
yield chunk
return async_generator()
- self.llm_event = LLMEvent(
- init_timestamp=init_timestamp,
- params=kwargs
- )
# v1.0.0+ responses are objects
try:
self.llm_event.returns = response.model_dump()
@@ -228,6 +227,122 @@ class LlmTracker:
return response
+ def _handle_response_cohere(self, response, kwargs, init_timestamp):
+ # TODO: """Handle responses for Cohere versions >v5.4.0"""
+ from cohere.types.non_streamed_chat_response import NonStreamedChatResponse
+ from cohere.types.streamed_chat_response import (
+ StreamedChatResponse,
+ StreamedChatResponse_CitationGeneration,
+ StreamedChatResponse_SearchQueriesGeneration,
+ StreamedChatResponse_SearchResults,
+ StreamedChatResponse_StreamEnd,
+ StreamedChatResponse_StreamStart,
+ StreamedChatResponse_TextGeneration,
+ StreamedChatResponse_ToolCallsGeneration,
+ )
+ # from cohere.types.chat import ChatGenerationChunk
+
+ # NOTE: Cohere only returns one message and its role will be CHATBOT which we are coercing to "assistant"
+ self.llm_event = LLMEvent(
+ init_timestamp=init_timestamp,
+ params=kwargs
+ )
+
+ def handle_stream_chunk(chunk):
+
+ # We take the first chunk and accumulate the deltas from all subsequent chunks to build one full chat completion
+ if isinstance(chunk, StreamedChatResponse_StreamStart):
+ self.llm_event.returns = chunk
+ self.llm_event.agent_id = check_call_stack_for_agent_id()
+ self.llm_event.model = kwargs.get("model", "command-r-plus")
+ self.llm_event.prompt = kwargs["message"]
+ self.llm_event.completion = ""
+ return
+
+ try:
+ if isinstance(chunk, StreamedChatResponse_StreamEnd):
+ # Streaming is done. Record LLMEvent
+ # self.llm_event.returns.finish_reason = chunk.is_finished
+ self.llm_event.completion = {
+ "role": "assistant", "content": self.llm_event.completion}
+ self.llm_event.end_timestamp = get_ISO_time()
+
+ self.client.record(self.llm_event)
+
+ elif isinstance(chunk, StreamedChatResponse_TextGeneration):
+ self.llm_event.completion += chunk.text
+ elif isinstance(chunk, StreamedChatResponse_ToolCallsGeneration):
+ pass
+ elif isinstance(chunk, StreamedChatResponse_CitationGeneration):
+ pass
+ elif isinstance(chunk, StreamedChatResponse_SearchQueriesGeneration):
+ pass
+
+ except Exception as e:
+ self.client.record(ErrorEvent(trigger_event=self.llm_event, exception=e))
+ kwargs_str = pprint.pformat(kwargs)
+ chunk = pprint.pformat(chunk)
+ logger.warning(
+ f"🖇 AgentOps: Unable to parse a chunk for LLM call. Skipping upload to AgentOps\n"
+ f"chunk:\n {chunk}\n"
+ f"kwargs:\n {kwargs_str}\n"
+ )
+
+ # NOTE: As of Cohere==5.x.x, async is not supported
+ # if the response is a generator, decorate the generator
+ if inspect.isasyncgen(response):
+ async def async_generator():
+ async for chunk in response:
+ handle_stream_chunk(chunk)
+ yield chunk
+ return async_generator()
+
+ elif inspect.isgenerator(response):
+ def generator():
+ for chunk in response:
+ handle_stream_chunk(chunk)
+ yield chunk
+ return generator()
+
+ # TODO: we should record if they pass a chat.connectors, because it means they intended to call a tool
+ # Not enough to record StreamedChatResponse_ToolCallsGeneration because the tool may have not gotten called
+
+ try:
+ self.llm_event.returns = response.dict()
+ self.llm_event.agent_id = check_call_stack_for_agent_id()
+ self.llm_event.prompt = []
+ if response.chat_history:
+ role_map = {
+ "USER": "user",
+ "CHATBOT": "assistant",
+ "SYSTEM": "system"
+ }
+
+ for i in range(len(response.chat_history) - 1):
+ message = response.chat_history[i]
+ self.llm_event.prompt.append({"role": role_map.get(
+ message.role, message.role), "content": message.message})
+
+ last_message = response.chat_history[-1]
+ self.llm_event.completion = {
+ "role": role_map.get(last_message.role, last_message.role), "content": last_message.message}
+ self.llm_event.prompt_tokens = response.meta.tokens.input_tokens
+ self.llm_event.completion_tokens = response.meta.tokens.output_tokens
+ self.llm_event.model = kwargs.get("model", "command-r-plus")
+
+ self.client.record(self.llm_event)
+ except Exception as e:
+ self.client.record(ErrorEvent(trigger_event=self.llm_event, exception=e))
+ kwargs_str = pprint.pformat(kwargs)
+ response = pprint.pformat(response)
+ logger.warning(
+ f"🖇 AgentOps: Unable to parse response for LLM call. Skipping upload to AgentOps\n"
+ f"response:\n {response}\n"
+ f"kwargs:\n {kwargs_str}\n"
+ )
+
+ return response
+
def override_openai_v1_completion(self):
from openai.resources.chat import completions
@@ -288,6 +403,35 @@ class LlmTracker:
# Override the original method with the patched one
litellm.acompletion = patched_function
+ def override_cohere_chat(self):
+ import cohere
+ import cohere.types
+
+ original_chat = cohere.Client.chat
+
+ def patched_function(*args, **kwargs):
+ # Call the original function with its original arguments
+ init_timestamp = get_ISO_time()
+ result = original_chat(*args, **kwargs)
+ return self._handle_response_cohere(result, kwargs, init_timestamp)
+
+ # Override the original method with the patched one
+ cohere.Client.chat = patched_function
+
+ def override_cohere_chat_stream(self):
+ import cohere
+
+ original_chat = cohere.Client.chat_stream
+
+ def patched_function(*args, **kwargs):
+ # Call the original function with its original arguments
+ init_timestamp = get_ISO_time()
+ result = original_chat(*args, **kwargs)
+ return self._handle_response_cohere(result, kwargs, init_timestamp)
+
+ # Override the original method with the patched one
+ cohere.Client.chat_stream = patched_function
+
def _override_method(self, api, method_path, module):
def handle_response(result, kwargs, init_timestamp):
if api == "openai":
@@ -331,11 +475,14 @@ class LlmTracker:
module = import_module(api)
if api == 'litellm':
module_version = version(api)
+ if module_version is None:
+ logger.warning(f'🖇 AgentOps: Cannot determine LiteLLM version. Only LiteLLM>=1.3.1 supported.')
+
if Version(module_version) >= parse('1.3.1'):
self.override_litellm_completion()
self.override_litellm_async_completion()
else:
- logger.warning(f'🖇 AgentOps: Only litellm>=1.3.1 supported. v{module_version} found.')
+ logger.warning(f'🖇 AgentOps: Only LiteLLM>=1.3.1 supported. v{module_version} found.')
return # If using an abstraction like litellm, do not patch the underlying LLM APIs
if api == 'openai':
@@ -350,6 +497,18 @@ class LlmTracker:
for method_path in self.SUPPORTED_APIS['openai']['0.0.0']:
self._override_method(api, method_path, module)
+ if api == 'cohere':
+ # Patch cohere vx.x.x+ methods
+ module_version = version(api)
+ if module_version is None:
+ logger.warning(f'🖇 AgentOps: Cannot determine Cohere version. Only Cohere>=5.4.0 supported.')
+
+ if Version(module_version) >= parse('5.4.0'):
+ self.override_cohere_chat()
+ self.override_cohere_chat_stream()
+ else:
+ logger.warning(f'🖇 AgentOps: Only Cohere>=5.4.0 supported. v{module_version} found.')
+
def stop_instrumenting(self):
self.undo_override_openai_v1_async_completion()
self.undo_override_openai_v1_completion()
diff --git a/examples/cohere.py b/examples/cohere.py
new file mode 100644
index 00000000..8e1ef3aa
--- /dev/null
+++ b/examples/cohere.py
@@ -0,0 +1,30 @@
+import cohere
+import agentops
+from dotenv import load_dotenv
+load_dotenv()
+
+agentops.init()
+co = cohere.Client()
+
+chat = co.chat(
+ chat_history=[
+ {"role": "USER", "message": "Who discovered gravity?"},
+ {
+ "role": "CHATBOT",
+ "message": "The man who is widely credited with discovering gravity is Sir Isaac Newton",
+ },
+ ],
+ message="What year was he born?",
+)
+
+print(chat)
+
+stream = co.chat_stream(
+ message="Tell me a short story"
+)
+
+for event in stream:
+ if event.event_type == "text-generation":
+ print(event.text, end='')
+
+agentops.end_session('Success')
diff --git a/examples/multi_agent_example.ipynb b/examples/multi_agent_example.ipynb
index 65cb7841..b59e0216 100644
--- a/examples/multi_agent_example.ipynb
+++ b/examples/multi_agent_example.ipynb
@@ -2,17 +2,22 @@
"cells": [
{
"cell_type": "markdown",
- "source": [
- "# Multi-Agent Support\n",
- "This is an example implementation of tracking events from two separate agents"
- ],
+ "id": "a2e266428cefc683",
"metadata": {
"collapsed": false
},
- "id": "a2e266428cefc683"
+ "source": [
+ "# Multi-Agent Support\n",
+ "This is an example implementation of tracking events from two separate agents"
+ ]
},
{
"cell_type": "code",
+ "execution_count": null,
+ "id": "7c566fac57d3b6ce",
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"import agentops\n",
@@ -23,53 +28,53 @@
"import logging\n",
"\n",
"from IPython.display import display, Markdown"
- ],
- "metadata": {
- "collapsed": false
- },
- "id": "7c566fac57d3b6ce",
- "execution_count": null
+ ]
},
{
"cell_type": "code",
+ "execution_count": null,
+ "id": "9f8c52496c04693",
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"load_dotenv()\n",
"OPENAI_API_KEY = os.getenv('OPENAI_API_KEY', \"<your_openai_key>\")\n",
"AGENTOPS_API_KEY = os.getenv('AGENTOPS_API_KEY', \"<your_agentops_key>\")\n",
"logging.basicConfig(level=logging.DEBUG) # this will let us see that calls are assigned to an agent"
- ],
- "metadata": {
- "collapsed": false
- },
- "id": "9f8c52496c04693",
- "execution_count": null
+ ]
},
{
"cell_type": "code",
+ "execution_count": null,
+ "id": "af062552554d60ce",
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"agentops.init(AGENTOPS_API_KEY)\n",
"openai_client = OpenAI(api_key = OPENAI_API_KEY )"
- ],
- "metadata": {
- "collapsed": false
- },
- "id": "af062552554d60ce",
- "execution_count": null
+ ]
},
{
"cell_type": "markdown",
- "source": [
- "Now lets create a few agents!"
- ],
+ "id": "95d212546aaf1f82",
"metadata": {
"collapsed": false
},
- "id": "95d212546aaf1f82"
+ "source": [
+ "Now lets create a few agents!"
+ ]
},
{
"cell_type": "code",
+ "execution_count": null,
+ "id": "727e3cc26ce3ec3",
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"@track_agent(name='qa')\n",
@@ -85,108 +90,108 @@
" res = openai_client.chat.completions.create(model='gpt-3.5-turbo', messages=[{\"role\": \"system\", \"content\": \"You are a software engineer and only output python code, no markdown tags.\"},\n",
" {\"role\": \"user\", \"content\": prompt}], temperature=0.5)\n",
" return res.choices[0].message.content"
- ],
- "metadata": {
- "collapsed": false
- },
- "id": "727e3cc26ce3ec3",
- "execution_count": null
+ ]
},
{
"cell_type": "code",
+ "execution_count": null,
+ "id": "79b75d65de738522",
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"qa = QaAgent()\n",
"engineer = EngineerAgent()"
- ],
- "metadata": {
- "collapsed": false
- },
- "id": "79b75d65de738522",
- "execution_count": null
+ ]
},
{
"cell_type": "markdown",
+ "id": "69dd3af9206308cc",
+ "metadata": {
+ "collapsed": false
+ },
"source": [
"Now we have our agents and we tagged them with the `@track_agent` decorator. Any LLM calls that go through this class will now be tagged as agent calls in AgentOps.\n",
"\n",
"Lets use these agents!"
- ],
- "metadata": {
- "collapsed": false
- },
- "id": "69dd3af9206308cc"
+ ]
},
{
"cell_type": "code",
- "outputs": [],
- "source": [
- "generated_func = engineer.completion(\"Write a python function that accepts two numbers and multiplies them together, then divides by two. No example.\")"
- ],
+ "execution_count": null,
+ "id": "69e76061a626549",
"metadata": {
"collapsed": false
},
- "id": "69e76061a626549",
- "execution_count": null
+ "outputs": [],
+ "source": [
+ "generated_func = engineer.completion(\"Write a python function that accepts two numbers and multiplies them together, then divides by two. No example.\")"
+ ]
},
{
"cell_type": "code",
- "outputs": [],
- "source": [
- "display(Markdown('```python\\n' + generated_func + '\\n```'))"
- ],
+ "execution_count": null,
+ "id": "830b86dac47dceb3",
"metadata": {
"collapsed": false
},
- "id": "830b86dac47dceb3",
- "execution_count": null
+ "outputs": [],
+ "source": [
+ "display(Markdown('```python\\n' + generated_func + '\\n```'))"
+ ]
},
{
"cell_type": "code",
- "outputs": [],
- "source": [
- "generated_test = qa.completion(\"Write a python unit test that test the following function: \\n \" + generated_func)"
- ],
+ "execution_count": null,
+ "id": "63c9d0d457aee91a",
"metadata": {
"collapsed": false
},
- "id": "63c9d0d457aee91a",
- "execution_count": null
+ "outputs": [],
+ "source": [
+ "generated_test = qa.completion(\"Write a python unit test that test the following function: \\n \" + generated_func)"
+ ]
},
{
"cell_type": "code",
- "outputs": [],
- "source": [
- "display(Markdown('```python\\n' + generated_test + '\\n```'))"
- ],
+ "execution_count": null,
+ "id": "a88ffcbd2015d422",
"metadata": {
"collapsed": false
},
- "id": "a88ffcbd2015d422",
- "execution_count": null
+ "outputs": [],
+ "source": [
+ "display(Markdown('```python\\n' + generated_test + '\\n```'))"
+ ]
},
{
"cell_type": "markdown",
- "source": [
- "Perfect! It generated the code as expected, and in the DEBUG logs, you can see that the calls were made by agents named \"engineer\" and \"qa\"!"
- ],
+ "id": "1bd312ed049a5511",
"metadata": {
"collapsed": false
},
- "id": "1bd312ed049a5511"
+ "source": [
+ "Perfect! It generated the code as expected, and in the DEBUG logs, you can see that the calls were made by agents named \"engineer\" and \"qa\"!"
+ ]
},
{
"cell_type": "markdown",
- "source": [
- "Lets verify one more thing! If we make an LLM call outside of the context of a tracked agent, we want to make sure it gets assigned to the Default Agent."
- ],
+ "id": "cbd0817a31756397",
"metadata": {
"collapsed": false
},
- "id": "cbd0817a31756397"
+ "source": [
+ "Lets verify one more thing! If we make an LLM call outside of the context of a tracked agent, we want to make sure it gets assigned to the Default Agent."
+ ]
},
{
"cell_type": "code",
+ "execution_count": null,
+ "id": "122e923cb07fd5f4",
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"res = openai_client.chat.completions.create(\n",
@@ -195,22 +200,17 @@
" {\"role\": \"user\", \"content\": \"Say hello\"}]\n",
")\n",
"res.choices[0].message.content"
- ],
- "metadata": {
- "collapsed": false
- },
- "id": "122e923cb07fd5f4",
- "execution_count": null
+ ]
},
{
"cell_type": "markdown",
- "source": [
- "You'll notice that we didn't log an agent name, so the AgentOps backend will assign it to the Default Agent for the session!"
- ],
+ "id": "a30909020c6a1ada",
"metadata": {
"collapsed": false
},
- "id": "a30909020c6a1ada"
+ "source": [
+ "You'll notice that we didn't log an agent name, so the AgentOps backend will assign it to the Default Agent for the session!"
+ ]
}
],
"metadata": {
@@ -229,7 +229,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
- "version": "2.7.6"
+ "version": "3.12.2"
}
},
"nbformat": 4,
diff --git a/pyproject.toml b/pyproject.toml
index 0e815c02..49250903 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "agentops"
-version = "0.1.9"
+version = "0.1.10"
authors = [
{ name="Alex Reibman", email="[email protected]" },
{ name="Shawn Qiu", email="[email protected]" },
diff --git a/tests/openai_handlers/_test_single_openaiv1.py b/tests/openai_handlers/_test_single_openaiv1.py
new file mode 100644
index 00000000..ad3e0046
--- /dev/null
+++ b/tests/openai_handlers/_test_single_openaiv1.py
@@ -0,0 +1,25 @@
+from openai import OpenAI, AsyncOpenAI
+import openai
+import agentops
+from dotenv import load_dotenv
+load_dotenv()
+client = OpenAI()
+
+async_client = AsyncOpenAI()
+
+# Assuming that initializing will trigger the LlmTracker to override methods
+agentops.init(tags=['mock agent', openai.__version__])
+
+# Now the client.chat.completions.create should be the overridden method
+print('Chat completion')
+chat_completion = client.chat.completions.create(
+ messages=[
+ {
+ "role": "user",
+ "content": "Say this is a test",
+ }
+ ],
+ model="gpt-3.5-turbo",
+)
+
+agentops.end_session('Success')