Skip to content

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI

the rest

reibs38011714c495
diff --git a/.github/CODE_OF_CONDUCT.md b/.github/CODE_OF_CONDUCT.md
new file mode 100644
index 00000000..cb6b9efa
--- /dev/null
+++ b/.github/CODE_OF_CONDUCT.md
@@ -0,0 +1,20 @@
+# Code of Conduct
+
+Be kind and be mindful. We're here to accelerate together.
+
+All participants in our project's community are expected to follow this code of conduct, which covers our behavior as contributors, maintainers, and community members.
+
+## Our Pledge
+
+
+We pledge to make participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
\ No newline at end of file
diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
new file mode 100644
index 00000000..74601edc
--- /dev/null
+++ b/.github/CONTRIBUTING.md
@@ -0,0 +1,15 @@
+# Contributing Guide
+
+Thank you for your interest in contributing to our project!
+
+## Setting Up the Development Environment
+
+Create a virtual environment `python -m venv env` and install requirements `pip install -e .`
+
+## Making a Pull Request
+
+1. Fork the repository on GitHub.
+2. Clone your forked repository to your local machine.
+3. Make your changes and commit them to your forked repository.
+4. Push your changes to your forked repository on GitHub.
+5. Create a new pull request from your forked repository to our original repository.
\ No newline at end of file
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
new file mode 100644
index 00000000..5f0338a3
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -0,0 +1,26 @@
+---
+name: 🐛 Bug Report
+about: Noticed something off with Agentops? Let us know!
+title: ''
+labels: 'bug'
+assignees: ''
+---
+
+## 🐛 Bug Report
+
+**🔎 Describe the Bug**
+Give a clear and concise description of the bug.
+
+**🔄 Reproduction Steps**
+List the steps to reproduce the behavior.
+
+**🙁 Expected Behavior**
+Describe what you expected to happen.
+
+**📸 Screenshots**
+If applicable, add screenshots to help explain the problem.
+
+**🔍 Additional Context**
+Provide any other context about the problem here.
+
+Thank you for helping us improve Agentops!
diff --git a/.github/ISSUE_TEMPLATE/documentation_update.md b/.github/ISSUE_TEMPLATE/documentation_update.md
new file mode 100644
index 00000000..5a4bdb2d
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/documentation_update.md
@@ -0,0 +1,21 @@
+---
+name: 📚 Documentation Update
+about: Suggest an improvement or addition to our documentation
+title: ''
+labels: 'documentation'
+assignees: ''
+
+---
+
+## 📚 Documentation Update
+
+**📘 Describe the current state of documentation**
+What information is missing or could be improved?
+
+**📖 Suggested Improvement**
+Describe your idea for the new/improved documentation.
+
+**🔍 Additional Context**
+Any extra information to help us understand the change?
+
+Thank you for helping us improve Agentops documentation!
\ No newline at end of file
diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md
new file mode 100644
index 00000000..3ab7460b
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/feature_request.md
@@ -0,0 +1,27 @@
+---
+name: 🚀 Feature Request
+about: Got an idea to improve Agentops? We're all ears!
+title: ''
+labels: 'enhancement'
+assignees: ''
+---
+
+## 🚀 Feature Request
+
+**💡 Got a brilliant idea?**
+Briefly describe your feature suggestion. We're all about improving agent observablity and performance.
+
+**🤔 Is your feature request related to a problem?**
+If your feature request is related to a problem you're facing, describe it here. The more we understand your struggle, the better we can address it.
+
+**🎉 Describe the solution you'd like**
+What's your solution? Be as detailed as possible.
+
+**📚 Have you considered any alternatives?**
+Share any alternative solutions or features you've considered.
+
+**🖼️ Can you provide a visual mock-up or sketch?**
+If possible, provide a visual mock-up or sketch of your feature request. Visual aids can help us better understand your vision.
+
+**🔍 Additional context**
+Any extra information or context to help us understand better? Share it here.
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
new file mode 100644
index 00000000..b179923d
--- /dev/null
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -0,0 +1,26 @@
+---
+name: 📥 Pull Request
+about: Contribute to Agentops by submitting a pull request
+title: ''
+labels: 'pull request'
+assignees: ''
+---
+
+## 📥 Pull Request
+
+**📘 Description**
+Briefly describe the changes you've made.
+
+**🔄 Related Issue (if applicable)**
+If this PR is related to an existing issue, reference it here.
+
+**🎯 Goal**
+Explain the intention behind this change.
+
+**🔍 Additional Context**
+Any extra information or context to help us understand the change?
+
+**🧪 Testing**
+Describe the tests you performed to validate your changes.
+
+Thank you for your contribution to Agentops!
diff --git a/.github/stale.yml b/.github/stale.yml
new file mode 100644
index 00000000..61841c19
--- /dev/null
+++ b/.github/stale.yml
@@ -0,0 +1,4 @@
+# Configuration for probot-stale - https://github.com/probot/stale
+
+# Number of days of inactivity before an issue becomes stale
+daysUntilStale: 60
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 00000000..eeb33f95
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,164 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+# Usually these files are written by a python script from a template
+# before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+# For a library or package, you might want to ignore these files since the code is
+# intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+# However, in case of collaboration, if having platform-specific dependencies or dependencies
+# having no cross-platform support, pipenv may install dependencies that don't work, or not
+# install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+# This is especially recommended for binary packages to ensure reproducibility, and is more
+# commonly ignored for libraries.
+# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+# in version control.
+# https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+# and can be added to the global gitignore or merged into this file. For a more nuclear
+# option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+.vscode/
+.benchmarks/
+.DS_Store
\ No newline at end of file
diff --git a/LICENSE b/LICENSE
new file mode 100644
index 00000000..7a64b51e
--- /dev/null
+++ b/LICENSE
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 AgentOps-AI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
diff --git a/agentops/__init__.py b/agentops/__init__.py
new file mode 100755
index 00000000..bb509241
--- /dev/null
+++ b/agentops/__init__.py
@@ -0,0 +1,4 @@
+from .agentops import AgentOps
+from .event import Event
+from .config import Configuration
+from .logger import AgentOpsLogger
\ No newline at end of file
diff --git a/agentops/agentops.py b/agentops/agentops.py
new file mode 100644
index 00000000..253505a5
--- /dev/null
+++ b/agentops/agentops.py
@@ -0,0 +1,113 @@
+"""
+AgentOps client module that provides a client class with public interfaces and configuration.
+
+Classes:
+ AgentOps: Provides methods to interact with the AgentOps service.
+"""
+
+from .config import Configuration
+from .event import Session, Event
+from .worker import Worker
+from uuid import uuid4
+import json
+from typing import Optional, Dict
+import functools
+import inspect
+
+
+class AgentOps:
+ """
+ Client for AgentOps service.
+
+ Args:
+ api_key (str): API Key for AgentOps services.
+ tags (Dict[str, str], optional): Tags for the sessions that can be used for grouping or sorting later (e.g. {"llm": "GPT-4"}).
+ config (Configuration, optional): A Configuration object for AgentOps services. If not provided, a default Configuration object will be used.
+
+ Attributes:
+ session (Session, optional): A Session is a grouping of events (e.g. a run of your agent).
+ config (Configuration): A Configuration object for AgentOps services.
+ """
+
+ def __init__(self, api_key: str, tags: Optional[Dict[str, str]] = None, config: Configuration = Configuration()):
+ self.config: Configuration = config
+ self.config.api_key = api_key
+ self.start_session(tags)
+
+ def record(self, event: Event):
+ """
+ Record an event with the AgentOps service.
+
+ Args:
+ event (Event): The event to record.
+ """
+
+ if not self.session.has_ended():
+ self.worker.add_event(
+ {'session_id': self.session.get_session_id(), **event.__dict__})
+ else:
+ print("This event was not recorded because the previous session has been ended. Start a new session to record again")
+ print(event.__dict__)
+
+ def record_action(self, event_name: str, tags: Optional[Dict[str, str]] = None):
+ """
+ Decorator to record an event before and after a function call.
+ Args:
+ event_name (str): The name of the event to record.
+ tags (dict, optional): Any tags associated with the event. Defaults to None.
+ """
+ def decorator(func):
+ @functools.wraps(func)
+ def wrapper(*args, **kwargs):
+ func_args = inspect.signature(func).parameters
+ arg_names = list(func_args.keys())
+ arg_values = dict(zip(arg_names, args))
+ arg_values.update(kwargs)
+
+ try:
+ output = func(*args, **kwargs)
+
+ # Record the event after the function call
+ self.record(Event(event_type=event_name,
+ params=arg_values,
+ output=output,
+ result="SUCCESS",
+ tags=tags))
+
+ except Exception as e:
+ # Record the event after the function call
+ self.record(Event(event_type=event_name,
+ params=arg_values,
+ output=None,
+ result='FAIL',
+ tags=tags))
+
+ # Re-raise the exception
+ raise
+
+ return output
+
+ return wrapper
+
+ return decorator
+
+ def start_session(self, tags: Optional[Dict[str, str]] = None):
+ """
+ Start a new session for recording events.
+
+ Args:
+ tags (Dict[str, str], optional): Tags that can be used for grouping or sorting later. Examples could be {"llm": "GPT-4"}.
+ """
+ self.session = Session(str(uuid4()), tags)
+ self.worker = Worker(self.config)
+
+ def end_session(self, end_state: Optional[str] = None, rating: Optional[str] = None):
+ """
+ End the current session with the AgentOps service.
+
+ Args:
+ end_state (str, optional): The final state of the session.
+ rating (str, optional): The rating for the session.
+ """
+ self.session.end_session(end_state, rating)
+ self.worker.end_session(self.session)
diff --git a/agentops/config.py b/agentops/config.py
new file mode 100644
index 00000000..16f2f266
--- /dev/null
+++ b/agentops/config.py
@@ -0,0 +1,107 @@
+"""
+AgentOps configuration.
+
+Classes:
+ Configuration: Stores the configuration settings for AgentOps clients.
+"""
+
+from typing import Optional
+
+
+class Configuration:
+ """
+ Stores the configuration settings for AgentOps clients.
+
+ Args:
+ api_key (str, optional): API Key for AgentOps services
+ endpoint (str, optional): The endpoint for the AgentOps service. Defaults to 'https://agentops-server-v2.fly.dev'.
+ max_wait_time (int, optional): The maximum time to wait in milliseconds before flushing the queue. Defaults to 1000.
+ max_queue_size (int, optional): The maximum size of the event queue. Defaults to 100.
+ """
+
+ def __init__(self, api_key: Optional[str] = None, endpoint: Optional[str] = 'https://agentops-server-v2.fly.dev',
+ max_wait_time: Optional[int] = 1000, max_queue_size: Optional[int] = 100):
+ self._api_key: str | None = api_key
+ self._endpoint = endpoint
+ self._max_wait_time = max_wait_time
+ self._max_queue_size = max_queue_size
+
+ @property
+ def api_key(self) -> str:
+ """
+ Get the API Key for AgentOps services.
+
+ Returns:
+ str: The API Key for AgentOps services.
+ """
+ return self._api_key
+
+ @api_key.setter
+ def api_key(self, value: str):
+ """
+ Set the API Key for AgentOps services.
+
+ Args:
+ value (str): The new API Key.
+ """
+ self._api_key = value
+
+ @property
+ def endpoint(self) -> str:
+ """
+ Get the endpoint for the AgentOps service.
+
+ Returns:
+ str: The endpoint for the AgentOps service.
+ """
+ return self._endpoint
+
+ @endpoint.setter
+ def endpoint(self, value: str):
+ """
+ Set the endpoint for the AgentOps service.
+
+ Args:
+ value (str): The new endpoint.
+ """
+ self._endpoint = value
+
+ @property
+ def max_wait_time(self) -> int:
+ """
+ Get the maximum wait time for the AgentOps service.
+
+ Returns:
+ int: The maximum wait time.
+ """
+ return self._max_wait_time
+
+ @max_wait_time.setter
+ def max_wait_time(self, value: int):
+ """
+ Set the maximum wait time for the AgentOps service.
+
+ Args:
+ value (int): The new maximum wait time.
+ """
+ self._max_wait_time = value
+
+ @property
+ def max_queue_size(self) -> int:
+ """
+ Get the maximum size of the event queue.
+
+ Returns:
+ int: The maximum size of the event queue.
+ """
+ return self._max_queue_size
+
+ @max_queue_size.setter
+ def max_queue_size(self, value: int):
+ """
+ Set the maximum size of the event queue.
+
+ Args:
+ value (int): The new maximum size of the event queue.
+ """
+ self._max_queue_size = value
diff --git a/agentops/event.py b/agentops/event.py
new file mode 100644
index 00000000..135d5d2c
--- /dev/null
+++ b/agentops/event.py
@@ -0,0 +1,92 @@
+from .helpers import get_ISO_time
+from typing import Optional, Dict, Literal
+
+"""
+AgentOps events.
+
+Classes:
+ Event: Represents discrete events to be recorded.
+ Session: Represents a session of events, with a start and end state.
+"""
+
+
+class Event:
+ """
+ Represents a discrete event to be recorded.
+
+ Args:
+ event_type (str): Type of the event, e.g., "API Call". Required.
+ params (str, optional): The parameters passed to the operation.
+ output (str, optional): The output of the operation.
+ result (str, optional): Result of the operation, e.g., "success", "fail", "indeterminate".
+ tags (Dict[str, str], optional): Tags that can be used for grouping or sorting later. e.g. {"llm": "GPT-4"}.
+
+
+ Attributes:
+ timestamp (float): The timestamp for when the event was created, represented as seconds since the epoch.
+ """
+
+ def __init__(self, event_type: str,
+ params: Optional[str] = None,
+ output: Optional[str] = None,
+ result: Optional[str] = None,
+ tags: Optional[Dict[str, str]] = None
+ ):
+ self.event_type = event_type
+ self.params = params
+ self.output = output
+ self.result = result
+ self.tags = tags
+ self.timestamp = get_ISO_time()
+
+
+class Session:
+ """
+ Represents a session of events, with a start and end state.
+
+ Args:
+ session_id (str): The session id is used to record particular runs.
+ tags (Dict[str, str], optional): Tags that can be used for grouping or sorting later. Examples could be {"llm": "GPT-4"}.
+
+ Attributes:
+ init_timestamp (float): The timestamp for when the session started, represented as seconds since the epoch.
+ end_timestamp (float, optional): The timestamp for when the session ended, represented as seconds since the epoch. This is only set after end_session is called.
+ end_state (str, optional): The final state of the session. Suggested: "Success", "Fail", "Indeterminate"
+ rating (str, optional): The rating for the session.
+
+ """
+
+ def __init__(self, session_id: str, tags: Optional[Dict[str, str]] = None):
+ self.session_id = session_id
+ self.init_timestamp = get_ISO_time()
+ self.tags = tags
+
+ def end_session(self, end_state: Optional[str], rating: Optional[str] = None):
+ """
+ End the session with a specified state and rating.
+
+ Args:
+ end_state (str, optional): The final state of the session. Suggested: "Success", "Fail", "Indeterminate"
+ rating (str, optional): The rating for the session.
+ """
+ self.end_state = end_state
+ self.rating = rating
+ self.end_timestamp = get_ISO_time()
+
+ def get_session_id(self) -> str:
+ """
+ Get the session id.
+
+ Returns:
+ str: The id of the session.
+ """
+ return self.session_id
+
+ def has_ended(self) -> str:
+ """
+ Returns whether the session has been ended
+
+ Returns:
+ bool: Whether the session has been ended
+ """
+ return hasattr(self, "end_state")
diff --git a/agentops/helpers.py b/agentops/helpers.py
new file mode 100644
index 00000000..f8752c82
--- /dev/null
+++ b/agentops/helpers.py
@@ -0,0 +1,5 @@
+import time
+from datetime import datetime
+
+def get_ISO_time():
+ return datetime.fromtimestamp(time.time()).isoformat(timespec='milliseconds') + 'Z'
\ No newline at end of file
diff --git a/agentops/http.py b/agentops/http.py
new file mode 100644
index 00000000..70cf0ed8
--- /dev/null
+++ b/agentops/http.py
@@ -0,0 +1,88 @@
+import json
+from enum import Enum
+from typing import Optional
+import requests
+from requests.adapters import Retry, HTTPAdapter
+
+JSON_HEADER = {
+ "Content-Type": "application/json; charset=UTF-8",
+ "Accept": "*/*"
+}
+
+retry_config = Retry(total=5, backoff_factor=0.1)
+
+
+class HttpStatus(Enum):
+ SUCCESS = 200
+ INVALID_REQUEST = 400
+ TIMEOUT = 408
+ PAYLOAD_TOO_LARGE = 413
+ TOO_MANY_REQUESTS = 429
+ FAILED = 500
+ UNKNOWN = -1
+
+
+class Response:
+
+ def __init__(self, status: HttpStatus = HttpStatus.UNKNOWN, body: Optional[dict] = None):
+ self.status: HttpStatus = status
+ self.code: int = status.value
+ self.body = body
+ if not self.body:
+ self.body = {}
+
+ def parse(self, res: requests.models.Response):
+ res_body = res.json()
+ self.code = res.status_code
+ self.status = self.get_status(self.code)
+ self.body = res_body
+ return self
+
+ @staticmethod
+ def get_status(code: int) -> HttpStatus:
+ if 200 <= code < 300:
+ return HttpStatus.SUCCESS
+ elif code == 429:
+ return HttpStatus.TOO_MANY_REQUESTS
+ elif code == 413:
+ return HttpStatus.PAYLOAD_TOO_LARGE
+ elif code == 408:
+ return HttpStatus.TIMEOUT
+ elif 400 <= code < 500:
+ return HttpStatus.INVALID_REQUEST
+ elif code >= 500:
+ return HttpStatus.FAILED
+ return HttpStatus.UNKNOWN
+
+
+class HttpClient:
+
+ @staticmethod
+ def post(url: str, payload: bytes, api_key: str = None, header=None) -> Response:
+ result = Response()
+ try:
+ # Create request session with retries configured
+ request_session = requests.Session()
+ request_session.mount(url, HTTPAdapter(max_retries=retry_config))
+
+ if api_key != None:
+ JSON_HEADER["X-Agentops-Auth"] = api_key
+
+ res = request_session.post(url, data=payload,
+ headers=JSON_HEADER, timeout=20)
+
+ result.parse(res)
+ except requests.exceptions.Timeout:
+ result.code = 408
+ result.status = HttpStatus.TIMEOUT
+ except requests.exceptions.HTTPError as e:
+ try:
+ result.parse(e.response)
+ except:
+ result = Response()
+ result.code = e.response.status_code
+ result.status = Response.get_status(e.response.status_code)
+ result.body = {'error': str(e)}
+ except requests.exceptions.RequestException as e:
+ result.body = {'error': str(e)}
+ return result
diff --git a/agentops/logger.py b/agentops/logger.py
new file mode 100755
index 00000000..43f9fae1
--- /dev/null
+++ b/agentops/logger.py
@@ -0,0 +1,111 @@
+import logging
+import re
+from .agentops import AgentOps
+from .event import Event
+
+class AgentOpsLogger():
+ """
+ A utility class for creating loggers and handlers configured to work with the AgentOps service.
+
+ This class provides two static methods for creating a logger or a handler that sends log
+ records to the AgentOps service. The logger and handler are configured with a specific
+ AgentOps client and name.
+
+ Example Usage:
+
+ >>> from agentops import AgentOps
+ >>> client = AgentOps(...)
+ >>> logger = AgentOpsLogger.get_agentops_logger(client, 'my_logger')
+ >>> logger.info('This is an info log')
+
+ This will send an 'info' log to the AgentOps service.
+ """
+
+ @staticmethod
+ def get_agentops_logger(client: AgentOps, name: str, level=logging.DEBUG):
+ """
+ Create and return a logger with an AgentOpsHandler.
+
+ Args:
+ client (AgentOps): The AgentOps client to which the logs will be sent.
+ name (str): The name for the logger and handler.
+ level (int, optional): The minimum severity level to log. Defaults to logging.DEBUG.
+
+ Returns:
+ logging.Logger: A logger configured with an AgentOpsHandler.
+ """
+ logger = logging.getLogger(name)
+ logger.setLevel(level)
+ handler = AgentOpsHandler(client, name)
+ handler.setLevel(level)
+ logger.addHandler(handler)
+ return logger
+
+ @staticmethod
+ def get_agentops_handler(client: AgentOps, name: str):
+ """
+ Create and return an AgentOpsHandler.
+
+ Args:
+ client (AgentOps): The AgentOps client to which the logs will be sent.
+ name (str): The name for the handler.
+
+ Returns:
+ AgentOpsHandler: A new AgentOpsHandler with the given client and name.
+ """
+ return AgentOpsHandler(client, name)
+
+
+class AgentOpsHandler(logging.Handler):
+ """
+ Custom logging handler for sending logs to the AgentOps service.
+
+ This handler extends the built-in logging.Handler class to send log records to AgentOps.
+ It also removes ANSI color codes from log messages before sending them.
+ """
+
+ def __init__(self, client: AgentOps, name: str):
+ """
+ Initialize the handler with a specific AgentOps client and name.
+
+ Args:
+ client (AgentOps): The AgentOps client to which the logs will be sent.
+ name (str): The name for this handler.
+ """
+ super().__init__()
+ self.name = name
+ self.client = client
+
+ @staticmethod
+ def remove_color_codes(s: str) -> str:
+ """
+ Remove ANSI color codes from a string.
+
+ Args:
+ s (str): The string from which color codes will be removed.
+
+ Returns:
+ The same string, but without any color codes.
+ """
+ ansi_escape = re.compile(r"\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])")
+ return ansi_escape.sub("", s)
+
+ def emit(self, record):
+ """
+ Process a log record and send it to the AgentOps client.
+
+ This method is called whenever a log record needs to be processed.
+
+ Args:
+ record (logging.LogRecord): The log record to process.
+ """
+ log_entry = self.format(record)
+ log_entry = self.remove_color_codes(log_entry)
+
+ if record.levelno == logging.ERROR:
+ result = "fail"
+ else:
+ result = "indeterminate"
+
+ self.client.record(
+ Event(f'{self.name}:{record.levelname}', output=log_entry, result=result))
diff --git a/agentops/worker.py b/agentops/worker.py
new file mode 100644
index 00000000..f2e98838
--- /dev/null
+++ b/agentops/worker.py
@@ -0,0 +1,57 @@
+import json
+import threading
+import time
+from .http import HttpClient
+from .config import Configuration
+from .event import Event, Session
+from typing import Dict
+
+
+class Worker:
+ def __init__(self, config: Configuration) -> None:
+ self.config = config
+ self.queue: list[Dict] = []
+ self.lock = threading.Lock()
+ self.stop_flag = threading.Event()
+ self.thread = threading.Thread(target=self.run)
+ self.thread.start()
+
+ def add_event(self, event: str) -> None:
+ with self.lock:
+ self.queue.append(event)
+ if len(self.queue) >= self.config.max_queue_size:
+ self.flush_queue()
+
+ def flush_queue(self) -> None:
+ with self.lock:
+ if len(self.queue) > 0:
+ events = self.queue
+ self.queue = []
+
+ payload = {
+ "events": events
+ }
+
+ HttpClient.post(f'{self.config.endpoint}/events',
+ json.dumps(payload).encode("utf-8"),
+ self.config.api_key)
+
+ def end_session(self, session: Session) -> None:
+ self.stop_flag.set()
+ self.thread.join()
+ self.flush_queue()
+
+ with self.lock:
+ payload = {
+ "session": session.__dict__
+ }
+
+ HttpClient.post(f'{self.config.endpoint}/sessions',
+ json.dumps(payload).encode("utf-8"),
+ self.config.api_key)
+
+ def run(self) -> None:
+ while not self.stop_flag.is_set():
+ time.sleep(self.config.max_wait_time / 1000)
+ if self.queue:
+ self.flush_queue()
diff --git a/setup.py b/setup.py
new file mode 100644
index 00000000..fc4b996c
--- /dev/null
+++ b/setup.py
@@ -0,0 +1,18 @@
+from setuptools import setup, find_packages
+
+setup(
+ name='agentops',
+ version='0.1',
+ author='Alex Reibman',
+ author_email='[email protected]',
+ packages=find_packages(),
+ install_requires=[
+ 'requests'
+ ],
+ extras_require={
+ 'dev': [
+ 'pytest',
+ 'requests_mock'
+ ],
+ },
+)
\ No newline at end of file
diff --git a/tests/test_canary.py b/tests/test_canary.py
new file mode 100644
index 00000000..5f268eca
--- /dev/null
+++ b/tests/test_canary.py
@@ -0,0 +1,38 @@
+import pytest
+import requests_mock
+import time
+
+from agentops import AgentOps, Event, Configuration
+
+
+def mock_req():
+ with requests_mock.Mocker() as m:
+ url = 'https://agentops-server-v2.fly.dev'
+ m.post(url + '/events', text='ok')
+ m.post(url + '/sessions', text='ok')
+ yield m
+
+class TestCanary:
+ def setup_method(self):
+ self.url = 'https://agentops-server-v2.fly.dev'
+ self.api_key = "random_api_key"
+ self.config = Configuration(max_wait_time=5)
+ self.client = AgentOps(api_key=self.api_key, config=self.config)
+
+ def teardown_method(self):
+ self.client.end_session(end_state="success")
+
+ def test_agent_ops_record(self, mock_req):
+ # Arrange
+ event_type = 'test_event_type'
+ # Act
+
+ self.client.record(Event(event_type))
+ time.sleep(0.1)
+
+ # Assert
+ assert len(mock_req.request_history) == 1
+ request_json = mock_req.last_request.json()
+ assert mock_req.last_request.headers['X-Agentops-Auth'] == self.api_key
+ assert request_json['events'][0]['event_type'] == event_type
diff --git a/tests/test_logger.py b/tests/test_logger.py
new file mode 100755
index 00000000..0b04b690
--- /dev/null
+++ b/tests/test_logger.py
@@ -0,0 +1,92 @@
+import pytest
+import requests_mock
+import time
+from agentops import AgentOps, AgentOpsLogger, Configuration
+
+
+def mock_req():
+ with requests_mock.Mocker() as m:
+ url = 'https://agentops-server-v2.fly.dev'
+ m.post(url + '/events', text='ok')
+ m.post(url + '/sessions', text='ok')
+ yield m
+
+
+class TestLogger:
+ def setup_method(self):
+ self.url = 'https://agentops-server-v2.fly.dev'
+ self.api_key = "random_api_key"
+ self.config = Configuration(max_wait_time=5)
+ self.client = AgentOps(api_key=self.api_key, config=self.config)
+
+
+ def teardown_method(self):
+ self.client.end_session(end_state="success")
+
+ def test_info(self, mock_req):
+ # Arrange
+ self.event_type = "info"
+ self.logger = AgentOpsLogger.get_agentops_logger(
+ self.client, self.event_type)
+ test_message = "Test info message"
+
+ # Act
+ try:
+ self.logger.info(test_message)
+ except Exception as e:
+ pytest.fail(f"test_info failed with {e}")
+
+
+ time.sleep(0.1)
+
+ # Assert
+ assert len(mock_req.request_history) == 1
+ assert mock_req.last_request.headers['X-Agentops-Auth'] == self.api_key
+ request_json = mock_req.last_request.json()
+ assert request_json['events'][0]['event_type'] == f"{self.event_type}:INFO"
+ assert request_json['events'][0]['output'] == test_message
+
+ def test_error(self, mock_req):
+ # Arrange
+ test_message = "Test error message"
+ self.event_type = "error"
+ self.logger = AgentOpsLogger.get_agentops_logger(
+ self.client, self.event_type)
+
+ # Act
+ try:
+ self.logger.error(test_message)
+ except Exception as e:
+ pytest.fail(f"test_error failed with {e}")
+
+ time.sleep(0.1)
+
+ # Assert
+ assert len(mock_req.request_history) == 1
+ assert mock_req.last_request.headers['X-Agentops-Auth'] == self.api_key
+ request_json = mock_req.last_request.json()
+ assert request_json['events'][0]['event_type'] == f"{self.event_type}:ERROR"
+ assert request_json['events'][0]['output'] == test_message
+
+ def test_warn(self, mock_req):
+ # Arrange
+ test_message = "Test warn message"
+ self.event_type = "warning"
+ self.logger = AgentOpsLogger.get_agentops_logger(
+ self.client, self.event_type)
+
+ # Act
+ try:
+ self.logger.warning(test_message)
+ except Exception as e:
+ pytest.fail(f"test_warn failed with {e}")
+
+ time.sleep(0.1)
+
+ # Assert
+ assert len(mock_req.request_history) == 1
+ assert mock_req.last_request.headers['X-Agentops-Auth'] == self.api_key
+ request_json = mock_req.last_request.json()
+ assert request_json['events'][0]['event_type'] == f"{self.event_type}:WARNING"
+ assert request_json['events'][0]['output'] == test_message
diff --git a/tests/test_session.py b/tests/test_session.py
new file mode 100644
index 00000000..eaa9ff19
--- /dev/null
+++ b/tests/test_session.py
@@ -0,0 +1,132 @@
+import pytest
+import requests_mock
+import time
+
+from agentops import AgentOps, Event, Configuration
+
+
+def mock_req():
+ with requests_mock.Mocker() as m:
+ url = 'https://agentops-server-v2.fly.dev'
+ m.post(url + '/events', text='ok')
+ m.post(url + '/sessions', text='ok')
+ yield m
+
+
+class TestSessions:
+ def setup_method(self):
+ self.api_key = "random_api_key"
+ self.event_type = 'test_event_type'
+
+ def test_session(self, mock_req):
+ # Arrange
+ config = Configuration(max_wait_time=5)
+ client = AgentOps(api_key=self.api_key, config=config)
+
+ # Act
+ client.record(Event(self.event_type))
+
+ # Assert
+ assert len(mock_req.request_history) == 0
+
+ # Act
+ client.record(Event(self.event_type))
+ time.sleep(0.1)
+
+ # Assert
+ assert len(mock_req.request_history) == 1
+ assert mock_req.last_request.headers['X-Agentops-Auth'] == self.api_key
+ request_json = mock_req.last_request.json()
+ assert request_json['events'][0]['event_type'] == self.event_type
+
+ # Act
+ end_state = "Succeed"
+ client.end_session(end_state)
+ time.sleep(0.1)
+
+ assert len(mock_req.request_history) == 2
+ assert mock_req.last_request.headers['X-Agentops-Auth'] == self.api_key
+ request_json = mock_req.last_request.json()
+ assert request_json['session']['rating'] == None
+ assert request_json['session']['end_state'] == end_state
+ assert request_json['session']['tags'] == None
+
+ def test_tags(self, mock_req):
+ # Arrange
+ tags = {'llm': 'GPT-4'}
+ config = Configuration(max_wait_time=5)
+ client = AgentOps(api_key=self.api_key, config=config, tags=tags)
+
+ # Act
+ client.record(Event(self.event_type))
+ time.sleep(0.1)
+
+ # Assert
+ assert len(mock_req.request_history) == 1
+ assert mock_req.last_request.headers['X-Agentops-Auth'] == self.api_key
+ request_json = mock_req.last_request.json()
+ assert request_json['events'][0]['event_type'] == self.event_type
+
+ # Act
+ end_state = "Succeed"
+ client.end_session()
+ time.sleep(0.1)
+
+ assert len(mock_req.request_history) == 2
+ assert mock_req.last_request.headers['X-Agentops-Auth'] == self.api_key
+ request_json = mock_req.last_request.json()
+ assert request_json['session']['rating'] == None
+ assert request_json['session']['end_state'] == None
+ assert request_json['session']['tags'] == tags
+
+
+class TestRecordAction:
+ def setup_method(self):
+ self.url = 'https://agentops-server-v2.fly.dev'
+ self.api_key = "random_api_key"
+ self.event_type = 'test_event_type'
+ self.config = Configuration(max_wait_time=5)
+ self.client = AgentOps(self.api_key, config=self.config)
+
+ def teardown_method(self):
+ self.client.end_session(end_state="success")
+
+ def test_record_action_decorator(self, mock_req):
+ @self.client.record_action(event_name=self.event_type, tags={'foo': 'bar'})
+ def dummy_func(x, y):
+ return x + y
+
+ # Act
+ dummy_func(3, 4)
+
+ # Assert
+ assert len(mock_req.request_history) == 1
+ assert mock_req.last_request.headers['X-Agentops-Auth'] == self.api_key
+ request_json = mock_req.last_request.json()
+ assert request_json['event']['event_type'] == self.event_type
+ assert request_json['event']['params'] == {
+ 'args': [3, 4], 'kwargs': {}}
+ assert request_json['event']['output'] == 7
+ assert request_json['event']['result'] == 'SUCCESS'
+ assert request_json['event']['tags'] == {'foo': 'bar'}
+
+ def test_record_action_decorator(self, mock_req):
+ # Arrange
+ @self.client.record_action(event_name=self.event_type, tags={'foo': 'bar'})
+ def dummy_func(x, y):
+ return x + y
+
+ # Act
+ dummy_func(3, 4)
+ time.sleep(0.1)
+
+ # Assert
+ assert len(mock_req.request_history) == 1
+ assert mock_req.last_request.headers['X-Agentops-Auth'] == self.api_key
+ request_json = mock_req.last_request.json()
+ assert request_json['events'][0]['event_type'] == self.event_type
+ assert request_json['events'][0]['params'] == {'x': 3, 'y': 4}
+ assert request_json['events'][0]['output'] == 7
+ assert request_json['events'][0]['result'] == 'SUCCESS'
+ assert request_json['events'][0]['tags'] == {'foo': 'bar'}
diff --git a/tox.ini b/tox.ini
new file mode 100644
index 00000000..37402e4f
--- /dev/null
+++ b/tox.ini
@@ -0,0 +1,18 @@
+# tox (https://tox.readthedocs.io/) is a tool for running tests
+# in multiple virtualenvs. This configuration file will run the
+# test suite on all supported python versions. To use it, "pip install tox"
+# and then run "tox" from this directory.
+
+[tox]
+envlist = py37, py38, py39, mypy
+
+[testenv]
+deps =
+ pytest
+ requests_mock
+ coverage
+ mypy: mypy
+commands =
+ coverage run --source agentops -m pytest
+ coverage report -m
+ mypy: mypy agentops
\ No newline at end of file