Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI
Python agentagentopsagents-sdkaianthropicautogencost-estimationcrewaievalsevaluation-metricsgroqlangchainllmmistralollamaopenaiopenai-agents
Task: Migrate Computed Fields from v2 to v4 API
Description
Develop a strategy and implementation for migrating computed fields from v2 to v4 API. In v2, these fields were computed by "meters", but in v4, they need to be computed from OpenTelemetry data in Clickhouse.
Requirements
- Identify all computed fields in v2 endpoints
- Develop a strategy for computing these fields from OpenTelemetry data
- Implement the computation logic
- Ensure backward compatibility with existing clients
- Optimize queries for performance
Computed Fields to Migrate
- LLM calls (count, tokens, cost)
- Tool calls (count, types)
- Session duration
- Session cost
- Session statistics (success rate, error rate)
- Agent performance metrics
Implementation Details
- Create a new file at
agentops/api/utils/computed_fields.py - Implement functions for computing each field from OpenTelemetry data
- Use span attributes and events to extract the required data
- Optimize queries for performance
- Ensure backward compatibility with existing clients
OpenTelemetry Data Mapping
- Map v2 computed fields to OpenTelemetry concepts:
- LLM calls: Spans with
span.kind=clientandai.model.nameattribute - Tool calls: Spans with
span.kind=clientand specific attributes - Session duration: Difference between first and last span timestamp in a trace
- Session cost: Sum of costs from LLM call spans
- Session statistics: Derived from span status codes and attributes
- LLM calls: Spans with
Query Optimization
- Create materialized views in Clickhouse to precompute common aggregations
- Use efficient query patterns to minimize data transfer
- Implement caching for frequently accessed data
Testing
- Create unit tests for the computation logic
- Test with real data in Clickhouse
- Test performance with large datasets
- Test backward compatibility with existing clients
Dependencies
- Clickhouse client implementation
- Trace, log, and metric endpoints implementation
Estimated Time
12-16 hours