🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
Evaluations
- LLM as a judge needs mapping of data to prompt variables - done by the user
- the code evaluators write a hardcoded mapping to the database - handled by the server. The actual data is mapped by the user in code
- Observation filters
- During the evaluator setup users can set all kind of observation filters to select a sample
- Filters used in rules are limited to what the
InMemoryFilterServicecan process
- JobConfigurationID (legacy) and Rule ID (new) are the same (see eval migration)