:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(aio): tell agents llma-evaluation-run takes generation evals only (#129079)
Bernat Torresad70cac6db74
diff --git a/products/ai_observability/mcp/tools.yaml b/products/ai_observability/mcp/tools.yamlindex 68db78f22e9f..37aaa3206084 100644--- a/products/ai_observability/mcp/tools.yaml+++ b/products/ai_observability/mcp/tools.yaml@@ -847,10 +847,14 @@ tools:idempotent: falsetitle: Run evaluationdescription: >- Manually trigger an evaluation run for a specific $ai_generation event. Enqueues a Temporal workflow that- asynchronously executes the evaluation and stores the result as a '$ai_evaluation' event. Returns a- workflow_id and status "started". Pair with execute-sql to query results: SELECT * FROM events WHERE event =- '$ai_evaluation'.+ Manually trigger an evaluation run for a specific $ai_generation event. Only works for evaluations whose+ target is 'generation'; a 'trace' or 'session' evaluation is rejected with evaluation_target_mismatch. To+ preview hog code on traces or sessions, use llma-evaluation-test-hog. To run an enabled trace or session+ evaluation on one specific trace or session, use llma-evaluation-backfill-estimate and then+ llma-evaluation-backfill-create with a condition on $ai_trace_id (or $ai_session_id), a window that contains+ it, and rerun_existing true. Enqueues a Temporal workflow that asynchronously executes the evaluation and+ stores the result as a '$ai_evaluation' event. Returns a workflow_id and status "started". Pair with+ execute-sql to query results: SELECT * FROM events WHERE event = '$ai_evaluation'.param_overrides:evaluation_id:aliases:diff --git a/services/mcp/schema/generated-tool-definitions.json b/services/mcp/schema/generated-tool-definitions.jsonindex dc81a9fecd75..b0760884109b 100644--- a/services/mcp/schema/generated-tool-definitions.json+++ b/services/mcp/schema/generated-tool-definitions.json@@ -9328,7 +9328,7 @@}},"llma-evaluation-run": {- "description": "Manually trigger an evaluation run for a specific $ai_generation event. Enqueues a Temporal workflow that asynchronously executes the evaluation and stores the result as a '$ai_evaluation' event. Returns a workflow_id and status \"started\". Pair with execute-sql to query results: SELECT * FROM events WHERE event = '$ai_evaluation'.",+ "description": "Manually trigger an evaluation run for a specific $ai_generation event. Only works for evaluations whose target is 'generation'; a 'trace' or 'session' evaluation is rejected with evaluation_target_mismatch. To preview hog code on traces or sessions, use llma-evaluation-test-hog. To run an enabled trace or session evaluation on one specific trace or session, use llma-evaluation-backfill-estimate and then llma-evaluation-backfill-create with a condition on $ai_trace_id (or $ai_session_id), a window that contains it, and rerun_existing true. Enqueues a Temporal workflow that asynchronously executes the evaluation and stores the result as a '$ai_evaluation' event. Returns a workflow_id and status \"started\". Pair with execute-sql to query results: SELECT * FROM events WHERE event = '$ai_evaluation'.","category": "AI observability","feature": "llm_analytics","summary": "Run evaluation",diff --git a/services/mcp/schema/tool-definitions-all.json b/services/mcp/schema/tool-definitions-all.jsonindex ab7ce06cba89..febbc2ba34d9 100644--- a/services/mcp/schema/tool-definitions-all.json+++ b/services/mcp/schema/tool-definitions-all.json@@ -9594,7 +9594,7 @@}},"llma-evaluation-run": {- "description": "Manually trigger an evaluation run for a specific $ai_generation event. Enqueues a Temporal workflow that asynchronously executes the evaluation and stores the result as a '$ai_evaluation' event. Returns a workflow_id and status \"started\". Pair with execute-sql to query results: SELECT * FROM events WHERE event = '$ai_evaluation'.",+ "description": "Manually trigger an evaluation run for a specific $ai_generation event. Only works for evaluations whose target is 'generation'; a 'trace' or 'session' evaluation is rejected with evaluation_target_mismatch. To preview hog code on traces or sessions, use llma-evaluation-test-hog. To run an enabled trace or session evaluation on one specific trace or session, use llma-evaluation-backfill-estimate and then llma-evaluation-backfill-create with a condition on $ai_trace_id (or $ai_session_id), a window that contains it, and rerun_existing true. Enqueues a Temporal workflow that asynchronously executes the evaluation and stores the result as a '$ai_evaluation' event. Returns a workflow_id and status \"started\". Pair with execute-sql to query results: SELECT * FROM events WHERE event = '$ai_evaluation'.","category": "AI observability","feature": "llm_analytics","summary": "Run evaluation",