Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(aio): tell agents llma-evaluation-run takes generation evals only (#129079)

Bernat Torresad70cac6db74
diff --git a/products/ai_observability/mcp/tools.yaml b/products/ai_observability/mcp/tools.yaml
index 68db78f22e9f..37aaa3206084 100644
--- a/products/ai_observability/mcp/tools.yaml
+++ b/products/ai_observability/mcp/tools.yaml
@@ -847,10 +847,14 @@ tools:
idempotent: false
title: Run evaluation
description: >
- Manually trigger an evaluation run for a specific $ai_generation event. Enqueues a Temporal workflow that
- asynchronously executes the evaluation and stores the result as a '$ai_evaluation' event. Returns a
- workflow_id and status "started". Pair with execute-sql to query results: SELECT * FROM events WHERE event =
- '$ai_evaluation'.
+ Manually trigger an evaluation run for a specific $ai_generation event. Only works for evaluations whose
+ target is 'generation'; a 'trace' or 'session' evaluation is rejected with evaluation_target_mismatch. To
+ preview hog code on traces or sessions, use llma-evaluation-test-hog. To run an enabled trace or session
+ evaluation on one specific trace or session, use llma-evaluation-backfill-estimate and then
+ llma-evaluation-backfill-create with a condition on $ai_trace_id (or $ai_session_id), a window that contains
+ it, and rerun_existing true. Enqueues a Temporal workflow that asynchronously executes the evaluation and
+ stores the result as a '$ai_evaluation' event. Returns a workflow_id and status "started". Pair with
+ execute-sql to query results: SELECT * FROM events WHERE event = '$ai_evaluation'.
param_overrides:
evaluation_id:
aliases:
diff --git a/services/mcp/schema/generated-tool-definitions.json b/services/mcp/schema/generated-tool-definitions.json
index dc81a9fecd75..b0760884109b 100644
--- a/services/mcp/schema/generated-tool-definitions.json
+++ b/services/mcp/schema/generated-tool-definitions.json
@@ -9328,7 +9328,7 @@
}
},
"llma-evaluation-run": {
- "description": "Manually trigger an evaluation run for a specific $ai_generation event. Enqueues a Temporal workflow that asynchronously executes the evaluation and stores the result as a '$ai_evaluation' event. Returns a workflow_id and status \"started\". Pair with execute-sql to query results: SELECT * FROM events WHERE event = '$ai_evaluation'.",
+ "description": "Manually trigger an evaluation run for a specific $ai_generation event. Only works for evaluations whose target is 'generation'; a 'trace' or 'session' evaluation is rejected with evaluation_target_mismatch. To preview hog code on traces or sessions, use llma-evaluation-test-hog. To run an enabled trace or session evaluation on one specific trace or session, use llma-evaluation-backfill-estimate and then llma-evaluation-backfill-create with a condition on $ai_trace_id (or $ai_session_id), a window that contains it, and rerun_existing true. Enqueues a Temporal workflow that asynchronously executes the evaluation and stores the result as a '$ai_evaluation' event. Returns a workflow_id and status \"started\". Pair with execute-sql to query results: SELECT * FROM events WHERE event = '$ai_evaluation'.",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "Run evaluation",
diff --git a/services/mcp/schema/tool-definitions-all.json b/services/mcp/schema/tool-definitions-all.json
index ab7ce06cba89..febbc2ba34d9 100644
--- a/services/mcp/schema/tool-definitions-all.json
+++ b/services/mcp/schema/tool-definitions-all.json
@@ -9594,7 +9594,7 @@
}
},
"llma-evaluation-run": {
- "description": "Manually trigger an evaluation run for a specific $ai_generation event. Enqueues a Temporal workflow that asynchronously executes the evaluation and stores the result as a '$ai_evaluation' event. Returns a workflow_id and status \"started\". Pair with execute-sql to query results: SELECT * FROM events WHERE event = '$ai_evaluation'.",
+ "description": "Manually trigger an evaluation run for a specific $ai_generation event. Only works for evaluations whose target is 'generation'; a 'trace' or 'session' evaluation is rejected with evaluation_target_mismatch. To preview hog code on traces or sessions, use llma-evaluation-test-hog. To run an enabled trace or session evaluation on one specific trace or session, use llma-evaluation-backfill-estimate and then llma-evaluation-backfill-create with a condition on $ai_trace_id (or $ai_session_id), a window that contains it, and rerun_existing true. Enqueues a Temporal workflow that asynchronously executes the evaluation and stores the result as a '$ai_evaluation' event. Returns a workflow_id and status \"started\". Pair with execute-sql to query results: SELECT * FROM events WHERE event = '$ai_evaluation'.",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "Run evaluation",