Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(tracing): point agents at endpoint health in mcp tools (#126397)

Jon McCallum005c4eff73ba
diff --git a/products/tracing/backend/service_health_query_runner.py b/products/tracing/backend/service_health_query_runner.py
index 9e8980a81de4..d550b829dd8a 100644
--- a/products/tracing/backend/service_health_query_runner.py
+++ b/products/tracing/backend/service_health_query_runner.py
@@ -309,7 +309,7 @@ class ServiceHealthQueryRunner:
names = [row["service"] for row in unhealthy[:NEXT_SERVICES_LIMIT]]
query: dict[str, Any] = {"dateRange": _range(self.current), "serviceNames": names, "limit": 20}
if compare_filter := aggregate_compare_filter(self.params.compare):
- query["compareFilter"] = compare_filter
+ query = {**query, "compareFilter": compare_filter, "includeHealth": True}
calls = [{"tool": "apm-spans-aggregate", "why": _clip(f"operations behind {', '.join(names)}"), "query": query}]
worst = next((row for row in unhealthy if row["error_count"]), None)
if worst:
diff --git a/products/tracing/backend/tests/test_service_health_query_runner.py b/products/tracing/backend/tests/test_service_health_query_runner.py
index 5c3a7473afc3..087e218859df 100644
--- a/products/tracing/backend/tests/test_service_health_query_runner.py
+++ b/products/tracing/backend/tests/test_service_health_query_runner.py
@@ -402,12 +402,16 @@ class TestServiceHealthQueryRunner(_TraceSpansTestBase):
aggregate["query"]["dateRange"], {"date_from": "2026-06-02T08:00:00Z", "date_to": "2026-06-02T09:00:00Z"}
)
self.assertEqual(aggregate["query"]["compareFilter"], {"compare": True})
+ self.assertTrue(aggregate["query"]["includeHealth"])
error_spans = calls["query-apm-spans"]["query"]
self.assertEqual((error_spans["serviceNames"], error_spans["statusCodes"]), (["worker"], [2]))
self.assertEqual((error_spans["flatSpans"], error_spans["rootSpans"]), (True, False))
for call in result["next"]:
self.assertLessEqual(len(call["why"]), MAX_REASON_LENGTH)
self.assertEqual(self._run()["next"], [])
+ no_baseline = self._run(environment=None, compare=ServiceHealthCompare.NONE)["next"][0]
+ self.assertEqual(no_baseline["tool"], "apm-spans-aggregate")
+ self.assertNotIn("includeHealth", no_baseline["query"])
def test_a_truncated_scan_does_not_invent_silent_or_new_services(self):
with patch("products.tracing.backend.service_health_query_runner.SERVICE_SCAN_LIMIT", 3):
diff --git a/products/tracing/mcp/prompts/apm-spans-aggregate.md b/products/tracing/mcp/prompts/apm-spans-aggregate.md
index 8c5d274e9d4f..e5b259417af0 100644
--- a/products/tracing/mcp/prompts/apm-spans-aggregate.md
+++ b/products/tracing/mcp/prompts/apm-spans-aggregate.md
@@ -1,4 +1,4 @@
-Aggregate trace span statistics grouped by `(service_name, name)` over a date window.
+Aggregate trace span statistics grouped by `(service_name, name)` over a date window. Set `groupByEndpoint` to group HTTP server spans by route, and `includeHealth` to get a `health_status` per row against a baseline.
All parameters go inside `query` — top-level fields are rejected:
@@ -20,6 +20,8 @@ Use to answer:
- "What's the p95 or p99 latency of `GET /users`?"
- "How many errors did the checkout service emit in the last day?"
- "Did `POST /orders` get slower this week vs last week?" (with `compareFilter`)
+- "Which endpoints regressed today?" (with `groupByEndpoint` and `includeHealth`)
+- "Which operations got slower in this service?" (with `includeHealth`)
For per-call-tree breakdowns (parent → child relationships), use `apm-spans-tree` instead. For time-bucketed trends ("when did it change?"), use `apm-spans-sparkline` instead — `compareFilter` only contrasts two static windows.
diff --git a/products/tracing/skills/exploring-apm-traces/SKILL.md b/products/tracing/skills/exploring-apm-traces/SKILL.md
index 6e3967d21b36..d8120dfcb167 100644
--- a/products/tracing/skills/exploring-apm-traces/SKILL.md
+++ b/products/tracing/skills/exploring-apm-traces/SKILL.md
@@ -104,8 +104,9 @@ To rebuild the tree:
1. `apm-service-health` scores every service in one call: requests, error rate, p50/p95/p99, the change from a baseline window, and a `status` (`silent`, `regressed`, `degraded`, `new`, `recovered`, `improved`, `ok`) with a `reason`. Rows come worst first.
2. Pick the baseline with `compare`: `previous_period` (default), `previous_day`, `previous_week`, or an explicit `baselineDateRange` next to `query`. Scope with `environment` and `serviceNames`.
-3. Run the calls in `next`. They are prefilled: `apm-spans-aggregate` for the operations behind the regressed and degraded services, `apm-spans-sparkline` for when a silent service stopped, `query-apm-spans` for error spans.
-4. Do not build a service summary from `apm-spans-aggregate` rows or from one `apm-spans-count` call per service. Percentiles of operation rows do not add up to a service percentile.
+3. Run the calls in `next`. They are prefilled: `apm-spans-aggregate` with `includeHealth` for the operations behind the regressed and degraded services, `apm-spans-sparkline` for when a silent service stopped, `query-apm-spans` for error spans.
+4. For HTTP services, call `apm-spans-aggregate` with `groupByEndpoint: true` and `includeHealth: true` to see which routes regressed. It counts only HTTP server spans, so it returns no rows for queue consumers and workflow workers.
+5. Do not build a service summary from `apm-spans-aggregate` rows or from one `apm-spans-count` call per service. Percentiles of operation rows do not add up to a service percentile.
#### Daily service health report
diff --git a/services/mcp/schema/generated-tool-definitions.json b/services/mcp/schema/generated-tool-definitions.json
index 4383c4c21231..a4e0682abc5f 100644
--- a/services/mcp/schema/generated-tool-definitions.json
+++ b/services/mcp/schema/generated-tool-definitions.json
@@ -1242,7 +1242,7 @@
"feature_flag": "tracing"
},
"apm-spans-aggregate": {
- "description": "Aggregate trace span statistics grouped by `(service_name, name)` over a date window.\n\nAll parameters go inside `query` — top-level fields are rejected:\n\n```json\n{ \"query\": { \"serviceNames\": [\"api\"], \"dateRange\": { \"date_from\": \"-1h\" } } }\n```\n\nReturns one row per `(service_name, name)` pair with the following metrics:\n\n- `count` — number of spans matched\n- `total_duration_nano`, `avg_duration_nano`, `p50_duration_nano`, `p95_duration_nano`, `p99_duration_nano`, `p999_duration_nano` — duration stats in nanoseconds (1 second = 1,000,000,000 ns). `p999_duration_nano` is the 99.9th percentile and is only meaningful for `(service, name)` groups with enough spans; on low-volume operations it collapses to the max\n- `error_count` — spans with OTel status code `Error` (status_code = 2)\n\nRows are ordered by `total_duration_nano` DESC. By default only the top 100 rows are returned — this keeps the response small, which matters because high span-name cardinality (e.g. untemplated URL paths) can otherwise produce an enormous payload. The top rows already surface the heaviest operations. When more rows genuinely exist the response sets `has_more: true` and `next_offset`; page through with `offset`, or raise `limit` (hard max 5000). Prefer narrowing with `serviceNames`/`filterGroup` over pulling a large page.\n\nUse to answer:\n\n- \"Which services/operations consume the most time?\"\n- \"What's the p95 or p99 latency of `GET /users`?\"\n- \"How many errors did the checkout service emit in the last day?\"\n- \"Did `POST /orders` get slower this week vs last week?\" (with `compareFilter`)\n\nFor per-call-tree breakdowns (parent → child relationships), use `apm-spans-tree` instead. For time-bucketed trends (\"when did it change?\"), use `apm-spans-sparkline` instead — `compareFilter` only contrasts two static windows.\n\n# Comparison window\n\nSet `query.compareFilter.compare: true` to also fetch a comparison window. The response then includes a `compare` array of the same shape as `results`.\n\n- Omit `compare_to` (or set null) to compare against the immediately previous period of equal length (e.g. `dateRange: -1d` → compares vs the day before).\n- Set `compare_to: \"-7d\"` to compare against the window 7 days earlier (same length as the primary window).\n\nWhen `compare` is true, `compare` is always an array — empty (`[]`) if the comparison window matched no spans (for example an operation that didn't exist yet), not null. `compare` is `null` only when no comparison was requested. So an empty array is a real answer (\"nothing in the prior window\"), not a failure.\n\n# Data narrowing\n\n## Property filters\n\nUse `query.filterGroup` to narrow results to spans matching specific attributes. Only include filters that are essential to the user's question.\n\nFilter `type` values:\n\n- `span` — built-in span fields (trace_id, span_id, duration, name, kind, status_code, is_root_span). A `duration` filter value is in milliseconds (1 second = `1000`).\n- `span_attribute` — span-level attributes (e.g. \"http.method\", \"http.status_code\")\n- `span_resource_attribute` — resource-level attributes (e.g. k8s labels, deployment info)\n\nUse `apm-attributes-list` and `apm-attribute-values-list` to discover available attribute keys/values before guessing.\n\nSupported operators:\n\n- String: `exact`, `is_not`, `icontains`, `not_icontains`, `regex`, `not_regex`\n- Numeric: `exact`, `gt`, `lt`\n- Existence (no value needed): `is_set`, `is_not_set`\n\nAn attribute filter (`span_attribute` or `span_resource_attribute`) matches only spans that have the attribute, also for negated operators such as `is_not` and `not_icontains`. To find spans without the attribute, use `is_not_set`.\n\n## Time period\n\nUse `query.dateRange` to control the time window. Default is the last hour (`-1h`). Examples: `-1h`, `-6h`, `-1d`, `-7d`.\n\n# Parameters\n\n## query.dateRange\n\nDate range for the primary window. Defaults to the last hour.\n\n- `date_from`: Start of the range. Accepts ISO 8601 timestamps or relative formats: `-1h`, `-6h`, `-1d`, `-7d`, `-30d`.\n- `date_to`: End of the range. Same format. Omit or null for \"now\".\n\n## query.compareFilter\n\nOptional comparison-window configuration. Omit when you only need the primary window.\n\n- `compare` (boolean): set to true to enable comparison.\n- `compare_to` (string, optional): relative offset for the comparison window (e.g. `-1d`, `-7d`). Defaults to the immediately previous period of equal length.\n\n## query.serviceNames\n\nList of service names to restrict the aggregation to. Use `apm-services-list` to discover available services.\n\n## query.filterGroup\n\nA flat list of property filters applied to both the primary and comparison windows. See the \"Property filters\" section.\n\n## query.limit\n\nMax rows to return, ordered by `total_duration_nano` DESC. Defaults to 100; hard max 5000. Keep this small — a high value on high-cardinality span names returns a very large response. Narrow with `serviceNames`/`filterGroup` instead of raising it when you can.\n\n## query.offset\n\nRow offset for pagination. Combine with `limit` and the `next_offset` from a previous response to page through results beyond the first page.\n\n## query.groupByEndpoint\n\nSet to `true` to get per-endpoint health for HTTP servers. Rows then group by `METHOD route` (for example `GET /users/{id}`) from the `http.route` attribute, in place of the span name.\n\n- Only HTTP server spans (span kind server with an HTTP method) are counted. Queue consumers, workflow activities and client spans are left out.\n- A span without `http.route` keeps its span name as the row key.\n- Use it when span names are raw paths (`/users/1`, `/users/2`) or a bare method (`GET`), which split or merge routes.\n\n## query.includeHealth\n\nSet to `true` to get `health_status` and `health_reason` on every row. Each row is compared with the same row in the `compareFilter` window, or in the previous period of the same length when `compareFilter` is off.\n\n- `regressed`: the error rate or p95 jumped against the baseline.\n- `degraded`: the error rate is high, with no clear jump.\n- `new`: the row has no spans in the baseline.\n- `recovered` and `improved`: the error rate or p95 went down.\n- `ok`: none of the above.\n\nIt runs a second window query for the routes on the current page. Without `compareFilter`, the response `compare` stays null. With `compareFilter`, `compare` holds the baseline rows for those same routes.\n\n# Examples\n\n## Top operations by total time in the last hour\n\n```json\n{\n \"query\": {}\n}\n```\n\n## p95 latency by operation in the last day\n\n```json\n{\n \"query\": {\n \"dateRange\": { \"date_from\": \"-1d\" }\n }\n}\n```\n\nInspect the `p95_duration_nano` field on each result.\n\n## Compare this hour vs the previous hour\n\n```json\n{\n \"query\": {\n \"compareFilter\": { \"compare\": true }\n }\n}\n```\n\nReturns `results` for the last hour and `compare` for the hour before. Diff `total_duration_nano`, `avg_duration_nano`, etc. to find regressions.\n\n## Compare today vs same time last week\n\n```json\n{\n \"query\": {\n \"dateRange\": { \"date_from\": \"-1d\" },\n \"compareFilter\": { \"compare\": true, \"compare_to\": \"-7d\" }\n }\n}\n```\n\n## Aggregate only error spans\n\n```json\n{\n \"query\": {\n \"filterGroup\": [{ \"key\": \"status_code\", \"operator\": \"exact\", \"type\": \"span\", \"value\": 2 }],\n \"dateRange\": { \"date_from\": \"-6h\" }\n }\n}\n```\n\n## Endpoint health for one service\n\n```json\n{\n \"query\": {\n \"serviceNames\": [\"api-gateway\"],\n \"groupByEndpoint\": true,\n \"includeHealth\": true,\n \"dateRange\": { \"date_from\": \"-1d\" }\n }\n}\n```\n\nEach row is one route. Start with the rows whose `health_status` is `regressed` or `degraded`, and read `health_reason` for the cause.\n\n## Aggregate within one service\n\n```json\n{\n \"query\": {\n \"serviceNames\": [\"api-gateway\"],\n \"dateRange\": { \"date_from\": \"-1d\" }\n }\n}\n```\n\n# Reminders\n\n- Duration values in results are in nanoseconds. Divide by 1,000,000 for ms, 1,000,000,000 for seconds. A `duration` filter value is in milliseconds.\n- Results are ordered by `total_duration_nano` DESC and default to the top 100 rows. Check `has_more`/`next_offset` to page; raise `limit` (max 5000) only when you truly need the long tail.\n- Use `apm-attributes-list` and `apm-attribute-values-list` to discover attribute keys/values before filtering.\n- Use `apm-services-list` to discover services before filtering by service name.\n- For parent → child breakdowns, use `apm-spans-tree` instead.",
+ "description": "Aggregate trace span statistics grouped by `(service_name, name)` over a date window. Set `groupByEndpoint` to group HTTP server spans by route, and `includeHealth` to get a `health_status` per row against a baseline.\n\nAll parameters go inside `query` — top-level fields are rejected:\n\n```json\n{ \"query\": { \"serviceNames\": [\"api\"], \"dateRange\": { \"date_from\": \"-1h\" } } }\n```\n\nReturns one row per `(service_name, name)` pair with the following metrics:\n\n- `count` — number of spans matched\n- `total_duration_nano`, `avg_duration_nano`, `p50_duration_nano`, `p95_duration_nano`, `p99_duration_nano`, `p999_duration_nano` — duration stats in nanoseconds (1 second = 1,000,000,000 ns). `p999_duration_nano` is the 99.9th percentile and is only meaningful for `(service, name)` groups with enough spans; on low-volume operations it collapses to the max\n- `error_count` — spans with OTel status code `Error` (status_code = 2)\n\nRows are ordered by `total_duration_nano` DESC. By default only the top 100 rows are returned — this keeps the response small, which matters because high span-name cardinality (e.g. untemplated URL paths) can otherwise produce an enormous payload. The top rows already surface the heaviest operations. When more rows genuinely exist the response sets `has_more: true` and `next_offset`; page through with `offset`, or raise `limit` (hard max 5000). Prefer narrowing with `serviceNames`/`filterGroup` over pulling a large page.\n\nUse to answer:\n\n- \"Which services/operations consume the most time?\"\n- \"What's the p95 or p99 latency of `GET /users`?\"\n- \"How many errors did the checkout service emit in the last day?\"\n- \"Did `POST /orders` get slower this week vs last week?\" (with `compareFilter`)\n- \"Which endpoints regressed today?\" (with `groupByEndpoint` and `includeHealth`)\n- \"Which operations got slower in this service?\" (with `includeHealth`)\n\nFor per-call-tree breakdowns (parent → child relationships), use `apm-spans-tree` instead. For time-bucketed trends (\"when did it change?\"), use `apm-spans-sparkline` instead — `compareFilter` only contrasts two static windows.\n\n# Comparison window\n\nSet `query.compareFilter.compare: true` to also fetch a comparison window. The response then includes a `compare` array of the same shape as `results`.\n\n- Omit `compare_to` (or set null) to compare against the immediately previous period of equal length (e.g. `dateRange: -1d` → compares vs the day before).\n- Set `compare_to: \"-7d\"` to compare against the window 7 days earlier (same length as the primary window).\n\nWhen `compare` is true, `compare` is always an array — empty (`[]`) if the comparison window matched no spans (for example an operation that didn't exist yet), not null. `compare` is `null` only when no comparison was requested. So an empty array is a real answer (\"nothing in the prior window\"), not a failure.\n\n# Data narrowing\n\n## Property filters\n\nUse `query.filterGroup` to narrow results to spans matching specific attributes. Only include filters that are essential to the user's question.\n\nFilter `type` values:\n\n- `span` — built-in span fields (trace_id, span_id, duration, name, kind, status_code, is_root_span). A `duration` filter value is in milliseconds (1 second = `1000`).\n- `span_attribute` — span-level attributes (e.g. \"http.method\", \"http.status_code\")\n- `span_resource_attribute` — resource-level attributes (e.g. k8s labels, deployment info)\n\nUse `apm-attributes-list` and `apm-attribute-values-list` to discover available attribute keys/values before guessing.\n\nSupported operators:\n\n- String: `exact`, `is_not`, `icontains`, `not_icontains`, `regex`, `not_regex`\n- Numeric: `exact`, `gt`, `lt`\n- Existence (no value needed): `is_set`, `is_not_set`\n\nAn attribute filter (`span_attribute` or `span_resource_attribute`) matches only spans that have the attribute, also for negated operators such as `is_not` and `not_icontains`. To find spans without the attribute, use `is_not_set`.\n\n## Time period\n\nUse `query.dateRange` to control the time window. Default is the last hour (`-1h`). Examples: `-1h`, `-6h`, `-1d`, `-7d`.\n\n# Parameters\n\n## query.dateRange\n\nDate range for the primary window. Defaults to the last hour.\n\n- `date_from`: Start of the range. Accepts ISO 8601 timestamps or relative formats: `-1h`, `-6h`, `-1d`, `-7d`, `-30d`.\n- `date_to`: End of the range. Same format. Omit or null for \"now\".\n\n## query.compareFilter\n\nOptional comparison-window configuration. Omit when you only need the primary window.\n\n- `compare` (boolean): set to true to enable comparison.\n- `compare_to` (string, optional): relative offset for the comparison window (e.g. `-1d`, `-7d`). Defaults to the immediately previous period of equal length.\n\n## query.serviceNames\n\nList of service names to restrict the aggregation to. Use `apm-services-list` to discover available services.\n\n## query.filterGroup\n\nA flat list of property filters applied to both the primary and comparison windows. See the \"Property filters\" section.\n\n## query.limit\n\nMax rows to return, ordered by `total_duration_nano` DESC. Defaults to 100; hard max 5000. Keep this small — a high value on high-cardinality span names returns a very large response. Narrow with `serviceNames`/`filterGroup` instead of raising it when you can.\n\n## query.offset\n\nRow offset for pagination. Combine with `limit` and the `next_offset` from a previous response to page through results beyond the first page.\n\n## query.groupByEndpoint\n\nSet to `true` to get per-endpoint health for HTTP servers. Rows then group by `METHOD route` (for example `GET /users/{id}`) from the `http.route` attribute, in place of the span name.\n\n- Only HTTP server spans (span kind server with an HTTP method) are counted. Queue consumers, workflow activities and client spans are left out.\n- A span without `http.route` keeps its span name as the row key.\n- Use it when span names are raw paths (`/users/1`, `/users/2`) or a bare method (`GET`), which split or merge routes.\n\n## query.includeHealth\n\nSet to `true` to get `health_status` and `health_reason` on every row. Each row is compared with the same row in the `compareFilter` window, or in the previous period of the same length when `compareFilter` is off.\n\n- `regressed`: the error rate or p95 jumped against the baseline.\n- `degraded`: the error rate is high, with no clear jump.\n- `new`: the row has no spans in the baseline.\n- `recovered` and `improved`: the error rate or p95 went down.\n- `ok`: none of the above.\n\nIt runs a second window query for the routes on the current page. Without `compareFilter`, the response `compare` stays null. With `compareFilter`, `compare` holds the baseline rows for those same routes.\n\n# Examples\n\n## Top operations by total time in the last hour\n\n```json\n{\n \"query\": {}\n}\n```\n\n## p95 latency by operation in the last day\n\n```json\n{\n \"query\": {\n \"dateRange\": { \"date_from\": \"-1d\" }\n }\n}\n```\n\nInspect the `p95_duration_nano` field on each result.\n\n## Compare this hour vs the previous hour\n\n```json\n{\n \"query\": {\n \"compareFilter\": { \"compare\": true }\n }\n}\n```\n\nReturns `results` for the last hour and `compare` for the hour before. Diff `total_duration_nano`, `avg_duration_nano`, etc. to find regressions.\n\n## Compare today vs same time last week\n\n```json\n{\n \"query\": {\n \"dateRange\": { \"date_from\": \"-1d\" },\n \"compareFilter\": { \"compare\": true, \"compare_to\": \"-7d\" }\n }\n}\n```\n\n## Aggregate only error spans\n\n```json\n{\n \"query\": {\n \"filterGroup\": [{ \"key\": \"status_code\", \"operator\": \"exact\", \"type\": \"span\", \"value\": 2 }],\n \"dateRange\": { \"date_from\": \"-6h\" }\n }\n}\n```\n\n## Endpoint health for one service\n\n```json\n{\n \"query\": {\n \"serviceNames\": [\"api-gateway\"],\n \"groupByEndpoint\": true,\n \"includeHealth\": true,\n \"dateRange\": { \"date_from\": \"-1d\" }\n }\n}\n```\n\nEach row is one route. Start with the rows whose `health_status` is `regressed` or `degraded`, and read `health_reason` for the cause.\n\n## Aggregate within one service\n\n```json\n{\n \"query\": {\n \"serviceNames\": [\"api-gateway\"],\n \"dateRange\": { \"date_from\": \"-1d\" }\n }\n}\n```\n\n# Reminders\n\n- Duration values in results are in nanoseconds. Divide by 1,000,000 for ms, 1,000,000,000 for seconds. A `duration` filter value is in milliseconds.\n- Results are ordered by `total_duration_nano` DESC and default to the top 100 rows. Check `has_more`/`next_offset` to page; raise `limit` (max 5000) only when you truly need the long tail.\n- Use `apm-attributes-list` and `apm-attribute-values-list` to discover attribute keys/values before filtering.\n- Use `apm-services-list` to discover services before filtering by service name.\n- For parent → child breakdowns, use `apm-spans-tree` instead.",
"category": "Tracing",
"feature": "tracing",
"summary": "Aggregate span statistics",
diff --git a/services/mcp/schema/tool-definitions-all.json b/services/mcp/schema/tool-definitions-all.json
index 58ef91ae39da..e23bd84eedc2 100644
--- a/services/mcp/schema/tool-definitions-all.json
+++ b/services/mcp/schema/tool-definitions-all.json
@@ -1257,7 +1257,7 @@
"feature_flag": "tracing"
},
"apm-spans-aggregate": {
- "description": "Aggregate trace span statistics grouped by `(service_name, name)` over a date window.\n\nAll parameters go inside `query` — top-level fields are rejected:\n\n```json\n{ \"query\": { \"serviceNames\": [\"api\"], \"dateRange\": { \"date_from\": \"-1h\" } } }\n```\n\nReturns one row per `(service_name, name)` pair with the following metrics:\n\n- `count` — number of spans matched\n- `total_duration_nano`, `avg_duration_nano`, `p50_duration_nano`, `p95_duration_nano`, `p99_duration_nano`, `p999_duration_nano` — duration stats in nanoseconds (1 second = 1,000,000,000 ns). `p999_duration_nano` is the 99.9th percentile and is only meaningful for `(service, name)` groups with enough spans; on low-volume operations it collapses to the max\n- `error_count` — spans with OTel status code `Error` (status_code = 2)\n\nRows are ordered by `total_duration_nano` DESC. By default only the top 100 rows are returned — this keeps the response small, which matters because high span-name cardinality (e.g. untemplated URL paths) can otherwise produce an enormous payload. The top rows already surface the heaviest operations. When more rows genuinely exist the response sets `has_more: true` and `next_offset`; page through with `offset`, or raise `limit` (hard max 5000). Prefer narrowing with `serviceNames`/`filterGroup` over pulling a large page.\n\nUse to answer:\n\n- \"Which services/operations consume the most time?\"\n- \"What's the p95 or p99 latency of `GET /users`?\"\n- \"How many errors did the checkout service emit in the last day?\"\n- \"Did `POST /orders` get slower this week vs last week?\" (with `compareFilter`)\n\nFor per-call-tree breakdowns (parent → child relationships), use `apm-spans-tree` instead. For time-bucketed trends (\"when did it change?\"), use `apm-spans-sparkline` instead — `compareFilter` only contrasts two static windows.\n\n# Comparison window\n\nSet `query.compareFilter.compare: true` to also fetch a comparison window. The response then includes a `compare` array of the same shape as `results`.\n\n- Omit `compare_to` (or set null) to compare against the immediately previous period of equal length (e.g. `dateRange: -1d` → compares vs the day before).\n- Set `compare_to: \"-7d\"` to compare against the window 7 days earlier (same length as the primary window).\n\nWhen `compare` is true, `compare` is always an array — empty (`[]`) if the comparison window matched no spans (for example an operation that didn't exist yet), not null. `compare` is `null` only when no comparison was requested. So an empty array is a real answer (\"nothing in the prior window\"), not a failure.\n\n# Data narrowing\n\n## Property filters\n\nUse `query.filterGroup` to narrow results to spans matching specific attributes. Only include filters that are essential to the user's question.\n\nFilter `type` values:\n\n- `span` — built-in span fields (trace_id, span_id, duration, name, kind, status_code, is_root_span). A `duration` filter value is in milliseconds (1 second = `1000`).\n- `span_attribute` — span-level attributes (e.g. \"http.method\", \"http.status_code\")\n- `span_resource_attribute` — resource-level attributes (e.g. k8s labels, deployment info)\n\nUse `apm-attributes-list` and `apm-attribute-values-list` to discover available attribute keys/values before guessing.\n\nSupported operators:\n\n- String: `exact`, `is_not`, `icontains`, `not_icontains`, `regex`, `not_regex`\n- Numeric: `exact`, `gt`, `lt`\n- Existence (no value needed): `is_set`, `is_not_set`\n\nAn attribute filter (`span_attribute` or `span_resource_attribute`) matches only spans that have the attribute, also for negated operators such as `is_not` and `not_icontains`. To find spans without the attribute, use `is_not_set`.\n\n## Time period\n\nUse `query.dateRange` to control the time window. Default is the last hour (`-1h`). Examples: `-1h`, `-6h`, `-1d`, `-7d`.\n\n# Parameters\n\n## query.dateRange\n\nDate range for the primary window. Defaults to the last hour.\n\n- `date_from`: Start of the range. Accepts ISO 8601 timestamps or relative formats: `-1h`, `-6h`, `-1d`, `-7d`, `-30d`.\n- `date_to`: End of the range. Same format. Omit or null for \"now\".\n\n## query.compareFilter\n\nOptional comparison-window configuration. Omit when you only need the primary window.\n\n- `compare` (boolean): set to true to enable comparison.\n- `compare_to` (string, optional): relative offset for the comparison window (e.g. `-1d`, `-7d`). Defaults to the immediately previous period of equal length.\n\n## query.serviceNames\n\nList of service names to restrict the aggregation to. Use `apm-services-list` to discover available services.\n\n## query.filterGroup\n\nA flat list of property filters applied to both the primary and comparison windows. See the \"Property filters\" section.\n\n## query.limit\n\nMax rows to return, ordered by `total_duration_nano` DESC. Defaults to 100; hard max 5000. Keep this small — a high value on high-cardinality span names returns a very large response. Narrow with `serviceNames`/`filterGroup` instead of raising it when you can.\n\n## query.offset\n\nRow offset for pagination. Combine with `limit` and the `next_offset` from a previous response to page through results beyond the first page.\n\n## query.groupByEndpoint\n\nSet to `true` to get per-endpoint health for HTTP servers. Rows then group by `METHOD route` (for example `GET /users/{id}`) from the `http.route` attribute, in place of the span name.\n\n- Only HTTP server spans (span kind server with an HTTP method) are counted. Queue consumers, workflow activities and client spans are left out.\n- A span without `http.route` keeps its span name as the row key.\n- Use it when span names are raw paths (`/users/1`, `/users/2`) or a bare method (`GET`), which split or merge routes.\n\n## query.includeHealth\n\nSet to `true` to get `health_status` and `health_reason` on every row. Each row is compared with the same row in the `compareFilter` window, or in the previous period of the same length when `compareFilter` is off.\n\n- `regressed`: the error rate or p95 jumped against the baseline.\n- `degraded`: the error rate is high, with no clear jump.\n- `new`: the row has no spans in the baseline.\n- `recovered` and `improved`: the error rate or p95 went down.\n- `ok`: none of the above.\n\nIt runs a second window query for the routes on the current page. Without `compareFilter`, the response `compare` stays null. With `compareFilter`, `compare` holds the baseline rows for those same routes.\n\n# Examples\n\n## Top operations by total time in the last hour\n\n```json\n{\n \"query\": {}\n}\n```\n\n## p95 latency by operation in the last day\n\n```json\n{\n \"query\": {\n \"dateRange\": { \"date_from\": \"-1d\" }\n }\n}\n```\n\nInspect the `p95_duration_nano` field on each result.\n\n## Compare this hour vs the previous hour\n\n```json\n{\n \"query\": {\n \"compareFilter\": { \"compare\": true }\n }\n}\n```\n\nReturns `results` for the last hour and `compare` for the hour before. Diff `total_duration_nano`, `avg_duration_nano`, etc. to find regressions.\n\n## Compare today vs same time last week\n\n```json\n{\n \"query\": {\n \"dateRange\": { \"date_from\": \"-1d\" },\n \"compareFilter\": { \"compare\": true, \"compare_to\": \"-7d\" }\n }\n}\n```\n\n## Aggregate only error spans\n\n```json\n{\n \"query\": {\n \"filterGroup\": [{ \"key\": \"status_code\", \"operator\": \"exact\", \"type\": \"span\", \"value\": 2 }],\n \"dateRange\": { \"date_from\": \"-6h\" }\n }\n}\n```\n\n## Endpoint health for one service\n\n```json\n{\n \"query\": {\n \"serviceNames\": [\"api-gateway\"],\n \"groupByEndpoint\": true,\n \"includeHealth\": true,\n \"dateRange\": { \"date_from\": \"-1d\" }\n }\n}\n```\n\nEach row is one route. Start with the rows whose `health_status` is `regressed` or `degraded`, and read `health_reason` for the cause.\n\n## Aggregate within one service\n\n```json\n{\n \"query\": {\n \"serviceNames\": [\"api-gateway\"],\n \"dateRange\": { \"date_from\": \"-1d\" }\n }\n}\n```\n\n# Reminders\n\n- Duration values in results are in nanoseconds. Divide by 1,000,000 for ms, 1,000,000,000 for seconds. A `duration` filter value is in milliseconds.\n- Results are ordered by `total_duration_nano` DESC and default to the top 100 rows. Check `has_more`/`next_offset` to page; raise `limit` (max 5000) only when you truly need the long tail.\n- Use `apm-attributes-list` and `apm-attribute-values-list` to discover attribute keys/values before filtering.\n- Use `apm-services-list` to discover services before filtering by service name.\n- For parent → child breakdowns, use `apm-spans-tree` instead.",
+ "description": "Aggregate trace span statistics grouped by `(service_name, name)` over a date window. Set `groupByEndpoint` to group HTTP server spans by route, and `includeHealth` to get a `health_status` per row against a baseline.\n\nAll parameters go inside `query` — top-level fields are rejected:\n\n```json\n{ \"query\": { \"serviceNames\": [\"api\"], \"dateRange\": { \"date_from\": \"-1h\" } } }\n```\n\nReturns one row per `(service_name, name)` pair with the following metrics:\n\n- `count` — number of spans matched\n- `total_duration_nano`, `avg_duration_nano`, `p50_duration_nano`, `p95_duration_nano`, `p99_duration_nano`, `p999_duration_nano` — duration stats in nanoseconds (1 second = 1,000,000,000 ns). `p999_duration_nano` is the 99.9th percentile and is only meaningful for `(service, name)` groups with enough spans; on low-volume operations it collapses to the max\n- `error_count` — spans with OTel status code `Error` (status_code = 2)\n\nRows are ordered by `total_duration_nano` DESC. By default only the top 100 rows are returned — this keeps the response small, which matters because high span-name cardinality (e.g. untemplated URL paths) can otherwise produce an enormous payload. The top rows already surface the heaviest operations. When more rows genuinely exist the response sets `has_more: true` and `next_offset`; page through with `offset`, or raise `limit` (hard max 5000). Prefer narrowing with `serviceNames`/`filterGroup` over pulling a large page.\n\nUse to answer:\n\n- \"Which services/operations consume the most time?\"\n- \"What's the p95 or p99 latency of `GET /users`?\"\n- \"How many errors did the checkout service emit in the last day?\"\n- \"Did `POST /orders` get slower this week vs last week?\" (with `compareFilter`)\n- \"Which endpoints regressed today?\" (with `groupByEndpoint` and `includeHealth`)\n- \"Which operations got slower in this service?\" (with `includeHealth`)\n\nFor per-call-tree breakdowns (parent → child relationships), use `apm-spans-tree` instead. For time-bucketed trends (\"when did it change?\"), use `apm-spans-sparkline` instead — `compareFilter` only contrasts two static windows.\n\n# Comparison window\n\nSet `query.compareFilter.compare: true` to also fetch a comparison window. The response then includes a `compare` array of the same shape as `results`.\n\n- Omit `compare_to` (or set null) to compare against the immediately previous period of equal length (e.g. `dateRange: -1d` → compares vs the day before).\n- Set `compare_to: \"-7d\"` to compare against the window 7 days earlier (same length as the primary window).\n\nWhen `compare` is true, `compare` is always an array — empty (`[]`) if the comparison window matched no spans (for example an operation that didn't exist yet), not null. `compare` is `null` only when no comparison was requested. So an empty array is a real answer (\"nothing in the prior window\"), not a failure.\n\n# Data narrowing\n\n## Property filters\n\nUse `query.filterGroup` to narrow results to spans matching specific attributes. Only include filters that are essential to the user's question.\n\nFilter `type` values:\n\n- `span` — built-in span fields (trace_id, span_id, duration, name, kind, status_code, is_root_span). A `duration` filter value is in milliseconds (1 second = `1000`).\n- `span_attribute` — span-level attributes (e.g. \"http.method\", \"http.status_code\")\n- `span_resource_attribute` — resource-level attributes (e.g. k8s labels, deployment info)\n\nUse `apm-attributes-list` and `apm-attribute-values-list` to discover available attribute keys/values before guessing.\n\nSupported operators:\n\n- String: `exact`, `is_not`, `icontains`, `not_icontains`, `regex`, `not_regex`\n- Numeric: `exact`, `gt`, `lt`\n- Existence (no value needed): `is_set`, `is_not_set`\n\nAn attribute filter (`span_attribute` or `span_resource_attribute`) matches only spans that have the attribute, also for negated operators such as `is_not` and `not_icontains`. To find spans without the attribute, use `is_not_set`.\n\n## Time period\n\nUse `query.dateRange` to control the time window. Default is the last hour (`-1h`). Examples: `-1h`, `-6h`, `-1d`, `-7d`.\n\n# Parameters\n\n## query.dateRange\n\nDate range for the primary window. Defaults to the last hour.\n\n- `date_from`: Start of the range. Accepts ISO 8601 timestamps or relative formats: `-1h`, `-6h`, `-1d`, `-7d`, `-30d`.\n- `date_to`: End of the range. Same format. Omit or null for \"now\".\n\n## query.compareFilter\n\nOptional comparison-window configuration. Omit when you only need the primary window.\n\n- `compare` (boolean): set to true to enable comparison.\n- `compare_to` (string, optional): relative offset for the comparison window (e.g. `-1d`, `-7d`). Defaults to the immediately previous period of equal length.\n\n## query.serviceNames\n\nList of service names to restrict the aggregation to. Use `apm-services-list` to discover available services.\n\n## query.filterGroup\n\nA flat list of property filters applied to both the primary and comparison windows. See the \"Property filters\" section.\n\n## query.limit\n\nMax rows to return, ordered by `total_duration_nano` DESC. Defaults to 100; hard max 5000. Keep this small — a high value on high-cardinality span names returns a very large response. Narrow with `serviceNames`/`filterGroup` instead of raising it when you can.\n\n## query.offset\n\nRow offset for pagination. Combine with `limit` and the `next_offset` from a previous response to page through results beyond the first page.\n\n## query.groupByEndpoint\n\nSet to `true` to get per-endpoint health for HTTP servers. Rows then group by `METHOD route` (for example `GET /users/{id}`) from the `http.route` attribute, in place of the span name.\n\n- Only HTTP server spans (span kind server with an HTTP method) are counted. Queue consumers, workflow activities and client spans are left out.\n- A span without `http.route` keeps its span name as the row key.\n- Use it when span names are raw paths (`/users/1`, `/users/2`) or a bare method (`GET`), which split or merge routes.\n\n## query.includeHealth\n\nSet to `true` to get `health_status` and `health_reason` on every row. Each row is compared with the same row in the `compareFilter` window, or in the previous period of the same length when `compareFilter` is off.\n\n- `regressed`: the error rate or p95 jumped against the baseline.\n- `degraded`: the error rate is high, with no clear jump.\n- `new`: the row has no spans in the baseline.\n- `recovered` and `improved`: the error rate or p95 went down.\n- `ok`: none of the above.\n\nIt runs a second window query for the routes on the current page. Without `compareFilter`, the response `compare` stays null. With `compareFilter`, `compare` holds the baseline rows for those same routes.\n\n# Examples\n\n## Top operations by total time in the last hour\n\n```json\n{\n \"query\": {}\n}\n```\n\n## p95 latency by operation in the last day\n\n```json\n{\n \"query\": {\n \"dateRange\": { \"date_from\": \"-1d\" }\n }\n}\n```\n\nInspect the `p95_duration_nano` field on each result.\n\n## Compare this hour vs the previous hour\n\n```json\n{\n \"query\": {\n \"compareFilter\": { \"compare\": true }\n }\n}\n```\n\nReturns `results` for the last hour and `compare` for the hour before. Diff `total_duration_nano`, `avg_duration_nano`, etc. to find regressions.\n\n## Compare today vs same time last week\n\n```json\n{\n \"query\": {\n \"dateRange\": { \"date_from\": \"-1d\" },\n \"compareFilter\": { \"compare\": true, \"compare_to\": \"-7d\" }\n }\n}\n```\n\n## Aggregate only error spans\n\n```json\n{\n \"query\": {\n \"filterGroup\": [{ \"key\": \"status_code\", \"operator\": \"exact\", \"type\": \"span\", \"value\": 2 }],\n \"dateRange\": { \"date_from\": \"-6h\" }\n }\n}\n```\n\n## Endpoint health for one service\n\n```json\n{\n \"query\": {\n \"serviceNames\": [\"api-gateway\"],\n \"groupByEndpoint\": true,\n \"includeHealth\": true,\n \"dateRange\": { \"date_from\": \"-1d\" }\n }\n}\n```\n\nEach row is one route. Start with the rows whose `health_status` is `regressed` or `degraded`, and read `health_reason` for the cause.\n\n## Aggregate within one service\n\n```json\n{\n \"query\": {\n \"serviceNames\": [\"api-gateway\"],\n \"dateRange\": { \"date_from\": \"-1d\" }\n }\n}\n```\n\n# Reminders\n\n- Duration values in results are in nanoseconds. Divide by 1,000,000 for ms, 1,000,000,000 for seconds. A `duration` filter value is in milliseconds.\n- Results are ordered by `total_duration_nano` DESC and default to the top 100 rows. Check `has_more`/`next_offset` to page; raise `limit` (max 5000) only when you truly need the long tail.\n- Use `apm-attributes-list` and `apm-attribute-values-list` to discover attribute keys/values before filtering.\n- Use `apm-services-list` to discover services before filtering by service name.\n- For parent → child breakdowns, use `apm-spans-tree` instead.",
"category": "Tracing",
"feature": "tracing",
"summary": "Aggregate span statistics",