Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(tracing): add apm-service-health and reject malformed GET date ranges (#119975)

Daniel Visca86a32f059c2f
diff --git a/posthog/test/repo_invariants/signals_scout_tool_citations_baseline.txt b/posthog/test/repo_invariants/signals_scout_tool_citations_baseline.txt
index e4250b41769c..8488a6ce2b28 100644
--- a/posthog/test/repo_invariants/signals_scout_tool_citations_baseline.txt
+++ b/posthog/test/repo_invariants/signals_scout_tool_citations_baseline.txt
@@ -1,6 +1,7 @@
agent-feedback
alert-simulate
apm-attribute-breakdown
+apm-service-health
apm-spans-aggregate
apm-spans-duration-histogram
apm-spans-sparkline
diff --git a/products/signals/skills/signals-scout-apm/SKILL.md b/products/signals/skills/signals-scout-apm/SKILL.md
index 311576987dba..bb84908ccbfc 100644
--- a/products/signals/skills/signals-scout-apm/SKILL.md
+++ b/products/signals/skills/signals-scout-apm/SKILL.md
@@ -7,8 +7,8 @@ description: >
compatibility: >
PostHog Signals agent (Claude sandbox). Read-only analytics + signal_scout_internal:write
(scratchpad) + signal_scout_report:write (report channel), plus the apm-* tool family
- (query-apm-spans, apm-trace-get, apm-spans-aggregate, apm-spans-tree, apm-spans-count,
- apm-spans-sparkline, apm-spans-duration-histogram, apm-attribute-breakdown,
+ (apm-service-health, query-apm-spans, apm-trace-get, apm-spans-aggregate, apm-spans-tree,
+ apm-spans-count, apm-spans-sparkline, apm-spans-duration-histogram, apm-attribute-breakdown,
apm-services-list, apm-attributes-list, apm-attribute-values-list) and the bundled
exploring-apm-traces deep-dive skill.
allowed_tools:
@@ -60,30 +60,28 @@ Three cheap reads cold-start a run:
- `scout-scratchpad-search` (`text=apm`) — durable steering from past APM runs. **Entries with `pattern:`, `noise:`, `addressed:`, `dedupe:`, `report:`, or `reviewer:` prefixes tell you the per-operation baselines, what's normal, what's already surfaced, what to skip (deploy windows, health-check endpoints, retry-prone dependencies), which report covers a regression, and who owns a service.**
- `scout-runs-list` (last 7d) — what prior APM runs found and ruled out. Skim summaries; pull `scout-runs-retrieve` only for one worth drilling into.
-- `apm-services-list` — the live service inventory. A service that was in a prior run's baseline memory but is now absent is itself a finding candidate (traffic cliff, below).
+- `apm-service-health` — the live service inventory with a status per service (the discriminator engine, below). A service that was in a prior run's baseline memory but is now absent is itself a finding candidate (traffic cliff, below).
- `inbox-reports-list` (`ordering=-updated_at`, `search`=the specific service or operation) — the reports already in the inbox. Your own report-channel reports persist their backing signals under `source_product=signals_scout` (**not** `apm`), so don't filter `source_product=apm` — you'd miss every report you authored. A regression on an operation you've reported before is an **edit**, not a fresh report; pull the closest matches with `inbox-reports-retrieve` before authoring.
### The discriminator engine
-One call gives you the seasonality-matched baseline for every operation:
+One call scores every service against the same window one week ago:
```json
-apm-spans-aggregate
+apm-service-health
{
"query": {
"dateRange": { "date_from": "-1d" },
- "compareFilter": { "compare": true, "compare_to": "-7d" }
+ "compare": "previous_week"
}
}
```
-`results` is the last 24h, `compare` is the same 24h one week ago — both as one row per `(service_name, name)` with `count`, `error_count`, `p50_duration_nano`, `p95_duration_nano`. Join the two arrays on `(service_name, name)` and compute, per operation:
+Each `services` row has `requests` and `baseline_requests` (the denominator guard), `error_rate_pct` and `error_rate_delta_pp`, `p95_ms` and `p95_change_pct`, and a `status` with a `reason`. The server computes the service percentiles, so do not rebuild them from operation rows. Rows come worst first: `silent`, `regressed`, `degraded`. The `next` calls open the operations behind each regressed or degraded service. Exclude dev and staging with `environment` when the project labels it.
-- **error rate** `error_count / count`, now vs 7d-ago
-- **p95 latency** `p95_duration_nano`, now vs 7d-ago
-- **request volume** `count`, now vs 7d-ago (the denominator guard)
+The server does not apply the volume guard. For each `regressed` row, check that `requests` stayed within ~2× of `baseline_requests` before you treat it as a regression.
-A busy service returns hundreds of operations (the payload runs to 100KB+ and the harness persists it to a file) — **process it programmatically, don't eyeball it.** Sort operations by delta and keep only those where the rate moved but `count` stayed within ~2× (the guard that separates a real regression from a volume swing); a low-`count` operation has too small a sample for a stable percentile (see disqualifiers). Scope to a few services per run rather than pulling the whole project at once.
+Then drill into the operations of the regressed services only: run the `apm-spans-aggregate` call in the response's `next` (it is already scoped to those services, with `compareFilter` on the same baseline). With `environment` set, `next` is empty, so build that call yourself. Do not pull the whole project per operation and join the arrays — a busy project returns 100KB+ that you must then process by hand.
### Profile shape
@@ -94,7 +92,7 @@ A busy service returns hundreds of operations (the payload runs to 100KB+ and th
| `p95` up materially, `count` ~flat | latency regression — investigate |
| `p95` up **and** `count` up sharply | saturation under load — investigate, lower confidence |
| new `(service, name)` erroring, no 7d-ago row | new code path / recent deploy — investigate |
-| service in baseline memory, now ~0 spans | traffic cliff (instrumentation break or outage) — investigate |
+| service `status: silent` | traffic cliff (instrumentation break or outage) — investigate |
Always score the **latest complete** bucket/window — a partial current hour always reads as a drop in volume and a dip in p95.
@@ -104,7 +102,7 @@ Patterns to watch — starting points, not a checklist.
#### Error-rate regression
-From the discriminator engine, find operations where error rate stepped up materially while `count` held roughly steady. Confirm _when_ it started: `apm-spans-sparkline` with your service/operation filters for total counts, then the same call with `statusCodes: [2]` for error counts — error rate per bucket = errors / total; the bucket where the ratio jumps is the onset. Pull a representative failing trace: `query-apm-spans` with a `status_code = 2` filter and `orderBy: "duration"`, grab a `trace_id`, then `apm-trace-get` and read `exception.type` / `exception.message` straight off the error span's `attributes` map. Walk `parent_span_id` up to see the request path that led there. **`query-apm-spans` defaults to root spans only** (`rootSpans: true`), so when the regressed operation is a child span (a DB or `Client` call), set `flatSpans: true` (and `rootSpans: false`) or the `status_code = 2` + operation filter matches nothing — the aggregate flags the regression but you can never pull a sample to confirm it.
+From the discriminator engine, take the `regressed` services whose `reason` names the error rate, and find the operations in them where the error rate stepped up while `count` held roughly steady. Confirm _when_ it started: `apm-spans-sparkline` with your service/operation filters for total counts, then the same call with `statusCodes: [2]` for error counts — error rate per bucket = errors / total; the bucket where the ratio jumps is the onset. Pull a representative failing trace: `query-apm-spans` with a `status_code = 2` filter and `orderBy: "duration"`, grab a `trace_id`, then `apm-trace-get` and read `exception.type` / `exception.message` straight off the error span's `attributes` map. Walk `parent_span_id` up to see the request path that led there. **`query-apm-spans` defaults to root spans only** (`rootSpans: true`), so when the regressed operation is a child span (a DB or `Client` call), set `flatSpans: true` (and `rootSpans: false`) or the `status_code = 2` + operation filter matches nothing — the aggregate flags the regression but you can never pull a sample to confirm it.
#### Latency p95 regression
@@ -118,7 +116,7 @@ An operation (or a downstream `Client`-kind span calling another service) newly
#### Service traffic cliff
-Compare `apm-services-list` and per-service `apm-spans-sparkline` against baseline memory: a service that emitted a steady span volume and dropped to ~0 is an instrumentation break or an outage (the trace-side analog of a capture cliff — spans are not retroactive). Guard against reading a partial current bucket as a cliff: confirm the drop spans ≥2 complete buckets.
+A `silent` row in `apm-service-health` is a service that had a steady span volume and stopped: an instrumentation break or an outage (the trace-side analog of a capture cliff — spans are not retroactive). Its `reason` names the last span. The server calls a service silent only when its spans stopped for at least 10 minutes (two 5-minute buckets) where its own rate predicted more, so one quiet partial bucket does not count. Date the drop with `apm-spans-sparkline` on that service. Also check baseline memory for services absent from both windows.
### Save memory as you go
@@ -160,7 +158,7 @@ When in doubt, write memory instead of filing a report.
## MCP tools
-Direct (read-only): `apm-services-list`, `apm-spans-aggregate`, `apm-spans-sparkline`, `apm-spans-tree`, `apm-spans-duration-histogram`, `apm-attribute-breakdown`, `apm-attributes-list`, `apm-attribute-values-list`, `apm-spans-count`, `query-apm-spans`, `apm-trace-get`.
+Direct (read-only): `apm-service-health`, `apm-services-list`, `apm-spans-aggregate`, `apm-spans-sparkline`, `apm-spans-tree`, `apm-spans-duration-histogram`, `apm-attribute-breakdown`, `apm-attributes-list`, `apm-attribute-values-list`, `apm-spans-count`, `query-apm-spans`, `apm-trace-get`.
Inbox & reviewer routing (mechanics in `authoring-scouts` → `references/report-contract.md`):
diff --git a/products/tracing/backend/facade/api.py b/products/tracing/backend/facade/api.py
index 37ab68d2f872..ccadbc234396 100644
--- a/products/tracing/backend/facade/api.py
+++ b/products/tracing/backend/facade/api.py
@@ -17,7 +17,7 @@ Do NOT:
"""
from datetime import datetime
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Any
from posthog.schema import (
CachedTraceSpansAttributeBreakdownQueryResponse,
@@ -58,8 +58,14 @@ from products.tracing.backend.latency_heatmap_query_runner import (
run_latency_heatmap_query as _run_latency_heatmap_query,
)
from products.tracing.backend.self_time import annotate_self_time as _annotate_self_time
+from products.tracing.backend.service_health_query_runner import (
+ run_service_health_query as _run_service_health_query,
+ service_health_byte_cap_message as _service_health_byte_cap_message,
+)
from products.tracing.backend.symbol_stats_query_runner import run_symbol_stats_query as _run_symbol_stats_query
+from .contracts import ServiceHealthParams
+
if TYPE_CHECKING:
from posthog.models import Team, User
@@ -258,3 +264,11 @@ def fetch_trace_ai_events(*, team: "Team", user: "User | None", trace_id: str) -
the returned columns.
"""
return _fetch_trace_ai_events(team=team, user=user, trace_id=trace_id)
+
+
+def run_service_health_query(*, team: "Team", params: ServiceHealthParams) -> dict[str, Any]:
+ return _run_service_health_query(team=team, params=params)
+
+
+def service_health_byte_cap_message(*, team: "Team", params: ServiceHealthParams) -> str:
+ return _service_health_byte_cap_message(team=team, params=params)
diff --git a/products/tracing/backend/facade/contracts.py b/products/tracing/backend/facade/contracts.py
index 7a482010b208..3dcfedc0dc09 100644
--- a/products/tracing/backend/facade/contracts.py
+++ b/products/tracing/backend/facade/contracts.py
@@ -11,3 +11,21 @@ Characteristics:
Do NOT depend on Django models, DRF serializers, or request objects.
"""
+
+from posthog.dataclasses import frozen
+
+from products.tracing.backend.facade import enums
+
+
+@frozen
+class ServiceHealthParams:
+ date_from: str = "-1h"
+ date_to: str | None = None
+ compare: enums.ServiceHealthCompare = enums.ServiceHealthCompare.PREVIOUS_PERIOD
+ compare_date_from: str | None = None
+ compare_date_to: str | None = None
+ environment: str | None = None
+ service_names: tuple[str, ...] = ()
+ span_scope: enums.ServiceHealthSpanScope = enums.ServiceHealthSpanScope.ENTRY
+ order_by: enums.ServiceHealthOrderBy = enums.ServiceHealthOrderBy.STATUS
+ limit: int = 25
diff --git a/products/tracing/backend/facade/enums.py b/products/tracing/backend/facade/enums.py
index dc25d9a5eebe..4417d30849f0 100644
--- a/products/tracing/backend/facade/enums.py
+++ b/products/tracing/backend/facade/enums.py
@@ -11,3 +11,43 @@ products need to interpret contract objects also belong here.
Internal-only constants (DB magic values, feature flags, etc.)
should stay in the implementation (logic.py, models.py).
"""
+
+from posthog.enums import LabeledStrEnum
+
+
+class ServiceHealthCompare(LabeledStrEnum):
+ NONE = "none"
+ PREVIOUS_PERIOD = "previous_period"
+ PREVIOUS_DAY = "previous_day"
+ PREVIOUS_WEEK = "previous_week"
+ CUSTOM = "custom"
+
+
+class ServiceHealthBaseline(LabeledStrEnum):
+ NONE = "none"
+ PREVIOUS_PERIOD = "previous_period"
+ PREVIOUS_DAY = "previous_day"
+ PREVIOUS_WEEK = "previous_week"
+
+
+class ServiceHealthSpanScope(LabeledStrEnum):
+ ENTRY = "entry"
+ ALL = "all"
+
+
+class ServiceHealthOrderBy(LabeledStrEnum):
+ STATUS = "status"
+ ERROR_RATE = "error_rate"
+ P95 = "p95"
+ THROUGHPUT = "throughput"
+ ERROR_COUNT = "error_count"
+
+
+class ServiceHealthStatus(LabeledStrEnum):
+ SILENT = "silent"
+ REGRESSED = "regressed"
+ DEGRADED = "degraded"
+ NEW = "new"
+ RECOVERED = "recovered"
+ IMPROVED = "improved"
+ OK = "ok"
diff --git a/products/tracing/backend/presentation/date_window.py b/products/tracing/backend/presentation/date_window.py
index 4d731af56c63..637e341611ba 100644
--- a/products/tracing/backend/presentation/date_window.py
+++ b/products/tracing/backend/presentation/date_window.py
@@ -17,10 +17,13 @@ previously turned a typo'd window into stale data with no error.
"""
import re
+from zoneinfo import ZoneInfo
from dateutil import parser as dateutil_parser
from rest_framework import serializers
+from posthog.utils import relative_date_parse
+
# Friendly tracing window: an integer with a single short unit, e.g. "-30m", "90s", "-2w".
# Case-insensitive; the leading minus is optional because windows are always in the past.
_TRACING_WINDOW_RE = re.compile(r"^-?(?P<number>\d+)(?P<unit>[smhdw])$", re.IGNORECASE)
@@ -37,6 +40,14 @@ _UNIT_TO_RELATIVE_TOKEN = {"s": "s", "m": "M", "h": "h", "d": "d", "w": "w"}
_GLOBAL_RELATIVE_RE = re.compile(r"-?(?:\d+[hdwmqysHDWMQY](?:Start|End)?|[hdwmqysHDWMQY](?:Start|End))")
+def _in_range(window: str) -> str:
+ try:
+ relative_date_parse(window, ZoneInfo("UTC"))
+ except (OverflowError, ValueError):
+ raise serializers.ValidationError(f"Date range value {window!r} is out of range.")
+ return window
+
+
def normalize_tracing_window(value: str | None) -> str | None:
"""Normalize a single ``date_from``/``date_to`` value for the tracing API.
@@ -46,6 +57,8 @@ def normalize_tracing_window(value: str | None) -> str | None:
"""
if value is None:
return None
+ if not isinstance(value, str):
+ raise serializers.ValidationError(f"Invalid date range value {value!r}: dates must be strings.")
stripped = value.strip()
if not stripped:
return None
@@ -53,19 +66,22 @@ def normalize_tracing_window(value: str | None) -> str | None:
match = _TRACING_WINDOW_RE.match(stripped)
if match:
token = _UNIT_TO_RELATIVE_TOKEN[match.group("unit").lower()]
- return f"-{match.group('number')}{token}"
+ return _in_range(f"-{match.group('number')}{token}")
# Absolute ISO 8601 timestamps pass through untouched.
try:
- dateutil_parser.isoparse(stripped)
- return stripped
+ parsed = dateutil_parser.isoparse(stripped)
except ValueError:
- pass
+ parsed = None
+ if parsed is not None:
+ if not 1970 <= parsed.year <= 9998:
+ raise serializers.ValidationError(f"Date range value {value!r} is out of range.")
+ return stripped
# Preserve the broader global relative grammar (e.g. "-1dStart", "wStart", "-3q", "-1y")
# so anything that already worked keeps working.
if _GLOBAL_RELATIVE_RE.fullmatch(stripped):
- return stripped
+ return _in_range(stripped)
raise serializers.ValidationError(
f"Invalid date range value {value!r}. Use a relative window like '-30m' (minutes), "
diff --git a/products/tracing/backend/presentation/service_health_serializers.py b/products/tracing/backend/presentation/service_health_serializers.py
new file mode 100644
index 000000000000..423a93478f8c
--- /dev/null
+++ b/products/tracing/backend/presentation/service_health_serializers.py
@@ -0,0 +1,170 @@
+import datetime as dt
+from zoneinfo import ZoneInfo
+
+from drf_spectacular.utils import extend_schema_field
+from rest_framework import serializers
+
+from posthog.utils import relative_date_parse
+
+from ..facade.contracts import ServiceHealthParams
+from ..facade.enums import (
+ ServiceHealthBaseline,
+ ServiceHealthCompare,
+ ServiceHealthOrderBy,
+ ServiceHealthSpanScope,
+ ServiceHealthStatus,
+)
+from .date_window import normalize_tracing_window
+
+UTC = ZoneInfo("UTC")
+
+
+class _ServiceHealthDateRangeSerializer(serializers.Serializer):
+ date_from = serializers.CharField(
+ required=False, allow_null=True, help_text="Relative (-30m, -6h, -7d) or ISO 8601."
+ )
+ date_to = serializers.CharField(required=False, allow_null=True, help_text='Same format. Omit or null for "now".')
+
+
+class _ServiceHealthQuerySerializer(serializers.Serializer):
+ dateRange = _ServiceHealthDateRangeSerializer(
+ required=False, help_text="The window to score. Defaults to the last hour."
+ )
+ compare = serializers.ChoiceField(
+ choices=ServiceHealthBaseline.choices,
+ default=ServiceHealthBaseline.PREVIOUS_PERIOD,
+ help_text=(
+ "The baseline. previous_period (default) is the same length just before the window. previous_day and "
+ "previous_week shift the window back 1 or 7 days. none skips it. baselineDateRange overrides it."
+ ),
+ )
+ environment = serializers.CharField(
+ required=False,
+ allow_null=True,
+ allow_blank=True,
+ help_text="Only this environment, matched without case on deployment.environment.name, then deployment.environment, then env.",
+ )
+ serviceNames = serializers.ListField(
+ child=serializers.CharField(), required=False, default=list, max_length=100, help_text="Only these services."
+ )
+ spanScope = serializers.ChoiceField(
+ choices=ServiceHealthSpanScope.choices,
+ default=ServiceHealthSpanScope.ENTRY,
+ help_text="entry (default) counts root, server and consumer spans, one per request. all counts every span.",
+ )
+ orderBy = serializers.ChoiceField(
+ choices=ServiceHealthOrderBy.choices,
+ default=ServiceHealthOrderBy.STATUS,
+ help_text="status (default) puts silent, regressed and degraded first. The others sort high to low.",
+ )
+ limit = serializers.IntegerField(default=25, min_value=1, max_value=200, help_text="Max service rows, 1-200.")
+
+
+class ServiceHealthRequestSerializer(serializers.Serializer):
+ query = _ServiceHealthQuerySerializer(help_text="The service health query. Every field is optional.")
+ baselineDateRange = _ServiceHealthDateRangeSerializer(
+ required=False, help_text="An explicit baseline window. Overrides query.compare."
+ )
+
+ def validate(self, attrs: dict) -> dict:
+ for name, window in (
+ ("dateRange", attrs["query"].get("dateRange")),
+ ("baselineDateRange", attrs.get("baselineDateRange")),
+ ):
+ if window is None:
+ continue
+ start = normalize_tracing_window(window.get("date_from")) or "-1h"
+ end = normalize_tracing_window(window.get("date_to"))
+ if relative_date_parse(start, UTC) >= (relative_date_parse(end, UTC) if end else dt.datetime.now(UTC)):
+ raise serializers.ValidationError({name: "date_from must be before date_to."})
+ return attrs
+
+
+def service_health_params(validated_data: dict) -> ServiceHealthParams:
+ data = validated_data["query"]
+ date_range = data.get("dateRange") or {}
+ baseline = validated_data.get("baselineDateRange")
+ compare = ServiceHealthCompare.CUSTOM if baseline else ServiceHealthCompare(data["compare"])
+ compare_from, compare_to = (baseline.get("date_from"), baseline.get("date_to")) if baseline else (None, None)
+ return ServiceHealthParams(
+ date_from=normalize_tracing_window(date_range.get("date_from")) or "-1h",
+ date_to=normalize_tracing_window(date_range.get("date_to")),
+ compare=compare,
+ compare_date_from=normalize_tracing_window(compare_from),
+ compare_date_to=normalize_tracing_window(compare_to),
+ environment=(data.get("environment") or "").strip() or None,
+ service_names=tuple(data["serviceNames"]),
+ span_scope=ServiceHealthSpanScope(data["spanScope"]),
+ order_by=ServiceHealthOrderBy(data["orderBy"]),
+ limit=data["limit"],
+ )
+
+
+class _ServiceHealthTimeRangeSerializer(serializers.Serializer):
+ date_from = serializers.CharField(help_text="Start, ISO 8601 UTC, inclusive.")
+ date_to = serializers.CharField(help_text="End, ISO 8601 UTC, exclusive.")
+
+
+class _ServiceHealthWindowSerializer(serializers.Serializer):
+ current = _ServiceHealthTimeRangeSerializer(
+ help_text="The scored window. Reuse it as dateRange in follow-up calls."
+ )
+ baseline = _ServiceHealthTimeRangeSerializer(
+ allow_null=True, help_text="The baseline window. Null when compare is none."
+ )
+ timezone = serializers.CharField(help_text="Always UTC.")
+
+
+class _ServiceHealthSummarySerializer(serializers.Serializer):
+ services = serializers.IntegerField(help_text="Services scored, before the limit.")
+ requests = serializers.IntegerField(help_text="Requests over all scored services.")
+ ok = serializers.IntegerField(help_text="Services per status.")
+ degraded = serializers.IntegerField(help_text="Services per status.")
+ regressed = serializers.IntegerField(help_text="Services per status.")
+ silent = serializers.IntegerField(help_text="Services per status.")
+ new = serializers.IntegerField(help_text="Services per status.")
+ recovered = serializers.IntegerField(help_text="Services per status.")
+ improved = serializers.IntegerField(help_text="Services per status.")
+
+
+class _ServiceHealthRowSerializer(serializers.Serializer):
+ service = serializers.CharField(help_text="Service name.")
+ status = serializers.ChoiceField(
+ choices=ServiceHealthStatus.choices,
+ help_text=(
+ "silent: spans stopped. regressed: error rate or p95 worse than the baseline. degraded: error rate 5% or "
+ "more. new: no spans in the baseline. recovered, improved: the reverse of regressed."
+ ),
+ )
+ reason = serializers.CharField(allow_blank=True, help_text="Why the service has this status. Empty for ok.")
+ requests = serializers.IntegerField(help_text="Spans in the window, in the chosen spanScope.")
+ throughput_per_min = serializers.FloatField(help_text="Requests per minute.")
+ error_count = serializers.IntegerField(help_text="Requests with OTel status Error.")
+ error_rate_pct = serializers.FloatField(allow_null=True, help_text="Error share of requests, percent.")
+ error_rate_delta_pp = serializers.FloatField(allow_null=True, help_text="Change from the baseline, points.")
+ p50_ms = serializers.FloatField(allow_null=True, help_text="Median duration, ms.")
+ p95_ms = serializers.FloatField(allow_null=True, help_text="95th percentile duration, ms.")
+ p95_change_pct = serializers.FloatField(allow_null=True, help_text="p95 change from the baseline, percent.")
+ p99_ms = serializers.FloatField(allow_null=True, help_text="99th percentile duration, ms.")
+ baseline_requests = serializers.IntegerField(allow_null=True, help_text="Baseline requests. Null without one.")
+ last_seen = serializers.CharField(allow_null=True, help_text="Last span, ISO 8601 UTC.")
+
+
+@extend_schema_field({"type": "object", "additionalProperties": True})
+class _NextQueryField(serializers.JSONField):
+ pass
+
+
+class _ServiceHealthNextCallSerializer(serializers.Serializer):
+ tool = serializers.CharField(help_text="The tool to call next.")
+ why = serializers.CharField(help_text="What the call answers.")
+ query = _NextQueryField(help_text="The prefilled query argument.")
+
+
+class ServiceHealthResponseSerializer(serializers.Serializer):
+ window = _ServiceHealthWindowSerializer(help_text="The absolute UTC windows the server used.")
+ summary = _ServiceHealthSummarySerializer(help_text="Counts over all scored services.")
+ services = _ServiceHealthRowSerializer(many=True, help_text="One row per service, in orderBy order.")
+ next = _ServiceHealthNextCallSerializer(many=True, help_text="Prefilled follow-up calls.")
+ truncated = serializers.BooleanField(help_text="True when more services matched than limit.")
+ notes = serializers.ListField(child=serializers.CharField(), help_text="Caveats about this result.")
diff --git a/products/tracing/backend/presentation/views.py b/products/tracing/backend/presentation/views.py
index 24dc0e37a76c..8f8deee53da6 100644
--- a/products/tracing/backend/presentation/views.py
+++ b/products/tracing/backend/presentation/views.py
@@ -64,7 +64,9 @@ from ..facade.api import (
run_duration_histogram_query,
run_impact_query,
run_latency_heatmap_query,
+ run_service_health_query,
run_symbol_stats_query,
+ service_health_byte_cap_message,
)
from ..has_spans_query_runner import team_has_spans
from ..logic import (
@@ -79,6 +81,11 @@ from ..logic import (
)
from ..sparkline_query_runner import TraceSpansSparklineQueryRunner
from .date_window import normalize_tracing_date_range
+from .service_health_serializers import (
+ ServiceHealthRequestSerializer,
+ ServiceHealthResponseSerializer,
+ service_health_params,
+)
# Matches FEATURE_FLAGS.TRACING_AI_EVENTS in the frontend.
TRACING_AI_EVENTS_FEATURE_FLAG = "tracing-ai-events"
@@ -384,11 +391,34 @@ class _TracingTraceRequestSerializer(serializers.Serializer):
)
-class _TracingServiceNamesQuerySerializer(serializers.Serializer):
- search = serializers.CharField(required=False, help_text="Search filter for service names.")
+class _TracingDateQueryParamsSerializer(serializers.Serializer):
dateRange = serializers.CharField(
required=False,
- help_text='JSON-encoded date range, e.g. \'{"date_from": "-1h"}\'.',
+ help_text='JSON-encoded date range object, e.g. {"date_from": "-1h"}. Malformed JSON returns 400. '
+ "Or pass date_from and date_to.",
+ )
+ date_from = serializers.CharField(required=False, help_text="Range start: -1h, -7d or ISO 8601. Defaults to -1h.")
+ date_to = serializers.CharField(required=False, help_text='Range end, same format. Omit for "now".')
+
+
+class _TracingServiceNamesQuerySerializer(_TracingDateQueryParamsSerializer):
+ search = serializers.CharField(required=False, help_text="Search filter for service names.")
+
+
+def _date_range_from_query_params(request: Request) -> dict:
+ raw = request.GET.get("dateRange")
+ if raw:
+ try:
+ parsed = json.loads(raw)
+ except json.JSONDecodeError:
+ parsed = None
+ if not isinstance(parsed, dict):
+ example = json.dumps({"date_from": "-1h"})
+ message = f"dateRange must be a JSON object such as {example}, or pass date_from and date_to. Got {raw!r}."
+ raise serializers.ValidationError({"dateRange": message})
+ return normalize_tracing_date_range(parsed)
+ return normalize_tracing_date_range(
+ {"date_from": request.GET.get("date_from"), "date_to": request.GET.get("date_to")}
)
@@ -469,7 +499,7 @@ class _TracingErrorCountsResponseSerializer(serializers.Serializer):
)
-class _TracingAttributesQuerySerializer(serializers.Serializer):
+class _TracingAttributesQuerySerializer(_TracingDateQueryParamsSerializer):
search = serializers.CharField(required=False, help_text="Search filter for attribute names.")
search_values = serializers.BooleanField(
required=False,
@@ -514,7 +544,7 @@ class SpanPropertyType(models.TextChoices):
SPAN_RESOURCE_ATTRIBUTE = "span_resource_attribute", "span_resource_attribute"
-class _TracingValuesQuerySerializer(serializers.Serializer):
+class _TracingValuesQuerySerializer(_TracingDateQueryParamsSerializer):
key = serializers.CharField(help_text="The attribute key to get values for.")
attribute_type = serializers.ChoiceField(
choices=SpanPropertyType.choices,
@@ -1245,11 +1275,7 @@ class SpansViewSet(TeamAndOrgViewSetMixin, PydanticModelMixin, viewsets.ViewSet)
def service_names(self, request: Request, *args, **kwargs) -> Response:
tag_queries(product=ProductKey.TRACING, feature=Feature.QUERY)
search = request.GET.get("search", "")
- try:
- raw_date_range = json.loads(request.GET.get("dateRange", '{"date_from": "-1h"}'))
- except json.JSONDecodeError:
- raw_date_range = {"date_from": "-1h"}
- date_range = self.get_model(normalize_tracing_date_range(raw_date_range), DateRange)
+ date_range = self.get_model(_date_range_from_query_params(request), DateRange)
results = run_service_names_query(team=self.team, date_range=date_range, search=search)
return Response({"results": results}, status=status.HTTP_200_OK)
@@ -1703,6 +1729,36 @@ class SpansViewSet(TeamAndOrgViewSetMixin, PydanticModelMixin, viewsets.ViewSet)
status=status.HTTP_200_OK,
)
+ @validated_request(
+ ServiceHealthRequestSerializer,
+ responses={200: OpenApiResponse(response=ServiceHealthResponseSerializer)},
+ )
+ @action(detail=False, methods=["POST"], required_scopes=["tracing:read"])
+ def service_health(self, request: ValidatedRequest, *args, **kwargs) -> Response:
+ tag_queries(product=ProductKey.TRACING, feature=Feature.QUERY)
+ params = service_health_params(request.validated_data)
+ try:
+ result = run_service_health_query(team=self.team, params=params)
+ except CHQueryErrorTooManyBytes:
+ return Response(
+ {"detail": service_health_byte_cap_message(team=self.team, params=params)},
+ status=status.HTTP_400_BAD_REQUEST,
+ )
+
+ self._report_usage(
+ request,
+ "tracing service health queried",
+ {
+ "compare": params.compare.value,
+ "span_scope": params.span_scope.value,
+ "has_environment": bool(params.environment),
+ "service_names_count": len(params.service_names),
+ "services_count": result["summary"]["services"],
+ "truncated": result["truncated"],
+ },
+ )
+ return Response(ServiceHealthResponseSerializer(instance=result).data, status=status.HTTP_200_OK)
+
@extend_schema(
request=_TracingTreeRequestSerializer,
responses={200: _TracingTreeResponseSerializer},
@@ -1973,11 +2029,7 @@ class SpansViewSet(TeamAndOrgViewSetMixin, PydanticModelMixin, viewsets.ViewSet)
limit = int(request.GET.get("limit", "100"))
offset = int(request.GET.get("offset", "0"))
- try:
- raw_date_range = json.loads(request.GET.get("dateRange", "{}"))
- except json.JSONDecodeError:
- raw_date_range = {}
- date_range = self.get_model(normalize_tracing_date_range(raw_date_range), DateRange)
+ date_range = self.get_model(_date_range_from_query_params(request), DateRange)
attribute_type = request.GET.get("attribute_type", "span_attribute")
if attribute_type not in ("span_attribute", "span_resource_attribute"):
@@ -2007,11 +2059,7 @@ class SpansViewSet(TeamAndOrgViewSetMixin, PydanticModelMixin, viewsets.ViewSet)
limit = int(request.GET.get("limit", "100"))
offset = int(request.GET.get("offset", "0"))
- try:
- raw_date_range = json.loads(request.GET.get("dateRange", "{}"))
- except json.JSONDecodeError:
- raw_date_range = {}
- date_range = self.get_model(normalize_tracing_date_range(raw_date_range), DateRange)
+ date_range = self.get_model(_date_range_from_query_params(request), DateRange)
attribute_type = request.GET.get("attribute_type", "span_attribute")
if attribute_type not in ("span", "span_attribute", "span_resource_attribute"):
diff --git a/products/tracing/backend/service_health_query_runner.py b/products/tracing/backend/service_health_query_runner.py
new file mode 100644
index 000000000000..9e8980a81de4
--- /dev/null
+++ b/products/tracing/backend/service_health_query_runner.py
@@ -0,0 +1,375 @@
+import datetime as dt
+from typing import TYPE_CHECKING, Any
+from zoneinfo import ZoneInfo
+
+from posthog.schema import DateRange, IntervalType, PropertyGroupsMode
+
+from posthog.hogql import ast
+from posthog.hogql.constants import LimitContext
+from posthog.hogql.modifiers import create_default_modifiers_for_team
+from posthog.hogql.parser import parse_expr, parse_select
+from posthog.hogql.query import execute_hogql_query
+
+from posthog.clickhouse.client.connection import Workload
+from posthog.dataclasses import frozen
+from posthog.hogql_queries.utils.caller_context import map_in_caller_context
+from posthog.hogql_queries.utils.query_date_range import QueryDateRange
+
+from products.tracing.backend.facade.contracts import ServiceHealthParams
+from products.tracing.backend.facade.enums import (
+ ServiceHealthCompare as Compare,
+ ServiceHealthOrderBy as OrderBy,
+ ServiceHealthSpanScope,
+ ServiceHealthStatus as Status,
+)
+from products.tracing.backend.logic import TIME_BUCKET_DATE_RANGE_WHERE, fail_fast_scalar_settings
+
+if TYPE_CHECKING:
+ from posthog.models import Team
+
+UTC = ZoneInfo("UTC")
+
+SILENT_MIN_BASELINE_REQUESTS = 20
+SILENT_GAP = dt.timedelta(minutes=10)
+SILENT_MIN_EXPECTED_IN_GAP = 10
+COMPARE_MIN_REQUESTS = 100
+ERROR_RATE_DELTA_PP = 1.0
+ERROR_RATE_RATIO = 2.0
+P95_RISE_PCT = 50.0
+DEGRADED_ERROR_RATE_PCT = 5.0
+DEGRADED_MIN_REQUESTS = 20
+MAX_REASON_LENGTH = 80
+SERVICE_SCAN_LIMIT = 1000
+NEXT_SERVICES_LIMIT = 5
+
+_ENTRY_WHERE = "(is_root_span = 1 OR kind IN (2, 5))"
+_ENVIRONMENT = (
+ "coalesce(nullIf(resource_attributes['deployment.environment.name'], ''), "
+ "nullIf(resource_attributes['deployment.environment'], ''), resource_attributes['env'], '')"
+)
+_ENVIRONMENT_WHERE = "lower(" + _ENVIRONMENT + ") = {env}"
+_STATUS_ORDER = list(Status)
+_UNHEALTHY = (Status.REGRESSED, Status.DEGRADED)
+_SHIFT = {Compare.PREVIOUS_DAY: dt.timedelta(days=1), Compare.PREVIOUS_WEEK: dt.timedelta(days=7)}
+_AGGREGATE_COMPARE_FILTER: dict[str, dict[str, Any]] = {
+ Compare.PREVIOUS_PERIOD: {"compare": True},
+ Compare.PREVIOUS_DAY: {"compare": True, "compare_to": "-1d"},
+ Compare.PREVIOUS_WEEK: {"compare": True, "compare_to": "-7d"},
+}
+_ORDER_FIELD = {
+ OrderBy.ERROR_RATE: "error_rate_pct",
+ OrderBy.P95: "p95_ms",
+ OrderBy.THROUGHPUT: "requests",
+ OrderBy.ERROR_COUNT: "error_count",
+}
+
+
+Window = tuple[dt.datetime, dt.datetime]
+
+
+@frozen
+class ServiceWindowStats:
+ requests: int
+ errors: int
+ p50_ms: float
+ p95_ms: float
+ p99_ms: float
+ last_seen: dt.datetime
+
+ @property
+ def error_rate_pct(self) -> float:
+ return self.errors / self.requests * 100 if self.requests else 0.0
+
+
+def _iso(value: dt.datetime) -> str:
+ return value.astimezone(UTC).strftime("%Y-%m-%dT%H:%M:%SZ")
+
+
+def _range(window: Window) -> dict[str, str]:
+ return {"date_from": _iso(window[0]), "date_to": _iso(window[1])}
+
+
+def _ms(value: float) -> str:
+ return f"{value / 1000:.1f}s" if value >= 1000 else f"{value:.0f}ms"
+
+
+def _clip(text: str) -> str:
+ return text if len(text) <= MAX_REASON_LENGTH else text[: MAX_REASON_LENGTH - 1] + "…"
+
+
+def _round(value: float | None) -> float | None:
+ return None if value is None else round(value, 1)
+
+
+def _since(value: dt.datetime, reference: dt.datetime) -> str:
+ return value.strftime("%H:%MZ" if value.date() == reference.date() else "%Y-%m-%d %H:%MZ")
+
+
+def _p95_change_pct(current: ServiceWindowStats, baseline: ServiceWindowStats) -> float | None:
+ return (current.p95_ms - baseline.p95_ms) / baseline.p95_ms * 100 if baseline.p95_ms else None
+
+
+def classify_service(
+ *,
+ current: ServiceWindowStats | None,
+ baseline: ServiceWindowStats | None,
+ has_baseline: bool,
+ window_from: dt.datetime,
+ window_to: dt.datetime,
+ baseline_minutes: float | None = None,
+) -> tuple[Status, str]:
+ if current is None or not current.requests:
+ quiet_for = window_to - baseline.last_seen if baseline is not None else dt.timedelta(0)
+ minutes = baseline_minutes or max((window_to - window_from).total_seconds() / 60, 1.0)
+ expected = baseline.requests / minutes * SILENT_GAP.total_seconds() / 60 if baseline is not None else 0.0
+ if (
+ baseline is not None
+ and baseline.requests >= SILENT_MIN_BASELINE_REQUESTS
+ and quiet_for >= SILENT_GAP
+ and expected >= SILENT_MIN_EXPECTED_IN_GAP
+ ):
+ since = _since(baseline.last_seen, window_to)
+ return Status.SILENT, f"no spans since {since} (baseline {baseline.requests} req)"
+ return Status.OK, f"no spans; baseline {baseline.requests if baseline else 0} req is too few to call silent"
+
+ gap = window_to - current.last_seen
+ rate = current.requests / max((current.last_seen - window_from).total_seconds() / 60, 1.0)
+ if gap >= SILENT_GAP and rate * gap.total_seconds() / 60 >= SILENT_MIN_EXPECTED_IN_GAP:
+ return Status.SILENT, f"no spans since {_since(current.last_seen, window_to)} (was {rate:.1f}/min)"
+ if has_baseline and (baseline is None or not baseline.requests):
+ return Status.NEW, "no spans in baseline window"
+
+ rate_now = current.error_rate_pct
+ if baseline is not None and min(current.requests, baseline.requests) >= COMPARE_MIN_REQUESTS:
+ rate_before = baseline.error_rate_pct
+ p95_change = _p95_change_pct(current, baseline)
+ errors_up = rate_now - rate_before >= ERROR_RATE_DELTA_PP and rate_now >= ERROR_RATE_RATIO * rate_before
+ parts = [f"error rate {rate_before:.1f}% → {rate_now:.1f}%"] if errors_up else []
+ if p95_change is not None and p95_change >= P95_RISE_PCT:
+ parts.append(f"p95 {_ms(baseline.p95_ms)} → {_ms(current.p95_ms)} (+{p95_change:.0f}%)")
+ if parts:
+ return Status.REGRESSED, _clip("; ".join(parts))
+ if rate_before - rate_now >= ERROR_RATE_DELTA_PP and rate_before >= ERROR_RATE_RATIO * rate_now:
+ return Status.RECOVERED, f"error rate {rate_before:.1f}% → {rate_now:.1f}%"
+ if rate_now < DEGRADED_ERROR_RATE_PCT and baseline.p95_ms >= current.p95_ms * (1 + P95_RISE_PCT / 100) > 0:
+ return Status.IMPROVED, f"p95 {_ms(baseline.p95_ms)} → {_ms(current.p95_ms)} ({p95_change:.0f}%)"
+ if current.requests >= DEGRADED_MIN_REQUESTS and rate_now >= DEGRADED_ERROR_RATE_PCT:
+ before = f" (baseline {baseline.error_rate_pct:.1f}%)" if baseline is not None and baseline.requests else ""
+ return Status.DEGRADED, f"error rate {rate_now:.1f}%{before}"
+ return Status.OK, ""
+
+
+def aggregate_compare_filter(compare: Compare) -> dict[str, Any] | None:
+ return _AGGREGATE_COMPARE_FILTER.get(compare)
+
+
+class ServiceHealthQueryRunner:
+ def __init__(self, *, team: "Team", params: ServiceHealthParams) -> None:
+ self.team = team
+ self.params = params
+ self.modifiers = create_default_modifiers_for_team(team)
+ self.modifiers.convertToProjectTimezone = False
+ self.modifiers.propertyGroupsMode = PropertyGroupsMode.OPTIMIZED
+ self.current = self._resolve(params.date_from, params.date_to)
+ self.baseline = self._baseline_window()
+
+ def _resolve(self, date_from: str | None, date_to: str | None) -> Window:
+ query_date_range = QueryDateRange(
+ date_range=DateRange(date_from=date_from, date_to=date_to, explicitDate=True),
+ team=self.team,
+ interval=IntervalType.MINUTE,
+ now=dt.datetime.now(),
+ timezone_info=UTC,
+ exact_timerange=True,
+ )
+ return query_date_range.date_from().astimezone(UTC), query_date_range.date_to().astimezone(UTC)
+
+ def _baseline_window(self) -> Window | None:
+ compare, (start, end) = self.params.compare, self.current
+ if compare == Compare.NONE:
+ return None
+ if compare == Compare.CUSTOM:
+ return self._resolve(self.params.compare_date_from, self.params.compare_date_to)
+ shift = _SHIFT.get(compare, end - start)
+ return start - shift, end - shift
+
+ def window(self) -> dict[str, Any]:
+ baseline = _range(self.baseline) if self.baseline else None
+ return {"current": _range(self.current), "baseline": baseline, "timezone": "UTC"}
+
+ def byte_cap_message(self) -> str:
+ seconds = max(int((self.current[1] - self.current[0]).total_seconds() / 2), 1)
+ window = (
+ f"-{seconds // 3600}h"
+ if seconds % 3600 == 0
+ else f"-{seconds // 60}m"
+ if seconds % 60 == 0
+ else f"-{seconds}s"
+ )
+ baseline = " and a shorter baselineDateRange" if self.params.compare == Compare.CUSTOM else ""
+ return (
+ f'This query read more than the 10 GB cap. Retry with a shorter dateRange, at most {{"date_from": "{window}"}}{baseline}, '
+ "or pass serviceNames to read fewer services."
+ )
+
+ def _execute(self, sql: str, window: Window, service_names: list[str], **placeholders: ast.Expr) -> list:
+ dates: dict[str, ast.Expr] = {
+ "date_from": ast.Constant(value=window[0]),
+ "date_to": ast.Constant(value=window[1]),
+ }
+ where = [
+ parse_expr(TIME_BUCKET_DATE_RANGE_WHERE, placeholders=dates),
+ parse_expr("timestamp >= {date_from} AND timestamp < {date_to}", placeholders=dates),
+ ]
+ if self.params.span_scope == ServiceHealthSpanScope.ENTRY:
+ where.append(parse_expr(_ENTRY_WHERE))
+ if service_names:
+ names = ast.Tuple(exprs=[ast.Constant(value=name) for name in service_names])
+ where.append(parse_expr("service_name IN {names}", placeholders={"names": names}))
+ if self.params.environment:
+ environment = ast.Constant(value=self.params.environment.lower())
+ where.append(parse_expr(_ENVIRONMENT_WHERE, placeholders={"env": environment}))
+ response = execute_hogql_query(
+ query_type="TraceSpansServiceHealthQuery",
+ query=parse_select(sql, placeholders={"where": ast.And(exprs=where), **placeholders}),
+ team=self.team,
+ modifiers=self.modifiers,
+ workload=Workload.LOGS,
+ limit_context=LimitContext.QUERY,
+ settings=fail_fast_scalar_settings(),
+ )
+ return response.results or []
+
+ def _service_stats(self, window: Window) -> dict[str, ServiceWindowStats]:
+ rows = self._execute(
+ """
+ SELECT service_name, count() AS requests, countIf(status_code = 2),
+ quantiles(0.5, 0.95, 0.99)(duration_nano), max(timestamp)
+ FROM posthog.trace_spans
+ WHERE {where}
+ GROUP BY service_name
+ ORDER BY requests DESC, service_name ASC
+ LIMIT {limit}
+ """,
+ window,
+ list(self.params.service_names),
+ limit=ast.Constant(value=SERVICE_SCAN_LIMIT + 1),
+ )
+ return {
+ service: ServiceWindowStats(
+ requests=requests,
+ errors=errors,
+ p50_ms=(quantiles[0] or 0) / 1e6,
+ p95_ms=(quantiles[1] or 0) / 1e6,
+ p99_ms=(quantiles[2] or 0) / 1e6,
+ last_seen=last_seen.replace(tzinfo=UTC) if last_seen.tzinfo is None else last_seen,
+ )
+ for service, requests, errors, quantiles, last_seen in rows
+ }
+
+ def _row(
+ self, service: str, current: ServiceWindowStats | None, baseline: ServiceWindowStats | None, cut: bool = False
+ ) -> dict:
+ status, reason = classify_service(
+ current=current,
+ baseline=baseline,
+ has_baseline=self.baseline is not None and not cut,
+ window_from=self.current[0],
+ window_to=self.current[1],
+ baseline_minutes=(self.baseline[1] - self.baseline[0]).total_seconds() / 60 if self.baseline else None,
+ )
+ live = current if current is not None and current.requests else None
+ delta_pp = p95_change = None
+ if live is not None and baseline is not None and baseline.requests:
+ delta_pp = _round(live.error_rate_pct - baseline.error_rate_pct)
+ p95_change = _round(_p95_change_pct(live, baseline))
+ last_seen = live.last_seen if live else (baseline.last_seen if baseline else None)
+ minutes = max((self.current[1] - self.current[0]).total_seconds() / 60, 1 / 60)
+ return {
+ "service": service,
+ "status": status,
+ "reason": reason,
+ "requests": live.requests if live else 0,
+ "throughput_per_min": round((live.requests if live else 0) / minutes, 1),
+ "error_count": live.errors if live else 0,
+ "error_rate_pct": _round(live.error_rate_pct) if live else None,
+ "error_rate_delta_pp": delta_pp,
+ "p50_ms": _round(live.p50_ms) if live else None,
+ "p95_ms": _round(live.p95_ms) if live else None,
+ "p95_change_pct": p95_change,
+ "p99_ms": _round(live.p99_ms) if live else None,
+ "baseline_requests": (baseline.requests if baseline else 0) if self.baseline else None,
+ "last_seen": _iso(last_seen) if last_seen else None,
+ }
+
+ def _next_calls(self, rows: list[dict]) -> list[dict[str, Any]]:
+ unhealthy = [row for row in rows if row["status"] in _UNHEALTHY]
+ if self.params.environment or not unhealthy:
+ return []
+ names = [row["service"] for row in unhealthy[:NEXT_SERVICES_LIMIT]]
+ query: dict[str, Any] = {"dateRange": _range(self.current), "serviceNames": names, "limit": 20}
+ if compare_filter := aggregate_compare_filter(self.params.compare):
+ query["compareFilter"] = compare_filter
+ calls = [{"tool": "apm-spans-aggregate", "why": _clip(f"operations behind {', '.join(names)}"), "query": query}]
+ worst = next((row for row in unhealthy if row["error_count"]), None)
+ if worst:
+ query = {"dateRange": _range(self.current), "serviceNames": [worst["service"]], "statusCodes": [2]}
+ query = {**query, "flatSpans": True, "rootSpans": False, "limit": 10, "excludeAttributes": True}
+ calls.append(
+ {"tool": "query-apm-spans", "why": _clip(f"recent error spans in {worst['service']}"), "query": query}
+ )
+ return calls
+
+ def _notes(self, rows: list[dict], scan_truncated: bool) -> list[str]:
+ notes = {
+ "No baseline: set compare to get new, regressed, recovered and improved statuses.": self.baseline is None,
+ f"Only the {SERVICE_SCAN_LIMIT} busiest services were scored. Pass serviceNames for others.": scan_truncated,
+ "No spans matched. spanScope=all also counts internal and client spans. The environment filter reads "
+ "deployment.environment.name, deployment.environment, then env.": not rows,
+ }
+ return [note for note, applies in notes.items() if applies]
+
+ def run(self) -> dict[str, Any]:
+ windows = [self.current] if self.baseline is None else [self.current, self.baseline]
+ stats = map_in_caller_context(self._service_stats, windows, thread_name_prefix="tracing_service_health")
+ current, baseline = stats[0], stats[1] if len(stats) > 1 else {}
+ current_cut, baseline_cut = len(current) > SERVICE_SCAN_LIMIT, len(baseline) > SERVICE_SCAN_LIMIT
+ names = [*current, *([] if current_cut else sorted(set(baseline) - set(current)))]
+ rows = [
+ self._row(name, current.get(name), baseline.get(name), baseline_cut and name not in baseline)
+ for name in names
+ ]
+
+ rows.sort(
+ key=lambda row: (
+ _STATUS_ORDER.index(row["status"]),
+ -row["requests"],
+ -(row["baseline_requests"] or 0),
+ row["service"],
+ )
+ )
+ field = _ORDER_FIELD.get(self.params.order_by)
+ ordered = sorted(rows, key=lambda row: (row[field] is None, -(row[field] or 0))) if field else rows
+ page = ordered[: self.params.limit]
+ scan_truncated = max(len(current), len(baseline)) > SERVICE_SCAN_LIMIT
+ result: dict[str, Any] = {
+ "window": self.window(),
+ "summary": {
+ "services": len(rows),
+ "requests": sum(row["requests"] for row in rows),
+ **{status.value: sum(row["status"] == status for row in rows) for status in Status},
+ },
+ "services": page,
+ "next": self._next_calls(rows),
+ "truncated": len(rows) > self.params.limit or scan_truncated,
+ "notes": self._notes(rows, scan_truncated),
+ }
+ return result
+
+
+def run_service_health_query(*, team: "Team", params: ServiceHealthParams) -> dict[str, Any]:
+ return ServiceHealthQueryRunner(team=team, params=params).run()
+
+
+def service_health_byte_cap_message(*, team: "Team", params: ServiceHealthParams) -> str:
+ return ServiceHealthQueryRunner(team=team, params=params).byte_cap_message()
diff --git a/products/tracing/backend/tests/test_date_window.py b/products/tracing/backend/tests/test_date_window.py
index 8d553c1f36de..21cec7b093bf 100644
--- a/products/tracing/backend/tests/test_date_window.py
+++ b/products/tracing/backend/tests/test_date_window.py
@@ -61,6 +61,10 @@ class TestNormalizeTracingWindow:
("h",),
("-m",),
("d",),
+ ("-999999999d",),
+ ("-99999999999y",),
+ (5,),
+ (True,),
]
)
def test_unparseable_values_raise(self, value: str) -> None:
diff --git a/products/tracing/backend/tests/test_get_date_params.py b/products/tracing/backend/tests/test_get_date_params.py
new file mode 100644
index 000000000000..c5ced7f21215
--- /dev/null
+++ b/products/tracing/backend/tests/test_get_date_params.py
@@ -0,0 +1,83 @@
+import json
+
+from posthog.test.base import APIBaseTest
+from unittest.mock import patch
+
+from parameterized import parameterized
+
+VIEWS = "products.tracing.backend.presentation.views"
+
+ENDPOINTS: list[tuple[str, str, dict[str, str], object]] = [
+ ("service-names", "run_service_names_query", {}, []),
+ ("attributes", "run_attribute_names_query", {}, ([], 0)),
+ ("values", "run_attribute_values_query", {"key": "http.method"}, []),
+]
+
+
+class TestTracingGetDateParams(APIBaseTest):
+ def _get(self, path: str, runner: str, params: dict, return_value):
+ with patch(f"{VIEWS}.{runner}", return_value=return_value) as run:
+ response = self.client.get(f"/api/projects/{self.team.id}/tracing/spans/{path}/", params)
+ return response, run
+
+ @parameterized.expand(ENDPOINTS)
+ def test_malformed_date_range_is_a_400_with_an_example(self, path, runner, extra, return_value):
+ response, run = self._get(path, runner, {"dateRange": "{date_from: -1h}", **extra}, return_value)
+ self.assertEqual(response.status_code, 400, response.content)
+ self.assertIn('{"date_from": "-1h"}', json.dumps(response.json()).replace('\\"', '"'))
+ run.assert_not_called()
+
+ @parameterized.expand(ENDPOINTS)
+ def test_date_range_must_be_an_object(self, path, runner, extra, return_value):
+ response, run = self._get(path, runner, {"dateRange": '"-1h"', **extra}, return_value)
+ self.assertEqual(response.status_code, 400, response.content)
+ run.assert_not_called()
+
+ @parameterized.expand(ENDPOINTS)
+ def test_date_range_values_must_be_strings(self, path, runner, extra, return_value):
+ response, run = self._get(path, runner, {"dateRange": '{"date_from": 5}', **extra}, return_value)
+ self.assertEqual(response.status_code, 400, response.content)
+ run.assert_not_called()
+
+ @parameterized.expand(ENDPOINTS)
+ def test_explicit_date_option_is_still_accepted(self, path, runner, extra, return_value):
+ response, run = self._get(
+ path, runner, {"dateRange": '{"date_from": "-6h", "explicitDate": true}', **extra}, return_value
+ )
+ self.assertEqual(response.status_code, 200, response.content)
+ self.assertEqual(run.call_args.kwargs["date_range"].date_from, "-6h")
+
+ @parameterized.expand(ENDPOINTS)
+ def test_json_date_range_still_works(self, path, runner, extra, return_value):
+ response, run = self._get(path, runner, {"dateRange": '{"date_from": "-6h"}', **extra}, return_value)
+ self.assertEqual(response.status_code, 200, response.content)
+ self.assertEqual(run.call_args.kwargs["date_range"].date_from, "-6h")
+
+ @parameterized.expand(ENDPOINTS)
+ def test_flat_date_params(self, path, runner, extra, return_value):
+ response, run = self._get(
+ path, runner, {"date_from": "-30m", "date_to": "2026-06-02T09:00:00Z", **extra}, return_value
+ )
+ self.assertEqual(response.status_code, 200, response.content)
+ date_range = run.call_args.kwargs["date_range"]
+ self.assertEqual((date_range.date_from, date_range.date_to), ("-30M", "2026-06-02T09:00:00Z"))
+
+ @parameterized.expand(ENDPOINTS)
+ def test_date_range_wins_over_flat_params(self, path, runner, extra, return_value):
+ response, run = self._get(
+ path, runner, {"dateRange": '{"date_from": "-6h"}', "date_from": "-2d", **extra}, return_value
+ )
+ self.assertEqual(response.status_code, 200, response.content)
+ self.assertEqual(run.call_args.kwargs["date_range"].date_from, "-6h")
+
+ @parameterized.expand(ENDPOINTS)
+ def test_no_dates_defaults_to_the_last_hour(self, path, runner, extra, return_value):
+ response, run = self._get(path, runner, extra, return_value)
+ self.assertEqual(response.status_code, 200, response.content)
+ self.assertEqual(run.call_args.kwargs["date_range"].date_from, "-1h")
+
+ @parameterized.expand(ENDPOINTS)
+ def test_bad_flat_value_is_a_400(self, path, runner, extra, return_value):
+ response, run = self._get(path, runner, {"date_from": "last tuesday", **extra}, return_value)
+ self.assertEqual(response.status_code, 400, response.content)
+ run.assert_not_called()
diff --git a/products/tracing/backend/tests/test_service_health_endpoint.py b/products/tracing/backend/tests/test_service_health_endpoint.py
new file mode 100644
index 000000000000..7534dc0068b5
--- /dev/null
+++ b/products/tracing/backend/tests/test_service_health_endpoint.py
@@ -0,0 +1,211 @@
+import datetime as dt
+
+from posthog.test.base import APIBaseTest
+from unittest.mock import patch
+
+from parameterized import parameterized
+
+from posthog.clickhouse.client import sync_execute
+from posthog.errors import CHQueryErrorTooManyBytes
+
+from products.tracing.backend.facade.contracts import ServiceHealthParams
+from products.tracing.backend.facade.enums import ServiceHealthCompare, ServiceHealthOrderBy, ServiceHealthSpanScope
+from products.tracing.backend.tests.test_keyset_pagination import _TraceSpansTestBase
+
+RUNNER_PATH = "products.tracing.backend.presentation.views.run_service_health_query"
+
+
+def _url(team_id: int) -> str:
+ return f"/api/projects/{team_id}/tracing/spans/service_health/"
+
+
+class TestServiceHealthValidation(APIBaseTest):
+ @parameterized.expand(
+ [
+ ("missing_query", {}),
+ ("query_not_object", {"query": "x"}),
+ ("unknown_compare", {"query": {"compare": "yesterday"}}),
+ ("compare_object", {"query": {"compare": {"date_from": "-2d"}}}),
+ ("baseline_not_object", {"query": {}, "baselineDateRange": "-2d"}),
+ ("compare_wrong_type", {"query": {"compare": 3}}),
+ ("bad_span_scope", {"query": {"spanScope": "root"}}),
+ ("bad_order_by", {"query": {"orderBy": "name"}}),
+ ("reversed_baseline", {"query": {}, "baselineDateRange": {"date_from": "-1d", "date_to": "-2d"}}),
+ ("future_window", {"query": {"dateRange": {"date_from": "2099-01-01T00:00:00Z"}}}),
+ ("future_baseline", {"query": {}, "baselineDateRange": {"date_from": "2099-01-01T00:00:00Z"}}),
+ ("limit_too_small", {"query": {"limit": 0}}),
+ ("limit_too_large", {"query": {"limit": 1000}}),
+ ("bad_date", {"query": {"dateRange": {"date_from": "yesterday-ish"}}}),
+ ("date_range_not_object", {"query": {"dateRange": "-1h"}}),
+ ("numeric_date", {"query": {"dateRange": {"date_from": 5}}}),
+ ("huge_window", {"query": {"dateRange": {"date_from": "-999999999d"}}}),
+ (
+ "year_one",
+ {"query": {"dateRange": {"date_from": "0001-01-03T00:00:00Z", "date_to": "0001-01-10T00:00:00Z"}}},
+ ),
+ ]
+ )
+ def test_rejects_bad_input(self, _name, body):
+ with patch(RUNNER_PATH) as runner:
+ response = self.client.post(_url(self.team.id), body, format="json")
+ self.assertEqual(response.status_code, 400, response.content)
+ runner.assert_not_called()
+
+ def test_defaults(self):
+ with patch(RUNNER_PATH, side_effect=CHQueryErrorTooManyBytes("too many bytes")) as runner:
+ self.client.post(_url(self.team.id), {"query": {}}, format="json")
+ params = runner.call_args.kwargs["params"]
+ self.assertEqual(
+ params,
+ ServiceHealthParams(
+ date_from="-1h",
+ date_to=None,
+ compare=ServiceHealthCompare.PREVIOUS_PERIOD,
+ environment=None,
+ service_names=(),
+ span_scope=ServiceHealthSpanScope.ENTRY,
+ order_by=ServiceHealthOrderBy.STATUS,
+ limit=25,
+ ),
+ )
+
+ def test_parses_every_field(self):
+ body = {
+ "query": {
+ "dateRange": {"date_from": "-30m", "date_to": None},
+ "environment": "production",
+ "serviceNames": ["api"],
+ "spanScope": "all",
+ "orderBy": "p95",
+ "limit": 5,
+ },
+ "baselineDateRange": {"date_from": "-2d", "date_to": "-1d"},
+ }
+ with patch(RUNNER_PATH, side_effect=CHQueryErrorTooManyBytes("too many bytes")) as runner:
+ self.client.post(_url(self.team.id), body, format="json")
+ params = runner.call_args.kwargs["params"]
+ self.assertEqual(params.date_from, "-30M")
+ self.assertEqual(params.compare, ServiceHealthCompare.CUSTOM)
+ self.assertEqual((params.compare_date_from, params.compare_date_to), ("-2d", "-1d"))
+ self.assertEqual(params.service_names, ("api",))
+ self.assertEqual((params.span_scope, params.order_by, params.limit), ("all", "p95", 5))
+
+ @parameterized.expand(["none", "previous_period", "previous_day", "previous_week"])
+ def test_accepts_compare_names(self, compare):
+ with patch(RUNNER_PATH, side_effect=CHQueryErrorTooManyBytes("too many bytes")) as runner:
+ self.client.post(_url(self.team.id), {"query": {"compare": compare}}, format="json")
+ self.assertEqual(runner.call_args.kwargs["params"].compare, compare)
+
+ def test_byte_cap_returns_an_actionable_400(self):
+ with patch(RUNNER_PATH, side_effect=CHQueryErrorTooManyBytes("too many bytes")):
+ response = self.client.post(
+ _url(self.team.id), {"query": {"dateRange": {"date_from": "-7d"}}}, format="json"
+ )
+ self.assertEqual(response.status_code, 400, response.content)
+ detail = response.json()["detail"]
+ self.assertIn("10 GB", detail)
+ self.assertIn("serviceNames", detail)
+ self.assertIn('{"date_from": "-84h"}', detail)
+
+ def test_byte_cap_on_a_short_window_recommends_a_smaller_one(self):
+ with patch(RUNNER_PATH, side_effect=CHQueryErrorTooManyBytes("too many bytes")):
+ response = self.client.post(
+ _url(self.team.id),
+ {"query": {"dateRange": {"date_from": "-1h"}}, "baselineDateRange": {"date_from": "-3h"}},
+ format="json",
+ )
+ detail = response.json()["detail"]
+ self.assertIn('{"date_from": "-30m"}', detail)
+ self.assertIn("baselineDateRange", detail)
+
+ def test_byte_cap_on_a_one_minute_window_recommends_seconds(self):
+ with patch(RUNNER_PATH, side_effect=CHQueryErrorTooManyBytes("too many bytes")):
+ response = self.client.post(
+ _url(self.team.id), {"query": {"dateRange": {"date_from": "-1m"}}}, format="json"
+ )
+ self.assertIn('{"date_from": "-30s"}', response.json()["detail"])
+
+
+class TestServiceHealthResponse(_TraceSpansTestBase):
+ @classmethod
+ def setUpTestData(cls):
+ super().setUpTestData()
+ cls._recreate_trace_spans_tables()
+ rows: list[tuple] = []
+ for i in range(120):
+ for service, start, errors in (
+ ("worker", dt.datetime(2026, 6, 2, 7, 0), 0),
+ ("worker", dt.datetime(2026, 6, 2, 8, 0), 12),
+ ):
+ ts = start + dt.timedelta(seconds=30 * i)
+ rows.append(
+ (
+ f"019e8810-0000-0000-0000-{len(rows):012d}",
+ cls.team.id,
+ f"trace-{len(rows)}",
+ f"span-{len(rows)}",
+ "",
+ "consume",
+ 5,
+ ts,
+ ts + dt.timedelta(milliseconds=20),
+ ts,
+ 2 if i < errors else 0,
+ service,
+ {},
+ )
+ )
+ sync_execute(
+ "INSERT INTO trace_spans (uuid, team_id, trace_id, span_id, parent_span_id, name, kind, timestamp, "
+ "end_time, observed_timestamp, status_code, service_name, resource_attributes) VALUES",
+ rows,
+ )
+
+ def _post(self, **query) -> dict:
+ body = {
+ "query": {"dateRange": {"date_from": "2026-06-02T08:00:00Z", "date_to": "2026-06-02T09:00:00Z"}, **query}
+ }
+ response = self.client.post(_url(self.team.id), body, format="json")
+ self.assertEqual(response.status_code, 200, response.content)
+ return response.json()
+
+ def test_response_shape(self):
+ data = self._post()
+ self.assertEqual(set(data), {"window", "summary", "services", "next", "truncated", "notes"})
+ self.assertEqual(
+ data["window"],
+ {
+ "current": {"date_from": "2026-06-02T08:00:00Z", "date_to": "2026-06-02T09:00:00Z"},
+ "baseline": {"date_from": "2026-06-02T07:00:00Z", "date_to": "2026-06-02T08:00:00Z"},
+ "timezone": "UTC",
+ },
+ )
+ self.assertEqual(data["summary"]["services"], 1)
+ self.assertEqual(data["summary"]["regressed"], 1)
+ [row] = data["services"]
+ self.assertEqual(
+ list(row),
+ [
+ "service",
+ "status",
+ "reason",
+ "requests",
+ "throughput_per_min",
+ "error_count",
+ "error_rate_pct",
+ "error_rate_delta_pp",
+ "p50_ms",
+ "p95_ms",
+ "p95_change_pct",
+ "p99_ms",
+ "baseline_requests",
+ "last_seen",
+ ],
+ )
+ self.assertEqual(row["service"], "worker")
+ self.assertEqual(row["status"], "regressed")
+ self.assertEqual(row["error_rate_pct"], 10.0)
+ self.assertEqual(row["p95_ms"], 20.0)
+ self.assertTrue(row["last_seen"].endswith("Z"))
+ self.assertFalse(data["truncated"])
+ self.assertEqual(data["next"][0]["tool"], "apm-spans-aggregate")
diff --git a/products/tracing/backend/tests/test_service_health_query_runner.py b/products/tracing/backend/tests/test_service_health_query_runner.py
new file mode 100644
index 000000000000..5c3a7473afc3
--- /dev/null
+++ b/products/tracing/backend/tests/test_service_health_query_runner.py
@@ -0,0 +1,428 @@
+import datetime as dt
+
+import time_machine
+from posthog.test.base import BaseTest
+from unittest.mock import patch
+
+from django.test import SimpleTestCase
+
+from parameterized import parameterized
+
+from posthog.clickhouse.client import sync_execute
+
+from products.tracing.backend.facade.contracts import ServiceHealthParams
+from products.tracing.backend.facade.enums import (
+ ServiceHealthCompare,
+ ServiceHealthOrderBy,
+ ServiceHealthSpanScope,
+ ServiceHealthStatus,
+)
+from products.tracing.backend.service_health_query_runner import (
+ MAX_REASON_LENGTH,
+ ServiceHealthQueryRunner,
+ ServiceWindowStats,
+ aggregate_compare_filter,
+ classify_service,
+)
+from products.tracing.backend.tests.test_keyset_pagination import _TraceSpansTestBase
+
+CURRENT_FROM = dt.datetime(2026, 6, 2, 8, 0, 0)
+CURRENT_TO = dt.datetime(2026, 6, 2, 9, 0, 0)
+BASELINE_FROM = dt.datetime(2026, 6, 2, 7, 0, 0)
+PREVIOUS_DAY_FROM = dt.datetime(2026, 6, 1, 8, 0, 0)
+
+PRODUCTION = {"deployment.environment.name": "production"}
+
+
+def _stats(
+ requests: int,
+ errors: int = 0,
+ p95_ms: float = 10.0,
+ last_seen: dt.datetime | None = None,
+) -> ServiceWindowStats:
+ return ServiceWindowStats(
+ requests=requests,
+ errors=errors,
+ p50_ms=p95_ms,
+ p95_ms=p95_ms,
+ p99_ms=p95_ms,
+ last_seen=last_seen or CURRENT_TO - dt.timedelta(seconds=20),
+ )
+
+
+class TestClassifyService(SimpleTestCase):
+ @parameterized.expand(
+ [
+ ("ok", _stats(200), _stats(200), ServiceHealthStatus.OK),
+ (
+ "silent_no_current_spans",
+ None,
+ _stats(100, last_seen=CURRENT_FROM - dt.timedelta(minutes=10)),
+ ServiceHealthStatus.SILENT,
+ ),
+ ("small_baseline_gone_is_not_silent", None, _stats(5), ServiceHealthStatus.OK),
+ (
+ "silent_spans_stopped_inside_window",
+ _stats(300, last_seen=CURRENT_FROM + dt.timedelta(minutes=30)),
+ _stats(300),
+ ServiceHealthStatus.SILENT,
+ ),
+ ("new", _stats(50), None, ServiceHealthStatus.NEW),
+ ("regressed_error_rate", _stats(200, errors=10), _stats(200, errors=1), ServiceHealthStatus.REGRESSED),
+ ("regressed_p95", _stats(200, p95_ms=30.0), _stats(200, p95_ms=10.0), ServiceHealthStatus.REGRESSED),
+ ("error_rise_below_ratio", _stats(200, errors=24), _stats(200, errors=20), ServiceHealthStatus.DEGRADED),
+ ("regression_needs_volume", _stats(50, errors=10), _stats(50), ServiceHealthStatus.DEGRADED),
+ ("degraded", _stats(200, errors=20), _stats(200, errors=20), ServiceHealthStatus.DEGRADED),
+ ("recovered", _stats(200), _stats(200, errors=20), ServiceHealthStatus.RECOVERED),
+ ("improved", _stats(200, p95_ms=10.0), _stats(200, p95_ms=40.0), ServiceHealthStatus.IMPROVED),
+ ]
+ )
+ def test_status_rules(self, _name, current, baseline, expected):
+ status, reason = classify_service(
+ current=current, baseline=baseline, has_baseline=True, window_from=CURRENT_FROM, window_to=CURRENT_TO
+ )
+ self.assertEqual(status, expected, reason)
+ self.assertLessEqual(len(reason), MAX_REASON_LENGTH)
+
+ def test_regression_reason_names_the_rates(self):
+ _, reason = classify_service(
+ current=_stats(200, errors=10),
+ baseline=_stats(200, errors=1),
+ has_baseline=True,
+ window_from=CURRENT_FROM,
+ window_to=CURRENT_TO,
+ )
+ self.assertEqual(reason, "error rate 0.5% → 5.0%")
+
+ def test_silent_reason_names_the_last_span(self):
+ _, reason = classify_service(
+ current=_stats(300, last_seen=dt.datetime(2026, 6, 2, 8, 30, 0)),
+ baseline=None,
+ has_baseline=False,
+ window_from=CURRENT_FROM,
+ window_to=CURRENT_TO,
+ )
+ self.assertTrue(reason.startswith("no spans since 08:30Z"), reason)
+
+ def test_sparse_baseline_is_not_silent(self):
+ status, _ = classify_service(
+ current=None,
+ baseline=_stats(30, last_seen=CURRENT_FROM - dt.timedelta(minutes=30)),
+ has_baseline=True,
+ window_from=CURRENT_FROM,
+ window_to=CURRENT_TO,
+ baseline_minutes=60,
+ )
+ self.assertNotEqual(status, ServiceHealthStatus.SILENT)
+
+ def test_no_current_spans_is_not_silent_inside_the_gap(self):
+ status, _ = classify_service(
+ current=None,
+ baseline=_stats(300, last_seen=CURRENT_TO - dt.timedelta(minutes=5)),
+ has_baseline=True,
+ window_from=CURRENT_TO - dt.timedelta(minutes=5),
+ window_to=CURRENT_TO,
+ )
+ self.assertNotEqual(status, ServiceHealthStatus.SILENT)
+
+ def test_without_a_baseline_only_absolute_rules_apply(self):
+ status, _ = classify_service(
+ current=_stats(200, errors=10),
+ baseline=None,
+ has_baseline=False,
+ window_from=CURRENT_FROM,
+ window_to=CURRENT_TO,
+ )
+ self.assertEqual(status, ServiceHealthStatus.DEGRADED)
+ status, _ = classify_service(
+ current=_stats(200), baseline=None, has_baseline=False, window_from=CURRENT_FROM, window_to=CURRENT_TO
+ )
+ self.assertEqual(status, ServiceHealthStatus.OK)
+
+
+class TestAggregateCompareFilter(SimpleTestCase):
+ @parameterized.expand(
+ [
+ (ServiceHealthCompare.PREVIOUS_PERIOD, {"compare": True}),
+ (ServiceHealthCompare.PREVIOUS_DAY, {"compare": True, "compare_to": "-1d"}),
+ (ServiceHealthCompare.PREVIOUS_WEEK, {"compare": True, "compare_to": "-7d"}),
+ (ServiceHealthCompare.CUSTOM, None),
+ (ServiceHealthCompare.NONE, None),
+ ]
+ )
+ def test_maps_compare_to_the_aggregate_tool(self, compare, expected):
+ self.assertEqual(aggregate_compare_filter(compare), expected)
+
+
+class TestServiceHealthWindows(BaseTest):
+ @parameterized.expand(
+ [
+ (ServiceHealthCompare.PREVIOUS_PERIOD, "2026-06-02T06:00:00Z", "2026-06-02T08:00:00Z"),
+ (ServiceHealthCompare.PREVIOUS_DAY, "2026-06-01T08:00:00Z", "2026-06-01T10:00:00Z"),
+ (ServiceHealthCompare.PREVIOUS_WEEK, "2026-05-26T08:00:00Z", "2026-05-26T10:00:00Z"),
+ ]
+ )
+ @time_machine.travel("2026-06-02T10:00:00Z", tick=False)
+ def test_baseline_window(self, compare, expected_from, expected_to):
+ runner = ServiceHealthQueryRunner(team=self.team, params=ServiceHealthParams(date_from="-2h", compare=compare))
+ window = runner.window()
+ self.assertEqual(window["current"], {"date_from": "2026-06-02T08:00:00Z", "date_to": "2026-06-02T10:00:00Z"})
+ self.assertEqual(window["baseline"], {"date_from": expected_from, "date_to": expected_to})
+ self.assertEqual(window["timezone"], "UTC")
+
+ def test_custom_baseline_window(self):
+ params = ServiceHealthParams(
+ date_from="2026-06-02T08:00:00Z",
+ date_to="2026-06-02T09:00:00Z",
+ compare=ServiceHealthCompare.CUSTOM,
+ compare_date_from="2026-05-30T00:00:00Z",
+ compare_date_to="2026-05-30T06:00:00Z",
+ )
+ window = ServiceHealthQueryRunner(team=self.team, params=params).window()
+ self.assertEqual(window["baseline"], {"date_from": "2026-05-30T00:00:00Z", "date_to": "2026-05-30T06:00:00Z"})
+
+ def test_no_baseline(self):
+ params = ServiceHealthParams(date_from="-1h", compare=ServiceHealthCompare.NONE)
+ self.assertIsNone(ServiceHealthQueryRunner(team=self.team, params=params).window()["baseline"])
+
+
+def _spread(start: dt.datetime, count: int, minutes: int = 60) -> list[dt.datetime]:
+ step = dt.timedelta(seconds=minutes * 60 / count)
+ return [start + step * i for i in range(count)]
+
+
+class TestServiceHealthQueryRunner(_TraceSpansTestBase):
+ @classmethod
+ def setUpTestData(cls):
+ super().setUpTestData()
+ cls._recreate_trace_spans_tables()
+ rows: list[tuple] = []
+
+ def add(
+ service: str,
+ start: dt.datetime,
+ count: int,
+ *,
+ errors: int = 0,
+ duration_ms: float = 10.0,
+ name: str = "GET /",
+ kind: int = 2,
+ parent: str = "",
+ resource: dict | None = None,
+ minutes: int = 60,
+ ) -> None:
+ for i, ts in enumerate(_spread(start, count, minutes)):
+ index = len(rows)
+ rows.append(
+ (
+ f"019e8800-0000-0000-0000-{index:012d}",
+ cls.team.id,
+ f"trace-{index}",
+ f"span-{index}",
+ parent,
+ name,
+ kind,
+ ts,
+ ts + dt.timedelta(milliseconds=duration_ms),
+ ts,
+ 2 if i < errors else 0,
+ service,
+ PRODUCTION if resource is None else resource,
+ )
+ )
+
+ add("api", BASELINE_FROM, 200)
+ add("api", CURRENT_FROM, 200)
+ add("api", CURRENT_FROM, 200, errors=200, kind=1, parent="cGFyZW50", name="db query")
+ add("api", CURRENT_FROM, 40, resource={"deployment.environment": "staging"})
+ add("stage-svc", CURRENT_FROM, 30, resource={"env": "staging"})
+ add("worker", BASELINE_FROM, 200, errors=1, kind=5, parent="cGFyZW50", name="consume order")
+ add("worker", CURRENT_FROM, 150, errors=10, kind=5, parent="cGFyZW50", name="consume order")
+ add("worker", CURRENT_FROM, 50, errors=2, kind=5, parent="cGFyZW50", name="consume refund")
+ add("dispatch", BASELINE_FROM, 100)
+ add("billing", CURRENT_FROM, 50)
+ add("search", BASELINE_FROM, 200, duration_ms=10.0)
+ add("search", CURRENT_FROM, 200, duration_ms=30.0)
+ add("mailer", BASELINE_FROM, 200, errors=20)
+ add("mailer", CURRENT_FROM, 200, errors=20)
+ add("ledger", BASELINE_FROM, 200, errors=20)
+ add("ledger", CURRENT_FROM, 200)
+ add("cache", BASELINE_FROM, 200, duration_ms=40.0)
+ add("cache", CURRENT_FROM, 200, duration_ms=10.0)
+ add("cron", BASELINE_FROM, 150)
+ add("cron", CURRENT_FROM, 150, minutes=30)
+ add("nightly", PREVIOUS_DAY_FROM, 100)
+
+ sync_execute(
+ "INSERT INTO trace_spans (uuid, team_id, trace_id, span_id, parent_span_id, name, kind, timestamp, "
+ "end_time, observed_timestamp, status_code, service_name, resource_attributes) VALUES",
+ rows,
+ )
+
+ def _run(self, **overrides):
+ fields = {"date_from": "2026-06-02T08:00:00Z", "date_to": "2026-06-02T09:00:00Z", "environment": "production"}
+ params = ServiceHealthParams(**{**fields, **overrides})
+ return ServiceHealthQueryRunner(team=self.team, params=params).run()
+
+ def _by_service(self, result) -> dict:
+ return {row["service"]: row for row in result["services"]}
+
+ def test_window_is_echoed_in_utc(self):
+ result = self._run()
+ self.assertEqual(result["window"]["current"]["date_from"], "2026-06-02T08:00:00Z")
+ self.assertEqual(result["window"]["current"]["date_to"], "2026-06-02T09:00:00Z")
+ self.assertEqual(result["window"]["baseline"]["date_from"], "2026-06-02T07:00:00Z")
+ self.assertEqual(result["window"]["baseline"]["date_to"], "2026-06-02T08:00:00Z")
+
+ def test_each_status(self):
+ rows = self._by_service(self._run(limit=100))
+ self.assertEqual(
+ {service: row["status"] for service, row in rows.items()},
+ {
+ "api": ServiceHealthStatus.OK,
+ "worker": ServiceHealthStatus.REGRESSED,
+ "dispatch": ServiceHealthStatus.SILENT,
+ "billing": ServiceHealthStatus.NEW,
+ "search": ServiceHealthStatus.REGRESSED,
+ "mailer": ServiceHealthStatus.DEGRADED,
+ "ledger": ServiceHealthStatus.RECOVERED,
+ "cache": ServiceHealthStatus.IMPROVED,
+ "cron": ServiceHealthStatus.SILENT,
+ },
+ )
+
+ def test_row_metrics_use_units(self):
+ worker = self._by_service(self._run())["worker"]
+ self.assertEqual(worker["requests"], 200)
+ self.assertEqual(worker["error_count"], 12)
+ self.assertEqual(worker["error_rate_pct"], 6.0)
+ self.assertEqual(worker["error_rate_delta_pp"], 5.5)
+ self.assertEqual(worker["baseline_requests"], 200)
+ self.assertEqual(worker["throughput_per_min"], 3.3)
+ self.assertEqual(worker["p95_ms"], 10.0)
+ self.assertEqual(worker["p95_change_pct"], 0.0)
+ self.assertEqual(worker["reason"], "error rate 0.5% → 6.0%")
+
+ search = self._by_service(self._run())["search"]
+ self.assertEqual((search["p50_ms"], search["p95_ms"], search["p99_ms"]), (30.0, 30.0, 30.0))
+ self.assertEqual(search["p95_change_pct"], 200.0)
+
+ def test_silent_row_has_no_current_metrics(self):
+ dispatch = self._by_service(self._run())["dispatch"]
+ self.assertEqual(dispatch["requests"], 0)
+ self.assertEqual(dispatch["baseline_requests"], 100)
+ self.assertIsNone(dispatch["error_rate_pct"])
+ self.assertIsNone(dispatch["p95_ms"])
+ self.assertTrue(dispatch["reason"].startswith("no spans since 07:"), dispatch["reason"])
+
+ def test_entry_scope_skips_internal_spans(self):
+ entry = self._by_service(self._run())["api"]
+ all_spans = self._by_service(self._run(span_scope=ServiceHealthSpanScope.ALL))["api"]
+ self.assertEqual((entry["requests"], entry["error_count"]), (200, 0))
+ self.assertEqual((all_spans["requests"], all_spans["error_count"]), (400, 200))
+ self.assertEqual(all_spans["status"], ServiceHealthStatus.REGRESSED)
+
+ def test_environment_uses_the_fallback_chain(self):
+ rows = self._by_service(self._run(environment="Staging"))
+ self.assertEqual(set(rows), {"api", "stage-svc"})
+ self.assertEqual(rows["api"]["requests"], 40)
+ self.assertEqual(rows["stage-svc"]["requests"], 30)
+
+ def test_no_environment_counts_every_label(self):
+ params = ServiceHealthParams(date_from="2026-06-02T08:00:00Z", date_to="2026-06-02T09:00:00Z")
+ rows = self._by_service(ServiceHealthQueryRunner(team=self.team, params=params).run())
+ self.assertEqual(rows["api"]["requests"], 240)
+
+ def test_service_names_filter(self):
+ result = self._run(service_names=("worker", "dispatch"))
+ self.assertEqual({row["service"] for row in result["services"]}, {"worker", "dispatch"})
+
+ def test_previous_day_compare(self):
+ rows = self._by_service(self._run(compare=ServiceHealthCompare.PREVIOUS_DAY, limit=100))
+ self.assertEqual(rows["nightly"]["status"], ServiceHealthStatus.SILENT)
+ self.assertEqual(rows["api"]["status"], ServiceHealthStatus.NEW)
+
+ def test_custom_compare(self):
+ rows = self._by_service(
+ self._run(
+ compare=ServiceHealthCompare.CUSTOM,
+ compare_date_from="2026-06-01T08:00:00Z",
+ compare_date_to="2026-06-01T09:00:00Z",
+ limit=100,
+ )
+ )
+ self.assertEqual(rows["nightly"]["status"], ServiceHealthStatus.SILENT)
+
+ def test_no_compare(self):
+ result = self._run(compare=ServiceHealthCompare.NONE, limit=100)
+ rows = self._by_service(result)
+ self.assertNotIn("dispatch", rows)
+ self.assertEqual(rows["billing"]["status"], ServiceHealthStatus.OK)
+ self.assertIsNone(rows["worker"]["error_rate_delta_pp"])
+ self.assertIsNone(result["window"]["baseline"])
+ self.assertTrue(any("compare" in note for note in result["notes"]))
+
+ def test_order_by_status_puts_the_worst_first(self):
+ statuses = [row["status"] for row in self._run(limit=100)["services"]]
+ self.assertEqual(statuses[:4], [ServiceHealthStatus.SILENT] * 2 + [ServiceHealthStatus.REGRESSED] * 2)
+ self.assertEqual(statuses[-1], ServiceHealthStatus.OK)
+
+ @parameterized.expand(
+ [
+ (ServiceHealthOrderBy.ERROR_RATE, "mailer"),
+ (ServiceHealthOrderBy.ERROR_COUNT, "mailer"),
+ (ServiceHealthOrderBy.P95, "search"),
+ ]
+ )
+ def test_order_by_metric(self, order_by, expected_first):
+ self.assertEqual(self._run(order_by=order_by, limit=100)["services"][0]["service"], expected_first)
+
+ def test_order_by_throughput(self):
+ rows = self._run(order_by=ServiceHealthOrderBy.THROUGHPUT, limit=100)["services"]
+ requests = [row["requests"] for row in rows]
+ self.assertEqual(requests, sorted(requests, reverse=True))
+
+ def test_limit_truncates_and_summary_counts_everything(self):
+ result = self._run(limit=2)
+ self.assertEqual(len(result["services"]), 2)
+ self.assertTrue(result["truncated"])
+ self.assertEqual(result["summary"]["services"], 9)
+ self.assertEqual(result["summary"]["silent"], 2)
+ self.assertEqual(result["summary"]["regressed"], 2)
+ self.assertEqual(result["summary"]["degraded"], 1)
+ self.assertEqual(result["summary"]["new"], 1)
+
+ def test_next_points_at_the_drill_down(self):
+ result = self._run(environment=None)
+ calls = {call["tool"]: call for call in result["next"]}
+ self.assertEqual(set(calls), {"apm-spans-aggregate", "query-apm-spans"})
+ aggregate = calls["apm-spans-aggregate"]
+ self.assertEqual(aggregate["query"]["serviceNames"], ["search", "worker", "mailer"])
+ self.assertEqual(
+ aggregate["query"]["dateRange"], {"date_from": "2026-06-02T08:00:00Z", "date_to": "2026-06-02T09:00:00Z"}
+ )
+ self.assertEqual(aggregate["query"]["compareFilter"], {"compare": True})
+ error_spans = calls["query-apm-spans"]["query"]
+ self.assertEqual((error_spans["serviceNames"], error_spans["statusCodes"]), (["worker"], [2]))
+ self.assertEqual((error_spans["flatSpans"], error_spans["rootSpans"]), (True, False))
+ for call in result["next"]:
+ self.assertLessEqual(len(call["why"]), MAX_REASON_LENGTH)
+ self.assertEqual(self._run()["next"], [])
+
+ def test_a_truncated_scan_does_not_invent_silent_or_new_services(self):
+ with patch("products.tracing.backend.service_health_query_runner.SERVICE_SCAN_LIMIT", 3):
+ result = self._run()
+ statuses = {row["status"] for row in result["services"]}
+ self.assertFalse(statuses & {ServiceHealthStatus.SILENT, ServiceHealthStatus.NEW}, statuses)
+ self.assertTrue(result["truncated"])
+
+ def test_runs_with_fail_fast_settings(self):
+ with patch(
+ "products.tracing.backend.service_health_query_runner.execute_hogql_query",
+ side_effect=lambda **kwargs: type("R", (), {"results": []})(),
+ ) as execute:
+ self._run()
+ settings = execute.call_args.kwargs["settings"]
+ self.assertEqual(settings.max_execution_time, 30)
+ self.assertEqual(settings.max_bytes_to_read, 10_000_000_000)
+ self.assertEqual(settings.read_overflow_mode, "throw")
diff --git a/products/tracing/frontend/generated/api.schemas.ts b/products/tracing/frontend/generated/api.schemas.ts
index ad5782e73adc..3923e3a14a74 100644
--- a/products/tracing/frontend/generated/api.schemas.ts
+++ b/products/tracing/frontend/generated/api.schemas.ts
@@ -738,6 +738,262 @@ export interface _TracingServiceNamesResponseApi {
results: _TracingServiceNameApi[]
}
+export interface _ServiceHealthDateRangeApi {
+ /**
+ * Relative (-30m, -6h, -7d) or ISO 8601.
+ * @nullable
+ */
+ date_from?: string | null
+ /**
+ * Same format. Omit or null for "now".
+ * @nullable
+ */
+ date_to?: string | null
+}
+
+/**
+ * * `none` - None
+ * * `previous_period` - Previous Period
+ * * `previous_day` - Previous Day
+ * * `previous_week` - Previous Week
+ */
+export type ServiceHealthBaselineEnumApi =
+ (typeof ServiceHealthBaselineEnumApi)[keyof typeof ServiceHealthBaselineEnumApi]
+
+export const ServiceHealthBaselineEnumApi = {
+ None: 'none',
+ PreviousPeriod: 'previous_period',
+ PreviousDay: 'previous_day',
+ PreviousWeek: 'previous_week',
+} as const
+
+/**
+ * * `entry` - Entry
+ * * `all` - All
+ */
+export type ServiceHealthSpanScopeEnumApi =
+ (typeof ServiceHealthSpanScopeEnumApi)[keyof typeof ServiceHealthSpanScopeEnumApi]
+
+export const ServiceHealthSpanScopeEnumApi = {
+ Entry: 'entry',
+ All: 'all',
+} as const
+
+/**
+ * * `status` - Status
+ * * `error_rate` - Error Rate
+ * * `p95` - P95
+ * * `throughput` - Throughput
+ * * `error_count` - Error Count
+ */
+export type ServiceHealthOrderByEnumApi = (typeof ServiceHealthOrderByEnumApi)[keyof typeof ServiceHealthOrderByEnumApi]
+
+export const ServiceHealthOrderByEnumApi = {
+ Status: 'status',
+ ErrorRate: 'error_rate',
+ P95: 'p95',
+ Throughput: 'throughput',
+ ErrorCount: 'error_count',
+} as const
+
+export interface _ServiceHealthQueryApi {
+ /** The window to score. Defaults to the last hour. */
+ dateRange?: _ServiceHealthDateRangeApi
+ /** The baseline. previous_period (default) is the same length just before the window. previous_day and previous_week shift the window back 1 or 7 days. none skips it. baselineDateRange overrides it.
+ *
+ * * `none` - None
+ * * `previous_period` - Previous Period
+ * * `previous_day` - Previous Day
+ * * `previous_week` - Previous Week */
+ compare?: ServiceHealthBaselineEnumApi
+ /**
+ * Only this environment, matched without case on deployment.environment.name, then deployment.environment, then env.
+ * @nullable
+ */
+ environment?: string | null
+ /**
+ * Only these services.
+ * @maxItems 100
+ */
+ serviceNames?: string[]
+ /** entry (default) counts root, server and consumer spans, one per request. all counts every span.
+ *
+ * * `entry` - Entry
+ * * `all` - All */
+ spanScope?: ServiceHealthSpanScopeEnumApi
+ /** status (default) puts silent, regressed and degraded first. The others sort high to low.
+ *
+ * * `status` - Status
+ * * `error_rate` - Error Rate
+ * * `p95` - P95
+ * * `throughput` - Throughput
+ * * `error_count` - Error Count */
+ orderBy?: ServiceHealthOrderByEnumApi
+ /**
+ * Max service rows, 1-200.
+ * @minimum 1
+ * @maximum 200
+ */
+ limit?: number
+}
+
+export interface ServiceHealthRequestApi {
+ /** The service health query. Every field is optional. */
+ query: _ServiceHealthQueryApi
+ /** An explicit baseline window. Overrides query.compare. */
+ baselineDateRange?: _ServiceHealthDateRangeApi
+}
+
+export interface _ServiceHealthTimeRangeApi {
+ /** Start, ISO 8601 UTC, inclusive. */
+ date_from: string
+ /** End, ISO 8601 UTC, exclusive. */
+ date_to: string
+}
+
+export interface _ServiceHealthWindowApi {
+ /** The scored window. Reuse it as dateRange in follow-up calls. */
+ current: _ServiceHealthTimeRangeApi
+ /** The baseline window. Null when compare is none. */
+ baseline: _ServiceHealthTimeRangeApi | null
+ /** Always UTC. */
+ timezone: string
+}
+
+export interface _ServiceHealthSummaryApi {
+ /** Services scored, before the limit. */
+ services: number
+ /** Requests over all scored services. */
+ requests: number
+ /** Services per status. */
+ ok: number
+ /** Services per status. */
+ degraded: number
+ /** Services per status. */
+ regressed: number
+ /** Services per status. */
+ silent: number
+ /** Services per status. */
+ new: number
+ /** Services per status. */
+ recovered: number
+ /** Services per status. */
+ improved: number
+}
+
+/**
+ * * `silent` - Silent
+ * * `regressed` - Regressed
+ * * `degraded` - Degraded
+ * * `new` - New
+ * * `recovered` - Recovered
+ * * `improved` - Improved
+ * * `ok` - Ok
+ */
+export type ServiceHealthStatusEnumApi = (typeof ServiceHealthStatusEnumApi)[keyof typeof ServiceHealthStatusEnumApi]
+
+export const ServiceHealthStatusEnumApi = {
+ Silent: 'silent',
+ Regressed: 'regressed',
+ Degraded: 'degraded',
+ New: 'new',
+ Recovered: 'recovered',
+ Improved: 'improved',
+ Ok: 'ok',
+} as const
+
+export interface _ServiceHealthRowApi {
+ /** Service name. */
+ service: string
+ /** silent: spans stopped. regressed: error rate or p95 worse than the baseline. degraded: error rate 5% or more. new: no spans in the baseline. recovered, improved: the reverse of regressed.
+ *
+ * * `silent` - Silent
+ * * `regressed` - Regressed
+ * * `degraded` - Degraded
+ * * `new` - New
+ * * `recovered` - Recovered
+ * * `improved` - Improved
+ * * `ok` - Ok */
+ status: ServiceHealthStatusEnumApi
+ /** Why the service has this status. Empty for ok. */
+ reason: string
+ /** Spans in the window, in the chosen spanScope. */
+ requests: number
+ /** Requests per minute. */
+ throughput_per_min: number
+ /** Requests with OTel status Error. */
+ error_count: number
+ /**
+ * Error share of requests, percent.
+ * @nullable
+ */
+ error_rate_pct: number | null
+ /**
+ * Change from the baseline, points.
+ * @nullable
+ */
+ error_rate_delta_pp: number | null
+ /**
+ * Median duration, ms.
+ * @nullable
+ */
+ p50_ms: number | null
+ /**
+ * 95th percentile duration, ms.
+ * @nullable
+ */
+ p95_ms: number | null
+ /**
+ * p95 change from the baseline, percent.
+ * @nullable
+ */
+ p95_change_pct: number | null
+ /**
+ * 99th percentile duration, ms.
+ * @nullable
+ */
+ p99_ms: number | null
+ /**
+ * Baseline requests. Null without one.
+ * @nullable
+ */
+ baseline_requests: number | null
+ /**
+ * Last span, ISO 8601 UTC.
+ * @nullable
+ */
+ last_seen: string | null
+}
+
+/**
+ * The prefilled query argument.
+ */
+export type _ServiceHealthNextCallApiQuery = { [key: string]: unknown }
+
+export interface _ServiceHealthNextCallApi {
+ /** The tool to call next. */
+ tool: string
+ /** What the call answers. */
+ why: string
+ /** The prefilled query argument. */
+ query: _ServiceHealthNextCallApiQuery
+}
+
+export interface ServiceHealthResponseApi {
+ /** The absolute UTC windows the server used. */
+ window: _ServiceHealthWindowApi
+ /** Counts over all scored services. */
+ summary: _ServiceHealthSummaryApi
+ /** One row per service, in orderBy order. */
+ services: _ServiceHealthRowApi[]
+ /** Prefilled follow-up calls. */
+ next: _ServiceHealthNextCallApi[]
+ /** True when more services matched than limit. */
+ truncated: boolean
+ /** Caveats about this result. */
+ notes: string[]
+}
+
export interface _TracingSparklineQueryBodyApi {
/** Date range for the query. Defaults to last hour. */
dateRange?: _TracingDateRangeApi
@@ -1248,6 +1504,21 @@ export type TracingSpansAttributesRetrieveParams = {
* @minLength 1
*/
attribute_type?: TracingSpansAttributesRetrieveAttributeType
+ /**
+ * JSON-encoded date range object, e.g. {"date_from": "-1h"}. Malformed JSON returns 400. Or pass date_from and date_to.
+ * @minLength 1
+ */
+ dateRange?: string
+ /**
+ * Range start: -1h, -7d or ISO 8601. Defaults to -1h.
+ * @minLength 1
+ */
+ date_from?: string
+ /**
+ * Range end, same format. Omit for "now".
+ * @minLength 1
+ */
+ date_to?: string
/**
* Max results (default: 100).
* @minimum 1
@@ -1280,10 +1551,20 @@ export const TracingSpansAttributesRetrieveAttributeType = {
export type TracingSpansServiceNamesRetrieveParams = {
/**
- * JSON-encoded date range, e.g. '{"date_from": "-1h"}'.
+ * JSON-encoded date range object, e.g. {"date_from": "-1h"}. Malformed JSON returns 400. Or pass date_from and date_to.
* @minLength 1
*/
dateRange?: string
+ /**
+ * Range start: -1h, -7d or ISO 8601. Defaults to -1h.
+ * @minLength 1
+ */
+ date_from?: string
+ /**
+ * Range end, same format. Omit for "now".
+ * @minLength 1
+ */
+ date_to?: string
/**
* Search filter for service names.
* @minLength 1
@@ -1301,6 +1582,21 @@ export type TracingSpansValuesRetrieveParams = {
* @minLength 1
*/
attribute_type?: TracingSpansValuesRetrieveAttributeType
+ /**
+ * JSON-encoded date range object, e.g. {"date_from": "-1h"}. Malformed JSON returns 400. Or pass date_from and date_to.
+ * @minLength 1
+ */
+ dateRange?: string
+ /**
+ * Range start: -1h, -7d or ISO 8601. Defaults to -1h.
+ * @minLength 1
+ */
+ date_from?: string
+ /**
+ * Range end, same format. Omit for "now".
+ * @minLength 1
+ */
+ date_to?: string
/**
* The attribute key to get values for.
* @minLength 1
diff --git a/products/tracing/frontend/generated/api.ts b/products/tracing/frontend/generated/api.ts
index bb3b3372e4b6..ae4862a14c7c 100644
--- a/products/tracing/frontend/generated/api.ts
+++ b/products/tracing/frontend/generated/api.ts
@@ -17,6 +17,8 @@ import type {
PatchedTeamTracingConfigApi,
PatchedTracesRetentionRuleApi,
PatchedTracingViewApi,
+ ServiceHealthRequestApi,
+ ServiceHealthResponseApi,
TeamTracingConfigApi,
TracesRetentionRuleApi,
TracingRetentionRulesListParams,
@@ -539,6 +541,23 @@ export const tracingSpansServiceNamesRetrieve = async (
})
}
+export const getTracingSpansServiceHealthCreateUrl = (projectId: string) => {
+ return `/api/projects/${projectId}/tracing/spans/service_health/`
+}
+
+export const tracingSpansServiceHealthCreate = async (
+ projectId: string,
+ serviceHealthRequestApi: ServiceHealthRequestApi,
+ options?: RequestInit
+): Promise<ServiceHealthResponseApi> => {
+ return apiMutator<ServiceHealthResponseApi>(getTracingSpansServiceHealthCreateUrl(projectId), {
+ ...options,
+ method: 'POST',
+ headers: { 'Content-Type': 'application/json', ...options?.headers },
+ body: JSON.stringify(serviceHealthRequestApi),
+ })
+}
+
export const getTracingSpansSparklineCreateUrl = (projectId: string) => {
return `/api/projects/${projectId}/tracing/spans/sparkline/`
}
diff --git a/products/tracing/frontend/generated/api.zod.ts b/products/tracing/frontend/generated/api.zod.ts
index afa17ebe7542..1496c8b1913a 100644
--- a/products/tracing/frontend/generated/api.zod.ts
+++ b/products/tracing/frontend/generated/api.zod.ts
@@ -1045,6 +1045,77 @@ export const TracingSpansQueryCreateBody = /* @__PURE__ */ zod.object({
.describe('The tracing spans query to execute.'),
})
+export const tracingSpansServiceHealthCreateBodyQueryOneCompareDefault = `previous_period`
+export const tracingSpansServiceHealthCreateBodyQueryOneServiceNamesMax = 100
+
+export const tracingSpansServiceHealthCreateBodyQueryOneSpanScopeDefault = `entry`
+export const tracingSpansServiceHealthCreateBodyQueryOneOrderByDefault = `status`
+export const tracingSpansServiceHealthCreateBodyQueryOneLimitDefault = 25
+export const tracingSpansServiceHealthCreateBodyQueryOneLimitMax = 200
+
+export const TracingSpansServiceHealthCreateBody = /* @__PURE__ */ zod.object({
+ query: zod
+ .object({
+ dateRange: zod
+ .object({
+ date_from: zod.string().nullish().describe('Relative (-30m, -6h, -7d) or ISO 8601.'),
+ date_to: zod.string().nullish().describe('Same format. Omit or null for \"now\".'),
+ })
+ .optional()
+ .describe('The window to score. Defaults to the last hour.'),
+ compare: zod
+ .enum(['none', 'previous_period', 'previous_day', 'previous_week'])
+ .describe(
+ '\* `none` - None\n\* `previous_period` - Previous Period\n\* `previous_day` - Previous Day\n\* `previous_week` - Previous Week'
+ )
+ .default(tracingSpansServiceHealthCreateBodyQueryOneCompareDefault)
+ .describe(
+ 'The baseline. previous_period (default) is the same length just before the window. previous_day and previous_week shift the window back 1 or 7 days. none skips it. baselineDateRange overrides it.\n\n\* `none` - None\n\* `previous_period` - Previous Period\n\* `previous_day` - Previous Day\n\* `previous_week` - Previous Week'
+ ),
+ environment: zod
+ .string()
+ .nullish()
+ .describe(
+ 'Only this environment, matched without case on deployment.environment.name, then deployment.environment, then env.'
+ ),
+ serviceNames: zod
+ .array(zod.string())
+ .max(tracingSpansServiceHealthCreateBodyQueryOneServiceNamesMax)
+ .optional()
+ .describe('Only these services.'),
+ spanScope: zod
+ .enum(['entry', 'all'])
+ .describe('\* `entry` - Entry\n\* `all` - All')
+ .default(tracingSpansServiceHealthCreateBodyQueryOneSpanScopeDefault)
+ .describe(
+ 'entry (default) counts root, server and consumer spans, one per request. all counts every span.\n\n\* `entry` - Entry\n\* `all` - All'
+ ),
+ orderBy: zod
+ .enum(['status', 'error_rate', 'p95', 'throughput', 'error_count'])
+ .describe(
+ '\* `status` - Status\n\* `error_rate` - Error Rate\n\* `p95` - P95\n\* `throughput` - Throughput\n\* `error_count` - Error Count'
+ )
+ .default(tracingSpansServiceHealthCreateBodyQueryOneOrderByDefault)
+ .describe(
+ 'status (default) puts silent, regressed and degraded first. The others sort high to low.\n\n\* `status` - Status\n\* `error_rate` - Error Rate\n\* `p95` - P95\n\* `throughput` - Throughput\n\* `error_count` - Error Count'
+ ),
+ limit: zod
+ .number()
+ .min(1)
+ .max(tracingSpansServiceHealthCreateBodyQueryOneLimitMax)
+ .default(tracingSpansServiceHealthCreateBodyQueryOneLimitDefault)
+ .describe('Max service rows, 1-200.'),
+ })
+ .describe('The service health query. Every field is optional.'),
+ baselineDateRange: zod
+ .object({
+ date_from: zod.string().nullish().describe('Relative (-30m, -6h, -7d) or ISO 8601.'),
+ date_to: zod.string().nullish().describe('Same format. Omit or null for \"now\".'),
+ })
+ .optional()
+ .describe('An explicit baseline window. Overrides query.compare.'),
+})
+
export const tracingSpansSparklineCreateBodyQueryOneFilterGroupDefault = []
export const tracingSpansSparklineCreateBodyQueryOneRootSpansDefault = false
diff --git a/products/tracing/mcp/prompts/apm-service-health.md b/products/tracing/mcp/prompts/apm-service-health.md
new file mode 100644
index 000000000000..21b93201da23
--- /dev/null
+++ b/products/tracing/mcp/prompts/apm-service-health.md
@@ -0,0 +1,18 @@
+Score every service in one call: requests, throughput, error rate, p50/p95/p99 and a status, compared with a baseline window.
+
+Use this first for "how healthy are my services" or a daily report.
+Use `apm-spans-aggregate` next for the operations inside one service. Use `apm-spans-count` only for a raw count.
+
+All parameters go inside `query`, and every one is optional:
+
+```json
+{ "query": { "dateRange": { "date_from": "-24h" }, "compare": "previous_day", "environment": "production" } }
+```
+
+Each row has a `status`: `silent` (spans stopped), `regressed` (error rate or p95 got worse), `degraded` (error rate 5% or more), `new`, `recovered`, `improved` or `ok`, and a short `reason`.
+Rows come worst first.
+
+`window` gives the absolute UTC times the server used. Reuse `window.current` as `dateRange` in follow-up calls.
+`next` holds prefilled follow-up calls for regressed and degraded services. It is empty when `environment` is set, because those tools cannot repeat the environment fallback.
+`spanScope: "entry"` (default) counts one span per request. Use `"all"` to count every span.
+A 400 that names the 10 GB cap means the window is too wide. Use the window it suggests, or add `serviceNames`.
diff --git a/products/tracing/mcp/prompts/apm-spans-count.md b/products/tracing/mcp/prompts/apm-spans-count.md
index 809065db32c4..db352201b39d 100644
--- a/products/tracing/mcp/prompts/apm-spans-count.md
+++ b/products/tracing/mcp/prompts/apm-spans-count.md
@@ -12,7 +12,7 @@ All parameters go inside `query` — top-level fields are rejected:
- When the user asks "how many X spans are there?" and you don't need to see individual spans.
- To check whether a filter combination matches anything at all before committing to a full query.
-This counts **spans**, not traces. A single trace contains many spans, so a count of matching spans will exceed the number of matching traces.
+`count` is the number of matching **spans**. `traceCount` is the number of distinct traces whose root span matches. A trace contains many spans, so `count` is usually larger.
If the count would scan too much data (a wide date range with no filters), the tool returns a 400 asking you to narrow the window or add filters — narrow the `dateRange` or add `serviceNames` / `statusCodes` / `filterGroup`, then retry.
diff --git a/products/tracing/mcp/tools.yaml b/products/tracing/mcp/tools.yaml
index af54ba930202..f19af110ea47 100644
--- a/products/tracing/mcp/tools.yaml
+++ b/products/tracing/mcp/tools.yaml
@@ -68,6 +68,18 @@ tools:
include:
- results
- count
+ apm-service-health:
+ operation: tracing_spans_service_health_create
+ enabled: true
+ scopes:
+ - tracing:read
+ annotations:
+ readOnly: true
+ destructive: false
+ idempotent: true
+ title: Service health against a baseline
+ description_file: ./prompts/apm-service-health.md
+ feature_flag: tracing
apm-services-list:
operation: tracing_spans_service_names_retrieve
enabled: true
@@ -121,6 +133,7 @@ tools:
response:
include:
- count
+ - traceCount
apm-spans-duration-histogram:
operation: tracing_spans_duration_histogram_create
enabled: true
diff --git a/products/tracing/skills/exploring-apm-traces/SKILL.md b/products/tracing/skills/exploring-apm-traces/SKILL.md
index 404c3df993b2..6e3967d21b36 100644
--- a/products/tracing/skills/exploring-apm-traces/SKILL.md
+++ b/products/tracing/skills/exploring-apm-traces/SKILL.md
@@ -4,7 +4,7 @@ description: >
Investigates distributed application performance using PostHog APM (OpenTelemetry span) data via MCP.
Use when the user asks about service traces, slow HTTP/database spans, error spans, error-rate trends or
spikes, latency distributions, trace IDs, or span attributes — not AI observability traces or product logs.
- Uses posthog:query-apm-spans, posthog:apm-trace-get, posthog:apm-spans-sparkline,
+ Uses posthog:apm-service-health, posthog:query-apm-spans, posthog:apm-trace-get, posthog:apm-spans-sparkline,
posthog:apm-services-list, posthog:apm-attributes-list, and posthog:apm-attribute-values-list.
---
@@ -22,6 +22,7 @@ When the question asks for SLO burn, call `posthog:metric-list` before the APM t
| Tool | Purpose |
| -------------------------------------- | ------------------------------------------------- |
+| `posthog:apm-service-health` | Status per service against a baseline: start here |
| `posthog:query-apm-spans` | Search and filter spans (compact list view) |
| `posthog:apm-trace-get` | Get the full span list for one hex `trace_id` |
| `posthog:apm-spans-aggregate` | Per-operation aggregates (count, p50/p95, errors) |
@@ -99,6 +100,19 @@ To rebuild the tree:
## Investigation patterns
+### "How healthy are my services?" (start here)
+
+1. `apm-service-health` scores every service in one call: requests, error rate, p50/p95/p99, the change from a baseline window, and a `status` (`silent`, `regressed`, `degraded`, `new`, `recovered`, `improved`, `ok`) with a `reason`. Rows come worst first.
+2. Pick the baseline with `compare`: `previous_period` (default), `previous_day`, `previous_week`, or an explicit `baselineDateRange` next to `query`. Scope with `environment` and `serviceNames`.
+3. Run the calls in `next`. They are prefilled: `apm-spans-aggregate` for the operations behind the regressed and degraded services, `apm-spans-sparkline` for when a silent service stopped, `query-apm-spans` for error spans.
+4. Do not build a service summary from `apm-spans-aggregate` rows or from one `apm-spans-count` call per service. Percentiles of operation rows do not add up to a service percentile.
+
+#### Daily service health report
+
+1. `apm-service-health` with `dateRange: {"date_from": "-24h"}`, and `compare: "previous_day"`. Follow its `next` calls to the operations behind each unhealthy service.
+2. `logs-patterns-diff` for new error patterns in the services that are not `ok`.
+3. `logs-alerts-list` for alerts that fired in the window.
+
### "Where is time going?"
1. Every span from `apm-trace-get` carries `self_time_nano` — duration not covered by children. Sort by it: the top span is where wall-clock actually went. A parent with large `self_time_nano` is an **uninstrumented gap** (the work happened inside it, not in any recorded child).
@@ -116,7 +130,7 @@ To rebuild the tree:
### "Did the request hit service X?"
1. Run `print_summary.py` — it prints the set of services involved in the trace.
-2. If service X is missing, the request never reached it (or instrumentation is missing — check `apm-services-list` to confirm X has emitted spans recently at all).
+2. If service X is missing, the request never reached it (or instrumentation is missing — check `apm-service-health` with `serviceNames: ["X"]` to see whether X emitted spans recently, and when it stopped).
### "What's different about the bad spans?" (over-represented values)
diff --git a/services/mcp/schema/generated-tool-definitions.json b/services/mcp/schema/generated-tool-definitions.json
index 2cb99fa190f0..9cf118a19357 100644
--- a/services/mcp/schema/generated-tool-definitions.json
+++ b/services/mcp/schema/generated-tool-definitions.json
@@ -941,6 +941,21 @@
},
"feature_flag": "tracing"
},
+ "apm-service-health": {
+ "description": "Score every service in one call: requests, throughput, error rate, p50/p95/p99 and a status, compared with a baseline window.\n\nUse this first for \"how healthy are my services\" or a daily report.\nUse `apm-spans-aggregate` next for the operations inside one service. Use `apm-spans-count` only for a raw count.\n\nAll parameters go inside `query`, and every one is optional:\n\n```json\n{ \"query\": { \"dateRange\": { \"date_from\": \"-24h\" }, \"compare\": \"previous_day\", \"environment\": \"production\" } }\n```\n\nEach row has a `status`: `silent` (spans stopped), `regressed` (error rate or p95 got worse), `degraded` (error rate 5% or more), `new`, `recovered`, `improved` or `ok`, and a short `reason`.\nRows come worst first.\n\n`window` gives the absolute UTC times the server used. Reuse `window.current` as `dateRange` in follow-up calls.\n`next` holds prefilled follow-up calls for regressed and degraded services. It is empty when `environment` is set, because those tools cannot repeat the environment fallback.\n`spanScope: \"entry\"` (default) counts one span per request. Use `\"all\"` to count every span.\nA 400 that names the 10 GB cap means the window is too wide. Use the window it suggests, or add `serviceNames`.",
+ "category": "Tracing",
+ "feature": "tracing",
+ "summary": "Service health against a baseline",
+ "title": "Service health against a baseline",
+ "required_scopes": ["tracing:read"],
+ "annotations": {
+ "destructiveHint": false,
+ "idempotentHint": true,
+ "openWorldHint": true,
+ "readOnlyHint": true
+ },
+ "feature_flag": "tracing"
+ },
"apm-services-list": {
"description": "List distinct service names that have emitted trace spans. Use to discover available services before filtering spans by service name.",
"category": "Tracing",
@@ -972,7 +987,7 @@
"feature_flag": "tracing"
},
"apm-spans-count": {
- "description": "Return a scalar count of trace spans matching a filter set. Use this as a cheap pre-flight before `query-apm-spans` — if the count is large, narrow the filters (or set `excludeAttributes`) before pulling rows.\n\nAll parameters go inside `query` — top-level fields are rejected:\n\n```json\n{ \"query\": { \"serviceNames\": [\"api\"], \"dateRange\": { \"date_from\": \"-1h\" } } }\n```\n\n# When to use\n\n- Before `query-apm-spans`, to confirm the filter set returns a tractable number of spans rather than pulling rows blind.\n- When the user asks \"how many X spans are there?\" and you don't need to see individual spans.\n- To check whether a filter combination matches anything at all before committing to a full query.\n\nThis counts **spans**, not traces. A single trace contains many spans, so a count of matching spans will exceed the number of matching traces.\n\nIf the count would scan too much data (a wide date range with no filters), the tool returns a 400 asking you to narrow the window or add filters — narrow the `dateRange` or add `serviceNames` / `statusCodes` / `filterGroup`, then retry.\n\n# Parameters\n\n## query.dateRange\n\nDate range for the count. Defaults to the last hour (`-1h`).\n\n- `date_from`: Start of the range. Accepts ISO 8601 timestamps or relative formats: `-1h`, `-6h`, `-1d`, `-7d`, `-30d`.\n- `date_to`: End of the range. Same format. Omit or null for \"now\".\n\n## query.serviceNames\n\nFilter by service names. Unlike `query-apm-spans`, an unfiltered count is cheap and useful for sizing up a filter before narrowing. Use `apm-services-list` to discover services.\n\n## query.statusCodes\n\nFilter by OTel span status codes (list of integers: `0` Unset, `1` OK, `2` Error) — **not** HTTP status codes. Use `[2]` to select error spans.\n\n## query.filterGroup\n\nProperty filters to narrow the count. Same format as `query-apm-spans` filters — each filter specifies `key`, `operator`, `type` (span/span_attribute/span_resource_attribute), and optionally `value`.\n\nA `span` filter on `duration` takes a value in milliseconds. For example, `{ \"key\": \"duration\", \"operator\": \"gt\", \"type\": \"span\", \"value\": \"30000\" }` counts spans longer than 30 seconds.\n\n# Examples\n\n## Count error spans in a service over the last day\n\n```json\n{\n \"query\": {\n \"serviceNames\": [\"api-gateway\"],\n \"statusCodes\": [2],\n \"dateRange\": { \"date_from\": \"-1d\" }\n }\n}\n```\n\n## Count how many spans match a name before fetching them\n\n```json\n{\n \"query\": {\n \"filterGroup\": [{ \"key\": \"name\", \"operator\": \"exact\", \"type\": \"span\", \"value\": \"redis_cluster.discovery\" }],\n \"dateRange\": { \"date_from\": \"-6h\" }\n }\n}\n```",
+ "description": "Return a scalar count of trace spans matching a filter set. Use this as a cheap pre-flight before `query-apm-spans` — if the count is large, narrow the filters (or set `excludeAttributes`) before pulling rows.\n\nAll parameters go inside `query` — top-level fields are rejected:\n\n```json\n{ \"query\": { \"serviceNames\": [\"api\"], \"dateRange\": { \"date_from\": \"-1h\" } } }\n```\n\n# When to use\n\n- Before `query-apm-spans`, to confirm the filter set returns a tractable number of spans rather than pulling rows blind.\n- When the user asks \"how many X spans are there?\" and you don't need to see individual spans.\n- To check whether a filter combination matches anything at all before committing to a full query.\n\n`count` is the number of matching **spans**. `traceCount` is the number of distinct traces whose root span matches. A trace contains many spans, so `count` is usually larger.\n\nIf the count would scan too much data (a wide date range with no filters), the tool returns a 400 asking you to narrow the window or add filters — narrow the `dateRange` or add `serviceNames` / `statusCodes` / `filterGroup`, then retry.\n\n# Parameters\n\n## query.dateRange\n\nDate range for the count. Defaults to the last hour (`-1h`).\n\n- `date_from`: Start of the range. Accepts ISO 8601 timestamps or relative formats: `-1h`, `-6h`, `-1d`, `-7d`, `-30d`.\n- `date_to`: End of the range. Same format. Omit or null for \"now\".\n\n## query.serviceNames\n\nFilter by service names. Unlike `query-apm-spans`, an unfiltered count is cheap and useful for sizing up a filter before narrowing. Use `apm-services-list` to discover services.\n\n## query.statusCodes\n\nFilter by OTel span status codes (list of integers: `0` Unset, `1` OK, `2` Error) — **not** HTTP status codes. Use `[2]` to select error spans.\n\n## query.filterGroup\n\nProperty filters to narrow the count. Same format as `query-apm-spans` filters — each filter specifies `key`, `operator`, `type` (span/span_attribute/span_resource_attribute), and optionally `value`.\n\nA `span` filter on `duration` takes a value in milliseconds. For example, `{ \"key\": \"duration\", \"operator\": \"gt\", \"type\": \"span\", \"value\": \"30000\" }` counts spans longer than 30 seconds.\n\n# Examples\n\n## Count error spans in a service over the last day\n\n```json\n{\n \"query\": {\n \"serviceNames\": [\"api-gateway\"],\n \"statusCodes\": [2],\n \"dateRange\": { \"date_from\": \"-1d\" }\n }\n}\n```\n\n## Count how many spans match a name before fetching them\n\n```json\n{\n \"query\": {\n \"filterGroup\": [{ \"key\": \"name\", \"operator\": \"exact\", \"type\": \"span\", \"value\": \"redis_cluster.discovery\" }],\n \"dateRange\": { \"date_from\": \"-6h\" }\n }\n}\n```",
"category": "Tracing",
"feature": "tracing",
"summary": "Count spans",
diff --git a/services/mcp/schema/tool-definitions-all.json b/services/mcp/schema/tool-definitions-all.json
index dd7c19b0b216..06d4a6fa02b4 100644
--- a/services/mcp/schema/tool-definitions-all.json
+++ b/services/mcp/schema/tool-definitions-all.json
@@ -956,6 +956,21 @@
},
"feature_flag": "tracing"
},
+ "apm-service-health": {
+ "description": "Score every service in one call: requests, throughput, error rate, p50/p95/p99 and a status, compared with a baseline window.\n\nUse this first for \"how healthy are my services\" or a daily report.\nUse `apm-spans-aggregate` next for the operations inside one service. Use `apm-spans-count` only for a raw count.\n\nAll parameters go inside `query`, and every one is optional:\n\n```json\n{ \"query\": { \"dateRange\": { \"date_from\": \"-24h\" }, \"compare\": \"previous_day\", \"environment\": \"production\" } }\n```\n\nEach row has a `status`: `silent` (spans stopped), `regressed` (error rate or p95 got worse), `degraded` (error rate 5% or more), `new`, `recovered`, `improved` or `ok`, and a short `reason`.\nRows come worst first.\n\n`window` gives the absolute UTC times the server used. Reuse `window.current` as `dateRange` in follow-up calls.\n`next` holds prefilled follow-up calls for regressed and degraded services. It is empty when `environment` is set, because those tools cannot repeat the environment fallback.\n`spanScope: \"entry\"` (default) counts one span per request. Use `\"all\"` to count every span.\nA 400 that names the 10 GB cap means the window is too wide. Use the window it suggests, or add `serviceNames`.",
+ "category": "Tracing",
+ "feature": "tracing",
+ "summary": "Service health against a baseline",
+ "title": "Service health against a baseline",
+ "required_scopes": ["tracing:read"],
+ "annotations": {
+ "destructiveHint": false,
+ "idempotentHint": true,
+ "openWorldHint": true,
+ "readOnlyHint": true
+ },
+ "feature_flag": "tracing"
+ },
"apm-services-list": {
"description": "List distinct service names that have emitted trace spans. Use to discover available services before filtering spans by service name.",
"category": "Tracing",
@@ -987,7 +1002,7 @@
"feature_flag": "tracing"
},
"apm-spans-count": {
- "description": "Return a scalar count of trace spans matching a filter set. Use this as a cheap pre-flight before `query-apm-spans` — if the count is large, narrow the filters (or set `excludeAttributes`) before pulling rows.\n\nAll parameters go inside `query` — top-level fields are rejected:\n\n```json\n{ \"query\": { \"serviceNames\": [\"api\"], \"dateRange\": { \"date_from\": \"-1h\" } } }\n```\n\n# When to use\n\n- Before `query-apm-spans`, to confirm the filter set returns a tractable number of spans rather than pulling rows blind.\n- When the user asks \"how many X spans are there?\" and you don't need to see individual spans.\n- To check whether a filter combination matches anything at all before committing to a full query.\n\nThis counts **spans**, not traces. A single trace contains many spans, so a count of matching spans will exceed the number of matching traces.\n\nIf the count would scan too much data (a wide date range with no filters), the tool returns a 400 asking you to narrow the window or add filters — narrow the `dateRange` or add `serviceNames` / `statusCodes` / `filterGroup`, then retry.\n\n# Parameters\n\n## query.dateRange\n\nDate range for the count. Defaults to the last hour (`-1h`).\n\n- `date_from`: Start of the range. Accepts ISO 8601 timestamps or relative formats: `-1h`, `-6h`, `-1d`, `-7d`, `-30d`.\n- `date_to`: End of the range. Same format. Omit or null for \"now\".\n\n## query.serviceNames\n\nFilter by service names. Unlike `query-apm-spans`, an unfiltered count is cheap and useful for sizing up a filter before narrowing. Use `apm-services-list` to discover services.\n\n## query.statusCodes\n\nFilter by OTel span status codes (list of integers: `0` Unset, `1` OK, `2` Error) — **not** HTTP status codes. Use `[2]` to select error spans.\n\n## query.filterGroup\n\nProperty filters to narrow the count. Same format as `query-apm-spans` filters — each filter specifies `key`, `operator`, `type` (span/span_attribute/span_resource_attribute), and optionally `value`.\n\nA `span` filter on `duration` takes a value in milliseconds. For example, `{ \"key\": \"duration\", \"operator\": \"gt\", \"type\": \"span\", \"value\": \"30000\" }` counts spans longer than 30 seconds.\n\n# Examples\n\n## Count error spans in a service over the last day\n\n```json\n{\n \"query\": {\n \"serviceNames\": [\"api-gateway\"],\n \"statusCodes\": [2],\n \"dateRange\": { \"date_from\": \"-1d\" }\n }\n}\n```\n\n## Count how many spans match a name before fetching them\n\n```json\n{\n \"query\": {\n \"filterGroup\": [{ \"key\": \"name\", \"operator\": \"exact\", \"type\": \"span\", \"value\": \"redis_cluster.discovery\" }],\n \"dateRange\": { \"date_from\": \"-6h\" }\n }\n}\n```",
+ "description": "Return a scalar count of trace spans matching a filter set. Use this as a cheap pre-flight before `query-apm-spans` — if the count is large, narrow the filters (or set `excludeAttributes`) before pulling rows.\n\nAll parameters go inside `query` — top-level fields are rejected:\n\n```json\n{ \"query\": { \"serviceNames\": [\"api\"], \"dateRange\": { \"date_from\": \"-1h\" } } }\n```\n\n# When to use\n\n- Before `query-apm-spans`, to confirm the filter set returns a tractable number of spans rather than pulling rows blind.\n- When the user asks \"how many X spans are there?\" and you don't need to see individual spans.\n- To check whether a filter combination matches anything at all before committing to a full query.\n\n`count` is the number of matching **spans**. `traceCount` is the number of distinct traces whose root span matches. A trace contains many spans, so `count` is usually larger.\n\nIf the count would scan too much data (a wide date range with no filters), the tool returns a 400 asking you to narrow the window or add filters — narrow the `dateRange` or add `serviceNames` / `statusCodes` / `filterGroup`, then retry.\n\n# Parameters\n\n## query.dateRange\n\nDate range for the count. Defaults to the last hour (`-1h`).\n\n- `date_from`: Start of the range. Accepts ISO 8601 timestamps or relative formats: `-1h`, `-6h`, `-1d`, `-7d`, `-30d`.\n- `date_to`: End of the range. Same format. Omit or null for \"now\".\n\n## query.serviceNames\n\nFilter by service names. Unlike `query-apm-spans`, an unfiltered count is cheap and useful for sizing up a filter before narrowing. Use `apm-services-list` to discover services.\n\n## query.statusCodes\n\nFilter by OTel span status codes (list of integers: `0` Unset, `1` OK, `2` Error) — **not** HTTP status codes. Use `[2]` to select error spans.\n\n## query.filterGroup\n\nProperty filters to narrow the count. Same format as `query-apm-spans` filters — each filter specifies `key`, `operator`, `type` (span/span_attribute/span_resource_attribute), and optionally `value`.\n\nA `span` filter on `duration` takes a value in milliseconds. For example, `{ \"key\": \"duration\", \"operator\": \"gt\", \"type\": \"span\", \"value\": \"30000\" }` counts spans longer than 30 seconds.\n\n# Examples\n\n## Count error spans in a service over the last day\n\n```json\n{\n \"query\": {\n \"serviceNames\": [\"api-gateway\"],\n \"statusCodes\": [2],\n \"dateRange\": { \"date_from\": \"-1d\" }\n }\n}\n```\n\n## Count how many spans match a name before fetching them\n\n```json\n{\n \"query\": {\n \"filterGroup\": [{ \"key\": \"name\", \"operator\": \"exact\", \"type\": \"span\", \"value\": \"redis_cluster.discovery\" }],\n \"dateRange\": { \"date_from\": \"-6h\" }\n }\n}\n```",
"category": "Tracing",
"feature": "tracing",
"summary": "Count spans",
diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.ts
index 234f0e1ae241..944cd64beb2a 100644
--- a/services/mcp/src/api/generated.ts
+++ b/services/mcp/src/api/generated.ts
@@ -102886,6 +102886,264 @@ export namespace Schemas {
policies?: ToolPolicyEntry[];
}
+ /**
+ * * `none` - None
+ * * `previous_period` - Previous Period
+ * * `previous_day` - Previous Day
+ * * `previous_week` - Previous Week
+ */
+ export type ServiceHealthBaselineEnum = typeof ServiceHealthBaselineEnum[keyof typeof ServiceHealthBaselineEnum];
+
+
+ export const ServiceHealthBaselineEnum = {
+ None: 'none',
+ PreviousPeriod: 'previous_period',
+ PreviousDay: 'previous_day',
+ PreviousWeek: 'previous_week',
+ } as const;
+
+ /**
+ * * `status` - Status
+ * * `error_rate` - Error Rate
+ * * `p95` - P95
+ * * `throughput` - Throughput
+ * * `error_count` - Error Count
+ */
+ export type ServiceHealthOrderByEnum = typeof ServiceHealthOrderByEnum[keyof typeof ServiceHealthOrderByEnum];
+
+
+ export const ServiceHealthOrderByEnum = {
+ Status: 'status',
+ ErrorRate: 'error_rate',
+ P95: 'p95',
+ Throughput: 'throughput',
+ ErrorCount: 'error_count',
+ } as const;
+
+ export interface _ServiceHealthDateRange {
+ /**
+ * Relative (-30m, -6h, -7d) or ISO 8601.
+ * @nullable
+ */
+ date_from?: string | null;
+ /**
+ * Same format. Omit or null for "now".
+ * @nullable
+ */
+ date_to?: string | null;
+ }
+
+ /**
+ * * `entry` - Entry
+ * * `all` - All
+ */
+ export type ServiceHealthSpanScopeEnum = typeof ServiceHealthSpanScopeEnum[keyof typeof ServiceHealthSpanScopeEnum];
+
+
+ export const ServiceHealthSpanScopeEnum = {
+ Entry: 'entry',
+ All: 'all',
+ } as const;
+
+ export interface _ServiceHealthQuery {
+ /** The window to score. Defaults to the last hour. */
+ dateRange?: _ServiceHealthDateRange;
+ /** The baseline. previous_period (default) is the same length just before the window. previous_day and previous_week shift the window back 1 or 7 days. none skips it. baselineDateRange overrides it.
+ *
+ * * `none` - None
+ * * `previous_period` - Previous Period
+ * * `previous_day` - Previous Day
+ * * `previous_week` - Previous Week */
+ compare?: ServiceHealthBaselineEnum;
+ /**
+ * Only this environment, matched without case on deployment.environment.name, then deployment.environment, then env.
+ * @nullable
+ */
+ environment?: string | null;
+ /**
+ * Only these services.
+ * @maxItems 100
+ */
+ serviceNames?: string[];
+ /** entry (default) counts root, server and consumer spans, one per request. all counts every span.
+ *
+ * * `entry` - Entry
+ * * `all` - All */
+ spanScope?: ServiceHealthSpanScopeEnum;
+ /** status (default) puts silent, regressed and degraded first. The others sort high to low.
+ *
+ * * `status` - Status
+ * * `error_rate` - Error Rate
+ * * `p95` - P95
+ * * `throughput` - Throughput
+ * * `error_count` - Error Count */
+ orderBy?: ServiceHealthOrderByEnum;
+ /**
+ * Max service rows, 1-200.
+ * @minimum 1
+ * @maximum 200
+ */
+ limit?: number;
+ }
+
+ export interface ServiceHealthRequest {
+ /** The service health query. Every field is optional. */
+ query: _ServiceHealthQuery;
+ /** An explicit baseline window. Overrides query.compare. */
+ baselineDateRange?: _ServiceHealthDateRange;
+ }
+
+ export interface _ServiceHealthTimeRange {
+ /** Start, ISO 8601 UTC, inclusive. */
+ date_from: string;
+ /** End, ISO 8601 UTC, exclusive. */
+ date_to: string;
+ }
+
+ export interface _ServiceHealthWindow {
+ /** The scored window. Reuse it as dateRange in follow-up calls. */
+ current: _ServiceHealthTimeRange;
+ /** The baseline window. Null when compare is none. */
+ baseline: _ServiceHealthTimeRange | null;
+ /** Always UTC. */
+ timezone: string;
+ }
+
+ export interface _ServiceHealthSummary {
+ /** Services scored, before the limit. */
+ services: number;
+ /** Requests over all scored services. */
+ requests: number;
+ /** Services per status. */
+ ok: number;
+ /** Services per status. */
+ degraded: number;
+ /** Services per status. */
+ regressed: number;
+ /** Services per status. */
+ silent: number;
+ /** Services per status. */
+ new: number;
+ /** Services per status. */
+ recovered: number;
+ /** Services per status. */
+ improved: number;
+ }
+
+ /**
+ * * `silent` - Silent
+ * * `regressed` - Regressed
+ * * `degraded` - Degraded
+ * * `new` - New
+ * * `recovered` - Recovered
+ * * `improved` - Improved
+ * * `ok` - Ok
+ */
+ export type ServiceHealthStatusEnum = typeof ServiceHealthStatusEnum[keyof typeof ServiceHealthStatusEnum];
+
+
+ export const ServiceHealthStatusEnum = {
+ Silent: 'silent',
+ Regressed: 'regressed',
+ Degraded: 'degraded',
+ New: 'new',
+ Recovered: 'recovered',
+ Improved: 'improved',
+ Ok: 'ok',
+ } as const;
+
+ export interface _ServiceHealthRow {
+ /** Service name. */
+ service: string;
+ /** silent: spans stopped. regressed: error rate or p95 worse than the baseline. degraded: error rate 5% or more. new: no spans in the baseline. recovered, improved: the reverse of regressed.
+ *
+ * * `silent` - Silent
+ * * `regressed` - Regressed
+ * * `degraded` - Degraded
+ * * `new` - New
+ * * `recovered` - Recovered
+ * * `improved` - Improved
+ * * `ok` - Ok */
+ status: ServiceHealthStatusEnum;
+ /** Why the service has this status. Empty for ok. */
+ reason: string;
+ /** Spans in the window, in the chosen spanScope. */
+ requests: number;
+ /** Requests per minute. */
+ throughput_per_min: number;
+ /** Requests with OTel status Error. */
+ error_count: number;
+ /**
+ * Error share of requests, percent.
+ * @nullable
+ */
+ error_rate_pct: number | null;
+ /**
+ * Change from the baseline, points.
+ * @nullable
+ */
+ error_rate_delta_pp: number | null;
+ /**
+ * Median duration, ms.
+ * @nullable
+ */
+ p50_ms: number | null;
+ /**
+ * 95th percentile duration, ms.
+ * @nullable
+ */
+ p95_ms: number | null;
+ /**
+ * p95 change from the baseline, percent.
+ * @nullable
+ */
+ p95_change_pct: number | null;
+ /**
+ * 99th percentile duration, ms.
+ * @nullable
+ */
+ p99_ms: number | null;
+ /**
+ * Baseline requests. Null without one.
+ * @nullable
+ */
+ baseline_requests: number | null;
+ /**
+ * Last span, ISO 8601 UTC.
+ * @nullable
+ */
+ last_seen: string | null;
+ }
+
+ /**
+ * The prefilled query argument.
+ */
+ export type _ServiceHealthNextCallQuery = { [key: string]: unknown };
+
+ export interface _ServiceHealthNextCall {
+ /** The tool to call next. */
+ tool: string;
+ /** What the call answers. */
+ why: string;
+ /** The prefilled query argument. */
+ query: _ServiceHealthNextCallQuery;
+ }
+
+ export interface ServiceHealthResponse {
+ /** The absolute UTC windows the server used. */
+ window: _ServiceHealthWindow;
+ /** Counts over all scored services. */
+ summary: _ServiceHealthSummary;
+ /** One row per service, in orderBy order. */
+ services: _ServiceHealthRow[];
+ /** Prefilled follow-up calls. */
+ next: _ServiceHealthNextCall[];
+ /** True when more services matched than limit. */
+ truncated: boolean;
+ /** Caveats about this result. */
+ notes: string[];
+ }
+
export interface SessionRecordingBulkDeleteRequest {
/**
* Session IDs of the recordings to delete (max 100 per call).
@@ -133384,6 +133642,21 @@ export namespace Schemas {
* @minLength 1
*/
attribute_type?: TracingSpansAttributesRetrieveAttributeType;
+ /**
+ * JSON-encoded date range object, e.g. {"date_from": "-1h"}. Malformed JSON returns 400. Or pass date_from and date_to.
+ * @minLength 1
+ */
+ dateRange?: string;
+ /**
+ * Range start: -1h, -7d or ISO 8601. Defaults to -1h.
+ * @minLength 1
+ */
+ date_from?: string;
+ /**
+ * Range end, same format. Omit for "now".
+ * @minLength 1
+ */
+ date_to?: string;
/**
* Max results (default: 100).
* @minimum 1
@@ -133416,10 +133689,20 @@ export namespace Schemas {
export type TracingSpansServiceNamesRetrieveParams = {
/**
- * JSON-encoded date range, e.g. '{"date_from": "-1h"}'.
+ * JSON-encoded date range object, e.g. {"date_from": "-1h"}. Malformed JSON returns 400. Or pass date_from and date_to.
* @minLength 1
*/
dateRange?: string;
+ /**
+ * Range start: -1h, -7d or ISO 8601. Defaults to -1h.
+ * @minLength 1
+ */
+ date_from?: string;
+ /**
+ * Range end, same format. Omit for "now".
+ * @minLength 1
+ */
+ date_to?: string;
/**
* Search filter for service names.
* @minLength 1
@@ -133437,6 +133720,21 @@ export namespace Schemas {
* @minLength 1
*/
attribute_type?: TracingSpansValuesRetrieveAttributeType;
+ /**
+ * JSON-encoded date range object, e.g. {"date_from": "-1h"}. Malformed JSON returns 400. Or pass date_from and date_to.
+ * @minLength 1
+ */
+ dateRange?: string;
+ /**
+ * Range start: -1h, -7d or ISO 8601. Defaults to -1h.
+ * @minLength 1
+ */
+ date_from?: string;
+ /**
+ * Range end, same format. Omit for "now".
+ * @minLength 1
+ */
+ date_to?: string;
/**
* The attribute key to get values for.
* @minLength 1
diff --git a/services/mcp/src/generated/tracing/api.ts b/services/mcp/src/generated/tracing/api.ts
index db1403209a1f..65616b6f3475 100644
--- a/services/mcp/src/generated/tracing/api.ts
+++ b/services/mcp/src/generated/tracing/api.ts
@@ -3,7 +3,7 @@
* MCP service uses these Zod schemas for generated tool handlers.
* To regenerate: hogli build:openapi
*
- * PostHog API - MCP 12 enabled ops
+ * PostHog API - MCP 13 enabled ops
* OpenAPI spec version: 1.0.0
*/
import * as zod from 'zod'
@@ -292,6 +292,15 @@ export const TracingSpansAttributesRetrieveQueryParams = () => zod.object({
.describe(
'Type of attributes: \"span_attribute\" for span-level attributes, \"span_resource_attribute\" for resource-level attributes.\n\n\* `span_attribute` - span_attribute\n\* `span_resource_attribute` - span_resource_attribute'
),
+ dateRange: zod
+ .string()
+ .min(1)
+ .optional()
+ .describe(
+ 'JSON-encoded date range object, e.g. {\"date_from\": \"-1h\"}. Malformed JSON returns 400. Or pass date_from and date_to.'
+ ),
+ date_from: zod.string().min(1).optional().describe('Range start: -1h, -7d or ISO 8601. Defaults to -1h.'),
+ date_to: zod.string().min(1).optional().describe('Range end, same format. Omit for \"now\".'),
limit: zod
.number()
.min(1)
@@ -802,10 +811,97 @@ export const TracingSpansServiceNamesRetrieveParams = () => zod.object({
})
export const TracingSpansServiceNamesRetrieveQueryParams = () => zod.object({
- dateRange: zod.string().min(1).optional().describe('JSON-encoded date range, e.g. \'{\"date_from\": \"-1h\"}\'.'),
+ dateRange: zod
+ .string()
+ .min(1)
+ .optional()
+ .describe(
+ 'JSON-encoded date range object, e.g. {\"date_from\": \"-1h\"}. Malformed JSON returns 400. Or pass date_from and date_to.'
+ ),
+ date_from: zod.string().min(1).optional().describe('Range start: -1h, -7d or ISO 8601. Defaults to -1h.'),
+ date_to: zod.string().min(1).optional().describe('Range end, same format. Omit for \"now\".'),
search: zod.string().min(1).optional().describe('Search filter for service names.'),
})
+export const TracingSpansServiceHealthCreateParams = () => zod.object({
+ project_id: zod
+ .string()
+ .describe(
+ "Project ID of the project you're trying to access. To find the ID of the project, make a call to \/api\/projects\/."
+ ),
+})
+
+export const tracingSpansServiceHealthCreateBodyQueryOneCompareDefault = `previous_period`
+export const tracingSpansServiceHealthCreateBodyQueryOneServiceNamesMax = 100
+
+export const tracingSpansServiceHealthCreateBodyQueryOneSpanScopeDefault = `entry`
+export const tracingSpansServiceHealthCreateBodyQueryOneOrderByDefault = `status`
+export const tracingSpansServiceHealthCreateBodyQueryOneLimitDefault = 25
+export const tracingSpansServiceHealthCreateBodyQueryOneLimitMax = 200
+
+export const TracingSpansServiceHealthCreateBody = () => zod.object({
+ query: zod
+ .object({
+ dateRange: zod
+ .object({
+ date_from: zod.string().nullish().describe('Relative (-30m, -6h, -7d) or ISO 8601.'),
+ date_to: zod.string().nullish().describe('Same format. Omit or null for \"now\".'),
+ })
+ .optional()
+ .describe('The window to score. Defaults to the last hour.'),
+ compare: zod
+ .enum(['none', 'previous_period', 'previous_day', 'previous_week'])
+ .describe(
+ '\* `none` - None\n\* `previous_period` - Previous Period\n\* `previous_day` - Previous Day\n\* `previous_week` - Previous Week'
+ )
+ .default(tracingSpansServiceHealthCreateBodyQueryOneCompareDefault)
+ .describe(
+ 'The baseline. previous_period (default) is the same length just before the window. previous_day and previous_week shift the window back 1 or 7 days. none skips it. baselineDateRange overrides it.\n\n\* `none` - None\n\* `previous_period` - Previous Period\n\* `previous_day` - Previous Day\n\* `previous_week` - Previous Week'
+ ),
+ environment: zod
+ .string()
+ .nullish()
+ .describe(
+ 'Only this environment, matched without case on deployment.environment.name, then deployment.environment, then env.'
+ ),
+ serviceNames: zod
+ .array(zod.string())
+ .max(tracingSpansServiceHealthCreateBodyQueryOneServiceNamesMax)
+ .optional()
+ .describe('Only these services.'),
+ spanScope: zod
+ .enum(['entry', 'all'])
+ .describe('\* `entry` - Entry\n\* `all` - All')
+ .default(tracingSpansServiceHealthCreateBodyQueryOneSpanScopeDefault)
+ .describe(
+ 'entry (default) counts root, server and consumer spans, one per request. all counts every span.\n\n\* `entry` - Entry\n\* `all` - All'
+ ),
+ orderBy: zod
+ .enum(['status', 'error_rate', 'p95', 'throughput', 'error_count'])
+ .describe(
+ '\* `status` - Status\n\* `error_rate` - Error Rate\n\* `p95` - P95\n\* `throughput` - Throughput\n\* `error_count` - Error Count'
+ )
+ .default(tracingSpansServiceHealthCreateBodyQueryOneOrderByDefault)
+ .describe(
+ 'status (default) puts silent, regressed and degraded first. The others sort high to low.\n\n\* `status` - Status\n\* `error_rate` - Error Rate\n\* `p95` - P95\n\* `throughput` - Throughput\n\* `error_count` - Error Count'
+ ),
+ limit: zod
+ .number()
+ .min(1)
+ .max(tracingSpansServiceHealthCreateBodyQueryOneLimitMax)
+ .default(tracingSpansServiceHealthCreateBodyQueryOneLimitDefault)
+ .describe('Max service rows, 1-200.'),
+ })
+ .describe('The service health query. Every field is optional.'),
+ baselineDateRange: zod
+ .object({
+ date_from: zod.string().nullish().describe('Relative (-30m, -6h, -7d) or ISO 8601.'),
+ date_to: zod.string().nullish().describe('Same format. Omit or null for \"now\".'),
+ })
+ .optional()
+ .describe('An explicit baseline window. Overrides query.compare.'),
+})
+
export const TracingSpansSparklineCreateParams = () => zod.object({
project_id: zod
.string()
@@ -1077,6 +1173,15 @@ export const TracingSpansValuesRetrieveQueryParams = () => zod.object({
.describe(
'Type of attribute: \"span\" for built-in span fields (e.g. name), \"span_attribute\" for span-level attributes, \"span_resource_attribute\" for resource-level attributes.\n\n\* `span` - span\n\* `span_attribute` - span_attribute\n\* `span_resource_attribute` - span_resource_attribute'
),
+ dateRange: zod
+ .string()
+ .min(1)
+ .optional()
+ .describe(
+ 'JSON-encoded date range object, e.g. {\"date_from\": \"-1h\"}. Malformed JSON returns 400. Or pass date_from and date_to.'
+ ),
+ date_from: zod.string().min(1).optional().describe('Range start: -1h, -7d or ISO 8601. Defaults to -1h.'),
+ date_to: zod.string().min(1).optional().describe('Range end, same format. Omit for \"now\".'),
key: zod.string().min(1).describe('The attribute key to get values for.'),
limit: zod
.number()
diff --git a/services/mcp/src/tools/generated/tracing.ts b/services/mcp/src/tools/generated/tracing.ts
index 22252070771d..a680f7355bdb 100644
--- a/services/mcp/src/tools/generated/tracing.ts
+++ b/services/mcp/src/tools/generated/tracing.ts
@@ -50,6 +50,9 @@ const apmAttributeValuesList = (): ToolBase<ReturnType<typeof ApmAttributeValues
path: `/api/projects/${encodeURIComponent(String(projectId))}/tracing/spans/values/`,
query: {
attribute_type: params.attribute_type,
+ dateRange: params.dateRange,
+ date_from: params.date_from,
+ date_to: params.date_to,
key: params.key,
limit: params.limit,
offset: params.offset,
@@ -79,6 +82,9 @@ const apmAttributesList = (): ToolBase<
path: `/api/projects/${encodeURIComponent(String(projectId))}/tracing/spans/attributes/`,
query: {
attribute_type: params.attribute_type,
+ dateRange: params.dateRange,
+ date_from: params.date_from,
+ date_to: params.date_to,
limit: params.limit,
offset: params.offset,
search: params.search,
@@ -90,6 +96,32 @@ const apmAttributesList = (): ToolBase<
},
})
+const ApmServiceHealthSchema = () => {
+ const TracingSpansServiceHealthCreateBody = orvalSchemas.TracingSpansServiceHealthCreateBody()
+ return TracingSpansServiceHealthCreateBody
+}
+
+const apmServiceHealth = (): ToolBase<ReturnType<typeof ApmServiceHealthSchema>, Schemas.ServiceHealthResponse> => ({
+ name: 'apm-service-health',
+ schema: ApmServiceHealthSchema(),
+ handler: async (context: Context, params: z.infer<ReturnType<typeof ApmServiceHealthSchema>>) => {
+ const projectId = await context.stateManager.getProjectId()
+ const body: Record<string, unknown> = {}
+ if (params.query !== undefined) {
+ body['query'] = params.query
+ }
+ if (params.baselineDateRange !== undefined) {
+ body['baselineDateRange'] = params.baselineDateRange
+ }
+ const result = await context.api.request<Schemas.ServiceHealthResponse>({
+ method: 'POST',
+ path: `/api/projects/${encodeURIComponent(String(projectId))}/tracing/spans/service_health/`,
+ body,
+ })
+ return result
+ },
+})
+
const ApmServicesListSchema = () => {
const TracingSpansServiceNamesRetrieveQueryParams = orvalSchemas.TracingSpansServiceNamesRetrieveQueryParams()
return TracingSpansServiceNamesRetrieveQueryParams.extend({
@@ -112,6 +144,8 @@ const apmServicesList = (): ToolBase<
path: `/api/projects/${encodeURIComponent(String(projectId))}/tracing/spans/service-names/`,
query: {
dateRange: params.dateRange,
+ date_from: params.date_from,
+ date_to: params.date_to,
search: params.search,
},
})
@@ -166,7 +200,7 @@ const apmSpansCount = (): ToolBase<ReturnType<typeof ApmSpansCountSchema>, Schem
path: `/api/projects/${encodeURIComponent(String(projectId))}/tracing/spans/count/`,
body,
})
- const filtered = pickResponseFields(result, ['count']) as typeof result
+ const filtered = pickResponseFields(result, ['count', 'traceCount']) as typeof result
return filtered
},
})
@@ -337,6 +371,7 @@ export const GENERATED_TOOLS: Record<string, () => ToolBase<ZodObjectAny>> = {
'apm-attribute-breakdown': apmAttributeBreakdown,
'apm-attribute-values-list': apmAttributeValuesList,
'apm-attributes-list': apmAttributesList,
+ 'apm-service-health': apmServiceHealth,
'apm-services-list': apmServicesList,
'apm-spans-aggregate': apmSpansAggregate,
'apm-spans-count': apmSpansCount,
diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-attribute-values-list.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-attribute-values-list.json
index 7efc1fbccb6d..581cd6be2fce 100644
--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-attribute-values-list.json
+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-attribute-values-list.json
@@ -6,6 +6,21 @@
"enum": ["span", "span_attribute", "span_resource_attribute"],
"type": "string"
},
+ "date_from": {
+ "description": "Range start: -1h, -7d or ISO 8601. Defaults to -1h.",
+ "minLength": 1,
+ "type": "string"
+ },
+ "date_to": {
+ "description": "Range end, same format. Omit for \"now\".",
+ "minLength": 1,
+ "type": "string"
+ },
+ "dateRange": {
+ "description": "JSON-encoded date range object, e.g. {\"date_from\": \"-1h\"}. Malformed JSON returns 400. Or pass date_from and date_to.",
+ "minLength": 1,
+ "type": "string"
+ },
"key": {
"description": "The attribute key to get values for.",
"minLength": 1,
diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-attributes-list.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-attributes-list.json
index f71dd9fb3440..454dc07a66e3 100644
--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-attributes-list.json
+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-attributes-list.json
@@ -6,6 +6,21 @@
"enum": ["span_attribute", "span_resource_attribute"],
"type": "string"
},
+ "date_from": {
+ "description": "Range start: -1h, -7d or ISO 8601. Defaults to -1h.",
+ "minLength": 1,
+ "type": "string"
+ },
+ "date_to": {
+ "description": "Range end, same format. Omit for \"now\".",
+ "minLength": 1,
+ "type": "string"
+ },
+ "dateRange": {
+ "description": "JSON-encoded date range object, e.g. {\"date_from\": \"-1h\"}. Malformed JSON returns 400. Or pass date_from and date_to.",
+ "minLength": 1,
+ "type": "string"
+ },
"limit": {
"description": "Max results (default: 100).",
"maximum": 100,
diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-service-health.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-service-health.json
new file mode 100644
index 000000000000..5505e5a5b7a7
--- /dev/null
+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-service-health.json
@@ -0,0 +1,113 @@
+{
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
+ "properties": {
+ "baselineDateRange": {
+ "description": "An explicit baseline window. Overrides query.compare.",
+ "properties": {
+ "date_from": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Relative (-30m, -6h, -7d) or ISO 8601."
+ },
+ "date_to": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Same format. Omit or null for \"now\"."
+ }
+ },
+ "type": "object"
+ },
+ "query": {
+ "description": "The service health query. Every field is optional.",
+ "properties": {
+ "compare": {
+ "default": "previous_period",
+ "description": "The baseline. previous_period (default) is the same length just before the window. previous_day and previous_week shift the window back 1 or 7 days. none skips it. baselineDateRange overrides it.\n\n* `none` - None\n* `previous_period` - Previous Period\n* `previous_day` - Previous Day\n* `previous_week` - Previous Week",
+ "enum": ["none", "previous_period", "previous_day", "previous_week"],
+ "type": "string"
+ },
+ "dateRange": {
+ "description": "The window to score. Defaults to the last hour.",
+ "properties": {
+ "date_from": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Relative (-30m, -6h, -7d) or ISO 8601."
+ },
+ "date_to": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Same format. Omit or null for \"now\"."
+ }
+ },
+ "type": "object"
+ },
+ "environment": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Only this environment, matched without case on deployment.environment.name, then deployment.environment, then env."
+ },
+ "limit": {
+ "default": 25,
+ "description": "Max service rows, 1-200.",
+ "maximum": 200,
+ "minimum": 1,
+ "type": "number"
+ },
+ "orderBy": {
+ "default": "status",
+ "description": "status (default) puts silent, regressed and degraded first. The others sort high to low.\n\n* `status` - Status\n* `error_rate` - Error Rate\n* `p95` - P95\n* `throughput` - Throughput\n* `error_count` - Error Count",
+ "enum": ["status", "error_rate", "p95", "throughput", "error_count"],
+ "type": "string"
+ },
+ "serviceNames": {
+ "description": "Only these services.",
+ "items": {
+ "type": "string"
+ },
+ "maxItems": 100,
+ "type": "array"
+ },
+ "spanScope": {
+ "default": "entry",
+ "description": "entry (default) counts root, server and consumer spans, one per request. all counts every span.\n\n* `entry` - Entry\n* `all` - All",
+ "enum": ["entry", "all"],
+ "type": "string"
+ }
+ },
+ "type": "object"
+ }
+ },
+ "required": ["query"],
+ "type": "object"
+}
diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-services-list.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-services-list.json
index 2520e54f4632..39129bfdcdf8 100644
--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-services-list.json
+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-services-list.json
@@ -1,8 +1,18 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
+ "date_from": {
+ "description": "Range start: -1h, -7d or ISO 8601. Defaults to -1h.",
+ "minLength": 1,
+ "type": "string"
+ },
+ "date_to": {
+ "description": "Range end, same format. Omit for \"now\".",
+ "minLength": 1,
+ "type": "string"
+ },
"dateRange": {
- "description": "JSON-encoded date range, e.g. '{\"date_from\": \"-1h\"}'.",
+ "description": "JSON-encoded date range object, e.g. {\"date_from\": \"-1h\"}. Malformed JSON returns 400. Or pass date_from and date_to.",
"minLength": 1,
"type": "string"
},